U.S. patent application number 16/029322 was filed with the patent office on 2019-01-10 for systems and methods for data-driven movement skill training.
The applicant listed for this patent is iCueMotion LLC. Invention is credited to Berenice Mettler May.
Application Number | 20190009133 16/029322 |
Document ID | / |
Family ID | 64904003 |
Filed Date | 2019-01-10 |
![](/patent/app/20190009133/US20190009133A1-20190110-D00000.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00001.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00002.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00003.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00004.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00005.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00006.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00007.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00008.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00009.png)
![](/patent/app/20190009133/US20190009133A1-20190110-D00010.png)
View All Diagrams
United States Patent
Application |
20190009133 |
Kind Code |
A1 |
Mettler May; Berenice |
January 10, 2019 |
SYSTEMS AND METHODS FOR DATA-DRIVEN MOVEMENT SKILL TRAINING
Abstract
A data-driven movement skill training system is disclosed. The
system uses movement skill assessment and diagnostics at distinct
levels of the human movement system hierarchy to specify training
goals for a user. The system may provide various augmentations that
are synthesized to help the user pursue the training goals. The
system may include features to track and/or manage training or
learning processes.
Inventors: |
Mettler May; Berenice; (San
Francisco, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
iCueMotion LLC |
San Francisco |
CA |
US |
|
|
Family ID: |
64904003 |
Appl. No.: |
16/029322 |
Filed: |
July 6, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62529412 |
Jul 6, 2017 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G16H 20/30 20180101;
G01B 21/04 20130101; A63B 24/0062 20130101; A63B 71/0622 20130101;
A63B 2220/10 20130101; A63B 2230/60 20130101; A63B 24/0075
20130101; A63B 2220/56 20130101; A63B 2220/803 20130101; A63B 69/40
20130101; A63B 2220/40 20130101; G09B 19/0038 20130101; A63B
2071/0625 20130101; A63B 24/0003 20130101; A63B 2220/54 20130101;
A63B 2220/806 20130101; A63B 2024/0071 20130101; A63B 2024/0065
20130101 |
International
Class: |
A63B 24/00 20060101
A63B024/00; A63B 71/06 20060101 A63B071/06; A63B 69/40 20060101
A63B069/40 |
Claims
1. An apparatus for movement skill training, the apparatus
comprising: a sensor system comprising one or more sensors
configured to obtain movement data for a subject performing an
activity; a processor system in communication with the one or more
sensors, the processor system having a microprocessor and memory
configured to: receive the movement data from the one or more
sensors, wherein the subject performs a primary movement unit
associated with the activity; identify one or more movement
patterns from the movement data, wherein the movement patterns are
associated with the subject performing the primary movement unit;
analyze the movement patterns to identify one or more skill
attributes descriptive of the subject performing the primary
movement unit; and assess the one or more skill attributes to
specify one or more training goals for the subject, wherein the
training goals are selected to address deficiencies in the skill
attributes.
2. The apparatus of claim 1, wherein the one or more sensors
comprise one or more inertial sensors, accelerometers, gyroscopes,
or inertial measurement units, and wherein the movement data
comprise one or more of velocity, rotational velocity, acceleration
or rotational acceleration descriptive of movement patterns.
3. The apparatus of claim 1, wherein the one or more sensors
comprise one or more of: a magnetometer configured to acquire
direction or orientation data descriptive of the movement patterns,
a transducer configured to acquire one or more of position,
velocity, pressure, strain, or torque data descriptive of the
movement patterns, an acoustic sensor configured to acquire
acoustic wave data descripting descriptive of the movement
patterns, a visual sensor or camera configured to acquire image
data descriptive of the movement patterns, and a video sensor
configured to acquire video data descriptive of the movement
patterns.
4. The apparatus of claim 1, wherein the one or more sensors are
configured to obtain the movement data from one or both of the
subject and an associated object used by the subject to perform the
primary movement unit, the movement data selected from one or more
of angle, angular velocity, direction, distance, force, linear
acceleration, position, pressure, rotation, rotational speed, and
speed of the object, or from one or more of strain, pressure and
torque on the object.
5. The apparatus of claim 1, wherein the one or more sensors are
further configured to obtain activity data descriptive of the
subject performing the activity over a number of sessions
distributed over a calendar period, the processor system being
further configured to assess outcomes related to performance of the
training goals over the calendar period, based on the activity data
and the skill attributes.
6. The apparatus of claim 1, wherein the activity performed by the
subject is selected from badminton, baseball, cricket, golf,
rehabilitative exercises, running, skiing, snowboarding, surfing,
surgery or other medical procedure, swimming, table tennis, tennis,
and volleyball.
7. (canceled)
8. The apparatus of claim 1, wherein the processing system is
configured to extract the one or more skill attributes from the one
or more movement patterns to define one or more skill elements, the
skill elements characterizing one or more of movement patterns for
the subject to form, movement patterns for the subject to
consolidate, and movement patterns for the subject to optimize.
9. The apparatus of claim 8, wherein the processing system is
configured to determine a skill status by applying criteria derived
from the skill attributes, the skill status defining one or more of
the movement patterns for the subject to form, the movement
patterns for the subject to consolidate, and the movement patterns
for the subject to optimize.
10. The apparatus of claim 9, wherein the processing system is
configured to: combine the skill elements with the skill status to
generate a skill profile describing an overall skill and
performance of the subject relative to population data for a
population of such subjects; and; analyze the skill attributes
taking into account the skill status to produce the training goals
using the population data to determine which of the movement skills
to be improved.
11. (canceled)
12. The apparatus of claim 10, wherein the processing system is
further configured for a user to select one or more of the training
goals based on the skill status and further configured to track and
update the one or more training goals based on changes in the one
or more skill elements relative to the population data.
13. The apparatus of claim 8, wherein the processing system is
configured to: derive one or more training elements from the skill
elements, wherein a skill attribute associated with one or more of
the skill elements is assigned to one of the training goals; and
generate a training schedule for the subject, wherein the training
schedule comprises the training elements and training goals to
which the associated skill elements are assigned.
14. (canceled)
15. The apparatus of claim 1, further comprising: a cueing system
configured to provide audible, visual, or haptic feedback cues to
the subject and an apparatus system programmed to provide activity
interactions concurrently with the subject performing the activity,
to support development of specific movement patterns in conjunction
with the cueing system.
16. The apparatus of claim 15, further comprising a communication
system configured to provide symbolic, verbal, or visual
instructions to the subject, wherein the apparatus system is
programmed to work conjointly with the feedback cues and
instructions to support the development of the specific movement
patterns.
17. (canceled)
18. The apparatus of claim 8, wherein the training goals identify
one of the movement patterns to form through modification of an
existing movement pattern of the subject, wherein the movement
pattern is absent from the movement patterns in the collected data
due to lack of differentiation among such existing movement
patterns.
19. (canceled)
20. The apparatus of claim 1, wherein the training goals identify
one of the movement patterns to consolidate by creating procedural
memory to enable repeatable execution of the movement pattern by
the subject, wherein the movement pattern is not sufficiently
defined in the collected data to allow reliable execution under
dynamic conditions and the deficiencies manifest as variability in
the movement pattern, lack of smoothness in the movement pattern,
inefficient performance of the movement pattern, or insufficient
flexibility in the movement pattern with changing conditions.
21. (canceled)
22. The apparatus of claim 1, wherein the training goals identify
one of the movement patterns to optimize to minimize strain on a
musculoskeletal system of the subject, wherein the pattern in the
collected data does not achieve a desired outcome and the
deficiencies result in excessive use of force when seeking the
desired outcome.
23. (canceled)
24. The apparatus of claim 1, wherein the collected data comprises
population data providing reference values for the skill attributes
and the training goals for the subject.
25. A method of training comprising: assessing movement skills of a
subject performing a task; identifying deficiencies in the movement
skills of the subject relative to population data for a population
of such subjects; specifying training goals for the subject to
address the deficiencies in the movement skills; providing feedback
augmentation selected to induce the subject to achieve the training
goals, using the population data to determine which of the
movements skills to be improved; and tracking a training process of
the subject wherein identifying the deficiencies comprises relating
the movement skills of the subject to the population data and
specifying the training goals comprises using the population data
to determine which of the movement skills to improve for a skill
level of the subject to attain a desired level of proficiency.
26. The method of claim 25, further comprising identifying aspects
of the movement skills to be improved and in what order, wherein
information extracted from the population data directs the training
process to the aspects to focus on first to produce progress in the
skill level.
27. (canceled)
28. The method of claim 30, further comprising selecting a training
element from the training plan, wherein selecting the training
element indicates to a monitoring system aspects of movement
performance characterizing movement skills of the subject to be
monitored and the monitoring system provides notification for
achievement of the at least one training goal, or a fraction
thereof.
29. The method of claim 30, wherein selecting the training element
indicates to an augmentation system aspects of movement performance
characterizing movement skills of the subject to be actively cued
and the augmentation system provides real-time feedback cues
concurrently with the movement performance, at discrete time
periods during execution of the movement performance, or following
an outcome of the movement performance.
30. The method of claim 25, further comprising developing a
training plan, wherein the training plan describes organization of
a training session in terms of training elements associated with
the training goals; wherein a plurality of the training elements
are compiled in a training list arranged as a training schedule;
and wherein the training schedule comprises at least one such
training session, each training session divided into a plurality of
sets, and each set assigned at least one of the training goals.
31-32. (canceled)
33. A closed-loop system for data-driven training, the system
comprising: a sensor configured to collect one or more of movement
data describing motion of a body of a user or equipment used by the
user during performance of a movement, physiological data collected
from muscle activity of the user during the performance, and data
collected from an outcome or effect of the performance by the user;
a movement processor configured to execute: an assessment loop
configured to collect the data from the sensor during the
performance by the user; a training loop configured to track
progress in at least one movement skill of the user during the
performance; and an augmentation loop configured by the training
loop to provide information to the user during the performance; a
cueing system comprising a cue processor configured to translate
the movement data into a cue signal and a cue generator configured
to translate the cue signal into a physical feedback stimulus
generated by a transducer, wherein the feedback stimulus is
selected from audio, visual, haptic, and symbolic; and a feedback
loop executed between the movement processor and cueing system,
wherein the cueing system operates in real time to provide the
feedback stimulus to the user during the performance.
34-35. (canceled)
36. The system of claim 33, wherein the system is configured to
track at least one such performance from a plurality of such users,
and wherein the system is further configured to track interactions
between the performances.
37. (canceled)
38. The system of claim 33, further comprising an extractor
configured to extract motion elements from a target motion of the
performance; wherein the augmentation loop collects the movement
data from the user and provides motion elements to the extractor;
wherein session data are provided to the extractor and a motion
model of the user is produced from output from the extractor, the
motion model comprising the session data; wherein skill assessment
and diagnostics are performed on the motion model to produce a
skill model, the skill model comprising reference skill data for
the at least one movement skill.
39-44. (canceled)
45. The system of claim 33, further comprising an instruction
module configured to receive a set of target skills from the user,
wherein the instruction module processes the target skills and
provides the processed target skills to the training loop.
46-47. (canceled)
48. The system of claim 38, wherein the cue processor implements a
finite state estimate comprising an approximation of the motion
model of the user; wherein the cue processor implements a cueing
law calculator that operates on the finite state estimate and the
data collected from the sensor to determine a cue to be delivered;
wherein a feedback synthesis model determines operation of the
cueing law calculator and the cueing law calculator determines what
the cue should communicate to the user.
49-55. (canceled)
56. The system of claim 33, wherein the augmentation loop provides
feedback to mimic a human information processing hierarchy of the
user, wherein the feedback comprises one or more of an instruction,
a notification, and a cue, or the physical feedback stimulus
provided in real time to the user.
57. (canceled)
58. The system of claim 56, wherein the feedback comprises the
instruction generated from at least one of: a motion model, wherein
the assessment loop comprises an extractor configured to extract
motion elements from a target motion of the performance by the user
and the motion model is produced from output of the extractor; a
skill model, wherein the skill model is produced from assessing the
motion model; and a diagnostic assessment, wherein the diagnostic
assessment identifies deficiencies in the performance of the
user.
59-61. (canceled)
62. The system of claim 58, wherein the instruction synthesizes one
or more cueing laws that govern the augmentation loop and provides
information about a training element and an associated training
goal to organize a training process for the user; wherein the
instruction is generated at an interval during the training
session, upon completing within a training set of the training
session, or after the training session; and wherein the instruction
is presented to the user verbally, symbolically, or
graphically.
63-67. (canceled)
68. The system of claim 56, wherein the feedback comprises the cue
provided in real time to the user, and wherein the cue comprises a
discrete audible, tactile, or visual signal selected to target
specific movement characteristics of the user to impact the
performance or the outcome of the performance.
69-70. (canceled)
71. The system of claim 56, wherein the physical feedback stimulus
comprises an audible, tactile, or visual stimulation of a muscle or
nerve of the user provided in real time to guide the movement of
the user and enhance a feature thereof.
72-73. (canceled)
74. The system of claim 56, wherein the notification is presented
verbally, symbolically, or graphically and provides information
about progress of the user towards a training goal.
75. (canceled)
76. The system of claim 56, wherein the feedback further comprises
an activity interaction provided by an apparatus concurrently with
the user performing the movement to support development of a
specific movement pattern in conjunction with the cueing
system.
77. The system of claim 76, wherein the apparatus comprises: a ball
machine configured to throw balls with different trajectories
selected for the subject to form a new stroke technique or adapt an
existing stroke technique; or an assistive robotic device or robot
manipulator used to physically guide movements of the subject.
78. The system of claim 33, wherein the assessment loop is
configured to update a diagnostic tool for identifying deficiencies
in the performance of the movement by the user; wherein the
deficiencies are synthesized into training goals for the user; and
further comprising a training agent configured to identify training
elements associated with the training goals, wherein the training
agent suggests the training goals to manage a training schedule for
the user.
79-83. (canceled)
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Patent
Application No. 62/529,412, filed Jul. 6, 2017, entitled "Systems
and Methods for Data Driven Movement Skill Training," which is
hereby incorporated by reference in its entirety.
FIELD
[0002] Disclosed are devices, systems, and methods for movement
skill training.
BACKGROUND
[0003] Humans rely on motion skills to perform daily tasks ranging
from actions essential to our autonomy to more specialized domains
requiring highly refined motion skills. Professional athletes,
musicians, surgeons, and even elite amateurs require thousands of
hours of systematical and focused training, as well as continued
training to maintain high skill levels. Even simple daily acts
involve complex coordination of a range of processes, from sensing
and motor-control to perception and cognition. Learning,
maintaining, and rehabilitating movement skills are valuable, but
at the same time, complex and challenging tasks. Acquiring and
maintaining specialized movement skills takes time. Progress of
movement skills does not develop linearly with training time.
Rather, skills progress following a power law with measurements
suggesting that some skills continue to improve for over 100,000
trials.
[0004] Different factors account for this slow progress in skills.
Movement performance relies on a broad range of functions (e.g.,
sensory, perceptual, planning, cognition). Many movement skills
within the category of complex movement are unnatural and therefore
require adaptation of innate movement skills to accommodate the
specific task requirements. Complex movement also involve the
coordination of large numbers of muscles and body segments. They
may take place over short time-frames, with critical phases
spanning 10th to 100th of a few millisecond. They often need to be
adapted during performance and synchronized with external events or
elements. Movements are typically learned by trial and error,
mostly by using some outcomes as feedback for corrections. Due to
these complexities, the specific details regarding movement
organization are stored in procedural memory and therefore are only
known implicitly. Explicit knowledge surrounding movement details
are typically not used during practice and execution. The fact that
complex movements often unfold quickly and involve many dimensions
make them hard or impossible to fully be perceived let alone
comprehended. For example, just the path of a piece of equipment,
such as a tennis racket, already involves three translational and
rotational variables (e.g., six degrees of freedom) with their
additional kinematic (speeds and angular rates) as well as dynamic
(accelerations) characteristics.
[0005] Movement complexity grows dramatically when the various body
segments and musculoskeletal and neuro-motor constraints are
included. To make matters even more complex, these variables are
constrained by the dynamics, which constrain their spatial and
temporal evolution. Finally, there are very few feedback stimuli,
or signals, available to a user during the training process. As a
result, for most people who don't have access to coaching, movement
skill relies on self-observation and tedious repetition. In many
domains, proficiency cannot be achieved without the assistance of
an expert coach or trainer.
[0006] Challenges also exist in characterizing and assessing
movement. First, human movement is variable. Each repeated trial of
the same task results in a slightly different execution. Second,
technique is idiosyncratic. Individuals with the same general level
of ability have a different approach and style. Third, movement is
fast. Often an action unfolds within a fraction of a second, with
relevant details only spanning a few milliseconds. Fourth, movement
is complex. It often focuses on the control of an end effector,
such as a tool (e.g., surgical instrument) or piece of equipment
(e.g., tennis racket, baseball bat, golf club), which need to be
controlled in three dimensional workspaces. The execution of such
movements, requires controlling the various limbs, joints, and
muscles, which add many more additional degrees of freedom.
[0007] Moreover, because the coordinated motion patterns are
typically too complex and execute too quickly to be perceived and
processed consciously, it is usually difficult to make training
interventions in real time. Athletes or operators usually do not
have a sufficiently explicit awareness of the details of their
motion execution. These characteristics explain why it is difficult
to improve skills once basic motion patterns are acquired. External
feedback from a trainer or coach become necessary in order to
improve.
[0008] Movement skills also depend on a perceptual understanding of
the external task elements. These characteristics are much harder
to assess from observations of the movement performance. They
manifest indirectly in the performance. A good instructor will call
attention to important perceptual cues and how these can be used to
inform the movement response characteristics.
[0009] Finally, one requirement for effective training is to
account for individual differences in body type, skill level,
health, etc. Such characteristics are much harder to take into
account during training. This can be particularly critical for
rehabilitation or when working with injured or aging athletes. A
training approach should also leverage the properties and natural
learning principles and processes of skill development.
[0010] Popular wearable and embedded devices currently primarily
focus on the identification and tracking of activity (e.g.,
FITBIT.RTM. activity tracker (available from Fitbit, Inc.) or
JAWBONE.RTM. fitness tracker (available from Aliphcom doing
business as Jawbone)). Popular examples of fitness trackers include
devices for counting steps and tracking distance covered. More
advanced capabilities can be found in devices that are specialized
for a particular sport. Tennis, badminton, and golf represent the
largest market segments (see, e.g., BABOLAT PLAY.TM. (from Babolat,
France), ZEPP.RTM. tennis swing analyzer (available from Zepp US,
Inc.), and the Smart Tennis Sensor (available from Sony)). These
products aim to provide a description of players' technical
performance. Typical features include tracking the type of actions;
reconstructing movements, such as the path of the tennis racket
during a stroke; tracking select outcome variables of actions such
as the racket head speed, the distribution of impacts on the string
bed, and the amount of spin.
[0011] The outputs of these assessments are typically provided
after a training or play session. The data is presented as
summaries of session performance, as well as time. The data is also
aggregated to provide statistical trends. The main shortcoming of
these products is that the analysis is based on outcome variables
(referred to as knowledge of results in the human skill literature)
and thus does not provide actionable information that can be
leveraged directly for training.
[0012] One of the most established frameworks for training is the
so-called "deliberate practice." Ericsson developed this framework
after reviewing bodies of evidences concerning the conditions of
optimal learning. He found that individualized practice with
training tasks, selected by a coach or teacher, with clear goals
designed to improve a particular aspect of performance, and
immediate and informative feedback was associated with best
learning.
[0013] Deliberate practice enables one to fully engage in a
training activity. This engagement can play a critical role. It has
been shown that the regular performance of an activity, without
deliberate practice, does not lead to improvements past the
competency level (Ericsson, 2007). This phenomenon is in part
explained by the fact that as part of the natural learning process,
the brain learns to automate a significant part of the performance.
The automatization itself limits the ability to make adjustments in
technique unless very deliberate efforts are applied to identify
weaknesses in performance and set goals to address those
weaknesses. This is the reason why people get set in their
technique and habits.
[0014] Besides immediate feedback, other potentially critical
aspects of deliberate practice include having training goals that
provide a gradual path toward the refinement of one's skills, and
providing opportunities to engage in a form of problem solving.
[0015] Skill acquisition follows an incremental process; therefore,
most people's skills can be considered at some intermediate level
that could be further developed. Each successive iteration along a
path to improving skills involves increasingly complex mental
representations and their supportive functions such as movement
coordination, and perception (Ericsson, 2009). It seems that skill
acquisition stabilizes along successive skill levels.
[0016] No method currently exists for generating goals from
performance measurements. In addition, no algorithms currently
exist that enable a data-driven, operationalized training process
that automatically determines and updates goals as an agent learns
skills for an activity.
SUMMARY
[0017] Data-driven movement skill training systems are disclosed
herein. The systems may use movement skill assessment and
diagnostics at distinct levels of the human movement system
hierarchy to specify training goals. The systems may then provide
different forms of augmentations synthesized to help pursue the
training goals. The system may also include a system to track
and/or manage the learning process.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] FIG. 1 is an illustration of a human augmentation system for
movement skill training or rehabilitation according to an
embodiment.
[0019] FIG. 2 is an illustration of an interaction between a stroke
motion and task and environment elements, including ball trajectory
relative to a court, the impact of the ball, and its bouncing
before the interception with the racket trajectory. FIG. 2 also
illustrates the gaze of the player along different points of the
ball trajectory and court locations, and shows a ball machine as an
apparatus that can be programmed to enable different forms of
interactions.
[0020] FIG. 3A is an illustration of a general movement trajectory
envelope delineating the movement phases that typically arise from
biomechanical and neuromotor constraints.
[0021] FIG. 3B is an illustration of the finite-state model
representation for the system shown in FIG. 3A, where each state
represents a movement phase.
[0022] FIG. 4 is an illustration of the primary movement unit for
six movement activities, along with corresponding phase segments.
The figure also highlights primary outcome quantities as vectors
(e.g., effect of a racket or club on a ball for tennis or golf,
effect of arm coordination on hand placement in rehabilitation,
propulsive force generated by a foot strike for running,
transversal acceleration used in turning while skiing, and
propulsive force generated by the pull phase in swimming).
[0023] FIG. 5 is an illustration of the progression across
different tennis stroke architectures corresponding to different
skill levels. The architecture is shown in terms of its constituent
movement phases.
[0024] FIG. 6 is an illustration of an overview of the movement
processing components that form the basis for data-driven skill
assessment, learning process analysis, and larger population
analysis.
[0025] FIG. 7 is an illustration of different outcome levels for
tennis and some of the outcome measurements: 1) stroke technique
and racket impact; 2) stroke primary outcomes; 3) shot trajectory
and type; 4) shot placement (relative to the court and opponent).
FIG. 7 also shows trajectories for two types of strokes e.g., flat
(FL) and topspin (TS).
[0026] FIG. 8 is an illustration of two players' respective ground
impact distributions from tennis shots, describing the
discretization of the task environment associated with the
ball-environment interactions. The skills at the shot level
manifest as different resolutions and precision in the interactions
with the task environment. FIG. 8 also shows court landmarks that
are relevant to the players' task environment perception and the
players' court motion.
[0027] FIG. 9 is an illustration of three interception types: 1) on
the descent; 2) at the apex; 3) on the rise of the trajectory
following the ground impact. FIG. 9 also illustrates the racket
string bed associated with the corresponding impact conditions and
examples of the return shot outcomes.
[0028] FIG. 10 is an illustration of the levels of assessment
highlighting the elements and outcomes for tennis and summarizing
the assessment and diagnostic components across the different
levels.
[0029] FIG. 11 is an illustration of the movement acquisition as an
evolutionary process during which movement patterns are learned
either from scratch or through differentiation of existing
patterns.
[0030] FIG. 12 is an illustration of movement pattern clusters
based on features extracted from movement data.
[0031] FIG. 13 is a tree diagram illustrating the evolutionary
relationship between movement patterns.
[0032] FIG. 14 is an illustration of the state space X (measured or
estimated dimensions derived from performance data that are
relevant to describe the movement behavior), highlighting the
classes associated with the movement patterns, and the mapping
associated with the outcome space or other attributes used in the
skill assessment. The figure also shows the embedding from V into a
subspace W that produces meaningful outcome categories (semantic
interpretation).
[0033] FIG. 15 is an illustration of a generic outcome-movement
pattern map showing the movement patterns in terms of associated
outcomes dimensions.
[0034] FIG. 16 is an illustration of a generic repertoire map
showing the movement pattern classes arranged in terms of outcome
dimensions that have been rescaled such as through the embedding g:
V->W.
[0035] FIG. 17 is an illustration of the skill profile of two
subjects A, B resulting from their repertoire of movement patterns
or skill elements, highlighting the difference in skill profile
that results in an overall skill gap as well as a gap in the
repertoire range.
[0036] FIG. 18 is an illustration of population subgroups based on
skill attributes with level lines associated with a performance or
skill objective function.
[0037] FIG. 19 is an illustration of distribution for an entire
population group described by the group distribution highlighting a
member (subject A, described by skill element distribution
(e.sub.1, e.sub.2)), and the tiers (low, medium, high, very high)
associated with an outcome function for the entire population
subgroup (e.sub.1,G, e.sub.2,G).
[0038] FIG. 20 is an illustration of distribution of motion
patterns produced by subject A described by two features (f.sub.1,
f.sub.2), showing the center of the ellipsoid (.mu..sub.1,
.mu..sub.2) and the axes given by the eigenvalues (e.sub.1,
e.sub.2), and highlighting the level lines of some outcome tiers
(low, medium, high, very high).
[0039] FIG. 21 is an illustration of a data-driven closed-loop
training system including its primary processes organized according
to three primary feedback loops.
[0040] FIG. 22 is an illustration of a Human Augmentation System.
The system encompasses three primary tiers of augmentation that
leverage the human information processing hierarchy: real-time
feedback (cue stimuli and activity interactions), intermittent
feedback, and visualization and instructions.
[0041] FIG. 23 is an illustration of the augmented
perception-action loop associated with the feedback cueing system.
Low-level signal and cues are emphasized.
[0042] FIG. 24 is an illustration of the main components used for
feedback augmentation in the augmented human movement system.
[0043] FIG. 25 is an illustration of process flow along the
training process shows an activity across the stack of processes of
an assessment and training loop (e.g., data acquisition and
processing, motion model, skill model, training goals, augmentation
laws) as a function of time. During each session, activity data is
collected and processed.
[0044] FIG. 26 is an illustration of a diagnostic system building
on the assessment system.
[0045] FIG. 27 is an illustration of a diagnostic system, which
combines a knowledge representation, observations, and an inference
mechanism to produce a diagnostic of the movement performance.
[0046] FIG. 28 is an illustration of the factors influencing stroke
quality (categorized as observations, uncertain factors, or
hypotheses) and their relationships.
[0047] FIG. 29 is an illustration of population analysis and player
or performer profiling.
[0048] FIG. 30 is an illustration of an assessment, diagnostics,
and training goals across the skill-model hierarchy, incorporating
player profile information.
[0049] FIG. 31 is an illustration of an assessment including a)
different levels of assessment, b) elements that describe each
level, c) criteria and quantities that can be used to determine the
skill characteristics at a given level, d) analysis or diagnostics
to identify the critical characteristics, e) the drivers and
mechanisms used to produce training interventions, and f) the
intervention or feedback form that can be used.
[0050] FIG. 32 is an illustration of the primary outcome
characteristics (i.e., pace and spin) for a player's groundstroke
repertoire with overall reference ranges from population analysis
(gray background tiles).
[0051] FIG. 33 is an illustration of spin envelope for the
groundstrokes (solid line) divided into forehand and backhand with
reference ranges from population analysis (dashed lines).
[0052] FIG. 34 is an illustration of a leaderboard for population
analysis based on the global score shown as a percentile rank from
highest to lowest computed from the skill profile of a population
of players.
[0053] FIG. 35 is an illustration of play activity summation over a
calendar period showing sets and sessions.
[0054] FIG. 36 is an illustration of movement outcome trends for a
specific motion pattern class with overall reference ranges from
population analysis (gray background tiles). The vertical bands
delineate the sets.
[0055] FIG. 37 is an illustration of the forward swing phase stroke
profile for forehand topspin stroke class.
[0056] FIG. 38 is an illustration of selected components of a skill
element including outcome, attributes, and other characteristics
forming a composite skill element score. Two polygons are
superposed to provide a comparison.
[0057] FIG. 39 is an illustration of the trends in movement
patterns and movement outcomes for an activity session delineated
in individual sets.
[0058] FIG. 40 is an illustration of the skill profile as a bar
graph for the values from a composite score across a repertoire of
groundstrokes.
[0059] FIG. 41 is an illustration of the acquisition stage for the
strokes in the groundstroke repertoire.
[0060] FIG. 42 is an illustration of impact timing statistics for a
player's groundstroke repertoire with overall reference ranges from
population analysis (gray background tiles).
[0061] FIG. 43 is an illustration of the integrated perspective on
the assessment and diagnostic process organized in terms of the
assessment levels (i.e., physical, pattern, task, and
competitive).
[0062] FIG. 44A is an illustration of a skill status screen showing
the skill elements arranged according to their acquisition stage
(Patterns to Form, Patterns to Consolidate, and Patterns to
Optimize).
[0063] FIG. 44B is an illustration of a skill status screen showing
how training activity over several training sessions (Set 1-3) lead
to a change in the skill status of skill elements.
[0064] FIG. 45A is an illustration of a training list showing
selected training elements e.sub.i and their associated training
goals g.sup.k.sub.i expressed in terms of attributes a.sub.i for an
epoch k. The list is indexed according to relevant criteria, such
as user preference or importance of the element to the activity or
to the skill acquisition process.
[0065] FIG. 45B is an illustration of a training schedule. The
training session is subdivided into sets (Set 1 . . . Set N). Each
set focuses on one or more training element e.sub.i (e.g., grouped
into one set with related aspects, e.g., forehand and backhand top
spin) and its associated goal selected from the skill elements in
the skill status.
[0066] FIG. 46 is an illustration of a state machine showing the
active training element and the criteria for the issuance of
notifications to the performer.
[0067] FIG. 47 is an illustration of a trend plot displaying the
progress along training goals (g.sub.1, g.sub.2, and g.sub.3) over
a specified time range shown here as seven sessions.
[0068] FIG. 48 is an illustration of a learning curve associated
with the data driven training process. The learning curve shows the
incremental improvement in some relevant attribute a.sub.i of a
skill element e.sub.i over the training activity (sets and
sessions).
[0069] FIG. 49 is a flow diagram illustrating a data-driven
training process according to an embodiment.
[0070] FIG. 50 is a flow diagram illustrating the movement modeling
processes of FIG. 49.
[0071] FIG. 51 is a flow diagram illustrating the processes of the
skill modeling and assessment of FIG. 49.
[0072] FIG. 52A is a flow diagram illustrating the skill assessment
processes of FIG. 49.
[0073] FIG. 52B is a flow diagram illustrating the skill status
process of FIG. 49.
[0074] FIG. 53 is a flow diagram illustrating the training goals
and feedback synthesis processes of FIG. 49.
[0075] FIG. 54A is a flow diagram illustrating the training goal
computation process accounting for skill status of FIG. 49.
[0076] FIG. 54B is a flow diagram illustrating the feedback
synthesis processes of FIG. 49.
[0077] FIG. 55A is a flow diagram illustrating the instructions
synthesis of FIG. 49.
[0078] FIG. 55B is a flow diagram illustrating the feedback and
cueing laws synthesis processes of FIG. 49.
[0079] FIG. 56 is a flow diagram illustrating the activity
management and monitoring processes of FIG. 49.
[0080] FIG. 57A is a flow diagram illustrating the system
configuration processes of FIG. 49.
[0081] FIG. 57B is a flow diagram illustrating the activity
monitoring process including the notification and user input of
FIG. 49.
[0082] FIG. 58 is illustration of a temporal structure and
organization of a typical activity session.
ELEMENTS OF MOVEMENT SKILL ACQUISITION
[0083] This section briefly reviews central elements of movement
skill acquisition and their execution. These elements highlight the
challenges involved with motor skill learning and how technology
can be used to augment the skill acquisition process. In
particular: how to generate knowledge about movement skills and the
associated learning process, how this knowledge can help determine
which quantities to use to track the movement learning process,
which information to feedback to the performer to help their
learning, in which form to communicate this information, and at
which time during the movement performance.
Motor Movement Skills Overview
[0084] Skill-based movement behaviors are usually fast,
coordinated, multi-dimensional movements. Delays in human's signal
transmission and processing limit the role of real-time feedback.
Therefore, the biological movement control system has to rely
extensively on "open-loop" control, meaning that trajectories are
implemented from pre-programmed profiles, which are stored in
procedural memory and therefore are largely unconscious. The
general motor program (GMP) explains how complex movements are
programmed. GMP describe the generalized rules that generate
spatial and temporal muscle patterns to produce a movement for the
collection of movement patterns in the repertoire. These programs
are generalized in the sense that GMP encompass the mechanisms
needed for adaptation to conditions within a given movement pattern
class.
[0085] Complex movements frequently involve a sequence of distinct
movement phases. Therefore, motor programs encompass mechanisms to
support the ordering and timing of these elements in a sequence
that forms a movement pattern. The movement phases are usually
formed to support various functional characteristics, such as
biomechanical constraints, task structure and various sensory
interactions with the environment. Movement segments can be
conceptualized as a movement directed towards a sub-goal, each with
its particular biomechanical and sensory-motor constraints. This
structure allows to breakdown complex movements into simpler
movement elements. It can also help in the acquisition of complex
movement skills, and support the flexibility and adaptability
needed to operate in dynamic and uncertain environments.
[0086] The human bandwidth limitation for closed-loop feedback
involving perceptual motor control is somewhere between 0.5 and 2
Hz, depending on the task. Above that bandwidth, intermittent
closed-loop control can be used. Movement phases typically
represent open-loop segments. Corrections can be implemented at
specific phase transition. These phase transitions are also
associated with functional features, such as when specific elements
of information are available. For example, in a tennis stroke, an
advanced player already has an idea of the intended outcome and
anticipates the conditions of the oncoming ball, at the initiation
of the stroke. At the end of the backswing phase, and before the
initiation of the forward swing, the player makes adjustments based
on the up-to-date information available from the oncoming ball
trajectory.
[0087] As will be appreciated by those skilled in the art, movement
skills often involve extensive interactions with the task and
environment elements. For example, in tennis these interactions
include producing the desired outcome in the task and dealing with
the range of impact conditions. See FIGS. 1 and 2 which illustrate
the interactions of a tennis player's racket with a delineation of
the racket stroke trajectory, and FIG. 9 which illustrates the
interception conditions that the performer has to accommodate to
best control the ball trajectory. The perceptual system usually
provides cues that are used to select the type of motion pattern
from the repertoire of learned movement patterns. Signals from the
sensory or perceptual system are used to modulate specific aspects
of the pattern, such as the timing of the stroke phases based on a
tennis ball's perceived speed. Training movement skills, therefore,
involves acquiring a comprehensive set of mechanisms. Movements are
not simply programs to steer body segments; they encompass numerous
mechanisms and capabilities to support the interactions and
adaptation to conditions. Therefore, skill acquisition also
includes learning how to extract relevant signs or cues from the
task environment, and developing plans for sequencing individual
movement patterns. The basic motor learning concepts are introduced
next.
[0088] Motor skills require integration of both sensory information
and motor responses to attain a particular goal. Goal-directed,
deliberate, instrumental, or intentional movements are movements
characterized by forethought with reference to the consequences
they produce. The outcome to be obtained is clear to the performer
and determines how they organize their movement pattern. Such
deliberate movements contrast reflexes or fixed action patterns.
Motor skills are categorized on a continuum defined by the dynamics
of the task and environment condition. On one end of the continuum
are the open skills, which take place in temporally and spatially
changing conditions; on the other end are the closed skills, which
take place under fixed, unchanging environmental conditions.
[0089] In open skills, a new movement formed to respond to a new
aspect of the task environment may either originate as a variation
of an existing pattern, or as a new movement that is formed as a
unique new pattern (see FIG. 11), albeit the new pattern may be
reusing components of the original pattern. Therefore, in open
skills, the user develops a repertoire of movement patterns that
match the range of environmental conditions and task requirements.
On the other hand, in closed skills, as the user learns to master
the task, the movement performance converges over time to a fixed
movement pattern that optimizes the outcome in relationship to the
task requirement. As described herein, the term "user" may refer to
a user of the data-driven training system, an agent using the
system, a subject to whom the system is applied, or a combination
thereof.
[0090] The movement segments that compose most complex movements
result from how the subject exploits the large number of degrees of
freedom (DOF). The high DOF in human motion result in redundant
movement solutions. For example, racket swinging can be achieved
through various combinations of joint motions such as wrist, elbow,
shoulder, hips, etc. Each DOF has its own specific displacement
range as well as other constraints such as speed or torque.
Different executions of the same general movement will cause
saturations at different stages of the overall trajectory and will
result in a different sequence of movement phases.
[0091] Furthermore, human subjects mostly learn through practice;
they essentially discover how to best exploit the rich movement
space to accomplish the desired outcome. As a result, complex
movement skill acquisition, and more specifically the development
of movement architecture, proceeds through stages, with each stage
making an increasing use of the available degrees of freedom (see
FIG. 5).
[0092] Typically, a deliberate movement is needed to produce a
particular outcome or change in the environment. Many skilled
movements involve the control of an end effector such as the hand,
foot, or a piece of equipment or instrument. Another class of
skilled motions are characterized by controlling the dynamics of
interactions with an environment such as in skiing or surfing.
These interaction behaviors involve the performance of particular
maneuvers to allow deliberate control of motion. Examples of
maneuvers include different turning techniques (stem, parallel,
carve) as well as other maneuvers such as rapid stopping, jumping,
etc. These maneuvers are movement units that can be used to
interact with the environment under different conditions or
purposes. Movement skill acquisition can be defined as the process
used by an individual to best change or maintain either their own
state, or the state of objects, in space.
[0093] These end effector motions encompass a variety of different
movement behaviors including reaching motions, such as those used
to grab an object or touch something, or interception and throwing
or hitting motions. All of these motions guide the end effector
along a path to a particular location in space. Most of the
reaching motions involve stationary end conditions. Interception
and hitting involve more dynamic end conditions. Most skillful
end-effector motions involve the precise control of its state at
various instances or phases of the movement (apex, contact,
interception, or throw) (see FIG. 5).
[0094] Reaching or intercepting motions rely heavily on visual
information. The output side of behavior, i.e., the control of
motion, therefore only describes part of the problem. The input
side of the behavior, which encompasses the sensory and perceptual
mechanisms, contributes to a complete understanding. These
movements are in part driven by motor program and functional
aspects such as the adaptation of the program to external task
elements or dynamics represents a fundamental aspect of the skill
acquisition. Goal-directed movements, such as in swing sports, are
organized around what can be considered a goal state. In tennis,
for example, the racket stroke motion is organized around the ball
interception or impact. However, because the movement has to
satisfy the constraints of the ball impact and the body and limb
biomechanics, it is achieved through a complex coordinated pattern
of motions. While the forward swing and impact phases are the most
critical, these ancillary phases are required to create the best
impact conditions needed to control the ball and also to adapt to
the dynamic conditions of the task.
[0095] In other activities such as skiing, individual movements
don't have such an explicit goal. Skiers use gravitational forces
and body biomechanics to generate a turning motion to steer and
control their path. These coordinated movements represent the
primary unit of motion. While they may not be a distinct goal state
such as in tennis or other swing sports, they often have a movement
phase such as the apex of a turn, which together with the local
environment interaction determine the primary outcome of the
movement pattern. Skilled human movements, such as the tennis
stroke, involve the sequencing of complex coordinated motions that
are executed based on internal states and external cues. Their
successful performance involves managing a range of contributions,
including the effects of the tool or equipment (e.g., the tennis
racket), the movement biomechanics, the interactions associated
with the activity (e.g., tennis ball impact), and the interactions
with the environment (e.g., aerodynamics or other medium) (see
FIGS. 2, 7, and 9).
[0096] For a detailed and comprehensive assessment of the
acquisition of movement skill, sufficient data from a description
of the movement interactions with the larger task and environment
elements may be required. Tracking and analyzing movement skills
has long relied on visual techniques. Using these techniques means
tediously observing video footage. Limitations in systematic
data-driven skill evaluation and modeling are due to various
complexities relating to the fundamental nature of complex
movements and other task environment characteristics that were
already discussed.
General Challenges and Requirements
[0097] Given the depth of hierarchical levels of the movement
system, the scope of motion analysis can encompass multiple levels.
For example, it could focus on low-level neuro-motor aspects, the
movement technique and structure, the optimization of outcomes, all
the way up to tactical and strategic levels (see FIG. 31). The
range of motion sensors, available either embedded or deployed in
the environment, can provide measurements of broad aspects of the
movement dynamics surrounding the users, actors and their
equipment. However, data alone is not sufficient to produce useful
and actionable insights.
[0098] Detailed and comprehensive analysis of movement skill, in
particular for open motor skills, has not been accomplished,
especially for a larger population, because of both the practical
issues of getting measurements, and the perceived complexities in
analysis and assessments. In tennis, for example, comprehensive
analysis has to consider the stroke motion as part of the larger
system of coordination and interactions that include the ball
trajectory, the footwork, going all the way to court motion, the
game tactics, etc. (see FIG. 7)
[0099] Therefore, one of the general challenges in data-driven
movement skill analysis has been the definition of a basic unit of
analysis that provides a meaningful level of skill characterization
and can be scaled to enable a more comprehensive scope of analysis
for a single individual, and also generalizes across a population
of performers. Basic analysis of the stroke motion usually focuses
on the racket trajectory (i.e. end effector or equipment). Since
that trajectory is the result of a kinematic chain that involves
the upper body and the driving motion that starts from the feet,
legs, and hips, by capturing the overall stroke pattern and its
movement phases it is possible to infer deeper relationships
between the larger biomechanical system and the end effector
motion. As more measurements are available to track the various
task elements and body segments, a more accurate and complete
description of movement performance can be achieved (see e.g.,
motion tracking cameras or distributed motion sensors on the
performers in FIG. 2). Ultimately, the depth of analysis depends on
the available measurements, however, capturing the movement phase
structure of the end effector motion that is used to produce the
primary outcome in a task (tennis racket, or ski) can already
provide comprehensive analysis and training interventions.
[0100] Another challenge in movement assessment and diagnostics is
that of variability in performance. Viewed through direct
observation, there is typically significant variability in human
performance on repeated trials, making it difficult to apply
quantitative models that describe an individual's technique and
skill both comprehensibly and in details. In addition, because of
individual differences in anatomy, style, fitness, and skill level,
movement produced by different people targeting the same general
outcome may turn out to be quite different. Therefore, it can be
helpful to be able to capture a user's unique elements and
features, and be able to continuously adapt the training method to
the user's evolving skill.
[0101] The differences between individuals manifest both in their
overall patterns and in their movement phase structure. The phase
structure, as already discussed however, depends on biomechanical
constraints, which are determined by individual characteristics
such as body type, physical strength, and motor coordination, and
therefore provides a more detailed understanding of an individual's
movement. For example, for a beginning tennis player, a forward
stroke will be a rudimentary movement including a forward swinging
motion implemented from the shoulder joint. Over the course of
skill acquisition and development, the brain will learn to better
take advantage of their physical potential, range of coordination
of their body segments, and other movement system components (FIG.
5).
[0102] Skilled behavior relies on organized strategies and builds
on the well-defined hierarchical organization of neurological
processes. The instances of observed movements belong to specific
classes of movement patterns that are used to support interactions
needed for a task performance. Therefore, capturing movements and
aggregating them within classes provides a solution to systematic
analysis even in the face of variations. These classes of movement
correspond to the movement units.
[0103] Therefore, to enable the systematic data-driven training
process, going from the skill assessment, to the diagnosis of skill
deficiencies, to defining training goals and protocols, and the
synthesis of various forms of feedback to help address those
deficiencies, it can be helpful to define a comprehensive modeling
language that captures the structure and organization of movement
and is grounded on the fundamental principles of human movement
science.
[0104] Following the example from natural language processing,
conceptually, the core technology focuses on decoding movement data
to extract relevant movement elements that can be used for skill
analysis. The relevant elements in natural speech processing are
the units of organization of speech production, known as phonemes.
The decoded phonemes can then be used to identify words and
eventually the meaning of a sound bite. To help extract movement
units that are useful for skill analysis and diagnosis of an
individual's movement technique, these units, similar to speech,
have to be related to the process used for movement production. The
result of this type of analysis can then be more readily translated
into instructions and used to synthesize augmentation systems.
[0105] In parallel, to the analytical questions, a data-driven
skill augmentation environment requires a system infrastructure to
operationalize the various processes. The basis of the
infrastructure is a data structure derived from the movement units
that support efficient handling, processing, tracking, and managing
of motion skill data. In addition, the data structure allows
codification of skill components and their functional
characteristics to design feedback mechanisms that target precise
aspects of the movement skill performance and learning.
[0106] The proposed modeling language and skill model and
accompanying technology infrastructure can accommodate the nuances
that naturally occur in human performance, and build on the
structural features inherent to the human movement system and its
various functional and learning mechanisms. Moreover, the methods
capture both the global skill components that give users its
versatile performance in an activity domain, and the specific skill
components needed for performance and adaptation to the specific
task elements and conditions. And finally, it can be generalized to
different activities and scaled to larger populations.
[0107] FIG. 4 shows examples of movement architecture for the
primary movement unit for other movement activities (tennis 441,
golf 442, rehabilitation 443, skiing 444, running 445, and swimming
446). The drawings also highlight the movement phases and the
primary outcome.
Motor Learning
[0108] Behaviors are produced through a process of selection of a
response (movement behavior), which is typically based on the
observable environment state. A successful outcome of a behavior
therefore depends on both the correct selection of the behavior
type and its correct execution. Learning is defined as a change in
behavior that results from experience. Learning is typically
improved through feedbacks that reinforce correct behavior (Law of
Effect).
[0109] Classic motor learning theory proposes that subjects have a
repertoire of responses, some are rewarded and hence strengthened,
increasing their probability of reoccurring (see Thorndike in Adams
1987). As a result of this process, the subjects develop and refine
their repertoire of behaviors. More recent theories have
investigated how movement pattern learning can be explained through
neuro-plasticity. For example, the Theory of Neuronal Group
Selection (Edelman, 1987) posits that the brain selectively
reinforces the formation of patterns based on how adaptive the
movement is given the prevailing environment or task conditions and
constraints. Patterns that best support the task at hand are
reinforced while unsuccessful ones are discarded. A movement that
has a positive adaptive value will be reused more frequently.
Through reuse, the pattern will be refined according to its
adaptive value.
[0110] The learning process therefore depends on availability of
signals that inform the subject of the success of its movement
behavior. Moreover, for complex behavior, information about the
outcome alone, or so-called knowledge of result, may not be
sufficient. For complex movements, it can be helpful to combine an
understanding of the movement technique--i.e., cognitive
level--with feedback on specific aspects of that technique during
and/or after the execution.
Augmented Skill Ecosystem
[0111] The data-driven training system builds on the augmented
skill ecosystem that was previously described in U.S. Patent
Application Publication No. 2017/0061817, which is hereby
incorporated by reference in its entirety.
[0112] FIG. 21 illustrates a data-driven closed-loop training
system including its primary processes organized according to three
primary feedback loops. The assessment loop 200 is configurable to
have five components. An extractor 201 extracts motion elements
from a target motion. The extracted motion elements can be directed
from an augmentation loop 202 which collects information from user
training or play. The augmentation loop 202 can have a feedback
loop between a movement process 222 and a cueing system 224.
Additionally, the augmentation loop 202 can receive information
from an instruction module 203. The instruction module 203 may
receive a set of target skills 204 from a user or a trainer.
Session data 226 can be provided to the extractor 201. The
extractor 201 output generates a motion model 205 which can then be
used for skill assessment and diagnostics 206 based on reference
skill data 207. A measurement process can be provided that maps
aspects of behavior or movement into one or more measurement
signals.
[0113] The system operationalizes the training process and creates
a systematic schedule that builds skills in following logical
development, consistent with human learning principles. The
training starts from a user's existing motor skills and proceeds by
shaping these skills towards the specified goal skills.
[0114] The Assessment Loop (AL) corresponds to the process of data
acquisition and processing associated with modeling subjects'
movement technique and skills, skill diagnosis, and the
organization of knowledge, for example in training lists and
training schedules/plans, as well as the synthesis of augmentation
laws. The Training Loop (TL) corresponds to interactions associated
with the management and organization of the training activity,
including reviewing the skill status, learning about the movement
technique, selecting training elements and goals for the session,
scheduling a training or performance session, and finally tracking
the progress of the training process. The inner most loop is the
Feedback Augmentation Loop (FL), which corresponds to actual
performance of the movement activity and includes the effect of
feedback cues communicated to the subject during the
performance.
[0115] The cueing system 124 can include two components: a cue
processor and a cue generator. The cue processor translates
movement data into cue signals. The cue processor implements a
finite state estimator and a cueing law calculator. The
finite-state estimator is an approximation of the user's movement
model (which is itself represented as a finite-state machine). The
cue generator translates cue signals into physical stimuli; the
system operates in real-time to provide feedback as the user
participates in an activity. The cueing law calculator takes the
state estimate and the motion data and operates on them to
calculate if a cue will be delivered and what the cue should
communicate. The feedback synthesis model determines how the cueing
law calculator operates, whereas the finite-state estimator is
defined by the user's current movement model. The cue generator
takes the cue signal and translates it into feedback stimuli
generated by a transducer (audio, visual, haptic, symbolic, or
other type). The form of transducer is determined by the platform
implementation details, user characteristics, equipment parameters,
environment status, and/or other concerns.
[0116] The system receives input from a user's physical movement
that takes place during a use or play session. The measurements can
capture a range of movement behavior that was performed to complete
the activity (e.g., all the motion associated with a tennis stroke,
all the motion associated with a golf swing, etc.), associated task
conditions, as well as the elements relevant to the broader
functional components such as perception of task elements.
DETAILED DESCRIPTION
[0117] FIG. 1 illustrates one embodiment of a human augmentation
system 101 applied to movement skill training or rehabilitation.
The system in this example, combines existing devices such as a
smart phone 102, a smart watch 103, or other processor in wired or
wireless communication with a motion tracking device 104 attached
to or embedded in the tennis racket 105. The device 104 streams
motion measurements to the smart watch 102 and/or phone 103 or
other processor. Motion measurements are typically obtained from
MEMS IMUs (e.g., available from ST Microelectronics and
InvenSense), which usually include 6-axes acceleration and angular
rates and 3-axes magnetometers, which are often used to estimate
absolute orientation in space (Attitude and Heading Reference
System or AHRS).
[0118] As described in U.S. Patent Application Publication No.
2017/0061817, the motion data is processed at different levels in
this system to render useful information for the subject's training
or rehabilitation. The processing is distributed across typical
internet of things (IoT) components, such as the
wearable/embeddable devices, smart devices and cloud
infrastructure. The segregation of these processes depend on the
temporal requirements, such as acceptable delays or latencies, the
required computational capacity, the availability of data, such as
subjects' history and even larger population data and meta-date.
Other factors include the streaming bandwidth and power
requirements. All of these factors combine to determine the best
network topology, data structure and management, as well as
hardware selection.
[0119] To render useful information from collected movement
measurement data collected, structural characteristics are
identified that can then be related to particular motor events or
actions. For computational analysis of technique and skills and
ultimately synthesis of effective feedback for training
instructions, it can be helpful to break down movement into
movement elements (see FIGS. 3A-5).
[0120] Movement characteristics can be represented as geometrical
and topological properties, which can be related to specific
aspects of movement organization and skill. For example, movement
characteristics can be observed in movement phase portraits such as
that of the racket angular rate. Ensembles of movement data can be
analyzed for patterns (e.g., using principle component analysis,
phase-space analysis, and nonlinear time series analysis techniques
such as state-space embedding). In addition, machine learning
techniques can be applied to analyze the distribution of features
and characteristics of the movement, as well as to aggregate and
classify the data to determine patterns which in turn can be used
to determine a deeper organization of the overall system. Given the
variety of movement types and the variability in human performance,
typically, the system is configurable to distinguish between
different movement types before proceeding to deeper analysis of
any individual movement or component thereof.
[0121] As shown in FIG. 2, which will be described in greater
details later, one or more motion sensors, either embedded or
deployed in the user's environment, can be used with the system to
provide measurements of movement dynamics encompassing one or more
users, actors, and their associated equipment (if any). As will be
appreciated by those skilled in the art, given the depth of
hierarchical levels of the movement system, the scope of motion
analysis can be conducted at multiple levels. For example, it could
focus on neuro-motor aspects, movement technique and structure, the
specific outcomes of these movements, all the way up to tactical
and strategic levels that describe how these movements are deployed
in a task (see FIG. 31). The illustration in FIG. 2 delineates
between different categories of measured or captured quantities.
The output side (measurements and observations) includes behavioral
quantities (movement such as the end effector, body segments;
visual attentions; muscle activation); task and environment
elements and objects. On the input side are motion tracking cameras
70, Gaze Tracking/AR device 80, and other sensor input.
[0122] Analysis of the intrinsic movement structure of the movement
technique and functional characteristics can be used for skill
analysis. This analysis can be formalized by focusing on the
interactions of the movement with the environment and task
elements. Operators or agents such as a tennis player organize
their behavior in relationship to environment and task
elements.
[0123] The resulting organization of the behavior combines the
effects associated with the natural organization of the human
movement system and the structure of the task and environment. FIG.
7 shows the different outcome levels, using tennis as an example,
and some of the outcome measurements and FIG. 8 shows how these
interactions produce the repertoire of strokes and their associated
shot distributions. A particular distinctive characteristic of
human behavior, which contrasts with robots and other engineered
system, is that human behavior can be considered relational, i.e.,
movement behavior is produced through the action-perception loop
and therefore is often anchored in a particular environment
features and elements. In tennis for example, the stroke, which can
be considered the primary movement unit, is directed at specific
target areas in the court environment.
[0124] The specific human court environment perception, and the
associated movement interactions, that can be formalized in terms
of the tennis stroke and their associated shots (see FIG. 7),
result in a specific discretization of the task environment as
shown in FIG. 8. The characteristics of this discretization depend
on the movement skills and the underlying motor, perceptual, and
cognitive processes. For example, beginning players, because of
their lack of control of the ball, may be able to consider only a
very large target area such as the entire opponent court half. As
the player improve, their perception of the environment and
associated movements become more precise and therefore lead to a
larger repertoire that spans the task environment with higher
resolution and thus allows higher task performance.
[0125] FIG. 2 illustrates an exemplar augmented activity for
tennis. The primary interaction is the tennis stroke, driving the
tennis racket 20 toward a ball impact 30. The activity environment
elements include the tennis court environment 50, with a net 52, as
well as marking on the court 51. One or more motion tracking
cameras 70 and/or other acoustic or RF motion sensors 90, can be
used to track the subject's motion on the court environment 50,
including the details of the individual body segments 15, the ball
30 and the racket 20. Other measurements can include the subject's
visual gaze 81, which direction changes depending on the focus of
visual attention, when tracking different visual cues, including
the ball's ground impact 32, or net crossing 31 as well as desired
court placement. The apparatus 40 shown in the same figure can be
programmed to enable different forms of interactions. In one tennis
example, the apparatus 40 is a ball machine that can be programmed
to support the development of specific stroke patterns and
therefore can be programmed in conjunction with the cueing
system.
Augmented Movement Performance
[0126] The systems and devices disclosed herein augment movement
skills at several levels, for example: 1) providing users feedback
for training, including providing signals during the performance;
2) enhancing the athletic experience during performance to help
focus; 3) providing protection from injury by helping users engage
in optimal techniques; and 4) developing training protocols which
are directed to developing skills related to the training.
[0127] Patterning characteristics are expected in many movement
activities. In tennis, for example, the same general stroke pattern
can be used to generate different amounts of top spin or pace.
However, to maximize these different outcomes, distinct patterns
have to be formed to fully exploit the biological capabilities. For
example, a stroke for a top spin or slice has characteristic
features in the temporal and spatial arrangement of movement
phases. Movement patterning is due to how changes in movement
outcomes or task conditions affect movement technique within a
particular operating region of the state-space. As the desired
outcome or task conditions change beyond a certain threshold, the
biomechanics and motor-control organize differently to best take
advantage of the system's capabilities. From a trajectory
optimization perspective, the changes in outcome and condition
alter the system's "operating point" and result in activation of a
different set of constraints. Due to the nonlinearity, this leads
to the emergence of a different motion pattern with distinct
dynamic characteristics. Patterning corresponds to a tendency for
the trajectories in each movement class of behavior to stay close
together in spatial and temporal terms. This closeness can be
described formally using techniques from nonlinear time series
analysis. Using these techniques, measurement data describing
racket state trajectories during a tennis stroke can be aggregated
and clustered to identify different stroke patterns, and
subsequently analyzed to determine their functional properties and
characteristics.
[0128] Such performance data for an activity taken in its totality,
for example from measurements of an entire tennis match, results in
a repertoire of distinct movement patterns. This repertoire of
distinct movement patterns is the result of the optimization of
movements technique, i.e., achieving the range of outcomes and
conditions required to be proficient in the particular activity.
For instance, in tennis an individual will develop a repertoire of
different strokes to optimize the desired outcomes (e.g., type and
amount of spin, strength, etc.) and accommodate the range of impact
conditions (ball height, speed, etc.; see FIG. 9). This repertoire
essentially plays the role of a vocabulary of motion pattern that
an individual can call upon when engaged in a particular activity.
For example, FIG. 8 illustrates distributions of shots associated
with different strokes and the effect of skills on the accuracy and
granularity of the discretization of the task environment, which in
this case is the tennis court.
[0129] The movement patterning and organization in repertoire,
therefore, have implications for the assessment of skills. The
skills of a particular tennis player, for example, can be assessed
by: 1) extracting characteristics about the entire repertoire of
strokes, e.g., how well they collectively achieve the range of
outcomes and conditions in the activity domain, 2) determining how
well and how consistently each class of strokes in the repertoire
achieves associated outcomes, and 3) determining how well the
strokes adapt to the impact conditions. The first analysis provides
a comprehensive assessment, and the last two emphasize the
technical implementation of the motion skills. Understanding human
movement from this analysis provides a deeper assessment and
diagnostic of the movement technique, that can be used to specify
training goals and various feedbacks to help correct and optimize
movement technique.
Improving Movement Learning
[0130] The following disclosure addresses the general question of
how to improve movement learning using information technology,
machine learning, and wearable devices. The disclosure also
addresses specific questions including how to formulate training
goals; how to manage the larger training process, in particular how
to break up larger training goals into a sequence of goals; and how
to dynamically update these goals based on data from the training
activity such as skill acquisition stage and trends. In addition,
the system determines what type of feedback to use to augment the
experience and accelerate the learning process, when to present the
feedback, how to determine the best type of feedback given the
learning stage, and how to distinguish between different skill
elements.
[0131] Furthermore, the disclosure also addresses how to best
represent information to augment a subject's training experience.
The resulting system takes into account what is learned by the
subject as they make progress in an activity domain, what aspects
of behavior to emphasize depending on learning stage, and also
accounts for the characteristics of human information processing to
provide feedback and information that can be processed and
assimilated efficiently.
[0132] The central requirement for deliberate training is the
specification of training goals and management of the training
process using these goals. These processes are usually handled by
human coaches or physical therapists. The contributions of this
disclosure are the algorithms and system that enable training to be
operationalized following a computational, data-driven process. The
disclosure addresses two central capabilities: the computation of
training goals, and scheduling and management of the training
process.
[0133] The general approach is to use movement data to assess skill
and identify deficiencies, followed by specification of training
goals to address these deficiencies. Regarding training process
management, the general approach is: i) leveraging the natural
structure and organization of the human skill learning process; ii)
using information from both individual subjects as well as from a
larger population to extract knowledge to guide that process while
accounting for individual characteristics.
[0134] The structure of the skill acquisition processes refers to
the type of changes taking place over time as a result of activity
(training or experience), which manifest as sequence of learning
patterns characterized by specific changes in movement skill
attributes and task performance. By applying population thinking,
i.e., considering the skill acquisition process across the diverse
group of subjects with different skill levels and movement
technique, as well as accounting for the wide range of factors that
affect this process, it is possible to extract knowledge about the
larger skill acquisition process, which in turn can be used to
guide training or rehabilitation.
[0135] Both of these goals require that skill be treated as an
explicit concept that can be expressed quantitatively, e.g.,
decomposed into skill elements, that can be computed from
performance data. Furthermore, this language of skill modeling
should be applicable to a diverse population of performers so that
spatiotemporal relationship in skill quantities can be extracted
across the same as well as different individuals. And finally, this
language should be valid across different forms of movement
activities.
[0136] To enable these goals, the skill development process is
formalized in terms of the hierarchical movement model detailed in
U.S. Patent Application Publication No. 2017/0061817. Humans become
proficient in a task or activity by developing a repertoire of
movement patterns needed to interact with the task and environment
elements involved in the overall goal of the task or activity. FIG.
11 illustrates the development of movement patterns over time. It
is expressed as the differentiation of existing patterns as well as
the formation of new patterns.
[0137] The model encompasses the repertoire of movement patterns,
and the movement structure associated with the movement patterns
used in the interactions with the task of environment. The specific
Movement Functional Structure (MFS) also makes it possible to
extract the wide range of movement skill attributes across the
levels of organization of the movement system and the task
structure.
[0138] Movement patterns that correspond to the primary movement
units are typically associated with primary interactions found in
an activity, some of those interactions produce specific outcomes
on the environment or task elements, and hence can be characterized
by their range of outcome and operating conditions. Therefore, the
motion patterns associated with these primary movement units can be
considered as the basic unit of skill, or skill element.
[0139] FIG. 6 gives an overview of the movement processing starting
from the extraction of movement units, their classification, the
movement model for each classes, following with the skill model
that is used to determine relevant skill attributes used in the
skill assessment and diagnostics. The figure also shows how these
skill elements are then aggregated to produce the repertoire, which
provides the basis for a subject's skill profile that can then be
used for the analysis of the skill development (learning curve) and
the population analysis.
[0140] The identification of these patterns in association with a
skill development, and their delineation over the longitudinal
acquisition process, makes it possible to relate the relevant
movement skill attributes across the larger population; which in
turn enables the systematic organization and management of the
training process.
[0141] The quantitative definition of a unit of skill also provides
the foundations to proceduralize training under an iterative
learning scheme, which specifies how skill assessment, diagnostics
and training goals are computed and updated over time. The system
also incorporates the movement performance augmentations defined in
U.S. Patent Application Publication No. 2017/0061817 (FIGS. 22 and
23) that are used to help induce changes in movement technique.
[0142] The central concepts needed for the realization of such a
training agent system are reviewed next.
Skill Elements and Skill Profile
[0143] The first capability includes the precise and comprehensive
assessment of an individual's movement skills, and more generally,
data-driven training includes tracking various attributes of these
skill elements. Using motion patterns as unit of skill enables the
formulation of quantifiable, incremental change in movement
technique, and its associated effect on measurable outcomes, as a
result of experience or training. The sum of all changes in skill
elements also ultimately produce incremental changes in some
overall skill level that captures the larger impact of skill on the
activity or task performance.
[0144] The skill element in the skill model represents the basic
unit of skill acquisition. It is defined as the primary outcome
associated with a particular class of movement pattern, and the
associated attributes, that describe the relevant movement
characteristics. These skill elements are derived from the movement
system hierarchy specified in U.S. Patent Application Publication
No. 2017/0061817. They encompass: (a) the repertoire of movement
pattern classes, where each class is described by a movement
pattern which is decomposed into phases; (b) the movement phases,
which are the manifestation of the movement functional structure
determined by the biomechanical constraints and other constraints
arising from the properties of the environment interactions.
[0145] The skill elements can be combined to form a subject's
comprehensive skill profile, which captures skill attributes
associated with the skill elements. An individual's skill profile
can be precisely and comprehensively characterized by the skill
element attributes that can be derived from the hierarchical
movement model and the functional structures underlying all
movement patterns in a repertoire used in a domain of activity.
[0146] The present disclosure extends this movement pattern
functional analysis and assessment, covered in U.S. Patent
Application Publication No. 2017/0061817 to encompass the
task-level performance, which is based on the fact that movement
pattern classes support the interactions needed to perform the
particular task or activity. Task performance metrics can be
computed from attributes of the repertoire of movement patterns.
Simple metrics can, for example, be determined from the use
frequency of the various movement patterns.
[0147] More detailed models for higher-level assessment can be
determined from the temporal sequence of movement patterns.
Spatiotemporal patterns at the level of the repertoire, i.e., what
movement patterns are used where and when, also enable the
description of the high-level decision-making processes associated
with planning and strategy which represent cognitive functions.
This extended task performance analysis provides tools to compare
players or performers, i.e., support the analysis competitive level
performance. They can also be extended using population analysis
(see concept of player profile).
[0148] Together, these elements make it possible to assess and
diagnose some of the subject's higher-level functions, including
the perceptual mechanisms, attention, and decision making. These
quantities enable a comprehensive and precise quantification of
skills, and therefore provide the basis for the computational
framework to drive training at different levels of the movement
system and task structure organization. For example, target
reference values for the various parameters of the skill model (see
Target Skills in FIG. 1) can be used to drive skill or performance
attributes at different levels from features of the movement
technique used to optimize outcomes to higher-level attributes such
as success rates of tennis shots in specific areas of the tennis
court.
Training Goals
[0149] To drive the training or rehabilitation process and enable
quantitative data-driven training, it can be helpful to specify
training goals. Training goals are a quantitative specification of
a subject's target changes in the movement that will produce the
desired increment in skill level. The training goal targets
actionable characteristics in movement technique and therefore
represents the drivers to achieve the larger skill level
targets.
[0150] The goals typically combine expected changes in movement
outcomes with the associated movement characteristic (functional
element). To produce effective drivers for training, the training
goals can be augmented by a range of instructions and feedback cues
as defined in U.S. Patent Application Publication No. 2017/0061817,
which can encompass different components of the information
processing levels to best target the various attributes of the
movement functional model.
[0151] To be useful, training goals should be: actionable,
sufficiently broad in scope, effective, and realistic. By
fulfilling these requirements, training goals enable subjects to
train deliberately and achieve predictable, quantifiable changes in
technique that result in improvements in skill level, relative to
the existing skill level, but also provide a path for the long-term
development of skills needed to attain the desired level of
proficiency.
[0152] To be actionable, the training goals have to represent
explicit changes in movement technique (and associated visual,
perceptual, etc. processes). This is achieved by building on the
movement and skill model just described.
[0153] To have sufficient scope, the training goals have to
encompass the various characteristics in movement behavior engaged
while operating in a particular activity or task. This is achieved
by comprehensive assessment enabled by the hierarchical model and
the movement functional structure.
[0154] To be effective, the training goals have to provide
actionable milestones that lead to an incremental improvement in
skill towards the next tier, and are aligned with the larger
developmental or skill acquisition path. This is achieved by
accounting for the larger skill development process.
[0155] Finally, to be realistic, the goals have to build on the
subject's current skill level and the individual conditions (e.g.,
constraints that are imposed by health, age, physical fitness,
etc.). This is achieved by accounting for the subject's specific
location within the global training or rehabilitation path and
specifying the training goals as precise incremental changes in
existing skill attributes.
Computation of Training Goals
[0156] The present disclosure describes how training goals are
identified and subsequently specified. The training goals are
specified as target values of skill element attributes. The target
skill values used to formulate training goals are computed from the
individual's performance data, and extended by population data. The
general approach is based on a statistical model describing the
individual's skill elements and skill profile.
[0157] The variability intrinsic in performance naturally results
in a range of values for these attributes. This statistical model
provides the basis for the individual's skill analysis (see FIG.
19). The movement diagnostic is performed through the inference of
relationships between specific movement technique features and
selected outcomes relevant to the task performance.
[0158] FIG. 19 shows the distribution for some example skill
attributes. Skill levels are captured through some objective
function which is shown in terms of its level lines (shown here as
low, medium, high, very high). The information specifies the
direction the attributes have to be changed to achieve a higher
skill level. The tiers can be derived from the individual's data or
the data obtained from a larger population.
Skill Acquisition Process and Training Process Management
[0159] The present disclosure further describes the computational
framework needed to determine training goals and manage the
training process. The framework is based on a skill development or
acquisition process model and, as already discussed, builds on the
movement and skill model elaborated in U.S. Patent Application
Publication No. 2017/0061817.
[0160] This training process model accounts for the development of
the skill as the acquisition of a repertoire of skill elements.
This process extends over larger periods of time and is influenced
by a broad range of factors. Characterizing the skill development
as a sequence of formations of movement patterns, i.e., skill
elements, it is possible to analyze the acquisition process, and
actually apply the gained knowledge to optimize an individual's
skill acquisition process.
[0161] The present disclosure extends the skill model to account
for the skill acquisition process. This process is formalized as
series of transformations in movement technique, which describe,
the longitudinal development or acquisition stages for each skill
element (characterizing the brain's and motor system's natural
learning process for the formation and consolidation of the
movement patterns), and how these manifest into the movement
functional structure, and overall skill profile. Typical learning
processes are described by learning curves. However, these don't
capture the details of structural changes associated with learning
complex movements.
[0162] Based on the proposed model, as an individual becomes more
proficient they can i) achieve more optimal behavior within an
existing MFS; or more radically, ii) develop a new functional
structure that better exploits the biomechanical capabilities and
other supporting processes needed for the interactions in the task.
The movement functional structure therefore provides the
characteristic that helps delineate between stages in skill
development, and also provides the basis to relate different
performers or subjects.
[0163] The overall goal is to evolve a subject's MFS along the
larger skill development process following stages that are best
suited for an individual, and their overall performance or skill
goals. The latter depends on a broad range of factors, including
desire/motivation, needs (e.g., for professionals), as well as the
various individual factors that are determined by biological and
health conditions.
[0164] The specific sequence of acquisition can, on the one hand,
be determined by the task requirements, specified by interactions
(outcomes and conditions) that can be relevant to the performance
of the task, and on the other hand, the individual's factors that
determine what is feasible given, for example, the current skill
level, the neuro-motor and physical factors involved in the
development of coordination. Learning process can be characterized
in terms of the skill acquisition stage, which provides the
information to determine the best type of intervention, drivers,
and activity to pursue the training goal.
[0165] Population data is able to capture the larger set of factors
and therefore provides useful information to help orient and
schedule this process, and at the same time account for these
individual factors, i.e., how different body types, injuries, or
health conditions affect the skill acquisition process, skill
profile, and overall performance.
Population Analysis
[0166] The details of the larger skill acquisition process are
determined based on movement data collected from a population of
performers. The population data provides understanding about the
global characteristics of the movement skill acquisition that
emerges when taking into account the broad range of factors
expressed in the population that affect this process. In essence,
it makes the extracted information actionable by contextualizing
it.
[0167] The general idea, therefore, is that learned global
population characteristics can help support individualization of
training and rehabilitation. The individualization is supported by
providing reference data that relates an individual's skill
attributes and skill profile to the larger population. This data
provides both local reference about the skill attributes, e.g., how
much specific attributes have to be improved to gain in skill
level. It also provides more global reference about the
longitudinal skill development from that local skill status, e.g.,
what aspects of the movement skills have to be optimized and in
what order to produce favorable long-term development (e.g., faster
progress in skill level and lower incidence of injuries).
Therefore, information extracted from a larger population can help
direct both the local performance and the more global, long-term
training process (what aspects to focus on first, etc.).
[0168] The present disclosure also details a computational model of
the skill assessment and diagnostics specific for population
analysis and the extended task performance level. The population
analysis builds on the skill profile derived from the movement
hierarchical model in U.S. Patent Application Publication No.
2017/0061817. The skill analysis from the population perspective is
defined under the concept of a player profile which describes the
skill attributes in the context of a larger population to capture
the type of player based on the type or style of movement
technique. The player profile can also encompass the higher-level
characteristics such as game strategy that captures how the
movement patterns are utilized or exploited in the settings of a
task or activity.
[0169] FIG. 30 describes the assessment and diagnostic process
incorporating the player profile which is applied across the
movement system hierarchy (see FIG. 6). The main components are:
(a) determination of the movement classes in the repertoire that
are in formation, consolidation, or optimization stage (based on
the individual's skill acquisition stage); (b) determination of how
these patterns are used in the performance of the task (e.g., based
on use frequency and game or performance strategy); (c)
identification of which aspects of the skill element needs to
improved, e.g., the quality of the primary outcomes, which can be
achieved by interventions at different levels of the motor control
hierarchy, from task level attention to deeper movement technique
(based on attributes).
[0170] The skill analysis incorporating the player profile,
enhanced by the reference values derived from the population
analysis, makes it possible to account for a broad set of factors
needed to support individualized training.
Training Agent System and Process Organization
[0171] Finally, the present disclosure also delves deeper into the
system architecture that supports data-driven augmented training.
In particular, the delineation between the different modalities of
augmentation (instructions, cues, apparatus), their deployment
across the human information processing system, and the data and
information management infrastructure.
[0172] FIG. 22 depicts the main elements of the augmentation system
architecture delineating the augmented activity (with feedback
cueing and/or apparatus interactions), the human system
augmentation loop (with communications and UI systems), and the
training management and configuration loop driven by the training
agent system (not shown, which performs the modeling, assessment,
and diagnostics to identify training elements that can be activated
as training goals). FIG. 22 also highlights the primary tiers of
augmentation that leverage human's natural information processing
levels.
[0173] These include "cognitive level" information, which is
communicated symbolically, verbally or visually (here as
instructions or notifications provided by a visual UI or natural
language such as a smart phone or smart watch, eyeglasses, etc.).
Instructions and other forms of information such as notifications
are provided by a communication system that can include a visual
display for text and graphical objects and natural language
processing system. Instructions are typically designed to help
subjects' understanding of their technique and performance.
[0174] The "feedback cue level" describes information communicated
via some cueing system (here an audible signal) but can also
include visual or haptic systems. And the "signal level," which
includes both cueing signals and activity interactions (e.g., ball
machine) that are typically delivered concurrently with the
movement performance.
[0175] FIG. 23 illustrates the augmented human performance
associated with the feedback cueing system emphasizing the
low-level signal and cues within a typical perception-action loop.
The movement data is processed in real-time by the cueing system to
compute cue stimuli designed to help performer improve a specific
aspect of the movement e.g., by acting as reinforcement signal.
[0176] Note that cueing can also be used to help focus attention to
relevant elements of the task environment including, for example,
the location of a task object (tennis ball) or features of that
object (ball trajectory) or features of the adversary's movement
that can be used to anticipate the result of the adversary's
movement. Anticipatory information can for example help the subject
select the movement pattern. Finally feedback cues also include cue
signals that can be used by the subject to time the execution of
the movement.
[0177] The various forms of feedback augmentations are computed by
algorithms that have been synthesized based on the subject movement
and skill model for the current epoch and historical records, and
can also include reference data from larger population.
[0178] The training process is formalized within a computational
framework with similarities to iterative learning. The framework
describes the management of data sets used to support skill
assessment and diagnostics, which include motion and skill model
(skill profile and the player profile), and the training goals and
synthesis of augmentations (instructions and cueing laws).
[0179] The data management process encompasses: i) the creation of
data sets, models, baselines; ii) tracking their validity; iii) and
updating these quantities to support an effective augmented
training process. FIG. 49 illustrates the top-level logic diagram
of this process and FIG. 25 illustrates the process flow diagram,
highlighting the activity across the stack of processes of the
assessment and training loop (data acquisition and processing,
motion model, skill model, training goals, augmentation laws) as a
function of time. During each session, activity data is collected
and processed.
[0180] As shown in FIG. 25, the motion model, skill model, training
goals, and augmentation laws are typically updated based on their
validity with respect to the new session data. Note that the
processing stack for the feedback augmentation (cueing system)
isn't shown here. For example at n-3 a full update is implemented
following changes in motion architecture. At session n-2 the motion
model is still valid and the remaining parameters don't need
updating. At n-1 the skill model is validated and progress on
training goals prompts an update in training goal and augmentation
laws. At session n the skill model is updated (skill status) and
new training goals are determined along with augmentation laws. At
session n+1 the complete update including the motion model.
[0181] The training process is delineated in sets, sessions, and
epochs. The former two are time periods needed to organize activity
and training (see FIG. 58). The epochs correspond to time periods
that correspond to the use and updating of the data sets supporting
the computation and processing of quantities. A new epoch starts
when the movement technique and performance has evolved beyond the
validity of the current models. Each epoch typically encompasses a
set of training goals for the range of movement classes in a
repertoire that will drive the next increment in skill level. The
more recent sets of movement data are used to create new motion and
skill model baselines, all the parameters used in the movement
processing algorithms (e.g., classification), and the other
algorithms supporting the computation of skill attributes, training
goals, and synthesis of various feedbacks.
[0182] Larger scale time periods beyond epochs are defined based on
patterns in the population data and typically would correspond to
the characteristics in player or performer subgroups. Transitions
between developmental stages typically involve deeper changes in
movement skills, such as the re-organization of the movement
functional structure (MFS).
[0183] The temporal structure introduced by this system, and
derived from the natural acquisition process structure--and of
course all the associated quantities--provides the basis for the
management of the training process. The structural patterns in the
acquisition process can inform how to compute trends, generate
reference data, as well as other critical capabilities of the
data-driven training system. Finally, the motion data, model, skill
profile, and all the training elements, when extracted over time
can also be used for bootstrapping recovery or rehabilitation
following injury or other causes of interruptions in training or
practice.
Human Movement Skills
[0184] Open Motor Skills and their Characteristics
[0185] Movement skill can be categorized into two primary groups.
The first, the so-called closed motor skills, involve a stable
environment where the performer initiates the action or movement.
These conditions allow selecting the best movement or action to
achieve the task objective. Closed motor skills therefore can
typically be learned and perfected in a systematic way by
identifying conditions and training movements in these
corresponding conditions.
[0186] The second, so-called open motor skills, involve a dynamic
environment with changing conditions and require responding to the
task and environmental conditions. These conditions also require a
broader range of movements and actions to adapt and achieve the
task goals. Open motor skills typically involve learning a large
repertoire of sensory-motor behaviors and associated perceptual
mechanisms, as well as planning mechanisms. The broad range in
system state and task conditions makes it difficult to understand
what movement patterns to train. The performance under dynamic
environments and conditions also makes it harder to create
meaningful training task conditions. Furthermore, it is difficult
to predict the specific range of conditions that need to be trained
because the behavior results from the dynamic interactions between
the performer and his or her environment.
[0187] Many human skills involve open systems which are
characterized by the exchanges of energy between the subject and
the environment (Davids, 2008; see, also Kugler 1982 in Davids p.
57). Skills in these systems require exploiting information both
related to the physical performance (energy flows) and the control
performance (system structure and behavior).
[0188] The neural system supporting motor control is organized
hierarchically to enable efficient encoding and programming of the
movements. A central theory in motor control is that to mitigate
the complexity associated with the large amount of degrees of
freedoms (DOF) (resulting from the numerous muscles and joints),
movement patterns take advantage of so-called muscle synergies (see
Bernstein, 1979). The synergies encode the coordination between
groups of muscles and joints and thereby reduce the DOF that need
to be controlled. They represent the functional elements of a
hierarchical and modular representation that can be efficiently
employed by the central nervous system to program and execute
complex movements.
[0189] In addition to the complexity associated with the DOF
problem, encoding and learning individual movements for every
desired outcome and possible condition would result in an
intractable amount of information to be stored. Another central
concept in motor control theory is that instead of learning
individual movement programs, humans and animals learn generalized
motor programs (GMP). These GMP specify the muscle activation
patterns for entire classes of motions. This concept was introduced
for movement control under the so-called schema theory (see
Schmidt, 1975). The GMP achieves efficiency by encoding common
movement and perceptual characteristics as a form of schema. This
provides flexibility by allowing variation in specific movement
characteristics needed to adapt to variations in conditions or
outcomes. As pointed out by Newell, it addresses the problem of
endless variability and novelty in performance (Newell 1991).
[0190] Furthermore, most movements in open motor skills are coupled
with dynamic elements of the task and environment. The resulting
combination of conditions and states dramatically increases the
complexity of learning to perform. For example, the tennis stroke
is an action directed at returning a ball that is itself moving
relative to the player and court, and the execution of a tennis
stroke also depends on the body's motion relative to the court and
the ball trajectory. Accounting for all these dimensions results in
a potentially intractably large amount of information that needs to
be extracted and encoded.
[0191] Coordination is defined over the organism and environment
interaction, not just the organism. The coupling between movement
and the environment was emphasized by the Ecological Theory of
behavior. Unfortunately, this "extended" coordination problem adds
considerable complexity. However, there are structural features
that emerge from the interactions in system components that human
and animal perceptual and control processes have evolved to
exploit, greatly simplifying the extraction of information and
internal representations needed to plan and organize behavior.
Gibson's work on visual perception demonstrated that some elements
of information involved in coordinating behavior relative to the
environment are, in fact, readily available from the visual
observations of environment (so-called direct perception, see
Gibson, 1979).
[0192] Therefore, one aspect of operating effectively in skilled
movement tasks is the automation of processes associated with
environment perception and organization of behavior to exploit the
natural structure of flow of information and behavior dynamics,
respectively. Such a strategy minimizes the amount of information
that needs to be explicitly processed from sensory signals,
information that needs to be programmed for the motor actions, and
information that needs to be stored.
Movement Patterns and Affordances
[0193] When operating in a new environment, or performing a new
task, the organism has to identify opportunities for new actions to
be learned. How are behaviors that are contributing to the task or
goals learned and/or identified? In ecological psychology, the
concept of affordances describes what the environment provides or
furnishes the animal, implying the complementarity interaction
between the animal and environment (Gibson, 1979). Based on this
affordance concept, the animal or human has to essentially learn to
recognize or perceive affordances. Therefore, learning skills can
be conceived as the process of learning to recognize affordances
and adapting behaviors to effectively exploit these
affordances.
[0194] Affordances can take a broad range of forms. They can be
static, such as chairs affording a person to sit, or dynamic, such
as stairs affording climbing. Researchers have developed and
adapted the concept of affordances to specific domains. Norman, for
example, adapted it to the domain of human computer interfaces
(HCI) where good interfaces convey action possibilities in forms
that are readily perceivable by users (Norman, 1999).
[0195] For skilled movement tasks, the affordances are specified
based on the dynamics of the agent-environment system. Such systems
are typically complex, high-dimensional nonlinear systems, with
numerous components interacting through their processes and
physical components, including body segments. Complex, nonlinear
dynamic systems are characterized by emergent behaviors (see Davids
2008 for emergent behavior in human movement). The physical system
and the muscle and sensory-motor supporting movement coordination,
along with the various processes needed to interact with the task
and environment elements, form a complex system. Therefore, the
overall evolution of the movement patterns and their properties are
emergent phenomenon.
[0196] This perspective was investigated for spatial guidance
behavior in (Mettler, 2015; see, also, Kauffman's 1993 and Van
Gelder and Port 1995 pp. 31 and 32) using the concept of
interaction patterns. Interaction patterns are agent-environment
dynamics that are exploited to achieve efficient learning and
programing of motion behavior. They have been shown to represent
behavioral invariants that satisfy properties of equivalence
relations (Kong & Mettler, 2013). Therefore, they provide an
efficient decomposition of the complex, high-dimensional
agent-environment dynamics into small sets of behaviors. Similar to
muscle synergies in body coordination, but here describing the
coordination of agent behavior relative to its environment.
[0197] These emergent interaction patterns can be exploited by
humans or animals, and provide functional capabilities needed to
achieve adaptive and robust performance in complex environments.
Besides helping with the organization of behavior, where they play
the role of unit of organization, the interaction patterns are
manifestations of the functional structure of sensory-motor
functions. Therefore, the interaction patterns also represent a
type of functional unit that helps with the organization of the
system-wide integration between different processes (control,
perception, and planning) (Mettler, 2017).
[0198] Therefore, when considering advanced open motion skills,
affordances can be formalized as emergent properties of a complex
dynamic system. The understanding of behaviors as interaction
patterns emerging from the agent-environment dynamics provides
additional insights about what is learned, and therefore helps
determine how this implicit knowledge acquired in a domain of
activity can be modeled. Sensory-motor skills condition the
interactions between the agent dynamics and the task and
environment elements, and therefore, viewed from larger perspective
they determine the affordances available to the operator or
agent.
[0199] Viewing behavior as properties of the agent-environment
dynamics suggests two components of the learning process. First,
learning involves recognizing the affordances that are enabled by a
subject's control and perceptual capabilities. Second, the subject
has to learn to exploit and further shape these available
affordances to improve the larger task performance, or more
generally, adapt to the conditions and contingencies arising in the
task environment.
[0200] And furthermore, for subjects to improve their skills in
open motor tasks, they should identify: 1) the potential for new
affordances, and 2) the opportunity of improving the quality of the
interactions. The first leads to the development of the repertoire
of behaviors, and the second leads to the refinement of movement
patterns.
[0201] When an agent approaches a movement learning task, their
existing sensory-motor capabilities determine the range of possible
affordances, i.e., unexploited, latent affordances. The agent then
should learn to identify and exploit these affordances. Once
identified, fully exploiting these latent affordances require
refining and optimizing the sensory-motor processes. The newly
acquired sensory-motor capabilities, in turn, create new
affordances. This process, therefore, describes the incremental
learning process and explains the extensive training and experience
needed to reach proficiency in a domain of activity.
Human Motor System
[0202] The human motor system has evolved to manage a variety of
movement tasks that involve interactions with environment elements,
while efficiently handling uncertainties, disturbances, and
contingencies arising during the performance. While the human
movement system has tremendous potential, systematic and dedicated
training is required for high levels of motor facility. This
requirement for training is similar in any domain of activity, such
as athletics, music, or vocational tasks. Movement task constraints
can be divided into extrinsic and intrinsic factors. Extrinsic
factors include the interactions with the environment such as the
foot strike or impact of the ball on the racket. Intrinsic factors
include the biomechanics, human motor control, and effects arising
from the manipulated equipment's dynamics. Most skilled behaviors
are so-called deliberate behaviors that are directed at achieving
specific outcomes. Therefore, learning skilled behavior in sports
or vocational activities involves learning to master these
interactions so as to achieve the desired outcomes or goals. The
coupling of the human movement system and the task environment has
to be considered as a coupled system. If the extrinsic and
intrinsic interactions were considered separately, the complexity
would be intractable.
[0203] An efficient and effective solution may be found in the use
of strategies that structure and organize movement behavior to
satisfy both the extrinsic and intrinsic factors. The brain evolves
specific organizational structures and functionalities to
efficiently work with these complexities. The brain and
sensory-motor mechanisms that optimally deal with the coupling of
the two domains, and achieve the sufficient level of adaptation,
are the result of natural selection. Movement skills represent a
critical aspect of a species fitness. As a result, the specific
structure and organization of the brain, including the nervous
system and larger biomechanical systems, support natural solutions
that enable efficient and adaptive behaviors. Therefore, a portion
of the movement system is genetically determined and are innate.
However, motor skills, especially in deliberate specialized
movement skills, are learned and perfected based on repeated
interactions within the task and environment. Finally, learning
movement skills involves changes in the cortex as a result of
neuroplasticity. These changes, however, follow a specific process
that is dictated by the organization of the various cortical
structures (cerebellum, parietal cortex, pre-motor and motor
cortices, and the prefrontal cortex). As a result, movement skills
are best acquired early in life when the brain is still
developing.
[0204] Three forms of units of behavior have been described for
complex movement behavior. At the top level, motion primitives are
related to the concept of "motor equivalence" which has been
identified as one of the fundamental characteristics of motor
behavior. The idea is that the same movement behavior can be
repeated in various contexts and without changing the overall form
of the motion. Therefore, segmentation of human movement behavior
into motion primitives has been most successful from invariant
characteristics in the performance that arise from symmetries and
equivalences in the problem space.
[0205] Furthermore, since complex movements are obtained from a
sequence of movement phases, the next level of primitive represents
the segments that can be combined sequentially to compose
movements. This is due to the brain's efficient encoding which
exploits principles of modularity. Finally, the last level of
decomposition is related to the so-called muscle synergies, which
represent the movement components that describe the parallel
combination of different muscle activations and the associated body
segments displacements. The top-level primitives are considered the
primary motion units, which support the interactions with the task
and environment elements, and the lower two levels, the movement
phases that specify the movement architecture and the synergies,
provide understanding of the functional properties in relationship
to the environment interactions and the biomechanical
constraints.
[0206] These elements of the movement system can be derived from
structural features extracted from measurements using pattern
analysis. There may be a great variety of movement patterns that
satisfy these constraints depending on the configuration and
conditions; however, they all typically share characteristic
features that enable the movement patterns to be identified and
segmented. Muscle synergies can be obtained from factorization
methods (e.g., principle component analysis or non-negative matrix
factorization). The general idea is that many movements can be
described as variations of a general model and once the general
category of movement is specified, some of the mechanisms needed to
achieve robust movement performance are those that allow adapting
to those movement pattern to changes in conditions and transferring
them to different contexts in a similar task or activity.
Learning Principles
[0207] In contrast to periodic and reflexive movements, which can
be generated from low-level motor functions, skilled movements
usually involve the deliberate expression of specific goals or
outcomes that rely on higher-level motor, perceptual and cognitive
functions. These movements may be completely self-initiated, e.g.,
picking up the phone to call someone; they may represent a stage in
the context of a larger task, e.g., opening the dish cabinet when
preparing food, or returning a tennis serve. As seen in these
examples, movements are rarely an isolated behavior but are part of
a larger set of interactions with the world and therefore are
typically part of a chain of behaviors.
[0208] Learning deliberate, skilled movements involves learning the
motor programs that determine the correct forms and sequences of
actions as well as the perceptual cues that provide information to
fine tune the movement characteristics that will enable to
successfully accomplish the intended goal or outcome (e.g.,
reaching to grab an object or imparting momentum to a ball).
Learning involves iterating on existing solutions as the task, or a
similar task, is repeated. Therefore, learning has to be able to
build on existing elements, and to change them incrementally to
improve the outcomes, efficiency, and overall task performance.
[0209] As will be appreciated by those skilled in the art, teaching
relies on two primary modalities: demonstration and practice.
Demonstration in theory should focus on instructions to help the
student understand what they need to know about the behavior or
movement. Practice refers to the process of performance repetition.
Studies have shown that demonstrations too often focus on the task
outcome rather than on an analysis of the coordination of movement.
Movement skill acquisition could therefore be accelerated by
providing specific signals delivered during performance. Two
signals in particular would be beneficial. First, signals that
highlight the structural elements used in the composition of
movement. Second, signals that indicate which characteristics of
these elements play a role in movement outcomes. However, these
signals have to be adapted to the skill level of the individual and
to his or her specific movement technique.
[0210] Users form an abstract understanding of movement
capabilities in terms of goals and outcomes. Users, for the most
part, learn in which contexts to use a particular movement pattern.
Therefore, at the highest level, people can assess how well they do
from knowledge of the range and quality of their movement pattern
repertoire. The technical details of movement skills are largely
unconscious. This is in part because movement execution is too fast
for humans to directly control their technique. Therefore, most
learning follows a trial and error process. Movements that achieve
goals are generally reinforced.
[0211] It is difficult to directly assess movement technique. Users
typically only determine technique indirectly through outcomes.
Therefore, it is hard to explicitly instruct the technical aspects
of the motion skill system. Trainers and coaches increasingly use
strategies to help users form so-called sense memory associated
with a correct movement technique. A feedback signal that validates
correct movement features can, through association, be used to
reinforce memory of what such a movement should feel like. This
form of feedback should hence accelerate the development and
consolidation of a particular skill.
[0212] Feedback types can be delineated in terms of their temporal
activity and the specific levels of the control hierarchy at which
they operate: Real-time feedback, taking place during performance;
feedback immediately following an action, such as based on
information from the movement outcome; and feedback at the end of a
training set or session. Inherent feedback associated with the
feel, look, sounds, etc., of movement performance, as well as the
movement outcome and interactions with the task and environment,
can provide large amounts of information that can be used to assess
performance and help train. Individuals, however, have to learn to
recognize and evaluate those sources of information. Natural
feedback describes feedback signals at each of these forms that are
inherently present in the task-environment and movement associated
with an activity. FIG. 24 illustrates the natural and augmented
feedback based on cues and interactions. The cueing system operates
by augmenting the natural cues that are available to the performer,
e.g., from the movement outcomes, task environment (task elements
and objects, adversary's movement, etc.). The augmented cue
environment is designed to help the human perform and train for the
task. Task interactions are produced by an apparatus that is
coupled with the activity and possibly with the subject. Augmented
feedback is information that is supplementary to inherent
information about the task or movement. The two major categories of
augmented feedback are recognized in the literature: knowledge of
result (KR) and knowledge of performance (KP). KR represents
post-performance information about the outcome or goal achieved. It
is sometimes called reinforcement. Note, however, that not all
movements have an outcome that is separable from the movement
performance. KP represents information about the movement technique
and patterning. This information is useful for the acquisition of
complex movement skills, such as those requiring high-dimensional
spatial and temporal coordination. Previously, it was difficult to
measure and track performance in many activities. The advent of
MEMS movement sensors has created a wide range of possibilities for
using information about movement kinematics and dynamics (kinetics)
from measurements.
Natural Feedbacks Supporting Learning and Performance
[0213] Several levels of natural feedback are involved in the
support of skilled movement learning and performance. One
consideration is that there exist different forms of feedback that
act at distinct levels across the hierarchical organization of the
nervous system. The cortical and subcortical systems are involved
in the formation and implementation of movement patterns, and in
the different feedback structures used to correct and modulate
movement. At the lowest level, the spinal and subcortical systems
physically implement movement by receiving commands from the
cortical and subcortical structures. Feedback encompasses the
information sensed at the level of the muscles, tendons and joints,
and provides modulation at the level of spinal circuits. Between
the spinal and subcortical is the system that controls posture.
Feedback at that level encompasses information from the vestibular
and proprioceptive systems, which also combines spinal and
cerebellar contributions.
[0214] At the center of the neuro-motor system is a specialized
system that deals with the formation of complex movement patterns,
especially the chunking and sequencing of movement phases. Feedback
mechanisms use information from cues extracted by visual, auditory,
and haptic sources. The task of this system is to fine tune and
synchronize behavior with external tasks and environment elements,
such as adapting timing of movement phases, or modulating phase
profiles. The phases are typically part of a sequence generated by
the cortical circuits. The highest structure is the cortical system
used for perception, planning and execution. This system combines
the various sources of sensory and perceptual information to build
representation that can be used to generate plans and monitor the
performance and outcomes of the behavior. This system can also
handle abstract information such as that in verbal or written
form.
[0215] The human information processing model helps provide an
understanding of what type of feedback information is most useful,
and for which components of movement behavior these feedbacks
apply. TABLE A summarizes the type of signals, cues/signs, and
symbols in tennis as an example. At the highest level,
knowledge-based behavior corresponds to the type of stroke and body
positioning, etc. to use given the information about the overall
situational awareness, such as adversary behaviors gained from
exteroceptive information. At the intermediate level, cues trigger
behavior. At the lowest level, signals are used to modulate muscle
responses.
TABLE-US-00001 TABLE A Example of signals, cues/signs and symbols
in tennis. Symbol: The type/class of stroke as well as the desired
ball placement. Cue/sign When to initiate a stroke phase and the
modulation of the stroke pattern based on the predicted ball impact
and bounce trajectory, etc. Signal Coordination of the muscles
during the stroke to conform to learned pattern based on
proprioceptive feedback.
[0216] At the highest level, the rule-based behavior involves
determining which pattern to activate based on the signs or cues
typically obtained from the exteroceptive information. At the
intermediate level, cues are used for time movement execution. For
example, the particular state of the ball extracted visually, such
as the impact, may be used to signal the instant to initiate the
backswing or the forward stroke, and modulate the strength of the
initial acceleration. Finally, at the lowest level, the skill-based
behavior corresponds to movement patterns. Signals are primarily
the proprioceptive information.
[0217] The delays and time constants of the sensory-motor system
are too large to provide continuous feedback corrections for
fast-paced skilled movements. The neuro-muscular time constant
(time from the signal to go from the motor cortex to the muscle
response) is of the order of 20-30 msecs; on the other hand, the
response time from visual or auditory stimuli to a physical
response is of the order of about 200 msecs.
[0218] Therefore, skilled movements rely on open-loop execution.
Feedback is structured, for example, for intermittent actions based
on specific cues and controlling the timing of phases. The largely
open-loop execution implies that segments have to be learned in
order to be reproduced accurately. And mechanisms to predict the
outcome of the movement help enable modulation of the movement at
the instant of execution. This natural functional structure of
movement can be used as a general model for the design of
augmentations to assist or train human movement behavior. In
principle, augmentations can be designed across all three levels.
Motion skills, assuming training within specific known outcomes,
primarily involve the skill-based and rule-based behaviors. The
symbol level is relevant to form mental models, for example
movement architecture and functional characteristics including the
environment cues. However, it is primarily relevant at the level of
task and competitive performance, such as planning and
strategy.
Augmented Feedback
[0219] FIG. 24 describes the main components used for real-time
feedback augmentation in the augmented human movement system. The
feedback augmentation comprises two primary forms of augmentation:
feedback cues and interactions. The cueing system achieves its
effect by augmenting the natural cues that are available to the
performer, e.g., from the analysis of the movement outcomes, the
real-time analysis of the movement technique, or event generating
cues that pertain to the task environment such as the behavior of
task elements and objects, opponents' movement or actions. One
consideration is that the natural feedback environment is usually
very sparse. Not many relevant quantities are directly observable
by the subject or operator. Therefore, augmentation can be
conceived as the supplementation of the useful signals and cues
that the brain can take advantage of to improve movement
performance and learning.
[0220] The augmented cue environment is designed to help the human
perform and train for the task. Task interactions are produced by
an apparatus that is coupled with the activity and possibly with
the subject to enhance the scope of conditions. The apparatus can
also include an assistive device that mechanically augments human
movement. Note also that this configuration also applies to
settings where humans operate in teleoperation such as a surgical
robot, where the subject interacts with the system through a visual
display and haptic interface, or even in the context of the
operation of prosthetics.
[0221] The forms of KP feedback that are most useful are those that
contribute to the understanding of the task or movement. This
explains why providing a type of normative reference trajectory,
e.g., to model after, is not necessarily useful. In that sense, KR
has the advantage that it provides objective information about the
implicit correctness of a movement.
[0222] Since human attention capacity is limited, it can be helpful
to select augmentations that also account for these limitations and
possibly organize these in ways that allows the brain to take
advantage of the mechanisms used to operate efficiently with
information (e.g., chunking).
[0223] Creating KP feedback contributes to understanding the task
or movement. This can be achieved by using movement kinematic and
dynamic measurements that produce KP that is connected to the
movement outcomes, as well as organized in terms of timing and
form, etc. in ways that are consistent with the movement's
functional dimensions, including biomechanics, motor control, and
sensing or perception mechanisms.
[0224] Therefore, to make it possible to generate feedback that
helps to provide an understanding of how the movement technique
contributes to specific outcomes and other attributes, a central
requirement for motion analysis and cueing platform technology is
the decomposition of movement into elemental movement units that
are grounded in biomechanics and motor control principles including
muscle synergies.
[0225] At the same time, using feedback that is structured based on
the natural functional organization of movement, helps better
overcome attention limitation since the movement units are part of
a coherent movement language. Finally, a technology that reinforces
and teaches this natural movement language will help acquire mental
models that enhance the subject's movement intelligence. This type
of cognitive enhancement would be difficult to develop using an ad
hoc notation system.
[0226] By working within natural functional elements and features,
it is also possible to factor out effects due to individual
differences. Focusing on the structural characteristics of
movements derived from performance data, and subsequently
identifying features within the functional elements that contribute
to the outcome, makes it possible to design feedback augmentation
that targets individual movement characteristics, but generalizes
over the range of skill and styles as well as differences that can
arise due to injuries and other factors.
Formal Movement Analysis
[0227] The following builds on the understanding of movement
structure and organization to briefly review select concepts
involved in the formal analysis of human movement.
[0228] Most tasks are composed of a series of stages, and each
stage involves complex movement patterns that are themselves
delineated in distinct phases. This understanding of how movement
behavior is composed provides the general approach to decomposing
movement performance data needed to support analysis of movement
skills. It is also beneficial to understand what aspects of skills
can be assessed from these different elements and levels of
analysis.
[0229] Movement analysis includes at least three components. The
first component involves decomposing the movement into primary
movement elements or units. Units are typically associated with
subtasks or subgoals that depend on the elements of the task and
environment giving rise to the task stages. These units manifest as
movement patterns that emerge from the functional characteristics
of the movement interactions with some elements of the task or
environment within a task stage, and therefore these units are also
named movement patterns in this disclosure. Second, is segmenting
these movement units into the sequence of movement phases. And
third, is decompose into components that can be combined in
parallel to achieve the coordination of the body segments and
muscles, i.e., muscle synergies.
[0230] Therefore, there are three primary levels of movement
organization, including i) the movement profiles and their
associated outcomes. This level corresponds, for example, to task
level description and represents the overall movement element or
unit such as a tennis stroke in tennis. Also, ii) the movement
profiles are usually composed of series of multiple phases. This
level corresponds to the biomechanical implementation, i.e., the
coordination of the limb segments and joints to achieve a complex
movement. And iii) The movement phase profiles can then be
decomposed into muscle synergies. This level corresponds to the
neuromuscular implementation, i.e., how the profiles are achieved
by superposition of muscle units. The muscle synergies represent
muscle activation patterns.
[0231] The first organizational level corresponds to the building
blocks developed by the brain through interactions with the
environment and task elements to partition the workspace
efficiently, and achieve a range of outcomes relevant to a task. It
is related to what could be considered the semantic
characteristics, i.e., the meaning of the movement elements in
relationship to the task goals, elements, contingencies, and the
range of conditions.
[0232] The second level, the phase segmentation, corresponds to the
functional structure of the movement, and is related to the
strategy used by the nervous system to achieve the particular
outcome given the available neuro-muscular system.
[0233] The third level, the muscle synergy, describes how the
various muscles are activated to achieve the movement profile at
the phase level. The synergies typically provide spatial and
temporal components that can be combined to achieve a variety of
movement. Therefore, it is expected that same set of synergies can
be reused by other movements. Yet, for example, in tennis the arm
segments configuration can be very different at different stroke
phases, therefore it is likely that different sets of synergies are
used in each phase.
[0234] As already discussed, complex human movements are
high-dimensional, i.e., their description requires large numbers of
state variables (position, velocities, angles). The
representational complexity is in part due to 3-dimensional (3D)
space which involves six degrees of freedom for the linear and
angular motions. This number is multiplied when multiple body
segments are involved, and becomes exponentially complex once
ligaments and muscles are accounted for. In addition, the effects
of dynamics dictate how these state variables evolve over time and
interact through the action of forces (both internal effects such
as inertial coupling and the external actions such as the muscles
or aerodynamics, etc.). For this reason, tracking even a single
segment or object in 3D space such as a tennis racket or forearm,
requires a dozen state variables.
[0235] Their temporal evolution is described through coupled
differential equations. These differential constraints and other
constraints on joint configuration, etc. result in geometrical
properties which can be exploited for analysis. From a control
standpoint, the formulation of movement programming in engineering
or robotics follows from the equations of motion, and the
specification of an initial state (starting configuration) and a
goal state (see FIG. 2). Such problems can be solved as a dynamic
program, or a two-point boundary value problem. The trajectory is
obtained by solving for a trajectory that minimizes a pre-specified
cost function (e.g., trajectory duration or energy). This
formulation leads to equations which provide conditions for an
optimal trajectory. Thus, for a given initial state and goal state
(e.g., that specifies the outcome), there typically exists a unique
optimal movement trajectory. The control and trajectory
optimization framework provide useful tools for the
conceptualization and analysis of movement. For example, it is
possible to define cost functions that characterize human
trajectories, such as energy or more general physical performance.
Furthermore, the calculus of variation used in trajectory
optimization make it possible to investigate relationships between
variations in trajectory and outcomes of the trajectory.
[0236] Movement measurements, such as from wearable motion sensors
or optical motion capture systems, are typically given in the form
of time series. Since these time series typically originate from
nonlinear dynamic processes, their analysis relies on an
understanding of the structural characteristics of the underlying
dynamics. These structural characteristics are associated with the
architecture of the movement, such as the movement phases in a
tennis stroke or golf swing. Insights can be gained using
computational visualization tools such as phase space; however, the
states may have too many dimensions to be practical. Therefore, the
data should be reduced. The behavioral data captured from the
available measurements results in a high-dimensional state space.
The dynamics driving the behavior, on the other hand, may be
lower-dimensional.
[0237] Dimensionality reduction is a class of unsupervised learning
techniques that can be used to discover the state dimension of the
underlying behavior. The goal is to transform the original movement
data time series which are described in terms of the
high-dimensional time series x.sub.t into a lower dimensional
description that preserves the geometric characteristics of the
underlying nonlinear movement dynamics. This can be done, for
example, using Taken's embedding theory. Examples of recent
applications of dimensionality reduction for movement analysis
include gait analysis.
Movement Patterns Analysis
[0238] Although movements are usually high-dimensional behaviors,
trained movements typically have specific patterns. Patterns have
the useful property that even though the behavior relies on many
degrees of freedom (DOF), they can be described by a few, dominant
DOFs. The patterns form a lower-dimensional system as a result of
the coordination provided by the neuro-motor processes, and other
perceptual and control mechanisms. The lower dimensions however can
hide a complex geometry and topology.
[0239] The movement architecture can be analyzed by focusing on the
low dimensional manifold that are associated with the particular
movements' dynamics pattern. Using a nonlinear dynamic systems
formulation gives access to analysis and modeling tools that, under
certain conditions, can reconstruct the pattern dynamics from
measurements of the behavior. The reconstructed dynamics can then
be analyzed to determine the underlying structure and geometry,
which can then be used to determine useful abstractions or
models.
[0240] Using mathematical tools used for the analysis of nonlinear
dynamic systems, the movement patterns can be described by a
nonlinear mapping F associated with discrete-time nonlinear
dynamics:
x.sub.t+1=F.sub.t(x.sub.t,t, .sub.t) [1]
where F.sub.t is a map, x.sub.t.di-elect cons..sup.n is the state
vector at discrete time t.di-elect cons., and .sub.t is a
time-dependent noise. A continuous time representation could also
be used. In the forthcoming discussion the dynamics are assumed to
be autonomous and use a constant map F.sub.t=F.
[0241] The nonlinear model of a movement pattern therefore can be
described by a map F that captures the combined effects of the
biomechanics, sensory, and motor-control processes. This model
assumes that the learned movements result in deterministic
dynamics. In this case, the dynamics are given as an ordinary
differential equation (ODE) {dot over (x)}=f(x(t), (t)), which
describes a vector field and is typically called the flow. The set
of initial conditions which result in the same asymptotic behavior
are referred to as the basin of attraction. Such nonlinear dynamic
models can describe a broad range of phenomena. The model could be
decomposed into subcomponents, giving access to the various
contributing systems and processes. For example, it may be possible
to explicitly model how the users adjust their movement pattern to
changes in conditions, such as adjustment of a forehand topspin
stroke for changes in ball height at impact. However, at this point
in time the behavior is regarded as a closed-loop behavior which
abstracts the various internal mechanisms.
[0242] The language of nonlinear dynamic systems makes it possible
to describe the collection of movement patterns that composes a
user's repertoire in a particular activity (tennis, skiing,
surgery, etc.) by a collection of distinct dynamics or maps
{F.sub..alpha., F.sub..beta., . . . , F.sub..gamma.}. In many
nonlinear time series, the movement system state variable x is
generally not directly observable. Instead, measurements y are
acquired for example through motion sensors. The observations, or
measurements can be defined as: y.sub.t=h(x.sub.t, .eta..sub.t),
where h is the output map and .eta..sub.t is measurement noise.
[0243] A property of movement at the highest level is referred to
as "motor equivalence." The fact that the brain generates movements
that are equivalent in terms of their accomplished outcomes
underscores the idea that at the highest level the brain encodes
outcomes and their relationship with task goals. The planning and
monitoring functions associated with goals are part of the brain's
executive system. For example, in tennis, the player selects a
stroke type based on the desired outcome and the conditions (ball
state including expected impact height, velocity, and spin of
ball). Even within the continuum of conditions and outcomes, it is
possible to recognize distinct classes of strokes. The invariant
characteristics in movement features enables the delineation
between movement classes, e.g., movements within one particular
class can be related through some smooth transformation such as
rigid-body translation and rotation, i.e., they are invariant under
this class of transformation. The overall movement class can be
subdivided into subclasses. For example, a hierarchical
decomposition would group movements based on relative
similarity.
[0244] In tennis, the overall stroke class can be subdivided into
dozens of subclasses based on movement where the levels represent
different types of features. For this example, a top hierarchical
level is called the category level. It differentiates between
groundstroke, volleys, serves, etc. The distinction between stroke
categories is made primarily based on the height of the impact
point. Further, subcategories can be created based on the side of
the impact, i.e., forehand or backhand. Even further subclasses can
be delineated based on the outcome (topspin, flat, slice), and
strength. Beyond these common classes, finer distinctions can then
be added based on additional aspects of stroke technique, such as
open or closed stance. Most of the stroke characteristics can be
determined entirely from the racket trajectory and therefore do not
require additional measurements such as the position of the player
on the court.
[0245] Each movement pattern class in a repertoire has different
geometrical characteristics and their domain may occupy a different
subspace of state-space (see FIG. 14). The shape and dimension are
a result of the dynamics, which is given by the transition map F.
The repertoire is the collection of these shapes or patterns. The
precise geometrical characteristics of the movement patterns can be
described via embedding theory. The idea is to determine the
subspace of DOF that fully describes the movement. The
dimensionality of the system and the geometry of the manifold that
contains the trajectory describe the movement class structure.
[0246] As is often the case in nonlinear system dynamics, the state
transition map F (the dynamics), the output map h, and the
dimensionality of the state vector n are not known. Techniques of
nonlinear time series analysis can (assuming deterministic dynamics
F and smooth output map h) estimate the dynamics associated with a
movement pattern from time series obtained from measurements of the
behavior. Repertoire characteristics can provide a variety of
information about skill. Movements are typically analyzed in
specific classes without considerations about the overall
repertoire structure. The movement repertoire for a particular
activity domain describes how a user organizes the outcomes and
technique in that task domain. The simplest way to classify
movements into repertoire is to extract features from the time
series and apply clustering techniques to determine classes.
[0247] Movement classification has been used in other applications
unrelated to skill modeling, such as activity detection or gesture
recognition. Gesture recognition is a growing aspect of natural
human-machine interfaces. The general goal in the latter
application is to determine motion primitives that provide a
low-dimensional description of the various movements that can occur
in that domain. The primitives can then be used to classify the
movements. The library of primitives can then be used by other
agents to identify the intent of a human or robotic agent and, for
example, allow collaboration between agents. The emphasis of
gesture classification is the identification of semantic
characteristics. In the present application, the goal is
classification based on characteristics that are related to
movement technique and outcomes. Typically, the higher categories
of the stroke classification can be considered in a semantic sense
(e.g., groundstroke vs. volley or backhand vs. forehand), and the
lower level classes are related to different techniques and
conditions (see FIG. 9).
[0248] A particular ensemble or repertoire of patterns in a domain
of activity arises through the effects of biomechanical,
neuro-muscular constraints, as well as task-related constraints. In
the most general sense, the patterns describe how an individual's
movement techniques are used to achieve an outcome. One aspect of
the movement characteristics is how they are broken down into
phases. Overall movement pattern characteristics, therefore, are
the result of the phase structure, and those can be used to
classify motion patterns.
[0249] The serial order in behavior, i.e., the task stages, and the
movement phase structure are usually distinct. As already
discussed, the serial order is associated with the activity level,
for example, characteristics related to the activity constraints
such as process stages, rules, etc. The movement phases, on the
other hand, are associated with the movement technique and are
related to characteristics and constraints of the movement system
and its interactions with the environment and task elements.
[0250] For example, in tennis, the movement stages associated with
the serial-order of behavior, include serving, then moving to the
anticipated return position, making adjustments in the positioning
as the ball returns, setting up for the stroke and engaging the
ball using the stroke type required for the desired outcome (see
FIG. 7-9). Each stage can be parsed to extract the primary movement
unit, and these movement patterns can then be analyzed to determine
the movement functional characteristics, i.e., how the movement
produces its specific outcomes while at the same time adapting to
conditions. The functional analysis is facilitated by further
decomposing the patterns into movement phases. The phase structure
of the primary movement patterns defines the topological
characteristics of the manifold, while the dynamics that drive the
phases define its geometrical characteristics.
Phase Segmentation
[0251] As already discussed, many complex movements are achieved by
combining several movement phases, leading to further temporal
structuring of the movement. Examples include the phases in
locomotion gait 445 or the phases in a tennis stroke 441 (see FIG.
4). Phase structuring of patterns typically arise from the
intrinsic movement constraints (biomechanics), some aspects of task
constraints, as well as functional factors related to motor-control
and decision mechanisms as discussed elsewhere. For example, in
gait, distinct phases are associated with the basic leg
biomechanics and mechanics of ground interactions.
[0252] In tennis, the general goal of the user is to return an
oncoming ball and further control the trajectory of that ball (see
FIG. 7). This is accomplished by imparting precise linear and
angular momentum to the ball with the racket. The user controls the
ball primarily by modulating the amount of momentum imparted to the
ball and selecting the precise interception point and time as the
ball enters the half-court (FIG. 9). For accomplished players, the
overall tennis stroke motion encompasses the kinetic chain formed
by the legs, hips, shoulder and elbow, and wrist. These segments
are coordinated to form a continuous movement starting from the
backswing all the way to the follow through and recovery. At closer
inspection, distinct phases can be recognized.
[0253] The exact phase characteristics depend heavily on skill
level. Beginner players primarily swing the racket from the
shoulder without very precise coordination with the rest of the
body segments. Advanced players exploit the entire body kinematics
to maximize the outcome. Ultimately, the phase characteristics
reflect the combination of the body segments' biomechanics and
neuro-motor strategies, including the muscle synergies that achieve
the highest outcome reliability with best use of the physical
capabilities. Different phases are associated with different
biomechanical functions. For example, in walking, synergies that
are activated at specific phases of the gait cycle (e.g., forward
propulsion, swing initiation, deceleration, etc.) have been
identified.
[0254] The role of constraints in creating distinct movement phases
can be explained using concepts from constraint optimal control. In
optimal control, trajectory segments are related to the concept of
singular arcs, which correspond to segments where different sets of
constraints are activated by the trajectory. In general, these
systems are best controlled using switched control laws. The
control law is determined based on a partitioning of the system's
state. As the system is driven by the control action, and travels
through the different partitions of the state-space, the control
strategy switches to best account for the local characteristics of
the dynamics.
[0255] Following the nonlinear dynamic systems description,
trajectory phasing can be described mathematically as a sequence of
dynamic models F.sub.1, F.sub.2, . . . , F.sub.N. The overall
trajectory is obtained by a series of initial values and asymptotic
behaviors, where the next set of initial values corresponds to the
terminal values of the previous phases (FIG. 3A). The dynamics
associated with each phase result from different joint and limb
segment configuration and force fields. Each dynamic model F.sub.i,
can therefore be assigned a state-space region specified by an
initial state set and a goal or subgoal set. For example, once the
dynamics are initiated from the initial set, the initial dynamics
F.sub.1 will take the state to its subgoal set .chi..sub.1, and
from there, assuming the state satisfies the next initial state
conditions for the next dynamics F.sub.2, and the dynamics are
triggered, the system dynamics will switch to the next phase, where
it will evolve to the next subgoal, etc. This process can be
cyclic, where the state transitions form a loop, such as for
periodic movements (see e.g., running 445 or swimming 446 depicted
in FIG. 4). In other activities such as tennis or skiing, the
behavior can be quasi-cyclic, where for example the same general
sequence of movement phase continues after a pause (see FIG. 3B).
The dynamics can also switch between patterns that have different
phase segments, such as a different stroke type or gait type, or
altogether different movement patterns, such as in skiing when
switching from a periodic turning sequence to a stopping maneuver,
or switching between different stroke patterns in tennis.
[0256] The switching between dynamics at the phase transitions are
typically determined by conditions on the terminal/initial states,
e.g., .chi..sub.1i=.chi..sub.0j. As already discussed, the dynamics
associated which each phase result from different joint and limb
segment configurations and force patterns. These force patterns are
determined by the spatio-temporal muscle activation patterns, i.e.,
muscle synergies. Within the phases, in particular for fast
motions, the force patterns are specified in open loop, therefore
the dynamics are specified by the force fields associated with the
muscle synergies. The brain ostensibly learns to compensate for
variations in initial conditions by adapting these force fields.
This makes it possible to produce fast corrections in movement
without relying on feedback. Feedback can be used intermittently,
e.g., during phase transitions or during specific movement phases
which can accommodate such effects, e.g., because of the slower
dynamics and availability of sensory information.
[0257] The muscle synergies describe the coordination between the
different muscle groups and limbs segments that are used to
implement movements. The synergies are a type of motor primitive
which is typically reserved for the neuro-muscular coordination. In
examples discussed earlier, various movement profiles observed in
an activity can be obtained through the combination of such
primitives. Decomposition into synergies therefore can help
determine the set of biomechanical and neurological components that
participate in movement skill. In turn, this information can be
used to gain understanding about the biological components, and
could be useful for physical performance and injury prevention.
[0258] Popular techniques are based on non-negative matrix
factorization, which decompose matrices that in this context
represent the data for a movement phase into a product of matrices.
Synergies have been characterized with a variety of measurements,
including movement profiles of the end points, joints and/or body
segments, as well as muscle and neurological activity, such as
provided by surface electromyography (EMG). The type of
measurements determines the accuracy of the results. For example,
simple end effector or body segment measurements may not provide
synergies that correlate strongly with the neuro-muscular activity.
Synergy analysis has not yet been integrated in clinical settings
where it could be used for assessment and rehabilitation. Since
synergies have been identified at different levels of the
neuro-motor hierarchy (motor cortex for grasping, brain stem for
posture and spinal cord for locomotion), the muscle synergy
analysis can provide a more precise picture of neuro-motor
deficits.
Movement Functional Structure and Primary Outcome
[0259] Some movements have an explicit outcome or goal. This goal
may be the movement's end state, i.e., .chi..sub.goal=.chi..sub.N,
or it could be the state at an intermediate phase such as a
subgoal. The latter is the case for the tennis stroke. While the
ball impact is the primary goal or outcome of the stroke, this
phase is not the actual end of the movement. The movement phase
following the impact, the follow through, is one part of the
overall movement pattern. Most complex movements involve many body
segments or degrees of freedom. Therefore, the state trajectory is
a multidimensional state vector and it can be helpful to add
distinctions between the different state trajectories that
participate in the action. Focal and corollary movements are
distinguishable; the focal movement is, for example, in a piano
performance, the finger movement that hits the key; the corollary
movement is, for example, the motion of all other fingers that are
part of the overall kinematic pattern involved in the task of
hitting the key.
[0260] Not every movement behavior has an explicit goal or outcome.
For example, most of the movements used in skiing 444 have as
purpose to control the skier's speed and direction. From a dynamic
system standpoint, this goal involves generating a centripetal
acceleration though the interaction of the skies with the terrain.
Depending on the skier's state and terrain conditions, different
motion pattern of the legs and hips, etc. are used to achieve the
best outcome (will be discussed elsewhere, see FIG. 4).
[0261] It is possible to define an optimal trajectory that takes
the system through the phase sequence achieving the goal condition
(outcome) while minimizing a performance objective such as jerk or
energy. Given the biomechanical constraints, muscle synergies,
etc., the optimal trajectory is associated with a specific phase
sequence. The conditions at the phase transitions, i.e., the set of
initial states, and subgoal states, .chi..sub.1i=.chi..sub.0j as
well as the dynamics F.sub.i describing the transitions, represent
characteristic features of the optimal trajectory (see FIG.
3A).
[0262] The absolute optimal trajectory is the global optimal
solution for a given outcome, while the local optimal trajectory
corresponds to a given phase structure. The latter, for example,
represents situations where due to a lack of flexibility or skills,
or the presence of an injury, only a limited set of configurations
and/or force fields is achievable. Therefore, movement phase
characteristics provide valuable information for injury prevention
and generally also for rehabilitation.
[0263] In optimal control theory, perturbation of the initial value
leads to neighboring optimal trajectories. This is guaranteed if
the initial value is within the so-called basin of attraction of
the system. A similar idea can be used for perturbations in the
dynamics F. Such perturbed dynamics lead to slightly different
asymptotic behaviors; however, for small enough perturbations the
trajectories stay close enough to the nominal trajectory that these
perturbed trajectories belong to the same movement pattern. The
range of perturbations in the initial values and dynamics for which
the trajectories remain in the basin of attraction defines the
admissible envelope. Perturbations in the dynamics and disturbances
are captured by the time dependent noise term .sub.t
[0264] FIG. 3A illustrates the trajectory envelope 113 for a
hypothetical movement pattern delineating the movement phases that
typically arise from biomechanical and neuromotor constraints. The
figure also highlights a primary outcome and its associated phase
(shown as a goal phase). It also shows an optimal trajectory across
the movement phases, and different envelopes (optimal, admissible,
feasible) resulting from the various movement constraints.
[0265] The trajectory envelope delineates a region of the
state-space over time and highlights the feasible envelope and the
envelope of admissible trajectories as well as the region for the
optimal trajectory's initial conditions (x*.sub.0i), and the
optimal trajectory (x*(t)). The structure of the movement both in
terms of patterning and the phase segmentation are given by its
spatio-temporal characteristics. Movement characteristics are
defined by the geometry and dimension of the manifold containing
the trajectory.
[0266] Several phases are shown in FIG. 3A including: movement
initiation, phase 1, phase 2, an intermediate goal phase, a
follow-on phase, and recovery phase. For tennis, these phases
correspond to the stroke initiation, backswing, back loop, forward
swing, impact, follow through and recovery. The goal phase in
tennis represents the impact phase, which is the phase during which
the primary outcome is produced.
[0267] These movement pattern characteristics are usually
determined from the topology of the movement pattern manifold
obtained from analyzing the nonlinear time series. A user may
choose "admissible movements" that belong to the same movement
pattern and still reach the goal conditions or outcome. This could
happen due to changes in movement goal conditions (impact height
and velocity), or imperfect initiation of the movement. The
suboptimal trajectories can still reach the desired end state or
outcome; however, they will typically require more physical effort,
may cause stress in some of the muscles or joints, or other
undesirable effects. The physical performance can be described
through models of the musculoskeletal system and cost functions
such as for energy consumption.
[0268] Movements belonging to the same pattern can therefore be
related through perturbations relative to a nominal trajectory.
Moreover, the trajectory perturbations also result in perturbations
in the primary outcome and any other secondary outcome
characteristic such as the different phase outcomes. Using this
data, it is therefore possible, for example through regression
analysis or sensitivity analysis, to determine relationships
between the trajectory perturbations (which correspond to the
movement technique) and perturbations in outcomes. This information
provides a quantitative basis to generate skill characteristics,
such as what aspects of the technique contributes favorably to the
outcomes and vice-versa what aspects are detrimental to good
outcomes. This knowledge in turn can be used for training and
eventually help synthesize feedback laws for real time cueing.
[0269] FIG. 3B is an illustration of the finite-state model
representation 114 for the system shown in FIG. 3A. By modeling
movement patterns as a sequence of phase segments with distinct
dynamics F.sub.i, the pattern dynamics can be abstracted as a
finite-state model (see FIG. 3B and FIG. 5). In the present case,
the finite states are the individual phase dynamics F.sub.i which
take the system from initial value x.sub.i0 to the next subgoal
state x.sub.i1. More generally, the initial and subgoal states are
represented by sets to account for the variations and disturbances
that are typically expected in human behavior. With this model, the
overall motion behavior is then given by some finite-state automata
which gets triggered from the initial state and initial movement
phase. The motion behavior combines both continuous dynamics and
discrete variables that capture phase transitions and mode
switching which may be associated with discrete decision variables.
Hybrid models can be used in many modern engineering applications
including robotics such as for autonomous systems, as well as
human-machine systems. Once the structure of the motion is
characterized, it can be described by finite-state models.
[0270] Statistical models, in contrast to deterministic models
where the current state uniquely determines the evolution of the
system (i.e., within the disturbance or model uncertainties),
describe the evolution of the probability density of future states.
Statistical models such as Dynamic Bayesian Networks have become
increasingly popular in data-driven approaches. Popular
applications in the movement domain are identification of human
activities. These approaches typically require learning the phase
of activities based on statistical pattern analysis; subsequently
using this knowledge to discretize the state space into discrete
states; and finally determining the state-transition probabilities.
A common model is the Hidden Markov Model (HMM). Most of the
notational systems focus on the discrete game structure and can be
used to analyze game plans but currently do not reach down to the
actual movement skill level.
[0271] Real-time movement phase estimation can be implemented by
someone trained in the art. For example, a multi-layer HMM
application to movement could be based on similar models to those
used for real time speech recognition. Decoding sound recording for
speech recognition typically proceeds on multiple levels. Most of
those are associated with the levels of organization of the speech
production system. The units of decomposition of speech is based on
phones which combine to form the phonemes. The phonemes are the
basic building blocks used to form words. The phones are related to
features of the vocal movements. This model for movement
corresponds to having, at the top level, a movement phase model
which describes the probability distribution over possible
sequences of movement phases. At the midlevel, a phase model that
describes the composition of the movement phases in terms of
movement components (c.f. synergies). And finally, at the bottom
level, the movement model that describes the movement components
based on features in the available measurements (IMU unit or other
sensors).
Movement Skill Acquisition
[0272] Learning is about organization of information, which is a
process that proceeds in stages. The following reviews some of the
concepts related to the skill acquisition and its implications for
training and concludes with an outline of the roles of technology
to support learning complex movement skills.
Organization of Information
[0273] Organization of the learning process and codification and
organization of information associated with movement is dictated by
principles that can help mitigate complexity. These mechanisms are
primarily directed at exploiting structure in task and interactions
between the agent and the environment. Two major concepts have been
proposed for dealing with complexity associated with representation
of information: chunking and hierarchical representations. Chunking
describes a general memory structure that applies to different
domains.
[0274] Miller proposed the following "Chunking Hypothesis: A human
acquires and organizes knowledge of the environment by forming and
storing expressions, called chunks, which are structured
collections of the chunks existing at the time of learning." (Cited
in Newell, 1981). This hypothesis is based on research on
perceptual behavior and memory retrieval (see Miller, 1956) and
earlier work by DeGroot in chess. The general idea of chunking is
to achieve a more efficient encoding by combining individual bits
of information into wholes. Gobet, for example, describes it as "a
collection of elements having strong associations with one another,
but weak associations with elements within other chunks." For a
review see (Gobet, 2001). A central assumption behind chunking of
information is that the joint encoding reduces the latency of
information retrieval, and more generally provides more economical
information encoding and processing.
[0275] Chunks have been extensively studied in domains that involve
static and discrete quantities, such as the perception or
memorization of chessboard configuration. Early chunking theory has
been studied as part of human perception and more generally
information processing in (Miller, 1956). Many activities are
described by a complex spatial and temporal structure. Later, the
chunking theory has also been applied to improve our understanding
of motor learning and more generally skill acquisition. There exist
fewer investigations in the sensory-motor domain. In that domain,
chunking is primarily associated with the concept of "serial order
in behavior" introduced by Lashley (Lashley, 1951), and the general
hierarchical learning theory.
[0276] The hierarchical models conceive complex skills as a
"hierarchy of habits." This model was introduced by Bryan and
Harter (1897) studying Morse code learning. In that example, the
telegrapher learns letters first, followed by sequences of letters
to form syllables and words, and then phrases. This model applies
to many motor skill domains. In most movement skills such as
tennis, the elementary actions are movement phases (muscle
synergies) that can be combined to form gross movements. Learning
such skills, therefore, involves learning elementary movement
units, and combining those into larger movement elements that are
themselves nested into actions.
[0277] Lashley's serial order in behavior was a response to the
linear sequencing that was suggested based on association learning
theory (Terrace, 2001). Instead of a serial sequence, Lashley
argues that skilled behaviors are planned, and plans have a
hierarchical organization which combine multiple units of behavior
into larger units. Some units are related to a movement's
biomechanical and functional constraints, and others are related to
task constraints (e.g., subgoals).
[0278] Following the hierarchical representation, it is possible to
decompose the activity and the associated movements into a sequence
of elements, which are themselves decomposed into smaller elements.
Chunks are usually not made of arbitrary segments but have a
functional purpose. Chunks, therefore, combine specific sensory and
motor patterns that relate to the task environment interactions, as
well as the constraints of the organism.
[0279] For example, in tennis, major behavioral chunks can include
the "ready state," "reposition," "preparation," and "stroke
execution." Each chunk can be described by a set of movement
patterns with their associated perceptual process. During the ready
state, the player orients himself or herself, extracting cues from
the environment needed for court positioning, observes the motion
of the ball and the opponent, etc. This information allows
prediction of the location of anticipated ball interception
selecting the desired outcome and planning the sequence of actions
to achieve the desired outcome of the stroke. During repositioning,
the player acquires the new court position and may start to bring
the racket back (backswing). During the stroke preparation, the
player adjusts his or her posture and extracts updated information
about the ball and opponent needed to fine-tune posture and prime
the stroke execution. Just before the stroke execution, the player
obtains final ball trajectory information for the interception. The
execution of the forward swing is synchronized with the arriving
ball. Finally, after the execution of the stroke, the player
returns to a ready state.
[0280] The behavioral chunks forming the larger program are
typically subdivided into smaller sensory-motor units, starting
with the elements such as muscle synergies that are combined to
form larger movement patterns. For example, the stroke is composed
of a sequence of body and arm movements (described elsewhere).
Similarly, extracting information involves a type of perceptual
chunking which describe how the various sensory stimuli are
integrated to form the cues that can be used to predict the
intentions of the opponent, anticipate the ball trajectory, and
select and initiate the appropriate stroke type.
[0281] As an individual acquires experience in a task, they
assimilate the movement units into procedural memory. Hence, less
attention is required at the level of individual components that
form the chunks, allowing increasingly automated processing.
Proficient individuals are able to focus attention on task-relevant
information, which enable better planned, more systematically
organized behavior with fewer extraneous movements and smoother
movement execution that takes advantage of the subject's physical
performance.
[0282] The acquisition of open motor skills with their environment
interactions, therefore, can be conceived as the acquisition of a
library, or repertoire, of sensory-motor patterns, their associated
perceptual cues, and the larger motor programs used to deploy these
patterns and attain outcomes needed for successful task
performance.
[0283] Following the general learning theory, learning proceeds as
an evolutionary process. Behavioral patterns are associated with
actions that produce outcomes for the task. Valuable outcomes are
rewarded, and thereby produce reinforcement for learning patterns
that are successful, i.e., have a positive outcome on the task.
This process, however, depends on extensive practice and experience
in the specific task domain.
[0284] The chunking theory of learning also provides additional
understanding of the learning process. For example, it has been
used to explain the so-called Power Law of Learning (see Newell,
1981). This law describes the improvement in skill (measured as
response time) as a function of training and has been validated in
many domains besides perceptual motor tasks, hence it is widely
accepted as a universal law. However, the law has received
criticisms, in particular that it does not explain qualitative
changes in movement dynamics with practice (see Newell, 1991). As
described in that reference, these may be due to the limited tasks
used in studies (few degrees of freedom and limited perceptual
environment).
[0285] The learning time to reach proficiency depends on the task
environment and complexity of interactions associated with the
movement production. To help appreciate this, it is useful to
consider Newell's notion of environmental exhaustion, which can
help describe the number of chunks required to cover the range of
task conditions. Many unique chunks are required in task domains
where there are many unique configurations (see for example,
chess). Natural environments tend to have a statistical
distribution of conditions, with large amounts of similar or
related configurations and fewer unique configurations. This
fractal, or self-similar, structure in natural environments means
that these can be expressed as hierarchical structures that take
advantage of modularity of the representation. However, even though
efficient representations may exist, the individuals still have to
experience the range of conditions to form an understanding of the
patterns and develop a sufficiently rich repertoire. This explains
why surgeons or athletes require thousands of hours of practice
before they are proficient, and also, why they keep improving with
additional experience (assuming the experience is sufficiently
varied and rich).
Learning Process and Stages
[0286] Finally, it can be beneficial to understand the brain
processes involved in learning skills, and, in particular, what
changes take place in the brain, and how the brain processes and
stores information as a function of the different stages of
acquisition. Fitts proposed three major stages of acquisition
(Fitts, 1964). The cognitive stage (also called verbal stage) is
characterized by a conscious effort required to understand and
control the movements. As a result, in this stage, movements are
slow, they lack dynamic coordination, and have low success rate.
Problem solving, by way of cognitive processes, is a critical
aspect for the development of mental models, or representations
that could be used to support this stage (Ericsson 2009). During
the associative stage, the movements are partly automated.
Conscious efforts are fewer but are still required to monitor and
improve performance. Finally, in the autonomous stage, movements
are stored in procedural memory which allows automatic execution.
Movement in this stage may still require visual inputs to ensure
accurate and consistent execution. However, these inputs are also
automated and focus on very specific elements, i.e., cues.
[0287] The type of knowledge gained by subjects as they learn to be
proficient in a task is directly related to the structure of the
task, and the structure of the interactions between the movement
and the task and environment elements. For spatial behaviors, the
critical aspect is the structure of the interaction between the
subject and the task environment and elements (see e.g., the
interaction patterns in Mettler 2015).
[0288] These interactions combine the perceptual mechanisms used to
extract information from the environment and the dynamics governing
the agent's motion. Ecological principles (see Gibson 1979) suggest
that humans and animals exploit information that can be obtained
directly from the perceptual environment without relying on complex
internal models. However, the brain can also learn more subtle
patterns relevant to a task (see e.g., the squash study of
Abernethy et al. 2001). Cues are determined by those features in an
organism's perceptual environment that are directly relevant to the
movement guidance and coordination with respect to the task
environment. Cues can be viewed as sparse sensory stimuli such as,
for example, in Tau theory (see Lee 1998).
[0289] As individuals familiarize themselves with a task, they
develop a repertoire of automatic behaviors and mechanisms to
deploy these behaviors (Ashby 2010). The repertoire represents a
library of sensory-motor patterns that is stored in the brain's
long-term memory. The structure associated with the task and
interaction between the movement and task elements suggests that
sensory-motor patterns are grouped hierarchically. The top
sensory-motor chunks define larger categories of behavior, such as
ground strokes and volleys; the intermediate level, which include
the various stroke classes in a category; and at the lower level of
the hierarchy are components of behavior which include muscle
synergies and are shared by different classes.
[0290] The hierarchical and modular encoding has been known from
early studies of the neural visual processing and encoding, and has
been verified in the domain of movement encoding and control
(Poggio 2004). For example, movement patterns within related
movement classes (e.g., tennis forehand slice and top spin) share
similar sub-movements. The movement phases result from the
activation of muscle synergies that are encoded in part in the
spinal circuits. Multiple studies have demonstrated the modular
encoding of movement (Mussa-Ivaldi 1999).
[0291] Each class of movement learned has some operating range that
defines the range of validity of the learned patterns. However,
there are limits to generalization. These are due both to the
neural encoding mechanisms (Kawato 1999), but also due to the
structural and functional characteristics of the state-space.
Therefore, to cover the range of outcomes and conditions typical of
open motor skill, multiple movement classes may be employed.
Hierarchical representations are used to efficiently encode these
movement classes into motor programs.
Movement Skill Acquisition
[0292] In summary, movement skill acquisition results from the need
to adapt to the task and environment, and thus learning proceeds
incrementally with exposure and experience performing a task.
Therefore, it is possible to conceive skill acquisition as an
evolutionary process (see FIG. 11). The specific skill elements are
classes of movement patterns that are evolving with their usage in
the task or activity. Learning and perfecting skills are the result
of an iterative process that takes place as these elements are
repeated under different conditions, and modified based on the
observed outcomes and effectiveness to the overall task goals and
performance.
[0293] The acquisition process can thus be described as the
evolution that involves two primary dimensions: 1) the
diversification of the movement patterns to respond to the range of
requirements and conditions called for by open motor tasks; 2) the
refinement and optimization of individual movement patterns, which
corresponds to the changes in those movements over the stages of
acquisition.
[0294] The process therefore can be analyzed by tracking the
movement repertoire over time. At any given time, an individual's
skills are described by a repertoire with one or more classes of
movement patterns (FIG. 11). The repertoire reflects both aspects
of how the individual deals with the task and environmental
structure, as well as the individual's perceptual and motor control
abilities. Each pattern class can be at a different stage of
acquisition.
[0295] Sensory-motor patterns serve as units of behavior used for
organizing and planning the behavior toward the larger task goals
(see Mettler, 2015). Identifying the repertoire of patterns
therefore also provides the elements needed to analyze the skills
at the planning level.
Challenges in Movement Acquisition and Training
[0296] The comprehensive understanding of movement skill
acquisition highlights several challenges to efficient training.
Forming a repertoire of movement patterns, and the associated
perceptual and planning processes that enable task proficiency and
versatility, depends on the training process and in particular the
information available to support and guide this process. Without a
coach, human skill development depends primarily on a trial and
error approach. Typical information to guide the process includes
the movement outcome. This so-called "knowledge of result" has been
shown to help learning. However, this alone typically does not
contain sufficient information to efficiently teach users how to
improve their movement. It can also make individuals dependent on
this (see Newell, Schmidt) type of feedback. Many common movements
can be learned efficiently through trial and error; however, for
complex movements such as found in surgery, music and many sports,
trial and error are limited because many of the movements involved
in these activities are unnatural. Additional information is needed
for the discovery of the correct or optimal technique. Furthermore,
some activities such as surgery also don't afford much
opportunities for trial and error.
[0297] This situation is also similar for rehabilitation because
the physical impairments acquired from an injury or disease can add
constraints that make movements challenging to develop under
natural conditions. For those situations, trial and error learning
can be extremely time consuming and cannot guarantee that correct
movement patterns will be discovered.
[0298] Therefore, movement skill training has depended on the
expertise of a coach. For rehabilitation, the patient depends on
the availability of a physical therapist. The traditional role of a
coach is to help focus training efforts on correct technique, and
attend to relevant aspects of the task and performance. However,
even expert coaches are subject to limitations in perceptual and
information processing. Most skilled movements involve coordinating
many degrees of freedom that take place over short time scales
(hundredth to even tens of milliseconds). These movements, such as
a tennis stroke or golf swing, are highly dynamic behaviors that
combine temporal and spatial dimensions into complex patterns.
[0299] Furthermore, motion patterns depend on complex biomechanical
constraints and muscle synergies. These depend on musculoskeletal
constraints, as well as the physical fitness and general health of
an individual. Therefore, the training approach should be able to
account for individual characteristics both in the movement
technique and in the longitudinal skill development process. It
takes great experience for a coach to be able to analyze movement
and identify relevant characteristics of these patterns while
taking into account the individual's constraints.
[0300] In contrast to closed motor skills, in which the conditions
can be controlled, open motor skills require a broad movement
repertoire in order to accommodate the varying conditions
associated with the task and environment and produce the range of
outcomes that help to control and pursue the task goals. In
addition, not all movements associated with a task have the same
importance to the task performance. Some movements are part of a
basic repertoire that cover the general performance and conditions,
and other movements are more specialized and allow actions in more
specific conditions.
[0301] Skill acquisition is a parallel process where at any given
time, a subject's repertoire will contain multiple movement
patterns, each at different stages of development. The two primary
directions in the skill acquisition process are: 1) the development
of a sufficiently broad repertoire to cover the task requirements
and conditions, and 2) the refinement of the movement technique
within each class of the repertoire to achieve better outcomes
and/or movement performance as well as adjust to conditions. These
two directions are referred in this document as the longitudinal
and vertical dimensions of skill acquisition. The longitudinal
dimension represents the stage of development or acquisition, which
is determined by skill characteristics in specific classes of
movement. The vertical dimension represents the aspects of movement
skills that have to be developed to cover the task conditions. At
any given time, the training can be directed at refining a movement
or diversifying the repertoire. The two dimensions are typically
interrelated. The differentiation of the repertoire in the vertical
direction often develops from the longitudinal process of
refinement of an existing movement.
[0302] The expanding repertoire, with its collection of movement
classes, and their associated outcomes, each at different stages of
development, results in a complex picture for anyone to operate and
train with, let alone comprehend. Therefore, efficient movement
skill acquisition can depend on the availability of appropriate
feedback targeting each movement type as well as on a systematic
method to prioritize and plan training. Human subjects also should
generate this information themselves, which requires mental
workload. Extracting useful information for training depends on
understanding how these movements satisfy the task constraints and
help meet its goals.
[0303] Finally, human skills rely on multiple levels of human
information processing, including signals, cues, and knowledge. The
knowledge level supports reasoning about technique, such as
particular details of the movement's spatial configuration. It also
supports game strategy, taking into account the environment and
task elements, etc. The cue level supports the efficient processing
of information; for example, the visual perceptual system learns to
focus on aspects of the scene and action that provide the most
valuable information for the performance. The signal level
typically encompasses the information used by brain processes to
control movement such as proprioception or actual visual
stimuli.
[0304] Human training does not effectively use the entire scope of
information levels. The range of information involved in the
movement skill process cannot easily be processed. Most human
training takes place through, and is codified and communicated,
using hands-on demonstrations and natural language. These
modalities work reasonably well for the cognitive aspects of
skills, such as introducing a new movement pattern. Many critical
aspects, however, relate to movement characteristics that are too
fast to observe, difficult to express verbally, or need to be
generated concurrently with the unfolding movement to be effective.
Even professional coaches cannot reliably generate cues and signals
during performance to support the training process. This is in part
due to limitations in human information processing, but also
because it requires excessive attention and mental workload for a
coach to simultaneously analyze and cue movement performance.
Roles of Technology
[0305] Technology can play a role in several areas of skill
acquisition. Technology provides a means of collecting
comprehensive information about human behavior that exceeds humans'
sensory processes' spatial and temporal resolution. For example,
the combination of distributed sensors in the form of wearable,
implantable, and remote sensors can capture comprehensive
dimensions of movement performance. This includes the movement of
an end effector such as a piece of equipment, an individual body
segment, muscle activity, as well as subjects' visual attention and
task relevant quantities (see FIGS. 2 and 24).
[0306] Information technology enables the deployment of analytical
and computational resources beyond humans' information processing
capabilities. Algorithms can be designed to estimate various
unmeasurable quantities, which can be used to provide feedback on
outcomes ("knowledge of results"), as well as more complex aspects
of performance such as those involved with the fast and
high-dimensional dynamics, and coordination with the environment
and task elements. This functional understanding can be used to
design feedback augmentations that target movement technique
("knowledge of performance"). Information technology enables
scalable deployment of analytical and computational resources
across larger populations, where it can be deployed to identify
patterns in movement technique and skill acquisition processes that
can take into account broad range of individual factors. However,
to be effective, these different augmentations and feedbacks should
be provided within a system that is compatible with the natural
movement mechanisms and learning process.
[0307] Aspects of technology for operation of the system-wide
data-driven training include: [0308] 1. Integration and analysis of
comprehensive performance data to assess an individual's movement
technique. [0309] 1.1 Diagnosing movement technique and identify
possible causes of outcome deficiencies, and other relevant
characteristics such as enabling efficient use of musculoskeletal
capabilities, as well as effects of fatigue, or onset of injury.
[0310] 1.2 Identifying most actionable characteristics in the
movement performance that can be used to drive training. [0311] 2.
Design of feedback augmentations that precisely target the specific
features of movement technique and help induce the changes needed
to achieve the training goals. [0312] 2.1 Selecting the feedback
augmentation and communication modality that are adapted to the
learning stage. [0313] 2.2 And, communicating feedback signals such
as real-time cues that leverage human information processing
capabilities. [0314] 2.3 Producing synergies between various forms
of communication, including visuals, natural language, and cues
across the human information processing hierarchy. Exploitation of
feedback signals and cues, as well as skill and performance
measurements, to stimulate attention and motivation. [0315] 3.
Operationalize training process driven by data to enable its
systematic and quantitative management. [0316] 3.1. Planning of the
training process through specification of training goals that are
based on the subject's skill and individual characteristics,
including fitness, physical strength, and health. [0317] 3.2
Tracking of the longitudinal and vertical dimensions of the skill
development process. [0318] 3.3 Tracking the effectiveness of the
different augmentation modalities and of training effectiveness for
the purpose of optimizing augmentation modalities and identifying
issues that interfere with progress, such as physical injuries or
psychological issues. [0319] 4. Combining data from a population of
subjects to discover global patterns in skill acquisition, movement
skills, and related factors such as injury, aging, etc. that can be
used to optimize performance training over larger training
cycles.
Core Technology Capabilities
[0320] Open motor skills require the development of a variety of
movement patterns to produce desired outcomes under changing task
and environment conditions. These movements and their associated
sensory-perceptual mechanisms are acquired from experience in a
task domain. Depending on the task or activity complexity, learning
motor skills can take several years.
[0321] Most of the motor skill acquisition in someone's life
follows a trial-and-error process. For advanced motor skills, which
rely on more complex movements, usually some forms of training
methods are used. Efficient training of open motor skills depends
on the availability of a range of feedbacks, including information
about the movement outcome (knowledge of results or KR), the
movement technique (knowledge of performance or KP), and overall
training progress and process.
[0322] As described previously, training in open motor skills
proceeds in two primary directions: the development of the range of
movement patterns that help to accommodate the range of conditions
and actions required by the task; and the development of optimal
movement techniques for each movement class to allow reliable and
efficient achievement of desired outcomes for a task. As a result,
training or rehabilitation requires emphasis on both variability in
conditions and outcome (see reference in Schmidt, 1975), and
mastery of specific conditions and outcomes.
[0323] The central idea of this technology is that the movement
performance at its various levels can be assessed computationally,
i.e. it can be computed, and then further diagnosed to identify
deficiencies at the various levels of the movement hierarchy, which
are needed to determine training goals. The training goals can then
be pursued through targeted training activities that can be
augmented by various feedback modalities. The following provides a
technical description of the capabilities needed to support
comprehensive data-driven skill assessment, and diagnostic and
training intervention for open motor skills. It introduces
definitions of the relevant quantities and processes that will be
formalized subsequently.
[0324] The section starts with the definition of relevant concepts
used to describe and quantify the skill acquisition process; its
assessments; the diagnosis and specification of training goals;
planning training; and finally, augmentations that can be used to
enhance training interventions. All of these concepts and
capabilities are described in general terms. They will be further
developed in the system's description and the process flow
description.
Movement Pattern Classes and Outcomes
[0325] As already discussed, the fundamental element of movement
behavior are the set of movement patterns that support the relevant
interactions with the environment and task elements. These are also
called primary movement units or skill elements. Most movement
patterns are directed at producing an outcome or action toward the
activity or task goal(s). The various movement patterns used by a
subject in a task can be identified and classified.
[0326] As can be appreciated from this description, the quality of
the skill assessment depends on the ability to extract relevant
movement patterns that characterize relevant interactions in the
task, and to classify these patterns according to their intrinsic
characteristics, i.e., the movement technique and movement phases,
and their relevance to the task, i.e., the movement outcomes and
the task conditions. This is in particular critical for open motor
skills, since the subject acquires a repertoire of movement
patterns to produce a broad range of outcomes under a range of
conditions. To ultimately provide feedback to help improve a
subject's skills, performance can be contextualized, which may
include identifying what movement technique is used under which
conditions and to produce what outcome.
[0327] FIG. 11 illustrates the acquisition and evolution of
movement patterns over time, highlighting the formation of movement
patterns either from scratch or through a process of
differentiation. At a different time in an individual's practice,
training or performance history (shown as stages S0, S1, . . . )
the movement skills can be described as a repertoire of movement
patterns (e.g., at S2 patterns P1-A, P1-B, P2-A, P2-B).
[0328] The width of the branches in FIG. 11 indicate the
variability in movement characteristics in a given pattern.
Beginner subjects tend to employ similar techniques to achieve a
range of outcomes and conditions. With experience, subjects learn
to perfect their control over the task conditions and can develop
movement techniques that are more specialized and yield higher
performance (more efficient, higher outcomes, more extreme
conditions). Therefore the general trend is for a subject to start
with a repertoire of a few movement patterns with fewer
capabilities, and with experience and training develop a larger
repertoire of more differentiated movement patterns.
[0329] A new pattern can form through differentiation of an
existing pattern (i.e., core pattern), shown here as a dashed line
that indicates the beginning of the differentiation process (e.g.,
differentiation of P1 into P1-A and P1-B at S1). Alternatively, the
pattern can form "de novo" such as illustrated for P3 at S3 in FIG.
11. Newly differentiated patterns next go through a consolidation
stage (shows as the bifurcation point at the end of the dashed
line, e.g., P1-A and P1-B at S2) where they each become distinct
patterns. After consolidation, patterns undergo a process of
optimization, as shown by a tapering of each branch into a tighter
distribution of patterns.
[0330] FIG. 12 shows several classes of movement patterns as
clusters for some parameterization such as features from the
measurement time histories. The clusters capture the pattern
differentiation that takes place as the individual improves their
skills. The example is based on the patterns at stage S3 in FIG.
11. The patterns that form following differentiation typically
appear as a mixture of two patterns, such as shown for P1-A1 and
P1-B2 in the original pattern P1-B. Patterns in an early stage of
consolidation show distinct features such as P2-A and P2-B.
[0331] FIG. 13 shows the family tree highlighting the evolutionary
relationship between movement patterns. Since some patterns form
through differentiation, it is possible to track the based on
features or attributes that are inherited. In FIG. 13, core pattern
to refer to the pattern that inherits the main attributes in the
development of the new patterns. The non-core patterns
differentiate to create new attributes that are distinct from the
core pattern.
[0332] Movement pattern classification is typically based on
movement profile features (e.g., racket angular rate or
acceleration). Movement outcomes are a consequence of the movement
performance and conditions, and therefore a function of the
movement characteristics (see FIGS. 3A and 3B). Consequently, some
movement profile features can be used to predict or estimate the
movement outcome. Viewed abstractly, therefore, the classification
task corresponds to identifying the structure of the extended
state-space X in FIG. 14. The state-space associated with the
entire human or system performance combines the typical states of
the systems, such as needed to describe the subject's or agent's
movement, as well as states that are associated with the task and
environment elements that participate in defining the conditions in
which a particular movement performance or pattern takes place.
Classification therefore can be conceived as the mapping from the
extended state space into its co-domain V.
[0333] FIG. 14 shows the mapping between the movement performance
state (state-space X) and the movement outcomes and other
attributes f.sub.i in V. The state space highlights the partitions
associated with various movement pattern classes. Movement patterns
are typically associated with features that originate from domain
characteristics (such as geometrical characteristics of the
manifold associated with task dynamics, interactions and various
constraints). The classification maps the state-space features to
the movement attribute space. Movement attributes include results
or outcomes (e.g., spin, pace, etc.), as well as other attributes
that can be used to assess the movement technique (consistency,
timing, smoothness, etc.), or performance (energy, etc.). These
attributes can be computed via analytical functions, estimated
statistically, generated using neural networks or even directly
measured (e.g., ball spin using computer vision). Each pattern has
a range of values for the particular outcome metric shown as a
partition.
[0334] Since the outcome is usually what the performer is trying to
achieve or control, and is often what they are most conscious and
deliberate about, it is helpful to depict the movement pattern
classification relative to the outcomes. For example, for tennis
strokes, a "stroke map" can be used to depict the different stroke
classes (forehand, backhand) as a function of the outcome: the spin
level (slice, flat, top spin) and pace (low, med, high) imparted on
the ball.
[0335] This example is illustrated in FIG. 15, where the dimension
O1 could represent stroke intensity and dimension O2 spin imparted
on the ball. Repertoire of movement patterns depicted relative to
the primary outcome dimensions (O1 and O2). The figure illustrates
the relationship between movement patterns and outcomes described
by the mapping f: X->V of FIG. 14.
[0336] The classes of strokes can be divided into subclasses. To be
intuitive, these subclasses have to represent different regimes or
conditions. FIG. 16 illustrates the relationship between the
movement patterns and their outcomes quantized based on ranges
defined by O.sub.11, O.sub.12, O.sub.13 for dimension O.sub.1, and
O.sub.21, O.sub.22, O.sub.23 for dimension O.sub.2. Such
relationships can be determined by embedding V into a subspace W
that produces meaningful outcome categories (semantic
interpretation), as illustrated in FIG. 14.
[0337] Furthermore, since the movement patterns and outcomes also
depend on task conditions, movement pattern classes can be
represented as a combination of outcomes and conditions. The
performer has to compensate for effects of conditions or even
exploit these conditions to their advantage in order to produce the
desired outcome. For example, in tennis, the ball comes into the
court with varying amounts of pace and spin. FIG. 9 shows three
interception types that are characterized by the impact
conditions.
[0338] Therefore, in addition to positioning the body to
successfully intercept the ball, the player has to adjust the
stroke execution to achieve the impact conditions that produce the
desired outcome. Typical adjustments of the stroke impact
conditions involve choosing the interception point relative to the
ball's impact on the ground, such as interception while the ball is
on the rise, when it is near or at the apex of the trajectory, or
when it is descending toward the ground. The conditions can have
dramatic effects on the ability to achieve certain outcomes. For
example, intercepting the ball on descent makes it easier to
produce top spin (because of the relative angle between the ball
velocity vector and the racket face orientation).
[0339] More advanced players are generally more conscious about the
conditions, since they will try to exploit the conditions to help
improve the outcomes, for example in FIG. 9 where backing off from
an oncoming ball affords the choice to intercept it on the descent,
which is advantageous for producing top spin. The subject can also
decide to intercept the ball on the rise or at the apex depending
on the desired outcomes at the different levels of the task (e.g.,
producing a shallow power shot deep in the court and down the line,
or clearing the player at the net). Therefore, extended repertoire
representation can include conditions as well as outcomes to
provide a more complete understanding of the subject's skills,
which in turn can be used to determine more complete and precise
training interventions.
Pattern Development and Learning Stages
[0340] To understand how to create meaningful interventions in the
skill development process, it is beneficial to understand the
brain's learning process. The acquisition of movement technique
proceeds according to relatively distinct stages, which can be
defined as follows: [0341] Pattern formation represents the first
stage of skill acquisition, the so-called cognitive stage. At this
stage, the subject forms a model of the movement such as the
outline of the movement spatial configuration. The movement at this
stage cannot be performed reliably because it relies on conscious
guidance and visual feedback, required to ensure that the movement
conforms to the model. [0342] Pattern consolidation refers to the
process of consolidation of the movement pattern from spatial
configuration (e.g., based on visual demonstration or verbal
description) into sensory-motor patterns that can be executed
dynamically without conscious effort. The movement patterns are
encoded as motor programs that can be performed in an open loop
(e.g., without visual feedback). This corresponds to the
acquisition of procedural memory. [0343] Pattern optimization
refers to the stage where a given movement pattern undergoes
further differentiation or refinement (e.g., by fine-tuning
technique and perceptual mechanisms), as well as developing
physical performance.
[0344] The acquisition stages manifest in movement characteristics
that are captured by the skill model. Therefore, the acquisition
stage can be assessed from statistics associated with the skill
attributes.
Movement Patterns Optimization
[0345] Note that the acquisition stages assume that skill
development takes place around a particular class of movement
pattern, and proceeds in successive stages from formation to
consolidation to optimization. It is helpful to realize that one
particular pattern may not be optimal in an absolute sense, but the
optimality of the movement is relative to a subject's particular
biological constraints (biomechanical system, physical strength,
health status). In this sense they can be considered as local
optima. Achieving globally optimal movement patterns in the
absolute sense requires building up various components involved in
the human movement system, including physical strength, the
neurological motor circuits that support response speed and
movement coordination, and other functions such as perceptual
mechanisms.
[0346] The overall capabilities of a subject, for example, manifest
in the range of possible movement architectures and their
corresponding functional capabilities. Therefore, movement skill
acquisition can be assessed and modeled by tracking the evolution
of the movement architecture, i.e., the sequence of movement phases
that makes up each movement pattern.
[0347] Movement patterns may go through several generations of
acquisition, each generation characterized by a specific movement
architecture and its associated functional characteristics (see
evolutionary process in FIG. 11). Within each generation, movement
patterns may progress through the stages of formation,
consolidation, and optimization. Newly acquired physical strength,
or other changes in constraints, can also prompt a new iteration in
movement architecture that will typically have to go through the
formation, consolidation, and optimizations stages.
[0348] When a particular movement pattern reaches the optimization
stage, limitations resulting from inefficiencies and other factors
will typically become apparent. Once the potential improvement
within the same pattern has been fully exploited, often the only
way to further improve the performance and outcome is to form a new
pattern. Therefore, it can be helpful to distinguish between the
training skills within the same generation of a pattern class, and
the training of a new pattern, or evolution of a pattern class into
a new generation (see movement architecture in FIG. 5). In some
cases, a new pattern generation emerges naturally from the
optimization. When a new pattern is formed, it will typically lead
to a momentary decrease in performance and consistency until it
consolidates and is eventually optimized.
[0349] This staggered acquisition process allows individuals to
perform optimally at the task level with their "suboptimal"
architecture. The skill development is interrelated with the body's
physical development. For example, a new movement architecture may
require physical strength and coordination that is not sustainable.
Therefore, some changes in movement technique can require
development of physical strength.
[0350] One factor that drives the evolution of the movement pattern
architecture is the opportunity to make movement more efficient.
Efficiency is determined by how well a performer is able to use his
or her biomechanics while protecting the body from wear and injury.
Typically, evolution of the movement pattern architecture follows a
development that proceeds from proximal to distal body segments.
Therefore, the architecture usually evolves to involve the
superposition of an increasing number of body segment motions.
[0351] In tennis, for example, the early generation of stroke
pattern is characterized by a simple backswing and forward swing
441 (see FIG. 4). The pattern is then refined as a performer learns
to exploit the multiple degrees of freedom afforded by their body
(legs, hip, torso, shoulders, elbow, wrist). The overall pattern
composed of multiple movement phases can be represented by a
finite-state machine (FIG. 5). For example, in tennis a typical
evolution in the stroke is starting from relatively simple,
lower-dimensional motions that exploit the basic biomechanical
capabilities, for example a basic backswing and forward swing
phases (see e.g., 4-state system in FIG. 5), to learning to exploit
and coordinate the larger degrees of freedom, for example using a
more elaborate backswing, with a back loop that transitions
optimally into the forward swing phase (e.g., 8-state system in
FIG. 5).
[0352] This process eventually extends into the entire available
body kinematic system. With training, subjects learn to exploit the
whole body kinetic chains, which involves movement that originates
at the feet, hips, torso, etc. Such movements are complex in the
sense of the spatio-temporal characteristics of multiple joints and
muscle groups. They also require more anticipation and therefore
rely on advanced perceptual skills and planning. Given these levels
of complexity, it is understandable why the complex movement skills
develop in stages.
[0353] The process of movement pattern refinement simultaneously
exposes the body to new and larger displacements with the potential
to create undesirable stresses on the joints, ligaments, tendons,
and muscles. Therefore, it is possible to conceive acquisition of
more advanced movement patterns as a process that's geared at
maximizing outcomes while minimizing strain and more generally
injury risks. Increased loads are also drivers for the development
of physical strength as well as musculoskeletal structure.
[0354] The process of movement pattern refinement simultaneously
exposes the body to new and larger displacements with the potential
to create undesirable stresses on the joints, ligaments, tendons,
and muscles. Therefore, it is possible to conceive acquisition of
more advanced movement patterns as a process that's geared at
maximizing outcomes while minimizing strain and more generally
injury risks. Increased loads are also drivers for the development
of physical strength as well as musculoskeletal structure.
Repertoire Development and Pattern Differentiation
[0355] FIG. 13. illustrates the evolutionary relationship between
movement patterns. Each movement pattern is identified in terms of
its ancestor(s) or parent(s) (shown in bold). The patterns shown
correspond to the ones in FIG. 11, ordered by which stage it was
formed along the evolutionary process (S1-S5).
[0356] As shown in FIG. 13, each movement also can be assigned a
degree of significance to the task that specifies how relevant the
pattern is to the task performance and goals, and is indicated as
primary, secondary, tertiary, etc. As described earlier, movement
patterns can either be formed de novo, or through differentiation
from an existing pattern. In the former case, the new pattern
typically fills a new need for the task performance, such as a
volley. In the latter case, new patterns typically form to expand
the range of outcomes or conditions. For example, in tennis, a
generic forehand stroke can evolve into several subclasses to
achieve specific ball spin and pace such as to better control the
outcome of the stroke (see FIGS. 11 and 12).
[0357] At the onset of skill learning in a new activity domain, a
subject typically starts with some rudimentary movement
capabilities. These early movements are typically adapted from the
repertoire they have acquired in other activity domains, or by
combining general movement primitives that are available from their
neuro-motor repertoire. At the beginning (S0 in FIG. 11), consider
the two movement patterns P1 and P2. For example, these could
represent a forehand and backhand stroke. At this very early stage,
the movements are not yet specialized. Beginners typically employ a
few movement patterns they try to adapt over a broad range of
outcomes and conditions. For example, in tennis, beginners may have
one forehand and one backhand stroke pattern to accommodate a broad
range of conditions such as return balls from an opponent under a
variety of conditions (e.g., pace, spin, impact point, interception
height, etc.).
[0358] Since beginner movement patterns have to accommodate broad
conditions, they cannot exploit the subject's movement capabilities
in an optimal fashion, i.e., using the same general movement
pattern for a range of conditions compromises its performance.
Therefore, to achieve optimal performance over a range of
conditions, multiple specialized movement patterns have to be
formed. These are optimized both for the perceptual conditions, and
the biomechanical movement conditions needed to support the range
of outcome.
[0359] With more experience, the subject learns to exploit his or
her biomechanics and to identify conditions in which movement
patterns can be specialized to produce more reliable outcomes. For
example, in tennis, a player may learn to generate top-spin on a
return, enabling a more aggressive return stroke with increased
pace that requires tighter control over timing and conditions, or
to return with a slice, which tolerates a broader striking
area.
[0360] With more extensive experience, the player also learns to
link the stroke patterns with the larger task hierarchy, in
particular they focus on improving the task performance, i.e.,
producing outcomes at the task level. In tennis this includes the
precise placement of shots on the court; at the same time, also
broadening the regions that can be targeted, while also learning to
target these regions from a range of impact location and
conditions. The development of the repertoire at the task
performance level can be assessed from the discretization of the
court environment shown in FIG. 8.
[0361] Movement specialization or differentiation at the pattern
level is illustrated in FIG. 11 at time S1, where the P1 patterns
start its differentiation into two distinct patterns P1-A and P1-B.
At the early stage of this differentiation process, the movements
still have overlap in their characteristics, as shown as mixture in
FIG. 12 for P2-B. Therefore, there will be variability in the
technique and unreliability in the performance.
[0362] Eventually, as shown at S2 in FIG. 11, the two patterns
begin to be sufficiently differentiated to represent distinct
movements in terms of their technique. As described elsewhere, the
movement technique is formed by sequencing movement phases that
build on muscle synergies. The development of movement technique,
therefore, also relies on the development of physical strength
along with motor coordination.
[0363] As the different functions supporting movement are formed, a
subject can begin optimizing their movement. At S3, following the
same process as for P1-A, P1-B differentiates into more specialized
patterns. Patterns can be further differentiated as a result of
ongoing refinement or optimization of the technique. For example,
S4 shows the optimization of pattern P1-A. Optimization requires
narrowing down on operating conditions and technique; therefore,
the patterns begin to have more restricted domains of operation
which leads to two new sub-patterns P1-A1 and P1-A2.
[0364] As a result of the learning process with different learning
stages (formation, consolidation, and eventually optimization),
subjects operating in an open skill domain expand their repertoire,
and, at any given time, subjects will have movements that are at
different stages of development. Even a relatively proficient
player in a sport may be required to form a new movement pattern,
or change an existing pattern to such a degree that it loses much
of its relationships with the original pattern.
[0365] To help describe the various phenomena in skill acquisition
process and computational processes in its analysis, it is useful
to define different time periods. The following terminology are
used: [0366] Epoch refers to time periods associated with a data
set that is associated with a particular model (see assessment loop
described later). [0367] Learning/acquisition stage refers to time
periods associated with transitions in a subject's neurological
learning process for a particular movement pattern (formation,
consolidation, and optimization). [0368] Developmental stage refers
to time periods associated with evolutionary milestones in the
development of a player's larger movement pattern repertoire.
[0369] Generations refer to time periods associated with
differentiation in a player's overall skill profile (e.g., as it
relates to other player subgroups), based on the aggregate
contribution of skill, technique, etc. This information can be
captured by the player subgroups through population analysis
described later.
Modeling Movement Pattern Development
[0370] As a movement pattern evolves, it can differentiate, and/or
new patterns can be formed from scratch. Therefore, several
patterns can coexist in the same class, i.e., supporting the same
outcome and task interactions (see FIG. 11). Sometimes, the classes
are differentiated by the conditions, i.e., they represent the same
outcomes but under different conditions. Typically, these patterns
evolve sufficiently to give rise to distinct classes for example
specialized in a specific range of outcome and conditions.
[0371] Therefore, when processing and analyzing movement skills as
a process of movement pattern evolution and development,
inheritance relationship between patterns can be considered. In the
following we define a core pattern (CP) as the primary pattern
descending from an ancestor, in contrast to new patterns that
emerge through differentiation. In FIG. 13 the core patterns are
shown by a solid edge to underscore that they inherit the main
attributes in the development of the new patterns. The non-core
patterns, linked by dashed edges in FIG. 13 differentiate to create
new attributes that are distinct from the core pattern.
[0372] The core pattern often corresponds to the predominant
technique in that class, for example that is further consolidated
in procedural memory. Under challenging conditions, the subject may
tend to fall back on that pattern. The core pattern may also be
more difficult to change because of its long-standing history.
[0373] This conceptualization of movement learning as an
evolutionary process combining the development of new patterns
through differentiation, as well as formation of patterns de novo,
is useful to assess the longitudinal skill acquisition process.
This involves relating the patterns through features that are
inherited as they differentiate and tracking the stage of learning
of the patterns through the differentiation process. The
hierarchical classification of patterns can determine the
hierarchical relationships between the classes. These structural
characteristics can be exploited to design training interventions,
and plan and manage the training process. For example,
interventions that help new patterns form through differentiation
and consolidation.
Movement Planning and Perceptual Mechanisms
[0374] For open motor skills, successful performance depends on
extracting various forms of information from the task environment
and elements. Many actions and movements need to be synchronized
with the task environment and elements. Learning movements also
involves learning the perceptual mechanisms used to extract
relevant information and using this information to plan or adapt
behavior.
[0375] Proficient movement technique and overall task proficiency
rely on the formation and optimization of perceptual mechanisms,
for example the ability to recognize the state of an incoming ball
and adjust the stroke to these conditions. For example: a fast,
high bouncing ball is returned with a slice, enabling a more
reliable but less offensive return. In addition, if a player can
extract early cues to estimate the return location (e.g., from an
opponent's body and racket swing) they can control the point by
selecting the return and positioning the body to precisely
intercept the ball in the strike zone to achieve the desired return
trajectory.
[0376] A broad movement repertoire allows a subject to select the
best actions needed to control the state of the activity based on
the task state and conditions. For example, a tennis player may
take advantage of a slower, shorter return to intercept the ball
earlier and produce large top-spin and pace as a way to surprise
the adversary with a deep return in the open side of the court.
Alternatively, under an offensive return from the adversary, the
player has less time to prepare a stroke and uses a slice to gain
time before the adversary's return. These changes reflect the
subject's ability to assess the situation and use this information
to control the task and achieve its goal, while at the same time
adapting to the environment and conditions.
Assessment of Open-Motor Skills
[0377] Most open motor tasks involve dynamic interactions with the
environment, combining different outcomes, at different levels of
the motor system hierarchy, levels of the information processing
hierarchy, and task structure hierarchy. Proficient performers are
able to combine these processes and components into an organized
whole. Therefore, skill assessment of open motor skill has to
encompass these different levels and components, which presents
some unique challenges both from an analytical and data
acquisition, or practical standpoint.
[0378] Tennis is a good example where every stroke is executed at
conditions that depend both on the player's control over their
position relative to the moving ball and the stroke technique.
Therefore, the assessment of skills encompasses different aspects
of the performance, and is enabled by defining outcomes that can be
defined based on how actions influence the task and environment
state over multiple levels of the motor system and task structure
hierarchy. A useful to way to formalize this analysis is to
investigate the various interactions between elements of the human
system, the participants, equipment, and environment and task
elements.
[0379] As an example, based on the tennis use case, FIG. 7 shows
relevant interactions in the larger system and illustrates the
following outcome levels, which are also shown in FIG. 2: [0380] 1.
Stroke/racket ball impact: impact conditions. [0381] 2. Impact and
shot primary outcome: ball velocity and spin. [0382] 3. Shot
trajectory and type relative to the environment elements, e.g., net
clearance, curvature, velocity, spin. [0383] 4. Shot placement
overall relative to the opponent and court landmarks.
[0384] These levels are defined based on the various interactions
between the agent and relevant task and environment elements and
form a nested closed-loop system. They underscore the general idea
that human behavior is relational, i.e., the behaviors are anchored
in specific object relations, which is both a result of how humans
perceive and conceptualize the environment (in contrast to machines
which are often based on discretization of certain dimensions).
[0385] Note that the outcomes at levels 2)-4) are all function of
the performer's ability to control the ball and manage the impact
conditions) (see FIG. 9). Therefore, controlling the ball and the
stroke execution conditions depends on the ability to perceive and
anticipate the task environment state, move on the court, prepare
the stroke, and establish appropriate posture.
[0386] Note also that behavior and performance in open-motor tasks
depend on the full range of human information processing: the
abstracted task level rules and organization, the discrete elements
and events associated with individual movement unit selection and
execution, and the continuous process of the physical movement
performance. It is possible to delineate the primary information
processing components corresponding to each outcome level.
[0387] These primary information processing components are
responsible for acquiring knowledge of the results and associated
knowledge of performance that in turn can be used to help improve
skills. Therefore, from a skill assessment standpoint, it is
critical to understand which outcome levels are processed at each
assessment level and, at the same time, provide knowledge of
performance that can be converted into an actionable training
intervention.
[0388] For example, stroke-impact and primary stroke outcome are
most directly related to motor control processing (body
coordination, and ball interception). The performer can assess
these outcomes through proprioception, including how the racket
"feels" at impact, and the resulting shot. But the latter does not
provide as much information about the movement technique or
knowledge of performance.
[0389] The shot trajectory and placement are most directly related
to the planned shot and game strategy but also depend on the
performer's performance and control of the ball and conditions. The
performer assesses these by perceiving the ball trajectory relative
to the court and the opponent and the impact on the game.
Information from this level helps improve the positioning and shot
selection and game strategy. However, training at this level relies
on sufficient facility to control the ball and achieve sufficiently
precise outcomes at levels 1)-3).
[0390] In activities that don't have an adversary, such as skiing
or surfing, the strategy level is concerned with the negotiating
the environment and conditions. This requires reading the terrain
and conditions, and planning the deployment of a sequence of
movement patterns.
[0391] The dynamic coupling and the multiple levels of information
processing make it very challenging to assess and produce effective
training interventions. The stroke and ball impact conditions
manifest directly in the stroke-impact quality 1), and primary
outcome 2), making this process the most directly observable;
however, it also depends on the ability to predict the ball
trajectory and anticipate and select the interception condition.
The other outcomes 2)-4), on the other hand, accumulate other
factors, making the diagnostic task difficult. With technology, it
is possible to tear these confounded contributions apart, thereby
delivering analysis and creating training interventions at the
appropriate outcome levels and the appropriate information
processing level.
[0392] The following describes the framework that was conceived to
make improved data-driven assessment and training possible.
Assessment and Diagnostics
[0393] Training relies on the ability to 1) assess motor skills,
which corresponds to the description of the movement outcomes and
characteristics in relationship to task requirements, and to 2)
diagnose skill, which corresponds to the identification of specific
aspects of the movement technique that are deficient and reduce
performance in the task through their effect on critical outcomes
(diagnostics). The gained knowledge can subsequently be used to
determine adequate interventions that address the specific skill
deficiencies and lead to a higher skill level and hence task
proficiency.
[0394] Skill assessment is responsible for characterizing the
movement performance. Producing an assessment is essentially the
challenge of defining metrics and features from collected movement
data that provide a concise and useful description of a subject's
performance outcome and technique (knowledge of result and
performance). For example: "The ball spin produced by the impact is
too low for the forehand top-spin high-strength (FHTSH) class."
[0395] Skill diagnostic is responsible for identifying the causes
of movement and task performance characteristics. It typically
focuses on the deficiencies that need to be addressed or corrected
to improve the skills towards a task performance. For the previous
example: "Racket height at forward stroke initiation is too high
and racket roll rate profile is too shallow."
Assessment Levels
[0396] Building on the components of the movement and skill models
in U.S. Patent Application Publication No. 2017/0061817
(illustrated in FIG. 6), skill assessment proceeds hierarchically,
accounting for movement performance at the different organizational
levels of the human movement system which also couple with the task
structure hierarchy. FIG. 10 illustrates the relationship between
the levels of the movement hierarchy and the task hierarchy. It
defines the following levels of assessment: [0397] Physical
performance level: The assessment at this level focuses on the
physical details of how the movement is produced. This level is
best analyzed at the level of movement phase segments, including
considerations such as the movement phases and relationship with
muscle synergies, the musculoskeletal constraints, and sensory and
perceptual processes used to execute and deploy the movement in
respective task conditions. [0398] Pattern performance level: The
assessment at this level focuses on how well the movement pattern
associated with the primary movement unit support the task and
environment interactions, and more specifically produce outcomes
that contribute to the task goals and adapt or take advantage of
conditions. This level is best analyzed through the movement
pattern and outcomes, for example in tennis the stroke and shot
relative to the court, as well as the oncoming shot and conditions
(see FIG. 9). [0399] Task performance level: The assessment at this
level focuses on the relationship between the acquired skill
elements and the task requirements. This level of assessment is
best analyzed through the repertoire. It includes considerations
such as what types of patterns have been acquired to support
critical task interactions such as producing the range of outcomes
and adapting to conditions, and how these outcomes and interactions
collectively contribute to the task or activity performance. In
analogy to robotics or trajectory planning, this level corresponds
to the assessment of the discretization of the task space, i.e.,
how the overall range of outcomes and conditions are quantized into
distinct patterns that collectively provide the skill elements to
perform the task proficiently. [0400] Competitive performance
level: The assessment at this level focuses on how the subject uses
their acquired skill elements in a task, while considering the
subject's strategy and more generally how they compare with other
performers. It is best analyzed at the level of repertoire but
taking into account how the movement patterns and capabilities are
utilized to support and enable competitive performance. The
assessment encompasses strategic characteristics that may, for
example, be used to outperform an adversary, both in a static way
as well as dynamic, which corresponds to modeling the temporal
relationships between movement patterns and events in the task and
environment, as well as participants.
[0401] Accounting for the hierarchical relationships between the
levels in the assessments makes it possible to leverage these
relationships in the design of training interventions.
Assessment Components
[0402] Assessment components refers to the different perspectives
that can be taken on the movement performance and skills and follow
from the assessment level analysis that was just discussed and is
summarized in FIG. 10. The following components can be considered:
[0403] Outcome characteristics: The outcome assessment corresponds
to the traditional knowledge of result and performance. The
outcomes capture specific qualities of the movement pattern, their
effects on the task environment, and the associated conditions in
which they are executed. Outcomes are defined and analyzed at
different levels of the movement system such as the different
outcome levels defined in FIGS. 7 and 8. [0404] Functional
characteristics: The assessment focuses on the underlying
mechanisms of the movement pattern classes and their effect on the
task. The functional analysis is usually connected to the various
outcome quantities and the range of conditions required for a task.
For example, functional analysis at the pattern level considers how
the movement phases combine to produce the movement pattern that
support the interaction with the task and environment level, and
produce the primary outcome for the task. Functional analysis also
encompasses the perceptual mechanisms, for example those used to
support synchronization with the environment and task elements. At
the physical performance level, the functional characteristics can
encompass the details of biomechanics and muscle activation (muscle
synergy). [0405] Perceptual characteristics: This assessment
highlights the quantities that can drive the subject's behavior
across the different assessment levels. For example, at the
physical performance level, perceptual quantities correspond to the
proprioceptive features of the movement phases that are critical to
the execution of a particular movement pattern. Perceptual
mechanisms are part of the functional characteristics, they are
separated as a component to emphasize their potential role as part
of cueing for example. [0406] Memory and learning characteristics:
Movement characteristics and skill level depend on the movement's
acquisition stage, which refers to specific milestones associated
with the brain's learning process. This assessment focuses on the
identification of the learning stage of a movement pattern, which
can help better select diagnostic tools and training interventions,
such as cueing to reinforce sensory-motor patterns, or
visualizations that can help form mental models.
[0407] FIG. 10 illustrates the different levels of assessment
highlighting the representative elements 280 of the model at each
level for the tennis example. The figure summarizes the assessment
and diagnostic components 290 that are applied across the different
levels. The illustration also conveys how the different levels are
nested into one another, going from the movement segments at the
bottom, which are used to form the stroke patterns; then, how these
patterns enable the shot interaction with the court environment;
next, how the different strokes and shots collectively discretize
the task space; and finally, the decision making and strategy
driving the task competitive performance.
[0408] FIG. 31 provides a different perspective with a description
of the: a) the levels of assessment, b) the central elements that
describe that level, c) the criteria and quantities that can be
used to determine the skill characteristics at that level, d) the
analysis or diagnostics to identify the critical characteristics,
and finally, e) the drivers and mechanisms used to produce training
interventions.
Assessment of Outcomes
[0409] The outcomes represent the primary result of the movement,
as viewed from the perspective of the task. As already discussed
and described in FIG. 10, outcomes can be defined at different
levels of the movement system hierarchy and task structure
hierarchy. Outcomes are quantities that provide relevant
information for the task performance and skill assessment. They are
typically designated based on the task requirements and available
measurements.
[0410] One type of outcome is success rates. Success and success
rate can be determined at different outcome levels (see FIGS. 2 and
7). For example, in tennis the success at the racket-ball
interaction level (Outcome 1) is determined by the racket impact
location and outcome level for the particular class such as spin
and pace. At the court interaction level (Outcome 3) it is
determined by the court impact location and state (see FIG. 8).
[0411] Every stroke class is characterized by the range of values
that characterize the functional model, which includes states at
phase transitions such as the racket states at the beginning of the
forward swing, or the racket orientation, the racket angular rate,
etc. at impact, etc. These characteristics can be used to determine
outcomes at different levels, including the ball spin and pace, but
also the shot trajectory. With the additional information about the
player position and orientation, it is also possible to predict and
estimate the ball impact location on the court.
[0412] An example of this in tennis includes a comprehensive motion
capture system that measures the task object (tennis ball) relative
to the task space in addition to the subject's body segments, body
pose, motion of the equipment, etc. It is possible to more directly
assess the outcome of the subject's motion. In addition to the
quality of the outcome, another attribute is the success rate of
movements for each specific movement class.
[0413] FIG. 8 shows respective shot placements based on ground
impact distributions for a player and an opponent. The skills at
the shot level manifest as different resolutions and precision in
interactions with the task environment. The task level performance,
which will be described next, while depending on the stroke, puts
more emphasis on the shot outcome level such as how the stroke used
by the player can control the ball relative to the court and
opponent (see FIG. 7).
[0414] The collection of movement classes in the repertoire and the
information extracted from the movements in the
repertoire--including the outcome, success rate, and other
metrics--form a subject's skill profile. The skill profile
represents a holistic description of the subject's skill, which can
be used to compare players as well as track how skill evolves over
time.
[0415] The movement class technique assessment looks at the overall
characteristics of the movement pattern. As described earlier, each
movement pattern can be described by a so-called core pattern (CP).
The idea is that the movement follows a motor program which has a
template with specific variability due to disturbances and
adjustments made to adapt to conditions. The CP therefore describes
the nominal movement performance.
[0416] Deviation from a CP can therefore be used to assess
technique and other attributes such as adaptability. Even under
perturbation, movement pattern should be distributed around the
nominal range of CP, i.e., normal range of variations. Movements
that exceed the normal range can represent poor execution or may
also be of a secondary pattern that may be due to differentiation
of the core pattern as part of the normal skill learning.
[0417] It is expected that as an individual's skills improve, the
range of variations of the CP decrease. This is in part due to the
specialization and optimization of the pattern as well as the
tighter compensation over effects of conditions.
[0418] Movement differentiation can be detected from the presence
of a secondary pattern grouping, distinct from the CP, within a
hierarchical movement pattern class. This type of differentiation
is especially likely in the early stages of skill acquisition, when
new patterns are derived from existing patterns.
Functional Assessment
[0419] Functional assessment was described in detail in U.S. Patent
Application Publication No. 2017/0061817. In the following, it is
extended to the different assessment levels and task hierarchy.
FIG. 2 illustrates an interaction between a stroke motion and the
task and environment elements, including the ball trajectory
relative to the court, the impact of the ball, and its bouncing
before the interception with the racket trajectory. The figure also
illustrates the gaze of the player along different points of the
ball trajectory and environment elements, and shows a ball machine
as an apparatus that can be programmed to enable different forms of
interactions.
[0420] FIG. 2 also illustrates details associated with the
functional characteristics of the stroke pattern and the
interaction with the environment to produce a desired outcome
(e.g., Outcomes 1-3). The interactions include for example adapting
to conditions, such as the timing of the movement phases relative
to the ball state following ground impact 32 (see also FIG. 9).
[0421] FIG. 2 also shows examples of visual cues that are used to
control the movement execution, such as the ball trajectory
curvature, the magnitude and angle of the bounce or impact 32. The
figure depicts the visual attention based on the gaze vector 81 to
some of these cues, as well as the elements that are relevant for
the Outcomes 1-3 indicated by labels 33-35.
[0422] Delving deeper, FIGS. 3A and 3B illustrate the movement as a
sequence of phases and highlights phase transition characteristics,
and phase profile characteristics. The phase profile
characteristics refer to the dynamics during the phase segment.
These characteristics are associated with the coordination of the
movement segment and muscle synergies. The figure also shows the
feasible envelope that results from musculoskeletal and other
constraints, an admissible envelope that represents movements that
produce acceptable outcomes but are suboptimal, and an optimal
envelope that represents the range of motions that produce the best
outcomes with the best use of the biological system.
[0423] The figure also introduces the concept of a goal phase,
which represents the phase associated with the primary interaction
relevant to the production of an outcome and environment and task
element interactions. The goal phase in tennis is the movement
segment that corresponds to the ball impact and extends throughout
the ball interaction or contact. This phase is critical in the
production of the outcome. Some details illustrating the functional
analysis for the forward swing phase is discussed in a later
section and illustrated in FIG. 37. The other phases (initiation
phase, phase 1, phase 2, follow-on phase, and recovery phase)
represent a sample of phases that can be used in a movement pattern
such as the tennis stroke.
[0424] With the designation of a primary outcome phase, it is
possible to conceive of the rest of the movement as the system
organizes around that goal phase to support the outcome. The
different segments play different roles in supporting the
production of an outcome, as well as supporting the adaptation to
conditions and interactions with the environment that may
contribute to a robust and versatile performance.
[0425] From this more general perspective, every phase could have
its own outcomes and interactions. In tennis for example, the
forward swing phase (phase 2) is the phase that is the next
critical one after the impact because the conditions achieved in
the goal phase (impact) are determined by that preceding phase.
Furthermore, in the case of the tennis stroke, the forward swing
phase lasts about 100 ms and therefore is too fast for the player
to make any corrections. Therefore, the forward swing phase is
largely determined by its initial conditions x(t=t.sub.02), which
in turn are determined by the back-loop phase (phase 1). Similar
general characteristics can be found in other movement
activities.
[0426] Transition characteristics are determined by the movement
configuration, including the state of the body segments and end
effector such as the racket. These conditions also include timing
characteristics, such as the synchronization with the environment
elements. For example, in tennis, a relevant timing is the
synchronization between the tennis stroke initiation phase and the
tennis ball state, which itself can be delineated into different
phases, such as the net crossing, ground impact, and various phases
before the ball impact (see conditions in FIG. 9). Next, the timing
of the forward swing phase initiation (phase 2) is determined
similarly by ball state and the anticipated impact conditions but
closer to the impact time. This synchronization and modulation of
the movement phases are instrumental in achieving an accurate
interception of the ball and producing the desired impact
conditions (target phase) that will lead to a successful outcome.
Note that similar considerations can be made about the rest of the
body segments and configuration.
[0427] The skill element therefore can be defined formally in terms
of these primary interactions and the skill characteristics and
determined from the various attributes of these interactions,
including: the movement functional characteristics (described by
the movement phase characteristics and perceptual and motor
interactions), the musculoskeletal characteristics, physical
performance, and different levels of the task and motor system
hierarchy.
Assessment of Learning Stages
[0428] The assessment of movement technique (knowledge of
performance) to determine what and how to train, ideally requires
taking into account some of the neurological properties of the
motor skill acquisition process. Learning stages are defined based
on motor learning theory, including memory representations and
cognitive strategies (see Rosenbaum 2010).
[0429] The following movement acquisition stages can be defined
from the three learning states described earlier: movement
formation, movement consolidation, and movement
refinement/optimization. The movement acquisition stages manifest
in movement characteristics and can be described as follows:
[0430] Patterns to form (e.g., FIG. 52B, step 322): Patterns are
missing from the repertoire or exist in unreliable form. The
missing patterns typically are due to the lack of differentiation
among existing motion patterns. For example, in tennis, the absence
of subclasses in backhand topspin represent gaps in possible
operating regimes and possible outcomes such as pace or spin. These
gaps in movement repertoire preclude flexible production of outcome
and adaptation to conditions and therefore manifest in task
performance.
[0431] Patterns to consolidate (e.g. FIG. 52B, step 323): Movement
phases are not yet sufficiently defined and integrated in the
movement pattern to allow reliable execution under dynamic
conditions. For example, the muscle synergies associated with the
phases are not yet fully automated and their transition are not
smooth. These deficiencies manifest as unreliable outcomes,
variability in movement pattern, lack of smoothness, inefficient
movement performance, and do not have sufficient flexibility to
deal with changing conditions. After early formation and
differentiation, patterns undergo automatization and refinement in
their structure. These changes reflect the brain's learning
mechanisms (e.g., procedural memory). The automatization allows
repeatability and reliability. The refinement of the pattern
structure is guided by the functional requirements, including
achieving better outcomes and physical efficiency, as well as
effectiveness with the task and environment constraints and
conditions.
[0432] Patterns to optimize (e.g., FIG. 53B, step 324): Movement
patterns do not achieve the outcomes efficiently and do not adapt
sufficiently to environment or task conditions. For example,
movement phases do not make optimal use of the subject's
biomechanics. These deficiencies for example, may result in
excessive use of force when seeking an increase in outcome.
[0433] Skill acquisition stages also manifest in physical changes,
including gaining sufficient strength and endurance to sustain good
technique over time.
[0434] The acquisition stage is captured by the concept of skill
status. For each existing class of movement patterns in the
repertoire, it is possible to assign a skill acquisition. The
acquisition stage can be determined based on quantitative criteria
or metrics. For example:
[0435] Missing patterns can be determined by the repertoire
completeness, i.e., how well the movements in the repertoire cover
the performance requirements associated with the task objectives
and environment conditions. Typical pattern analysis tools, such as
clustering combined with similarity measure (e.g., dendrogram) can
be used to identify new patterns within an existing movement class.
The degree of differentiation of a pattern relative to other
existing patterns can provide a measure of its development. [0436]
Patterns to consolidate can be identified by success rates,
variability in technique, and outcomes within a given class. At
this stage, movements also tend to display particular physical
performance characteristics, such as high jerk, lack of smoothness,
and timing variability. These patterns can also be identified by
inconsistency in movement phase structure, smoothness of phase
transitions, as well as unreliable timing of some movement phases
(e.g., forward swing acceleration profile). Finally, patterns at
this acquisition stage can also be identified from the lack of
flexibility in adapting outcomes to the range of conditions and
outcomes. [0437] Patterns to improve or optimize are already
formed, but the movement structure does not utilize the subject's
biomechanical potential efficiently, and does not achieve the
theoretical range of outcome and level of flexibility helpful to
deal with the range of conditions. Patterns to optimize are
primarily analyzed from the functional characteristics (feature
analysis described elsewhere), which provide a detailed
understanding of the relationship between the movement technique
and its relationship to outcomes. Frequently also relevant is
movement efficiency, i.e., the work required to produce an outcome.
One goal of movement optimization is refining movement technique to
use the least energy and produce the least strain on the
musculoskeletal system.
[0438] FIG. 41 provides an example of the acquisition stage
assignment for the skill elements in the groundstroke
repertoire.
TABLE-US-00002 TABLE 1 Qualitative characteristics used to
determine the acquisition stage of a movement class. Patterns
Patterns to Patterns to to Form Consolidate Optimize Outcome Poor
results Variability in Optimality outcome Flexibility Technique
Weak structure Variability in class Adaptability to Poor Unreliable
structure conditions differentiation (smoothness) Functional
Performance N.A. High Jerk Efficiency
TABLE-US-00003 TABLE 2 Quantitative criteria that can be used to
identify the acquisition stage of a movement class. Patterns
Patterns to Patterns to to Form Consolidate Optimize Primary
Smoothness Consistency Outcome level Criterion Timing Efficiency
Range of conditions Secondary Outcome Smoothness Timing Criterion
Outcome Smoothness Consistency Minimum SR > 50% SR > 70% SR
> 85% Requirements (success rate SR)
Population Analysis
[0439] Population analysis is valuable to understand the
contribution of the broad range of factors intervening in the skill
acquisition process. Population analysis can be used to determine
player types based on skill levels and a variety of other factors
such as body type, health, etc. The player type or profiling makes
it possible to generate appropriate reference outcome values by
accounting for groups of players with similar technique types and
skill levels. The player profiling at the same time enables
identification of the player characteristics or attributes, i.e.,
what skill attributes and other factors such as development stage,
are characteristic traits of a particular player group. The player
profile information can be used, for example, to determine weights
in the composite scores that determine the larger player
characteristics.
[0440] FIG. 29 illustrates the process of generating population
groups based on the performance and skill data from the
hierarchical movement model. The information extracted from the
population analysis makes it possible to determine performer
profiles.
[0441] FIG. 30 illustrates assessment across the skill-model
hierarchy, incorporating player profile information to generate
reference attribute values used to assess the skills at the
different levels of the movement system and performance hierarchy.
The reference values can be used to provide contextual information
to determine what training interventions to pursue.
Physical Performance Assessment
[0442] The movement physical implementation describes how each
movement is composed from distinct phase segments, where each
segment is typically associated with the coordination of a specific
set of body segments driven by so-called muscle synergies. The
performance criteria at this level includes how the biomechanical
system supports the phase segments, for example, which muscles and
joints are involved in the motion as a function of segment profile
dynamics and phase transitions.
[0443] The analysis at the movement phase level is based on
identifying the components of motion such as muscle synergies and
other musculoskeletal quantities. There are overlaps between the
segment level analysis and the functional movement analysis, in
particular when it comes to the critical movement phases, such as
the forward swing in the tennis stroke.
Pattern Performance Assessment
[0444] The individual movement segments combine to form the entire
movement pattern. This pattern represents the basic skill element
supporting the various task interactions. At that level, the
primary movement performance criteria are the movement outcomes
relevant to the task performance, as well as how movement adapts to
the task conditions. The analysis focuses on identifying features
or attributes that explain the relevant qualities of the outcomes
and conditions (e.g., using sensitivity analysis). These features
provide the quantities that can be manipulated through training
interventions to optimize movement technique. One question is to
determine the most actionable features or attributes and synthesize
feedback cueing or other augmentations such as instructions that
can be used to produce effective training intervention. Note that
other criteria relevant to movement technique and performance can
be considered, such as movement efficiency or injury risks.
[0445] Movements involve the spatial and temporal coordination of
multiple motion degrees of freedom. Detailed skill models focus on
the functional aspects of movement characteristics that support the
performance of the movement outcomes, the interaction with the
relevant task and environment elements, and the adaptation to
conditions.
[0446] More detailed assessments at the level of movement technique
can be performed by decomposing the patterns into segments. The
analysis of the movement's functional characteristics, for example,
determines movement efficiency in producing specific outcomes,
synchronization with task and environment event, and the ability to
compensate for conditions.
[0447] An example of a functional skill model for a stroke is the
coordination between racket roll and swing rate during the forward
swing phase, which describes the technique of the subject that may
be important for top spin. The model can be used to identify a
subject's "spin envelope," in a particular movement class (see
details the following sections, see also FIG. 33).
[0448] Similar models can be derived for other characteristics of
the forward swing and other phases of the stroke. For example, the
racket motion results from the superposition of several components
of body motions including the trunk, the shoulder, the forearm, and
the wrist. The ability of the subject to achieve the desired impact
conditions and result, as well as compensate for conditions,
depends on the proper timing and coordination of the body segments.
With sufficient measurements, it is possible to estimate the
contribution of the movement components of the different body
segments and determine outcome variables that characterize the
biomechanical performance based spatial or temporal profiles.
[0449] FIG. 9 illustrates the interception and impact conditions
and primary stroke and shot outcomes. These conditions affect the
outcomes but also represent characteristics of the movement pattern
classes, since these patterns are fundamental to the interactions
in the task. Therefore, movement classes are typically
characterized by the movement technique (stroke type), the stroke
outcomes, and the conditions under which the action takes place
(interception and impact conditions). Notice that the interception
conditions are determined by the players' movement on the court and
their ability to anticipate and plan their actions.
[0450] Temporal characteristics are also critical to the movement
performance (see analysis of ping pong stroke timing in Bootsma
1990). For the tennis example, two timing characteristics are
included for the assessment of the forward stroke: the instant of
peak racket angular rate relative to the impact, and the time of
the forward stroke initiation relative to the impact (see details
in subsequent sections, see also FIG. 42).
Task Performance Assessment
[0451] Individual movement patterns combine to form a complete
repertoire of skill elements that provides an individual subject
the range of interactions needed to effectively perform a task. At
the level of the task performance, the analysis determines how the
movement patterns are deployed in a task and how they collectively
contribute to task success.
[0452] At the repertoire or task level, the outcomes are related to
how movement patterns change the state of the task and adapt to
conditions and various contingencies that can arise in a task; for
example, producing shot placements that drive the game and adapt to
the opponent shots. Therefore, the repertoire of movement patterns
describes the movements or actions available to a subject in an
activity domain. Every subject acquires their own specific
repertoire, encompassing a particular range and quality of movement
patterns.
[0453] The elementary assessment at the task performance level is
based on assessing how complete the repertoire is relative to the
task requirements. Task requirements define the outcomes of actions
helpful for the task. At this assessment level, skill analysis
primarily focuses on identifying gaps in the repertoire. The
absence of outcome and associated motion patterns in a relevant
task area or condition can be used to identify "unformed patterns."
For example, in tennis, this may manifest as the absence of a
high-strength backhand top-spin class. The completeness of a
repertoire is determined by the extent to which it achieves a
sufficient discretization of the task space. Typically, as a
subject's skill level increases, the movement patterns become more
precise and therefore enable a more granular discretization of the
task environment (see FIG. 8). As the discretization level
increases, more optimal levels of task performance can be
achieved.
[0454] Note also that in some domains, the range of outcomes and
actions can depend on the style of play or even the personality of
the individual performer.
[0455] More advanced task performance analysis and assessment takes
a more comprehensive perspective and is achieved by tracking
attributes of entire sequence of actions. For example, in tennis,
relevant attributes include sequence of strokes, the length of the
rallies, what type of strokes are used, how they relate to the
other player's actions, including movement on the court, and the
overall performance of the activity or task. Statistical analysis
of the sequence movement patterns can also be used to provide
relevant information about the individual's skills and strategies,
such as the frequency distribution of which movement patterns the
subject uses in a task over a session, provides a signature of the
activity and the subject's strategy.
[0456] FIG. 8 also shows the type of quantities, such as player's
court motion and positioning, which can be used to model and assess
the subject's game strategy, i.e., how the player can position the
ball according to the opponent's pattern of play and position.
These high-level skills also depend on perception of the court, and
anticipation of the opponent's behavior. These dynamic
characteristics of how movement patterns are used can be modelled
by using techniques for learning temporal relationships and
dependencies in the performance data. Popular techniques include
Hidden Markov Models (HMM) or recurrent neural networks (RNN).
Competitive Performance Assessment
[0457] At the top level, the primary goal is to assess competitive
performance, which is typically performed at the level of a
population. The criteria therefore represent what can determine a
form of population fitness such as actual performer rankings
obtained from competitions. These may not always be available;
therefore it is also possible to compute rankings based on player
skill profiles, which can also take into account population
groupings. When available, competitive rankings can be used to
calibrate the ranking based on skill profile. The movement skill
attributes characteristics include those included in the skill
profile, how performers relate in terms of their individual
characteristics, and how these contribute to their competitive
performance. Analysis of the competitive performance provides
information about what aspect of the skill profile (skill element
and attributes) can be improved to make someone more competitive in
a task.
[0458] The skill profile is designed to capture the comprehensive,
composite characteristics of the individual's movement skills:
account for the performance (repertoire), and how well it serves
the task or activity; and the effectiveness of the individual
movement patterns in the repertoire (movement technique and
physical performance).
[0459] This can be accomplished though some composite cost or
objective function (see equation for Q(a.sub.i), below). The skill
profile can then be used to compare performers. FIG. 17 shows the
skill profile as a line graph with the contribution of the
different skill components, i.e., movement patterns to the
composite score. FIG. 40, which will be discussed later provides an
illustration of the skill profile for groundstroke repertoire.
Different objective functions can be used to emphasize different
aspects of the performance. For example, task performance,
efficiency, long-term injury risks. The illustration in FIG. 17
also highlights gaps in the repertoire, and the difference between
two subjects (C and A) as a skill profile gap.
[0460] The skill profile and composite cost can be analyzed to
determine which aspect of an individual's movement behavior, or
skill attribute, has the largest impact on the overall performance.
Since type of sensitivity information can be used as a guide to
determine what training elements to focus on. Here again, the
different objective functions that can be used for a skill profile
provide a way to look at training from different perspectives
(performance, efficiency, injury). They can also be combined in a
multi-objective analysis to find the tradeoffs, such as performance
vs. injury. In contrast to the analysis at the movement technique
level, this analysis provides a more holistic perspective on these
questions.
[0461] However, the skill profile is a static assessment in the
sense that it does not account for the dynamics, i.e., how these
skill elements are deployed as a task or activity unfolds, or in
response to an adversary. Usage frequency of motion patterns
provide simple model to assess strategy. The next level corresponds
to the statistics that describe the sequence of patterns such as
conditional probabilities of a pattern given the previous pattern
or the opponent's pattern. A more complete competitive analysis
accounts for the dynamics of the activity performance, i.e., the
transition between actions and their associated events, e.g.,
pattern X is used when return from opponent is of a particular
stroke type and ground impact conditions. Such a model takes into
account the chain of events in the outcomes, which corresponds to
determining a causal model. The dynamics capture the complete task
performance or game strategy building on the skill elements and
underlying details.
Population Analysis and Reference Values for Assessment
[0462] As already discussed, to capture the overall impact of the
wide range of factors that play out in someone's skills and
performance, and at the same time determine reference values for
the various attributes and characteristics, it can be beneficial to
take into account the data from a broad population of subjects.
[0463] The player or performer attributes provide information to
characterize player type. Player groups can be determined by
clustering the player attributes as illustrated in FIG. 18. Within
groups of performers that share similar characteristics, it is then
possible to analyze movement performance and skills across a
broader range of conditions and identify subtle variations in
technique that influence an individual's level within that
group.
[0464] FIG. 18 also indicates the relationship between the groups
and some skill level such as determined by the skill profile. This
information can then be used to determine the player profile (see
FIG. 29). FIG. 19 shows the distribution in attributes associated
with a score or cost function for an entire population group
described by the group distribution highlighting a member (subject
A, described by distribution (e.sub.1, e.sub.2)), and the tiers
({low, medium, high, very high}) associated with the composite
scoring function for the entire population subgroup (e.sub.1,G,
e.sub.2,G).
[0465] The statistics from these groups depend on how well
performer or subject subgroups sharing similar general movement
technique and other common factors can be identified. Population
level analysis can account for any possible relevant factors such
as body proportions, sizes, health conditions, age, etc. The
analysis could even be extended to genotype and thereby provide
insights into possible innate differences.
[0466] The population analysis enables performing absolute
assessments. The values obtained for the various skill attributes
relative to a larger group of performers help contextualize a
subject's performance. This allows more objective comparison
between the skill profiles of groups of players (FIG. 17) and can
be used to determine reference values.
[0467] The reference values from population analysis can be
incorporated in the assessment and diagnostic of the skill elements
and extended to the various levels of assessment. For example, the
composite score used to capture skill elements can be normalized by
reference ranges associated with the attributes for the subject's
subgroup.
[0468] Information from the population analysis can also be used to
rank players or performers, such as through leaderboards, which in
turn can provide additional source of incentives for training. The
leaderboard also enables the determination of which attributes in
the composite and profile cost function differentiate players or
performers. This corresponds to the competitive assessment (see
FIG. 31). Therefore, this information, for example, describes which
skill element and attributes have the largest impact on the
ranking, and can be also used to prioritize training.
[0469] Finally, the combination of the population analysis makes it
possible to find larger patterns in movement technique,
performance, and even skill acquisition. One aspect of the
assessment based on population data is the profiling of the player
or performer. Player profile can be determined to characterize the
player's performance or skills relative to the larger population.
This profiling can include ranking of a player, e.g., based on
different skill profile composites, as well as relating subgroups
of performers with different but related movement technique.
System-Level Assessment Considerations
[0470] These assessments are combined to provide a holistic
assessment using a composite analysis. The following summarizes how
the different elements integrate to produce a comprehensive
assessment that in turn can be exploited to achieve more effective
training interventions.
[0471] This section emphasizes the role of system-level thinking
and of critical quantities used for the assessment and diagnostics
and what characteristics provide the basis for their integration.
The system represented by FIG. 31 gives an overview of the holistic
understanding required for systematic skill training. FIG. 10
illustrate some of these quantities in the tennis use case.
[0472] The vertical arrows going up indicate the bottom-up
aggregation of the information and characteristics that participate
in the formation of the characteristics at the next level, where
additional elements also come into play. For example, at the
functional performance level, the movement phases combine into a
movement pattern that interacts with a task element to produce an
outcome. These characteristics are critical in understanding the
learning process, and therefore can be used to determine what
movement characteristics have to be developed first (e.g.,
difficulty rating: basic, intermediate, and high level).
[0473] The downward arrows indicate the top-down influence of the
higher-level assessment on informing the focus of the lower level
assessments. The higher levels can provide top-down information to
determine which specific assessments and characteristics drive
training. For example, the skill profile characteristics provide
understanding of which skill element and attribute have the most
effects on the current profile level. Therefore, acting on this
element and attribute will produce the most effect on the
performance at the profile level. These characteristics are
critical for understanding the task performance process, and
therefore can be used to determine what movement characteristics
are relevant for the task (e.g., core motion patterns, etc.).
Trend in Skill Acquisition
[0474] Skill status provides the basis for selection of skill
elements that should be exercised during training, in what order
these elements are trained, which goals are achievable, and what
forms of feedback augmentation are most appropriate for training
(see FIG. 22). Therefore, the skill status comprehension describes
an individual's skills and can be viewed as the state of the skill
acquisition process.
[0475] The determination of the acquisition stage also makes it
possible to more precisely analyze the progress someone is making
in an activity domain, which specific aspects are improving, and
which ones are more resistant to change.
[0476] The criteria applied for the acquisition stage provides
specific information about the skill element that can be used to
measure progress toward their improvement.
[0477] Given that skill evolves over time based on practice and
training activity--and also changes in fitness, health, etc. --the
skill status should be continuously evaluated. Continuous skill
evaluation makes it possible to adopt training activities that are
adapted to the subject's specific skill deficiencies and fitness
and health conditions.
[0478] Skill acquisition is a process that unfolds over time as a
function of exposure to the task. Thus, to determine future
training activity, it is also beneficial to be able to analyze the
trends of the different skill elements. The trends provide
information about the stability or susceptibility of these elements
to improve under a given training activity.
[0479] Motion patterns or skill elements have varying degrees of
stability. Some patterns are deeply solidified in a subject's
procedural memory, and therefore will show less variations from
session to session; other patterns are more malleable. Furthermore,
due to variability in human performance, movement patterns will
occasionally achieve superior outcomes and techniques. Therefore,
the skill analysis method should be able to capture such changes
that are inherent to movement behavior, be able to understand which
features are associated with improvements, and finally, have
feedback techniques to reinforce these features.
[0480] At any given time, it is possible to assess one's current
skill status and the trend of the repertoire of skill elements
relative to current and past times. Time windowing techniques can
be used to highlight skill status and trends at different times or
epochs in an individual's training history. Skill trends can be
analyzed for different time scales, e.g., within sets, from session
to session, etc.
[0481] Different time scales capture different aspects of the
movement skill process. For example, the longer-term trends (months
to years) can measure the physical characteristics associated with
movement skills such as strength, effects of wear, injury (both
development and recovery). The medium-term trends (weeks to months)
can measure the assimilation of the training goals and the
consolidation in procedural memory of the refinement and
optimization of movement patterns. Short-term trends (days to week)
measure the successful assimilation of the formation and
consolidation, or optimization, of movement patterns. Micro trends
(within sets or sessions) can measure the effectiveness of new
instructions and the effectiveness of feedback cues.
[0482] FIG. 47 shows a plot displaying the progress along several
training goals over a specified time range. The progress in the
figure is described as a normalized gap w.r.t. training goal (e.g.,
improving the top spin or consistency, success rate, etc.). When
the current training goal for a training element is attained (shown
as a star), the system generates a new goal (shown as a square).
The trend plots can be superposed for all the active training goals
or a specific subset (e.g., what a subject is currently focusing
on). The characteristics can be used to help identify which skill
elements to prioritize. For example, more focused effort can be put
on aspects that are difficult to improve, or on a training goal
that is close to completion to get it done and move to a new
training goal.
[0483] The trend can also show comprehensive skill elements as
described by their associated metrics (outcome, technique,
performance). The information from the skill status can be
converted to a numerical score or grade to provide a summative
assessment of skill and its evolution over time. Furthermore, it is
possible to decompose the total score into their respective
components, including outcome, technique, and performance.
Training Goals and Planning
[0484] One capability for data-driven training is the generation of
target values for the different movement skill attributes that can
then drive the training process and lead to improvements in the
associated aspects of movement (see 204 in FIG. 21). FIG. 30 gives
an overview of how target skills are generated across the levels of
the hierarchy. Target skills are used to determine training goals
that provide actionable drivers for training or rehabilitation.
[0485] FIG. 31 provides an overview of the integration of
assessment and diagnostics across the levels of the movement system
organization. It gives a description of the following: a) levels of
assessment, b) the central elements that describe that level, c)
criteria and quantities that can be used to determine the skill
characteristics at that level, d) analysis or diagnostics applied
to identify the critical characteristics to specify the training
goals, e) the drivers and mechanisms used to produce training
interventions, and f) the feedback modalities that can be used to
augment the training intervention.
Training Goals
[0486] The skill assessment attributes and metrics and the skill
status and trends provide the main elements to support the
quantitative, data-driven approach to training. The relevant step
to render an assessment actionable is to determine a training goal,
and preferably some specifications for the pursuit of that goal. As
already discussed, diagnostics are typically performed based on
some causal models. In the context of this invention, the causal
models are derived from the functional component of the assessment.
As described previously, the functional components explain how the
outcomes are produced at the different levels of the movement and
task structure organization. The specification of training goals is
also directly connected to the synthesis and selection of
appropriate feedbacks (instructions and real-time cueing).
[0487] In the proposed system, a skill element becomes a training
element once it gets assigned one or more training goals. Training
goals can target any attribute across the movement model hierarchy
(see e.g., FIG. 30). Training goals provide a way to direct and
drive training activity, as well as basic element needed for the
planning as well as the continued assessment and managements of the
training process.
[0488] FIG. 48 shows the learning curve associated with the data
driven training process. The learning curve shows the incremental
improvement in some relevant attribute a.sub.i of a skill element
e.sub.i over the training activity (sets and sessions). Typically,
multiple skill elements can be improved concurrently in one
training epoch. The training goals are expressed as a target change
in attribute a.sub.i of the skill element e.sub.i. When the
training goal is completed (or the underlying parameters such as
motion model, skill model, are not valid anymore) new baseline data
is generated and the training goal is updated. The figure also
illustrates the acceleration of the learning curve provided by an
update to models and augmentations, etc. As the model parameters
are tracked and incrementally updated, the training goals and
associated augmentations drive the learning process to achieve best
efficiency.
[0489] The training goals are identified based on the assessment
and diagnostics, which can include both the various skill and
performance attributes as well as the skill status. The various
sources of information from the assessment and diagnostics
determine the forms of augmentation that are most effective for
training the skill element (see FIG. 31).
[0490] The skill status (acquisition stage) provides relevant
information for specifying general training goals. For example:
[0491] The training goal for unformed patterns is directed at
helping subjects develop new movement patterns that help produce
desired outcomes, taking into account each subject's physical and
health status.
[0492] The training goal for pattern formation is directed at
helping subjects differentiate the existing movement into separate
patterns that can each better respond to task requirements
(outcomes and conditions). It can take into account the existing
pattern landscape in a class, e.g., the core pattern and the newly
differentiating patterns to help guide and reinforce the desired
attributes. The selection of which patterns to form may also depend
on a subject's physical and health status, e.g., patterns that are
causing stress or contribute to an injury.
[0493] The training goal for pattern consolidation is directed at
helping subjects refine movement patterns and create procedural
memory to enable automatic and repeatable execution.
[0494] The training goal for pattern optimization is directed at
helping subjects maximize outcomes, improve efficiency, and improve
ability to adapt to conditions.
[0495] The development stage also provides information to help
select appropriate augmentation forms and determine which movement
characteristic to emphasize. The augmentations, in particular
real-time feedback or apparatus, allow more effective learning and
therefore influence the training goals specification.
[0496] As can be appreciated by this description, training goals
can be determined based on a functional analysis of a subject's own
existing performance. The variability in performance ensures that
there is a range of performance level and associated attributes
contained in the data. A general approach for the training system
is to identify the best performance within the individual's range
of data, and then help the subject consolidate or optimize their
technique so that they operate at this new level. Incrementally,
with new data available from subsequent sessions, this process can
be pursued and the subject's performance therefore can be
incrementally improved.
[0497] This data-driven, analytical approach to formulating
training goals ensures that these goals are realistic for a
particular individual; however, working off an individual's own
data can be limiting. A broader sample of performance and attribute
can help form new movement patterns or techniques that are not
necessarily available in a subject's own repertoire. This is in
particular critical to extend the technique beyond what is
currently used by the individual. Population data extends the
performance, conditions, and the range of factors that are known to
contribute to skills.
[0498] With sufficient data from a population of performers and
data encompassing various other relevant factors (such as body
type, physical fitness, health, or age), this framework also makes
it possible to predict the time that may be needed for achieving
the goals, taking into account the particular feedback
augmentations.
Specification of Training Goals
[0499] The specification of more targeted training goals can be
based on skill analysis of the attributes across the hierarchical
model as shown in FIGS. 30 and 31. FIG. 30, for example,
illustrates example assessment, diagnostics, and training goals
across the skill-model hierarchy, incorporating player profile
information to generate reference values for the attributes used to
assess the skills at each level of the movement system and
performance hierarchy.
[0500] Training goals take different forms depending on the level
in the hierarchy (see assessment levels in FIG. 10). For example at
the physical level, the training goal synthesized for the
improvement of an outcome can be encoded as a change in features of
movement technique that has been shown to produce improvement in a
specific outcome.
[0501] At the pattern performance level, the training goal could
consist of improving movement technique in the deployment of the
stroke such as producing more precise court shot placement. The
training goal is specified in terms of skill attributes that have
been shown to produce improvement of shot level outcomes, such as
timing (FIG. 42).
[0502] The target values for the quantitative specification of
training goals can be determined from the statistical analysis. For
example, for the optimization of movement technique, the training
goals to improve outcomes can be determined from functional feature
analysis at the movement physical level. For example, see FIG. 37,
which illustrates key features for the forward swing phase along
with some example stroke phase profiles. FIG. 20 shows a model of
the statistical distribution for two technique features, which can
be used to analyze the forward swing phase. For example, the
features could be the angle of attack or phase length shown in FIG.
37, and the outcome could be the topspin imparted on the ball. The
level lines in FIG. 20 can be computed based on percentile ranking
from the individual's data.
[0503] A similar analysis can be conducted at higher levels, such
as taking into account any skill attribute that is relevant for the
task or activity performance. The training goal can be set to
achieve the next performance tier, or a fraction of the existing
variation in performance (see the ellipsoid e.sub.1, e.sub.2) in
FIG. 19, which illustrates the relationship between attributes
distribution and some measure of performance that is shown as level
lines in the context of a larger population or some selected
subgroup based on player profile information. The level lines in
this case can be computed based on percentile ranking from
population data.
[0504] The specification of training goals at higher levels follows
the characteristics from the movement system's and task structure's
hierarchical organization. As already discussed, these types of
training goals are derived from diagnostics, which are typically
performed using some causal models. The features and attributes,
and therefore the form and encoding used to specify training goals,
depend on the level of the movement and task hierarchy.
[0505] FIG. 43 gives an overview of integrated perspective on the
system's main components based on the tennis use case, organized in
terms of the levels of assessment (physical 510, pattern 520, task
530, and competitive 540), how the criteria can be expressed with
cost functions (512, 522, 532, and 542), and how these elements
relate across the different levels. Together with FIG. 10, they
highlight some of the key elements and quantities that can be used
to drive the diagnostic and ultimately the training process. In
particular see the assessment criteria at each level and the
diagnostic components shown in FIG. 10.
[0506] In this example, at the task level 530, the diagnostics are
concerned about how movement patterns are deployed across a larger
task environment (see FIGS. 8 and 10). As shown in FIG. 10, the
functional model at the task performance level can be formulated to
describe conditions that can be exploited to produce the desired
outcomes, including proper positioning on the court to control the
impact conditions (shown in FIG. 9). The training goals at that
level therefore can be specified based on deficiencies in these
functional characteristics.
[0507] The diagnostics and training goal specifications can also
include perceptual aspects, such as extracting the cues from the
environment and elements (court landmarks and ball trajectory)
needed to anticipate the oncoming ball, as well as generating
targets for the shots across desired court areas. Similarly, they
can include aspects of memory/learning, also shown in FIG. 10, such
as mental representations of these environment elements (see FIG.
8) and corresponding movement patterns.
[0508] The specification of training goals at the competitive level
follow a similar logic but focus on the dynamic characteristics,
i.e., the temporal sequence of shots driving the game. As already
discussed, the functional models at that level can for example be
formulated using Dynamic Bayesian Networks or Hidden Markov Models.
These models can then be used to assess the individual's strategy
from the temporal patterns in shots, and identify deficiencies that
are responsible for, for example the loss of points in a game. This
understanding can then be used to generate training goal
specifications that address these types of strategic or tactical
deficiencies.
Planning Training Activity
[0509] At any given time in a skill assessment cycle, the skill
status typically includes a repertoire of movement patterns, each
in one of three learning stages. The potentially large number of
movement types and the variety of challenges specific to learning
stage can make assessment and training challenging. A combination
of training goals is usually beneficial to effectively drive skill
training, including training goals for forming new patterns,
consolidating patterns, and optimizing patterns. Furthermore, there
are the questions of deciding which training goal to emphasize at
any given time, and keeping track of the changes in movement as
learning process unfolds.
[0510] Training should follow a systematic process that accounts
for the relative importance of the various skill elements to the
movement activity and, at the same time, accounts for the natural
skill acquisition process, i.e., how the brain naturally forms,
consolidates, and refines movements. The training process should be
able to distinguish between what aspects of skill to preserve and
build on, what aspects of skill to eliminate, and when to form and
consolidate new movement patterns.
[0511] Planning corresponds to the selection and scheduling of
training goals. Training activity can be planned using the
following criteria: [0512] 1. The significance of a movement
pattern and associated outcome to the particular domain of activity
(e.g., where the relevance of the tennis strokes is denoted in
terms of three categories: primary, secondary, and tertiary). Based
on this consideration, training should account for the importance
of a movement to task requirements and conditions. [0513] 2. The
relationship between movement patterns and, in particular, how some
patterns can be understood as derivatives of others (see
differentiation in FIG. 11 and evolutionary relationship in FIG.
13). Based on this consideration, training should emphasize
patterns that are fundamental to repertoire development. [0514] 3.
Available augmentation modalities. [0515] 4. Predicted difficulty
of each goal, and time required to achieve training goals.
[0516] The training elements can, for example, be arranged in a
list sorted in the order of priority that takes into account the
above criteria. The training list (see FIG. 45A) is a list of
training elements ordered by priority. The training list serves as
a type of "working memory" for the skill elements that a user wants
to focus on and track at a given period in a training activity.
[0517] For example, within each skill acquisition stage category,
it is possible to rank movement patterns with the highest
deficiency, as well as account for the hierarchical ordering of the
movement units and outcomes for each movement activity.
[0518] The elements of the training list can also be arranged in a
training schedule (see FIG. 45B). A typical schedule is defined by
time units such as a session subdivided into sets, and each set is
assigned with one or more training goals. The training schedule
makes it possible to organize the training activity for a session.
The sequence of training elements can be determined based on the
acquisition process, i.e., how the skill elements build on one
another and their respective acquisition stage.
[0519] Typically, the first set focuses on warming up, during which
movement patterns that are technically less challenging and
emphasize the range of motion and timing. Once warmed up, the
subsequent sets can focus on specific technical aspects. At the end
of a session, players can play freely or play points, which acts as
a test for how well the focused training activity is translated
into the task or activity performance. For each training goal in a
set, relevant aspects of the performance can be monitored and
augmented.
[0520] Planning can be done manually, with the assistance of an
expert, or by an algorithm. In one scenario, the user can select
training goal(s) to pursue based on skill status, trend, and
overall goals. In another scenario, a coach can use their domain
expertise in combination with the skill status and other quantities
to help select training goals. In yet another scenario, an
algorithm (training agent) can suggest and manage the training
goals and schedule.
[0521] FIG. 46 shows the state machine showing the active training
element and the criteria for the issuance of notifications to the
performer. Also shown are stopping conditions for the training
element, including, the number of strokes performed, the time
elapsed, the incremental (e.g., percentage) progress toward the
associated training goals. Typically, the subject is notified of
the incremental progress milestones and notified when the stopping
criteria of the training goal has been attained. At that point, the
next training element can be initiated.
[0522] FIG. 44A shows the skill status with elements ranked by
order of priority within each training stage category. The lists in
each acquisition stage category can be ordered based on the
contribution to the overall skill profile (based on the skill
element composite score).
[0523] FIG. 44B illustrates an example of skill status that shows
how training activity over several training sessions (e.g., Set
1-3) lead to a change in the skill status of skill elements. For
example, BHTSH increases its ranking within the pattern to form
(from 6.sup.th to 4.sup.th). Or the top skill element in the
"patterns to form" BHSLH improves and gets re-staged to "patterns
to consolidate." Similarly, another skill element BHFLM is upgraded
from "patterns to consolidate" to "patterns to optimize." (Note
that the training effect is exaggerated for purpose of
illustration.)
Other Approaches to Skill Assessment and Diagnostics
[0524] The system-level understanding of skill, with its different
levels of characteristics and assessments, essentially provides a
rich data set that can be processed using a variety of other
analytical techniques, in particular statistical modeling and
learning, including neural networks. The systems approach taken
here was motivated by the need to identify the different components
of a data-driven system and the various forms of assessment and
information. It is conceivable to generate these quantities using
statistical learning techniques, which can even help discover
additional skill attributes from patterns in the performance
data.
[0525] A well-known class of diagnostic processes is based on
so-called diagnostic expert systems. FIG. 26 shows an example of a
diagnostic system building on the assessment system. The assessment
system used to extract the various skill attributes can be used to
drive such a system. Such diagnostic networks can be configured to
generate the types of assessments presented herein (skill status,
skill profiles), as well as training goals and even feedbacks and
instructions and the configuration of the augmentation (cueing and
apparatus interaction laws).
[0526] Typical diagnostic expert systems reason backward, through
Bayesian inference, from observations to determine probable causes
of specific phenomena. Traditional expert systems are built around
a production system which provides the mechanisms to support user
interactions. The core component of these mechanisms are rules
(e.g., expressed using propositional logic), which are typically
deterministic.
[0527] FIG. 27 shows details of the diagnostic system. It combines
a knowledge representation, observations, and an inference
mechanism to produce a diagnostic of the movement performance. The
domain knowledge from an expert (e.g., tennis stroke motion and
game) is encoded in a representation (e.g., Bayesian Network). An
inference algorithm uses the Bayesian Network and the observations
to determine the most likely explanation for the observations,
i.e., diagnostic.
[0528] Complex behaviors such as human movement in open motor
tasks, depend on a broad range of factors (sensory, physical,
environmental, etc.); these relationships are complex and
uncertain. Statistical inference systems such as Bayesian Belief
Networks, which are graphical knowledge representation of a
decision problem, make it possible to capture non-deterministic
knowledge and uncertainties, as well as account for the larger
patterns in the combination of factors or attributes.
[0529] FIG. 28 shows an example of an influence diagram for tennis.
The diagram captures various factors across the different levels of
the movement system hierarchy, including perceptual processes,
court motion and positioning, stroke technique, and ball impact.
The observations correspond to the example metrics detailed in the
specifications. Other observations can be considered depending on
their availability as measurements. For example the ball trajectory
or the subject's gaze. The diagram can be structured as a Bayesian
Belief Network and used as part of the diagnostic system. Note that
the observations can also include general features.
[0530] The diagnostic system can combine expert knowledge, such as
shown in the influence diagram in FIG. 28, with detailed movement
functional analysis and direct diagnostics based on assessments.
Note also that while some features--for example, the skill
attributes illustrated for the tennis use case--are deterministic,
movement in the real world usually involves more complex
interactions such as adaptation to conditions. Therefore,
statistical models can provide deeper insights into movement
mechanisms. These models can be further extended using large
amounts of data available from a diverse population of subjects
spanning a broad range of skill levels, styles, and physical
attributes.
[0531] An instruction generator, for example, converts the
diagnostic results to verbal or visual communications (FIG. 27).
Information from the diagnostic system, when applied to the larger
control hierarchy, can also be used to analyze games or task
performance and even be used in real-time to recommend actions; for
example, which strokes to choose and which locations to target on
the court, given the current states of the system conditions.
[0532] Versatile and effective movement skills depend on the
seamless integration of all the functions or skill components
required to perform the task, including perceptual skills,
anticipation, planning (positioning), etc. Therefore, additional
measurements to capture body posture, as well as perceptual
functions such as gaze, may be required to assess a subject's
skills comprehensively (see described elsewhere). And, conversely,
feedback provided at all those levels is beneficial if it is
integrated systematically. TABLE 3 summarizes the primary elements
of feedback and instructions at different levels of the skill
hierarchy.
[0533] Other data-driven techniques such as deep learning, which
use multi-layered deep neural networks (DNN), can in theory produce
the data-processing capabilities described in this disclosure. The
main components of such as DNN may include: At the lowest level,
delineating between movement phases to produce movement functional
structural that would allow detailed characterization of the skills
and task performance. Next, learning the movement and broader
performance features (conditions and contingencies associated with
contextual details) that are associated with the pattern classes
and explain the repertoire structure and characteristics that
describe a player's performance. Furthermore, higher level layers
can identify the technique features that best delineate between
movement classes and outcomes at the task level to predict player
task performance. Finally, learning structured relationships
between features and other factors or conditions that explain a
subject's skill and performance at the task and competitive levels,
which includes the temporal relationships characterizing task
dynamics and for example game strategy.
Augmentations
[0534] The final category of capabilities for comprehensive
data-driven training are the augmentation methods described in
FIGS. 22-24. The general purpose of augmentation is to produce
various forms of feedbacks (instructions, cues, and signals) and
interactions that enhance the subject's performance and maximize
training effectiveness for a given set of training goals.
[0535] The augmentations achieve these effects by: 1) providing
information to the subject that help them assimilate the knowledge
and/or learning process associated with a training goal (e.g.,
forming new mental models); 2) providing reinforcements that help
induce specific changes in movement characteristics; and 3)
creating or extending interactions with the task or activity
performance that drive the operational envelope associated with the
range of conditions under which a subject can successfully produce
an outcome. The former is typically achieved through instructions,
the second through feedback cueing, and the third through the use
of an apparatus or cues in the task environment.
[0536] The human augmentation ideally follows an architecture that
builds on our knowledge of human information processing (see e.g.,
Rasmussen 1983). Feedback augmentation can operate at any of the
three primary information processing levels (see FIG. 22): the
knowledge, rule, and the signal level.
[0537] The knowledge level includes instructions that explain
training elements and training goals, bringing attention to
specific movement characteristics and explaining what and how to
correct these characteristics. This level of information is
typically communicated verbally, in writing, or through visual
representations. It helps form representations needed to monitor
and correct performance.
[0538] The rule level includes feedback cue stimuli that encode
information to help select the correct movement, or the timing of a
specific movement phase, and/or focus attention on relevant aspects
of the performance or environment. This level of feedback is
typically communicated through visual, audio, or haptic
signals.
[0539] The signal level includes continuous feedback, such as
sonifications of the movement based on specific parameters that can
be used to communicate relevant aspects or features of the movement
profile. This type of feedback can also include extraneous physical
effects such as a force field produced by an exoskeleton or often
robotic device. They may also include functional muscle
stimulation. Signal-level feedbacks are typically generated
concurrently with movement execution.
[0540] Through their combined actions, feedback creates
interactions that can stimulate subjects' learning process and/or
assist in the movement performance. It is useful to distinguish
between feedback that is produced about the subject's movement
performance, and feedback that is produced about the task
environment and its elements. The latter includes the interactions
enabled by an apparatus, e.g., robot manipulator in rehabilitation
or a ball-machine in tennis.
[0541] The following describes examples of specific forms of
feedback augmentations that are used to enable augmented training,
including Instructions and Notifications, Real-time Augmentation,
and Apparatus Augmentation (see FIGS. 22 and 24).
[0542] TABLE 3 details possible instructions and feedback across
the levels of the control hierarchy illustrated for tennis
including: game plan; task environment, orientation, positioning
and action selection; stroke-environment coordination; and stroke
execution (see influence diagram FIG. 28).
TABLE-US-00004 TABLE 3 Feedback and instructions at different
levels of skill hierarchy with examples for tennis. Level
Instructions Cueing Game plan Game rules, strategy, point Game
plan, point construction Task environment, Task elements and
Perceptual cueing orientation, positioning, orientation, etc.
(e.g., ball trajectory and action selection anticipation)
Positioning cueing Action selection (e.g., stroke selection)
Stroke-environment Elements of stroke-ball Perceptual cueing
coordination trajectory coordination Impact timing anticipation
Position adjustments Stroke execution Stroke architecture Racket
state at specific Relevant movement movement phases phases and
racket (phase transition configuration features) Outcome
validation
Instructions and Notifications
[0543] Instructions operate at the cognitive information processing
level, and are associated with the symbolic encoding of
information. Instructions can help contribute to the formation of
mental models or representations that support the skill acquisition
process. Instructions are typically communicated verbally or
visually.
[0544] Graphical instructions include plots, schematics describing
the spatial outline of a movement, maps, etc. For example, a
repertoire map (see FIG. 15) shows the distribution of different
movement classes depicted relative to their primary outcomes (e.g.,
pace and spin imparted on the ball). The graphical description can
be distilled based on a given set of movement pattern classes (see
FIG. 16).
[0545] For example in tennis, the repertoire of ground strokes can
be shown as a stroke map that highlights attributes such as the use
frequency of the movement pattern during a session; number of
movement executions; and statistics about outcome, success rate,
etc. This information can be extracted and displayed for different
time periods such as the current set or session. Additional
information can be communicated in the stroke map, such as the
relevance of the movement class to the task or the difficulty of
the movement pattern, which can be determined from the evolutionary
relationship shown in FIG. 13, as well as from the complexity of
the movement architecture (see e.g., the number of states of the
finite-state model in FIG. 5).
[0546] Another example of graphical instructions includes phase
profile plots for a particular movement class to highlight relevant
movement characteristics such as phase transition features. Or, an
illustration or simulation of the movement showing the spatial
configuration of the equipment over certain phases of the movement
execution. FIG. 37 shows an example of the forward swing phase
highlighting features associated with the spin outcome including
trajectory curvature at the beginning of the phase and angle of
attack at impact.
[0547] An example of verbal instructions would include validation
of an outcome or instructions describing which phase transition
feature to focus on. Or, it could walk through the movement phases
describing features that are critical to performance. Textual
instructions also include information layered on graphical
instructions or displayed on the screen of a smart watch to display
outcome information and progress toward training goals.
Instructions and notifications are communicated on a display such
as a smartwatch, smartphone, or tablet. It is also possible to use
verbal communication via a natural language processor. The training
agent determines when and what type of information is presented to
the subject.
[0548] Instructions and notifications provide the interactions
needed to run the training process. In an automated training mode,
training activity operates as an autonomous (or semi-autonomous)
program. As a training program, the system determines training
goals and schedules, then tracks and updates the training goals and
schedule based on the progress and trends. Notifications and
instructions are used to communicate information to run such a
program, provide instructions about the active training goals, how
they are pursued through the activity (e.g., drills), and when to
switch training goals, etc. Under an autonomous training program,
the training goals and schedules are updated dynamically.
Real-Time Augmentation
[0549] The disclosure builds on the real-time augmentation
technology for movement training described in U.S. Patent
Application Publication No. 2017/0061817. The three primary
categories of augmentation forms that can be used to help induce
changes in movement technique specified by training goals include:
[0550] Outcome validation: Signals provide instantaneous assessment
of the overall movement performance and outcomes. Validation cues
are generated immediately following the action to indicate a
successful outcome. The outcome validation is not limited to
movement outcomes, but can be used to reinforce other relevant
aspects of movement performance including smoothness, timing, etc.,
and those captured by performance criteria. [0551] Alerts: Alerts
augment the natural proprioceptive signals to enhance the subject's
sense of movement with respect to specific training goals. They can
also be used to implement injury prevention using the relationship
between movement characteristics and biomechanics. [0552] Outcome
improvement and optimization: Real-time audio feedback during the
movement execution helps reinforce and refine features of the
movement technique that contribute to the outcome.
[0553] A central aspect of learning good movement technique is
learning the sensory consequence of correct performance. Real-time
cueing, therefore, can provide validation signals that augment the
natural signals to reinforce learning the sensory consequence (see
FIG. 24). Feedback that validates movement features provide an
associative reinforcement of some sensory dimensions.
[0554] In addition, real-time augmentations can be designed to help
with: [0555] Training movement architecture: Real-time feedback
assists in the formation of new movement structure through the use
of visuals (e.g., simulation), as well as real-time cues e.g., that
signal the conformance of the pattern to a template. [0556] Forming
anticipatory perception: Provides signals to learn to identify
critical environment and task cues that are used to anticipate
critical states and conditions, such as timing of the movement
phases that enable synchronization of movement behavior with the
task elements or objects.
[0557] Real-time feedback augmentations are communicated by the
cueing system (see FIGS. 22 and 23) and include audible, visual,
and haptic signals.
Apparatus Augmentation
[0558] The natural variations in a training environment combined
with the variability in a subject's performance may not be
sufficient to expose the subject to all relevant conditions that
help drive skill acquisition. Particularly for deeply solidified
patterns, highlighting erroneous features in a movement or
providing feedback cues may not sufficiently change the movement
pattern. In these situations, it may be more effective to actively
produce new training conditions and thereby force the subject to
acquire new movement patterns.
[0559] Since movement skills are developed for the purpose of
adapting to task and environmental conditions, it is possible to
force the development of new patterns by manipulating task and
environmental elements and conditions. Varying the operating
conditions beyond the natural range can be used to force the
subject to develop new patterns and/or extend the range of
operation of a given pattern. For example, a tennis ball machine
can be used to produce ball trajectories that force the player to
form a new stroke technique or adapt an existing one beyond its
operating range.
[0560] An apparatus can also be used to help form new movement
patterns by physically guiding the movement. This technique is
already used in robotic movement rehabilitation.
Generalization to Other Activities
[0561] Since the training system is derived from the understanding
of human movement learning and movement organization and
performance, the training system can be implemented for a broad
range of movement domains, including sports such as tennis
(described in detail), rehabilitation, as well as professional
activities such as surgery. Most of the concepts and quantities
such as movement repertoire, their outcomes, etc. are derived from
the theory of open motor skills acquisition. The training system
can also be used for various forms of human-machine systems,
including tele-robotics, humans equipped with prosthetics, or other
forms of physical augmentations such as exoskeletons.
Human-Machine Systems
[0562] Since humans are increasingly integrated within
human-machine systems, the augmented training system can be
conceived as an integral part of such HM systems.
[0563] The robotic surgery system such as the da Vinci is an
example of such a HM system. Many of the relevant quantities
(operator inputs, manipulator or tool motion, visual gaze, etc.)
are measured and recorded; therefore, the training system can be
incorporated into the surgical robot's operating system. A
data-driven skill assessment and training system integrated into
such a robotic system can fulfill many functions, including: 1)
train surgeons for new procedures, where they would benefit from
accurate tracking of their skill learning process and feedback to
help that process; 2) opportunities to formalize the certifications
of surgeon training for different procedures, etc.
Report System Description
[0564] The following illustrates data visualizations for some of
the concepts and quantities described as part of the data-driven
analysis and training system in the context of a tennis
application. These plots show some of the elements of the
assessment and diagnostic processes illustrated in FIG. 30.
Overview of Data Visualizations
[0565] FIGS. 32-39 give a sample of the processed performance data.
Starting with FIG. 35, which illustrates the activity data for a
time period, highlighting sessions and sets over a calendar period.
FIG. 39 then provides a close-up into a specific session and shows
the event diagram that displays select stroke types ST used over
the session timeframe (12:13 to 12:50). It also displays the pace
SP and spin S outcomes as time histories TH to visualize trends in
those outcomes over the play duration.
[0566] FIG. 36 then gives a more detailed look at the period of
activity on a stroke-by-stroke basis 381. Additional outcome
quantities are illustrated first, including the impact variability
382 and success cumulative progress 383. Below, it includes the
separate time histories for the pace 384 and spin 385. The time
histories are filtered to smooth out the stroke-by-stroke
variations that can make these plots more difficult to read. Note,
however, that since there is no inherent continuity in outcomes
such as spin from one stroke to the next, the filtering can create
artifacts. The plots in FIG. 36 also highlight the reference tiers
360-364 for the outcome quantities to help their interpretation
(corresponding to low, medium, high, and very high values achieved
by a player population).
[0567] FIG. 37 illustrates details of the functional analysis at
the level of the stroke pattern. It displays the forward swing
movement segment phase for the forehand topspin medium (FHTSM)
stroke class, highlighting the path 710 of the racket relative to
the origin or impact point 720. The stroke analysis based on this
phase segment allows for the identification of features such as the
angle of attack 730, the curvature of the path at the beginning of
the forward swing 740 (transition from back loop phase), and the
length of the swing phase 750. The figure also illustrates sets of
segments corresponding to the core pattern 760 of this stroke
class, and a sub-pattern set 770 that represents the subclass of
strokes with the highest spin outcomes. This representation of the
stroke technique can, for example, be used to investigate the
efficiency with which the subject generates the spin outcome. The
results of this analysis provides the basis for the specification
of training goals and synthesis of real-time feedback and
instructions to help the subject form, consolidate, or optimize the
technique for that particular outcome.
[0568] Continuing with the functional analysis, FIG. 42 shows
impact timing for the different groundstroke classes GC, which is
defined as the timing relationship between the impact time and the
time of the peak acceleration (or angular rate) of the forward
swing movement phase T. Impact timing depends on the movement
technique, motor coordination, as well as proper anticipation of
the impact point and the player's preparation for the stroke.
Therefore, it provides critical information to diagnose stroke
technique.
[0569] FIG. 33 shows an aggregate view of the relationship between
the swing rate R (horizontal axis) and spin S (vertical axis)
produced for an ensemble of strokes in topspin, flat, and slice
classes C for a particular subject. The quantities define the
so-called spin envelope SE, which describes the range of spin S
that can be produced by the subject as a function of the racket
swing rate R. The spin represents the outcome and the swing rate
represents a movement technique attribute, which in this case can
be considered as the effort applied by the subject to produce the
outcome. The spin envelope is parameterized based on the slope of
the two linear boundaries (k.sub.max MX and k.sub.min MN), each are
depicted along with reference lines corresponding to low, medium,
high, and very high ranges, which again can be computed from a
population.
[0570] The data representation then shifts to FIG. 38, which
depicts the composite score for a specific stroke class (skill
element) as a radar chart, which is an illustration of the
skill-element composite score. It shows individual cost components
based on extracted performance and skill attributes (impact
precision IP, consistency CC, impact SR, efficiency EF, smoothness
SS). The composite score, which can be visualized as the area
covered by the polygon PG, represents the overall assessment of the
skill element (stroke class). This polygon compared to the less
opaque polygon CP illustrate what could be a comparison between two
players, or between different skill elements, or the same skill
element at different times in a subject training history.
[0571] FIG. 40 then takes a more comprehensive view and depicts the
overall skill profile as a bar graph of composite scores CS for the
groundstroke repertoire GR. This chart makes it possible to assess
the overall repertoire strength and weaknesses (see FIG. 17).
Similar to the skill element composite score, this skill profile
can be used for comparisons between different players or between
different times in a subject's training history. As already
discussed, different composite costs can be used to emphasize
different characteristics relevant to a task performance. FIG. 41
displays the acquisition stage of the strokes in the groundstroke
repertoire based on the criteria described in TABLE 1 and TABLE
2.
[0572] Finally, FIG. 34 shows the leaderboard, which synthesizes
the entire assessment at the population level. Note that these data
visualizations are a sample of quantities described in this
disclosure, and are used here to illustrate the types of quantities
that can be used for the assessment and diagnostics for different
levels and components, and how they can be used in conjunction with
reference ranges to support the identification of training goals
and eventually the feedback synthesis. These visualizations can
then also be used for tracking progress and for updating the
training elements and cueing laws, etc. as someone's skills evolve
relative to their own history as well as to that of a larger
population.
[0573] As already discussed, FIG. 43 gives an integrated
perspective on the system's main components, organized in terms of
the levels of assessment (physical 510, pattern 520, task 530 and
competitive 540). The figure highlights some elements and
quantities that drive the training process, in particular it
highlights examples of assessment criteria at each level, and how
the criteria relate across levels.
[0574] Starting from the physical performance level 510, the stroke
forward swing phase profile with features (shown with more detail
in FIG. 37) depicts an example of a skill model that can be used to
analyze a subject's movement technique, taking into account the
different assessment components (outcome, biomechanical,
functional, perceptual, memory, and learning). Each component can
be used to generate attributes for assessment and characterization
of the skill element (i.e., the stroke class). The efficiency
attribute EA captures the relationship between the spin outcome and
forward-swing energy. In some situations, the attributes can be
formally captured by a cost function 512, 522, 532, 542. FIG. 37
emphasizes the model describing the spin outcome and relevant
functional characteristics. Similar models for the biomechanical
characteristics, for example to identify features that can predict
joint loads or muscle strain, can be developed and then converted
to an attribute, e.g., injury index, that can be included for the
skill element composite score (see FIG. 38).
[0575] At the pattern performance level 520, FIG. 43 shows how
different attributes associated with the assessment components
contribute to create the overall skill element score (see FIG.
38).
[0576] At the task performance level 530, FIG. 43 shows how the
skill elements contribute to create the subject skill profile,
highlighting the forehand top-spin medium stroke FTSM depicted in
levels 510 and 520. It also shows how the skill profile is obtained
through a composite cost function 532 combining the skill elements
in the stroke repertoire.
[0577] At the competitive performance level 540, FIG. 43 shows how
an individual's skill compares at the level of a population. In
this example, the comparison is based on percentile rank computed
from the skill profile composite score. The figure highlights how
the individual's skill profile ranks SPR are relative to the
population PP.
[0578] The material illustrated in FIGS. 32-43 can be embedded
within a web-based or mobile app reporting system to allow a
subject to navigate their skill elements and characteristics. The
content below is organized into three sections: [0579] I. Activity
Session Report provides a description of the movement activity for
a given session in terms of the skill elements, how these are used
throughout the activity period, and various performance and skill
attributes. The session report can also include training elements
in the training list. The knowledge also provides the data to
generate training goals and to plan and schedule training
activities. [0580] II. Detailed Pattern Class Report is a
class-by-class detailed description of the various assessments,
including pattern level assessment, as well as functional analysis
and diagnostics at the level of the skill elements. The assessment
can also include historical trends of how different outcomes and
attributes of the individual skill elements evolved over the
subject's recorded activity history. The class-by-class description
can also provide information about active training elements as well
as suggested training goals. [0581] III. Comprehensive Player
Report provides a summary of the player's activity and how the
skill elements combine to create the subject's overall facility in
the domain of activity. This is illustrated here using the
repertoire, an overview of the different skill elements and their
outcomes and attributes, the skill profile, and skill status. The
player report can be augmented by population data to describe the
relationship to other players in the subgroup as well as related
subgroups that can represent longer-term skill targets for
training.
I. Activity Session Report
[0582] The activity session report focuses on the overall
description of the movement performance in a given session,
focusing on the activity performance characteristics. The purpose
of the session report is to convey understanding of high-level
patterns in the activity performance, such as the evolution of
various attributes over the period of the session; the use of
particular movement patterns; and the trend in their outcomes, such
as energy and success rate. The session report can enable the
identification of the onset of fatigue or loss of concentration.
This information can, for example, be used to help improve the
training session, or even fitness or physical strength.
Play Activity Summary
[0583] The activity summary for a session can be presented as a
table that describes statistics and trends for attributes of the
most frequently used movement patterns in the recorded session. The
statistics for the tennis use case can include: a) Pattern usage
frequency (%); b) Impact success rate; c) Pace (m/s); and Spin
(rpm). A trend symbol (up, down, or equal) and a trend value can be
appended next to each metric to highlight the change in the
respective metric for the session or relative to a selected time
period.
[0584] A similar table can be used to summarize the activity for
the training elements currently in a training list. The table may
include the activity level for each training element during the
session, when the element was created, the progress toward the goal
during the last session or relative to a selected time period, etc.
This information can be used to verify the effectiveness of
previous training goals, training lists, and training schedules,
and to help update the subsequent training plans. These summaries
can be linked to visualizations of the session activity that enable
more detailed insights into trends of select attributes of skill
elements or training elements.
Trends in Movement Patterns Usage
[0585] FIG. 39 depicts a time history TH of player movement pattern
usage. The usage trend plot depicts movement patterns on a
stroke-by-stroke basis, where each vertical line L is a stroke
occurrence. The movement class membership of a stroke, representing
a skill element, is indicated by the vertical position of each line
L. This example uses a subset of six stroke classes 30 to describe
the main movement pattern trends in this activity.
[0586] This data used for the usage trend can also be analyzed to
identify rally segment statistics, such as the average stroke
counts or rate of return for each class used during the rally. The
rate of return describes the probability of the opponent making a
return. This probability can be computed for a specific pattern
class. Furthermore, by analyzing rally ending strokes that lead to
either points or losses, it is possible to identify the strong or
deficient pattern classes, which can be used to identify
deficiencies in the repertoire.
Trends in Movement Outcomes
[0587] The movement outcome trends in this section shown in FIG. 39
focus on the evolution in primary stroke outcomes across the
different movement patterns during an activity session (pace SP and
spin S). The session report gives the breakdown of how the subject
used their time during a session, and therefore provides a
composite view of the activity in the session. This information can
reveal patterns in the technique and outcomes associated with the
activity performance at different stages, such as during the warm
up, while training on a specific training element.
[0588] The information in this chart can enable automatic
identification of the type of the sets in a session. Sets can be
identified by the intermittent rest, and for example, the
deliberate training sets will have specific features such as
concentration of strokes belonging to the same movement pattern, or
the stroke pattern transitions forming a repeating pattern.
[0589] Moreover, the chart, and the information underlying it can
convey information about the intensity or even the competitivity or
competitiveness of the play in a set. The information can also can
reveal patterns within a set, or across sets, that are related to
physiological or psychological processes, such as the onset of
fatigues, or deterioration in concentration.
[0590] These insights and knowledge can then be incorporated into
the system, and used to plan and schedule training sessions. For
example, this knowledge can help determine limits on the duration
of certain activities in sets, or the total number of repetitions
of particular skill elements in a set, or it can be used to set
dependencies in the sequence of training elements in an activity
period. All of these patterns can be identified using statistical
modeling techniques.
[0591] As already described, activity at the task and competitive
level can be further analyzed and assessed using statistical
algorithms, such as a Hidden Markov Model. For example, such
techniques can be used to build a state-machine that represents the
most likely transitions between movement patterns based on various
factors including a player's own prior activity. It can also
include information from opponent activity performance, and be set
up to capture the extended temporal patterns encompassing the task
and environment elements.
II. Detailed Pattern Class Reports
[0592] The pattern class report is organized at the level of the
individual skill element or movement pattern. It tracks the
multivariate attributes and characteristics for each movement
pattern, and therefore can provide insight into the skill
acquisition process of each movement pattern, and help identify the
specific deficiencies, which in turn can be used to help determine
training goals.
[0593] The play activity of the pattern class is presented as the
stroke counts by set, by session, and across the entire recorded
history (see FIG. 35). The bottom histogram 351 in FIG. 35 shows
the stroke counts by date over the entire recorded activity history
of a player. The shaded bar 354 on the histogram can be moved by
the user to select a set of consecutive dates to be presented in
top chart 355. In the top chart, sets are shown as stacked shaded
bars grouped by date 352. (This chart can also be used in the Play
Activity Summation section in the player report, with stacked bars
representing sets or movement pattern classes.)
[0594] The stroke counts 353 of the specific movement pattern
indicates how frequently the pattern has been used. If use
frequency is correlated with a decline in outcomes, for example, it
prompts the diagnosis to identify causes which in turn could be
used to formulate a training goal.
Movement Outcomes Trends
[0595] Movement outcome trends for a specific class are shown in
FIG. 36. It focuses on the longitudinal dimension of the movement
pattern development process by presenting the trends of a selection
of select movement outcomes and attributes (for example: pace 384,
spin 385, cumulative success progress 383, and impact variability
382) across the entire recorded activity history (see FIG. 35). The
plot background shades 70 in the x-axis delineates the different
sets. The plot background shades 360 in the y-axis encode the
information about the reference ranges or tiers (e.g., very high
361, high 362, medium 363, and low 364).
[0596] The success rate trend plot depicts the cumulative summation
of the impact success variable. The trend plot takes the form of a
stair function 370 (up one step for a successful impact, down one
for a missed impact). The dashed line 371 provides a reference for
100% success rate trend; a horizontal trend line would correspond
to a 50% success rate. A subject can easily determine success rate
trend by looking at the slope and contour of the trend line.
[0597] Impact variability is one of the class ensemble statistics.
It is calculated for every set and presented as a staircase
function across sets 377. The other trend plots (e.g., pace 384,
spin 383) depict the evolution of the movement outcomes over time
on a stroke-by-stroke basis. However, the time history can be
smoothed to remove large variations that can make the
interpretation more difficult.
[0598] At the scale of set and sessions, the diagram of the trends
enables the investigation of the range of variation of the movement
pattern. When multiple sets or sessions are combined as in FIG. 36,
it is possible to determine variations in movement pattern
performance as a function of the various types of sets, such as
training, free play, or competitive play. The long-term
longitudinal perspective also provides insights into the larger
skill development process.
[0599] Moreover, this visualization can be used to verify the
effectiveness of previous training goals, training lists, and
training schedules, and help update the subsequent training
plans.
Movement Functional Analysis
[0600] The movement functional analysis focuses on the details of
movement technique used by the player to achieve their outcomes
across the various movement patterns or skill elements. It also
encompasses other relevant mechanisms that are used to modulate the
outcomes or adapt to conditions. Functional analysis at the level
of movement phases provides detailed insights into the movement
technique that is valuable for the determination of training goals.
This is illustrated in FIG. 37 for the forehand topspin medium
(FHTSM) stroke class.
[0601] For example, the forward-swing phase, occurring immediately
before the target phase of impact, contributes to the realization
of desired movement outcomes of the motion pattern. Therefore, it
provides both information about the outcome, and the more general
organization of the movement. This phase lasts about 100 ms, which
means that most of this movement segment is executed in open-loop,
i.e., without opportunities for corrections. Therefore, its success
depends on the motor program stored in so-called procedural memory.
This program encodes the coordination and perceptual cues, the
muscle synergies that support the physical execution, and the
correct movement phase initiation and configuration (see FIG.
3A).
[0602] FIG. 37 presents the stroke trajectory profile of the
forward-swing phase of the forehand topspin class. The figure
compares core-pattern strokes with a subset of pattern strokes
identified as having the highest movement spin outcomes. As already
described, several features can be extracted for this movement
phase (angle of attack, curvature of the path at the beginning of
the forward swing, and the length and the elevation of the swing
phase).
[0603] The forward swing profile also provides a visual description
of the movement technique that can be used to generate visual
instructions, such as a target movement profile shape. Real-time
feedback cues can be generated to reinforce the desired features.
The efficacy of these cues can be enhanced by combining them with
visual descriptions of the target profile shape, which serves as a
template for a mental model. The integration between the
sensory-motor and cognitive levels can accelerate the
consolidation.
[0604] Other functional metrics can be defined that focus on the
overall range of outcomes. For example, FIG. 33 compares the
overall spin envelope SE, which is defined by the racket swing rate
R and imparted spin S. The spin envelope describes the efficacy of
the stroke technique as ratio or spin/swing rate. A larger angle
for the line delineating the envelope indicates that a player can
achieve a higher spin outcome with an equal racket swing rate. The
dashed lines DD correspond to the reference ranges from the
population analysis. The spin envelope helps identify the
deficiencies in stroke technique; as shown here the cause of the
spin deficiency is due to an insufficient racket roll rate at
impact. Generating larger roll rate at impact requires optimizing
movement coordination, i.e., the movement architecture between the
backswing and impact phase.
[0605] Another functional metric is the timing of the impact during
the forward swing. The timing metric is defined as the relation
between the instant of the peak racket swing rate and the impact.
Correct timing of the forward swing depends on the player's
anticipation of the interception, as well as other factors such as
anticipation of the ball trajectory, footwork, and preparation.
Composite Analysis
[0606] This section integrates the attributes statistics of the
movement pattern to determine a composite skill score using a cost
function, e.g., the weighted sum of the attributes:
Q(a.sub.i)=.SIGMA..sub.e.sup.Naw.sub.ea.sub.i,e/.SIGMA..sub.e.sup.Naw.su-
b.e, [2]
where w.sub.e are the weights indicating the relative importance of
the attributes.
[0607] The attributes can be normalized based on some
characteristic values. These values can also be obtained from the
individual's data reference ranges computed through population
analysis, with the advantage that the composite score then provides
more meaningful information.
[0608] A radar chart, as shown in FIG. 38, enables an intuitive
interpretation of the multivariate contributions of attributes to
each skill element. The figure shows a subset of select attributes
depicted as a dimension 10-50. Under certain conditions, the total
area of the polygon 60 formed by the outcome or attribute values
can be viewed as a description of the composite score of the
movement pattern.
[0609] The composite skill score can be used to rank the movement
patterns and can be combined across patterns to form the player
skill profile (see FIG. 40 and FIG. 17), which provides an overview
of the repertoire, enabling the identification of player strengths
and weaknesses.
[0610] This representation also enables the comparison of skill
elements over different time periods, or between different skill
elements. The two polygons 60, 70 shown in the chart can represent
the statistics of the current epoch versus that of the entire
recorded history, or the statistics of the player versus that of a
subgroup that the player belongs to.
III. Comprehensive Player Report
[0611] The example player report combines the different assessments
to create an overview of a player's overall skill status and skill
development progress. The player report is organized at the level
of the repertoire. It includes the following four sections:
Total Play Activity History
[0612] The play summation presents a player's activity statistics,
which is a summary of performance activity over the subject's
entire recorded history in terms of the following: 1) total number
of sets, 2) total number of sessions, 3) total time duration, 4)
last time of play, and 5) overall success rate.
Task/Repertoire Level Skill Assessment
[0613] The repertoire level skill assessment focuses on how
complete the repertoire is relative to the task requirements. The
repertoire completeness can be determined from use frequency
(stroke counts) and the overall movement outcomes of a performer's
repertoire relative to a nominal repertoire of motion patterns for
the task. In this example, the nominal groundstroke repertoire is
defined by a fixed number of groundstrokes expressed in terms of
the spin and pace. Each of these outcomes are discretized in three
levels ("slice," "flat," and "topspin" for the spin imparted on the
ball and "low," "medium," and "high" for the pace) (see FIG. 16 and
FIG. 32). In addition to these primary outcomes, the impact success
rate (defined based on sweet spot area) and impact location
variability are evaluated to measure the impact quality.
[0614] A more comprehensive assessment would cover different
outcome levels (see FIG. 7), extending across different shot types
as described by their trajectories and relationship to the court
(see FIG. 8) as well as the broader repertoire of strokes and
interception conditions (see FIG. 9). As previously discussed,
these levels can be assessed using additional data, such as
provided by vision-based tracking system.
[0615] FIG. 32 illustrates the overall movement outcomes using pace
and spin as example. Movement patterns are divided into backhand
and forehand, and sorted by the average outcome values. The data is
visualized as a histogram chart. The lighter color bars correspond
to the movement patterns without sufficient stroke counts and low
statistical significance.
[0616] Background shades in FIG. 32 indicate different
tiers/reference ranges (e.g., low, med, high, and very high). These
reference ranges can either be determined based on the player's own
statistics or derived from the population analysis that extracts
the statistics from a subgroup of players sharing similar movement
techniques and skill level. In this example, common reference
ranges for all movement pattern classes are depicted since the
emphasis is the overall repertoire. A more precise assessment can
be achieved by extracting reference ranges that are specific to
different pattern classes, including other relevant factors such as
impact conditions. The more detailed contextual information is
available, the more precise and actionable assessments can be
achieved.
Skill Status
[0617] Skill status captures the skill acquisition stages of the
movement patterns in the repertoire. FIG. 41 illustrates an example
for the groundstroke repertoire. Each movement pattern is
determined to be at one of the three stages: pattern formation,
pattern consolidation, and pattern optimization. The qualitative
characteristics and quantitative criteria that can be used to
identify the acquisition stages are listed in TABLE 1 and TABLE 2
respectively. Skill status can be presented as a table with
acquisition stages as columns, and movement pattern classes, or
skill elements, are arranged in a sorted order (see FIGS. 44A and
44B).
Player Skill Profile
[0618] The information of use frequency, movement outcomes, and
skill status of all the movement patterns can then be used to
determine the player's skill profile as a histogram of sorted
scores of motion patterns (see FIG. 17). The skill profile provides
the information to build a leaderboard (see FIG. 33) and the larger
population analysis.
[0619] Moreover, this section also presents the rally statistics,
such as the average number of strokes in a rally and the cadence
(number of strokes per minutes). This provides information to
identify the player style in the gameplay.
General System Description
[0620] The disclosure includes a system to help individuals train
or rehabilitate movement through and using targeted augmentations
designed to stimulate learning through feedbacks and interactions.
These augmentations are further adapted to the specific skill
deficiencies that occur at different stages of the movement
learning process, and account for the human information processing
hierarchy. The system builds on movement sensing, skill modeling
and diagnosis, and feedback synthesis, which are described
previously described in U.S. Patent Application Publication No.
2017/0061817.
[0621] The general goal of training augmentation is to help guide
the development of skills by providing feedback during training or
performance. Since skill learning is an ongoing, dynamic process, a
valuable feature of systematic data-driven skill training is the
capability to model and diagnose skills in a way that captures the
longitudinal and vertical dimensions of skill development. Recall,
the longitudinal skill dimensions refer to the process of skill
acquisition over time, through transformations of existing skill
elements, and the vertical skill dimensions refer to formation of
new skill elements.
Augmented Skill Ecosystem
[0622] The augmented skill platform is configurable to create an
integrated environment for training, maintaining, and
rehabilitating motion skills by combining motion capture
technology, skill modeling and analysis tools, and a set of
feedback modalities that can target precise aspects of movement
performance. The system trains movement techniques to optimize a
set of outcomes that are relevant to the activity over its domain
of operation. FIG. 2 illustrates the elements of the augmented
tennis activity environment that serves as a use case for this
disclosure.
[0623] Any task can be described by environment elements EE, and
task elements TE. For example, a person manipulates a device (e.g.,
tennis racket), end effector or piece of equipment, to interact
with the task elements TE (e.g., tennis ball). In addition, there
may be miscellaneous accessories Z such as shoes or clothing that
may be relevant for the description of the activity. The workspace
W is contained in the environment and is specified by various
constraints and rules that characterize the task's success and
performance (e.g., the tennis court and tennis game).
[0624] In tennis, the person is the player (or players); the task
environment is the tennis court; the task element is the tennis
ball; and the equipment is the tennis racket, and the accessories Z
are the shoes and other pieces of attire such as an arm or head
band. In addition, a variety of output devices can be included,
including graphical displays (e.g., LCD, OLED, etc.), haptic
devices (e.g., embedded in the racket grip), speakers. Finally,
consider a variety of input devices, including touch sensitive
display (user interface), keyboard, etc. The input and output
devices may be integrated in the form of a smart watch, tablet, or
a wearable device that can be worn by the person.
[0625] The overall elements, agents and other components used,
including the measurement, input and output devices, are referred
to as the augmented human system or simply the system S. Other
examples of systems that have this general setup include a robotic
system, a cybernetic system (e.g., a human fitted with a
prosthetic), and a human-machine system (human operating a robot
through tele-operation). For example, a robotic surgical system
such as the DaVinci.RTM. Surgical System (available from Intuitive
Surgical, Inc.) is a robot that is an example of an integrated
augmented movement skill system.
[0626] Measurements y that contribute to the recorded performance
data can be acquired from different components of the human actors,
equipment, or system. Typically, instrumentation is designed to
obtain measurements that encompass relevant variables for the
particular level of analysis. For example, as illustrated in FIG. 2
in the analysis of human tennis stroke path 25 and performance, the
states, or a subset of the racket motion may be sufficient.
However, to enable a complete analysis of the movement on the
court, the footwork, or the body motion such as the kinematic chain
or other movement units, additional measurements about the
environment and body segments 15 (e.g., arm, legs, feet, etc.) can
be added.
[0627] These measurements can be obtained using a variety of
technologies, including inertial measurement unit (IMU), visual or
optical tracking systems, etc. Examples include the use of video
cameras 70 that capture the broader agent behavior and the task
environment 50. Vision processing can also be used to extract
information about the motion of individual body segments 15.
[0628] Another category of performance data measurements is one
that captures physiological quantities. For example, a gaze
tracking system 80 to measure the visual attention. Thus, as shown
in FIG. 2, a user 10 (or player or other subject) holding a tennis
racket 20 which impacts a ball 30 during the swing or stroke of the
racket 25. One or more motion tracking or video cameras can be
attached to the performer, such as integrated with the gaze
tracking system. These so-called first-person cameras capture data
related to the interaction of the subject 10, the tennis racket 20,
the ball 30, as well as the motion of other participants such as
the opponent 53, and other relevant environment elements such as
the court 51 and net 52. Combined with measurements of the gaze
direction or vector 81, video cameras on the subjects and/or
environment make it possible to determine which elements or events
the performers are attending to at any given time, or at specific
instants of the performance such as during specific movement phases
or phase transitions 26, 27 on the path 25, opponent behaviors, or
task elements such as related to the ball trajectory 36.
[0629] Inertial sensors 21 or similar measurement units can be
embedded or affixed to the equipment; worn by the user, subject, or
other agent 10 to measure the movement of body or body segment 62;
or even placed on the user's, subject's, or agent's skin or
implanted in the body to measure muscle activity or neural signals
involved in the control of muscles 15.
[0630] Note that additional behavioral measurements such as gaze
can be used to analyze the perceptual functions. For example, the
gaze follows the ball trajectory 36, which has several notable
events during the motion, such as the ground impact 32, the racket
impact 30, and the interception by the opponent. The gaze
(described by gaze vector 81) also typically can fixate on target
areas on the court (outcome 3, ref 35), in between the court
(outcome 2, ref 34), as well as anticipated racket impact or
post-impact location (outcome 1, ref 33).
[0631] In addition to the measurements, data fusion and state
estimation techniques may be implemented to determine states x that
are not directly measured. For example, in most applications using
IMUs, the orientation of a body segment 15 or piece of equipment 20
requires an attitude estimator which combines angular rate data
from the gyroscopes, the accelerations from the accelerometer and
the magnetic field strength from the magnetometer. An example of
data fusion and estimation is the use of a vision-based tracking
algorithm, applied to video data from video cameras, combined with
IMU data from a device on either the body segment or equipment, to
extract body segment or equipment motion information. Such a data
fusion system can be used to provide an accurate estimation of
absolute pose of body segment or equipment. The combination of
motion processing such as based on IMU and computer vision enables
the extraction of video frames associated with certain events in
the agent-environment interactions. For example, the identification
of a phase transition 27, such as the forward swing initiation 26,
can be used to extract larger contextual information from the
environment such as the location of the ball or opponent at that
instant. Or vice-versa, a specific event in the task or environment
such as a ground impact of the ball 32, can be relevant in
assessing the agent strategy, taking into account visual attention
(gaze 81), body location of the subject 10, footwork (shoes or
feet) 60, and movement preparation, or particular phase segment
initiations 26, 27. These interactions provide the basis for the
task performance modeling, for example using Hidden Markov Models
(HMM).
[0632] In terms of outputs, various wearable devices can be
configured to generate a range of communication modalities such as
audio, haptic, or visual. These devices can operate along different
levels of the information processing hierarchy discussed earlier.
Such cueing devices can be worn on the body, skin; integrated in
the equipment such as in the racket grip 21, shoes 60; or even
implanted in the skin or body such as muscles 15. They can be
configured to provide different modalities of feedbacks such as
audio, haptic stimuli, or visual cues. Another class of output
devices include an augmented reality (AR) system 80 that can be
configured to provide visual cues superimposed on the natural
environment. Speakers, or visual signaling devices such as cones,
markers, etc. can also be deployed in the environment itself 50 or
on the object such as the tennis ball 30. Finally, implantable
devices can also be used as part of the augmented system and for
example provide functional muscle stimulation 15. Outputs can also
be communicated via the typical wearable devices, mobile and
portable devices and computers that are part of the augmented skill
ecosystem, such as smart watches, phones, or tablets.
[0633] Typical human cyber-physical systems are described formally
using hybrid system notation. This notation system combines
continuous and discrete quantities. For example, the movement of a
user, subject, or other agent may be governed by physical laws that
result in nonlinear continuous time differential equations.
Discrete variables may be used to evaluate conditions associated
with specific events, such as counting strokes in a tennis game or
scoring the game based on ball trajectory relative to the task
environment and rules. Categories of state variables include:
controlled variables, specific behavioral variables such as the
visual gaze vector, and features used as cues by the agent to make
decisions.
[0634] Actions are typically taken by the user and represent the
addition of force or energy to the system such as the racket ball
impact 30. Actions are typically applied to specific locations such
as the end effector or equipment. As already discussed, actions are
often motivated by a deliberate desire to achieve particular
outcomes 33-35. In tennis, for example, the player wants to impart
a specific effect on the ball (velocity and spin) 33, with the
ultimate goal of driving it to a specific location on the
opponent's court side 35. Events can be defined by particular state
conditions. For example, in tennis, a major event is the impact of
the ball on the racket 30. Events can be expressed formally by
constraints on the system states, e.g., racket acceleration
exceeding a threshold due to the impact, or alternatively, the
impact can be detected when the ball and racket velocity are equal.
Other relevant events in tennis include contact of the ball with
the ground and when the ball crosses the net (see FIGS. 7 and
9).
[0635] As already discussed previously, outcomes are defined as
quantities that capture the relevant characteristics of the agent's
behavior in the performance of task. To provide a concise
description, outcomes can be categorized hierarchically, e.g.,
primary outcomes, secondary outcomes, etc. (see FIGS. 7 and 10).
The definition of outcomes are a function of the scope and level of
the analysis. Expressed formally, outcomes are a subsect of the
system states (e.g., at specific times, defined by events) or a
function of the states. For example, in tennis, primary outcomes
are the characteristics associated with the racket-ball impact 30,
such as the spin of the ball when it leaves the racket or the
ball's velocity 33. Primary outcomes could also include the
location of the ball on the racket's string bed 30. Depending on
the level of analysis (and available measurements), more
comprehensive outcomes include the location of the ball's net
crossing 34 or impact on the court 35.
[0636] The skill of an agent A is the effectiveness with which the
agent is using its body and/or tool, equipment, etc., to achieve
desired task outcomes TO and more generally interact with, and/or
adapt to the environment elements EE and task elements TE.
[0637] Miscellaneous additional quantities that can be added to the
description of the task or activity performance include task or
game rules (e.g., rules of the tennis game), which provide the
basis to determine task success or completion and various task
performance characteristics, as well as various decision rules and
control laws for other computer-controlled or autonomous agents,
apparatus, or equipment or accessory. For example, control law,
rules, and algorithms that specify the behavior and actions of the
apparatus in the environment. These systems can include a
prosthetic limb, an apparatus that reacts to the environment or
task interactions, or even the various components of a robotics
system such as a surgical tele-robotic system.
[0638] Note that once formalized as a dynamic and augmented
agent-environment system, many movement activities include similar
elements such as a human agent, primary equipment, the environment
and its elements and, potentially, other human or robotic agents,
and apparatus. These elements participate in the activity and
combine to produce a scope of dynamic interactions. Such activities
also follow the same general organization and therefore can be
described using equivalent quantities and general modeling language
as described here for tennis.
Augmented Skill System Overview
[0639] The following provides a systems level description and
abstraction of such augmented human systems on which the
data-driven skill analysis and training system is built. FIG. 21
illustrates an overview of the system and is followed by a
description of the "augmented human system," and finally, the
general motion model, skill model, and the different augmentation
modalities illustrated in FIGS. 22, 23, and 24.
[0640] The iterative training process illustrated in FIG. 21
illustrates three primary feedback loops: 1) A skill assessment
loop (AL) 200 that tracks the overall progress in movement
performance in the task domain, updates information about the
user's skills, including motion models and skill models, as well as
diagnostic tools used to identify specific deficiencies in movement
technique that provide the basis for the synthesis of training
goals; 2) A training loop (TL) 208 that tracks the progress in
specific areas of the skill captured by training goals and
configures the augmentation system; 3) A feedback augmentation loop
(FL) 202 that provides relevant information during the movement
performance.
[0641] The identified motion and skill models, combined with the
diagnostic assessment, provide the basis for generating a set of
instructions, which can be used to organize the training process,
and synthesize cueing laws used to drive the augmentation. A user
receives two primary forms of feedback: instructions and real-time
cues. The instructions are typically generated during a session at
particular intervals, e.g., completion of a training set, or after
a training session. Instructions are typically presented in visual
form and emphasize more comprehensive aspects of performance and
skill.
[0642] The augmentation loop can be used to exercise movement on
movement characteristics that have been identified through the
diagnostic tools. The cueing process targets specific
characteristics to directly impact movement outcome and
performance. The cueing system computes feedback signals using
algorithms that are synthesized based on the motion and skill
models derived during the assessment. These cues are communicated
in real-time to the user. The assessment and augmentation feedback
are delivered following the hierarchical organization that takes
into account the hierarchical structure of skill development and
the temporal characteristics of the movement and skill
attributes.
[0643] The training assessment loop is managed by a training agent.
The augmentation loop is managed by a cueing agent. These agents
operationalize the two processes and are able to track progress at
these two levels and provide user with the interactions to run this
system (see FIG. 21).
Data-Driven Training System Capabilities
[0644] The motion model captures the comprehensive movement
performance through the movement repertoire which organizes the
range of movements as classes of movement patterns and their
associated outcomes. The repertoire model provides the ability to
identify gaps or weaknesses in patterns. Gaps in the repertoire,
i.e., missing motion patterns, manifest as the inability to produce
actions and outcomes in areas that are relevant to the task
performance. Gaps can also manifest as the inability to deal
successfully with the range of prevailing operating and task
conditions that are required to enable high level of task
performance or from contingencies or environmental disturbances.
Movement patterns are represented to describe relevant functional
characteristics, such as phases and their associated biomechanical
constraints.
[0645] The primary functions needed to support data-driven
augmented training include: [0646] 1. Assess and guide formation,
consolidation, and optimization of patterns at the level of the
repertoire. This function focuses on the actions and outcomes that
support task performance. [0647] 2. Assess and track the quality of
movement outcomes. Deficient patterns don't achieve the required
outcomes consistently or efficiently, or don't achieve them under a
sufficient range of conditions. [0648] 3. Diagnose movement
technique for deficient patterns, which corresponds to determining
aspects of the movement technique that are favorable or detrimental
to the outcomes. [0649] 4. Diagnose movement skills based on their
development stage to determine the appropriate types of training
(formation, consolidation, or optimization). [0650] 5. Formulate
training goals to address the specific deficiencies in skill
elements. [0651] 6. Determine appropriate forms of feedbacks across
the scope of human information processing levels, including:
instructions, real-time cues, and apparatus interactions. [0652] 7.
Monitor the learning process, track and update the skill models,
the derived training goals, augmentation forms, etc. based on
changes in a subject's movement skills and other factors including
health and fitness.
[0653] The system provides a range of feedback types that act as
drivers to modify subjects' behavior toward improving their skills.
The feedbacks are based on information and knowledge extracted from
the motion and skill models, as well as from the extended analysis
based on performer population, which make it possible to account
for broader factors.
[0654] The feedback, as already discussed, operate at various
levels of the human information processing systems. These encompass
a broad range of neuro-cognitive mechanisms. For example, the
highest-level feedbacks are based on drivers that are rooted in
social aspects of performance. These include leaderboards with
ranking, side-by-side comparisons between players (e.g., via the
skill profile, see FIG. 17), or role models that can be selected
from the population analysis.
[0655] TABLE 4 summarizes the drivers, derived from the data-driven
modeling and assessment, according to their levels of operation in
the hierarchy.
TABLE-US-00005 TABLE 4 Drivers for training derived from movement
and skill model across the hierarchical levels. Top-level Mid-level
Low-level drivers (cognitive) drivers (cues) drivers (signals)
Patterns to Quality of Feedback develop/form outcomes (e.g.,
augmentation (accommodate pace, spin, (functional conditions and
etc.) movement outcome Perceptual cues characteristics). types)
(movement Apparatus Outcomes types coordination (interactions and
relevance with environment that drive for task (e.g., and task
sensory effect of stroke elements) and motor outcome on Mid-level
cues/ processes, shot outcome, feedback e.g., expand etc.) (outcome
conditions, Movement validation) etc.) architecture Cue environment
(e.g., perceptual mechanisms) Population reference ranges
"Peer-pressure" (e.g., ranking or leaderboard).
[0656] With these functions, it is possible to operationalize the
training process as a training program with variable degrees of
user interaction. From manual--where the user uses the features to
guide his or her decisions, to completely automatic--where the
system guides the user through the training process generating and
updating the plan according to the evolving skill status.
[0657] Finally, the entire modeling, assessment and feedback
process can be extended by population-level analysis. The specific
features include: [0658] 1. Perform population skill analysis by
clustering the individuals based on their skill level, movement
technique, skill attributes, and other potential factors (health,
age, etc.). [0659] 2. Identify the subject's population subgroup
membership and the related groups with respect to skill
development. [0660] 3. Compare the subject's skill attributes to
the subgroup's. Statistics provide appropriate reference values to
help rate each performer within the group, perform diagnostics, and
specify training goals to drive and track the training process.
[0661] 4. Check the related subgroups to determine possible benefit
of forming a new movement technique, which would help the subject
transition into a "better" subgroup. [0662] 5. The population group
capturing the skill development provides the direction for the
orientation of the training, such as movement architecture
System Architecture
[0663] As described elsewhere, the system relies on a movement
capture and measurement system (shown in FIG. 2 and FIG. 21). This
system collects data from relevant movement quantities, including
movement of equipment and body segments; physiological quantities,
including electrical muscle activity (e.g., via surface or
implantable electrodes); and other relevant quantities from the
recorded performance data. Data also includes task relevant
quantities, such as outcome of the action or movements, as well as
its effect on the larger task outcomes. The system can track
multiple users and their interactions.
[0664] The three primary feedback loops 200, 202, 208 shown in FIG.
21 are closed around the augmented human system detailed in FIGS.
22 and 23. The human movement activity is augmented at three
primary feedback levels which are communicated to the user through
different modalities. The feedback forms are organized according to
the primary levels of human information processing and include:
instructions or notifications, feedback cues, and feedback cue
signals.
[0665] As already described, communication modalities include
audio, visual or haptic stimuli (potentially also direct functional
muscle stimulation or even stimulations of the subject's peripheral
and central nervous system). In addition, feedback augmentation
also includes activity interactions provided by an apparatus.
[0666] The purpose of these feedback augmentations are as follows:
[0667] Instructions provide information about the training elements
and the associated training goal. This information contributes to
the formation of mental representations. They are typically
communicated verbally, symbolically, or graphically. [0668]
Notifications provide information about progress with respect to a
training goal. These are considered at the knowledge level of human
information processing and can be communicated verbally,
symbolically, or graphically. [0669] Cues provide information to
highlight specific features about the performance or outcome. They
are typically communicated through discrete audible, tactile
(haptic), or visual signals and contribute to the formation of
rules that allow efficient processing of information both for motor
and perceptual functions. [0670] Signals provide real-time
information to guide movement and enhance relevant movement
features. They are typically communicated through continuous, or
piece-wise continuous audible, tactile (haptic), or visual signals
(potentially also direct functional muscle stimulation or even
stimulations of the subject's peripheral and central nervous
system). [0671] Apparatuses enable interactions at the activity
level to emphasize particular task states or conditions, such as a
ball machine that can throw balls with different trajectories
(pace, spin, height, depth, etc.) to the player. An apparatus can
also be used to physically guide movements, such as with an
assistive robotic device.
[0672] The system is configured to receive various inputs from the
user, coach, or physical therapist. The user interactions are
enabled by a graphical user interface (GUI) and/or natural language
interface (NLI). The GUI or NLI enable the user to browse or
interrogate the skill assessment and configure the training
process. For example, users can select which training elements to
track and which feedback forms (notifications, cues, signals) are
preferred. The user can also provide inputs related to the outcomes
or technique of the movement during performance. For example, they
can tag a particular action or movement they believe is relevant
for further analysis. Users can also rate individual sets in a
session, for example, based on their perceived training
effectiveness. These feedbacks about performance can be used to
highlight particular qualities during the assessment and diagnosis
process. For example, they can serve as additional assessment
signals.
[0673] The training loop is managed by a training agent which
provides various degrees of autonomy and provides functions to
assess skills along specific skill elements that have been
designated as training elements. The TL helps structure practice by
organizing training goals in schedules. It also manages the
configuration of the different components of the augmentation
system.
[0674] The feedback augmentation loop is managed by a cueing agent,
which tracks effectiveness of the selected cueing profiles (cueing
law, apparatus interactions).
General Operational Model
[0675] The general operation follows the block diagram shown in
FIG. 21 that lays out the three primary feedback loops introduced
earlier.
[0676] The Assessment Loop (AL) describes feedback that takes place
over longer periods, spanning one session to multiple sessions,
associated with the skill acquisition process. The unit of time for
the AL is an epoch, which as already discussed is defined by the
data set requirements for modeling and assessment, denoted by
superscript k. The primary functions of the AL are computing and
updating the movement models (M.sup.k) and skill model (skill
status S.sup.k). The skill status S.sup.k is a collection of skill
element extracted from the repertoire that are assessed with
respect to the skill learning stage. Information about movement and
skills are used to plan training activities and synthesize the
various forms of augmentations. The training activities are
codified by training elements and goals. These are represented as
projected changes in skill elements. Overall changes in skill are
measured as incremental changes in skill status .DELTA.S.sup.k.
[0677] The Training Loop (TL) encompasses feedback around training
elements, including selection of active training goals g=.DELTA.a,
the instructions relative to the training goals, and the tracking
and progress reported on the selected training goals. The unit of
time for the TL is the set, denoted by the subscript n. Changes in
training elements during a session are measured as changes toward
training goals .DELTA.s.sub.n.
[0678] The Feedback Augmentation Loop (FL) encompasses the feedback
during movement performance, including the various forms of cueing,
and those mediated by the apparatus to support the active training
goals. The FL focuses on the interactions that take place during
performance and directly impacts movement behavior such as provided
by real-time cueing. Users dispose of a range of instructions and
feedback modalities to augment their training or playing experience
including instructions, feedback cues, and/or apparatus
interactions.
[0679] Given the multiple hierarchical levels that contribute to
successful motion skill performance--limb segment coordination,
movement architecture, body posture, positioning, movement outcome;
all the way to movement planning, task, or game
strategy--augmentation can potentially encompass a wide scope of
skill components and interactions. As described elsewhere, the
feasible level of analysis and interactions depends on the
information that can be extracted from available measurements.
[0680] Typical training or play sessions can be described as
periods of performance interrupted by pauses (see FIG. 58). Pauses
subdivide the session into sets. Usually, users start a session by
planning their activity and setting active training goals. Not all
sessions are explicitly structured or planned. Even if this is the
case, users can improvise and at any time enable various
augmentations and access skill analysis and training management
features.
[0681] The main user interaction application supports browsing
functions to: review past and current data, view existing movement
skill status, select active training elements, view details of
training goals, and enable augmentation profiles.
Closed-Loop Data-Driven Training Process
[0682] The system components of the closed-loop training framework
of FIG. 21 operate according to three primary units of time. A
training epoch k is a time scale ranging from one to several
sessions. The set n associated with the training loop is a time
scale ranging from a few to many occurrences of a motion pattern
set, i.e., provide a unit of time to organize sessions. A set n can
have one or more active training elements. The time t corresponds
to the actual time and is typically associated with the
augmentation loop (real-time feedback from the cueing system or the
apparatus based on measurements y.sub.t).
[0683] The motion measurement data y is processed to determine the
movement state data x. The data can also include other behavioral
data (e.g., visual gaze) and task specific data (e.g., movement and
location of task elements and objects, and various types of
outcomes). Raw measurements are often extended through an
estimation process to determine relevant state information based on
available measurement data y.
[0684] Different data are emphasized depending on their role in the
system shown in FIG. 21, e.g., monitor the feedback augmentation
(y.sub.t), training loop (session Y.sub.n), or assessment loop
(.PSI..sup..kappa.).
[0685] The motion data is processed to extract the primary movement
units associated with the actions performed in a task or activity
(described elsewhere). This process can be formalized based on
human movement system theory or principles. The movement repertoire
R.sup.k can be obtained through classification of the ensemble of
movement units into a collection of movement patterns {P.sub.i},
which can be divided into classes (FIG. 12). The movement patterns
result from sensory-motor schemas or programs (described
elsewhere). Through motion modeling, these movement patterns can be
described by a sequence of movement phases that are related to the
functional characteristics, including muscle synergies,
biomechanical constraints, perceptual mechanisms, and task
constraints.
[0686] The result of the motion modeling at a given epoch k is a
set of motion models M.sup.k={.delta..sub.i, i=1 . . . N.sub.c}
that combines the movement repertoire, the phase decompositions,
and functional aspects that can, for example, be described by
finite-state machines or statistical models such as a hidden Markov
model (HMM) or some other model form learned from data (e.g.,
through deep learning). The elements of the motion model M.sup.k
provide the basis for skill assessment and diagnosis to extract
skill attributes encompassing competitive performance, task
performance, pattern performance, and physical performance levels
(see FIG. 14).
[0687] The movement pattern P.sub.i, the motion model
.delta..sub.i, and skill attributes a.sub.i in combination enable
the definition of skill element:
e.sub.i=(P.sub.i,.delta..sub.i,a.sub.i). [3]
[0688] The skill status S.sup.k contains a collection of sorted
skill elements. The skill elements are sorted based on their
acquisition stage. The collections correspond to the three
acquisition stages: formation, consolidation, and optimization:
S.sup.k=S.sup.k.sub.form.orgate.S.sup.k.sub.con.orgate.S.sup.k.sub.opt,
[4]
where e.g. S.sup.k.sub.form is the subset of skill elements that
contain motion patterns satisfying the criteria discussed earlier
for the formation stage.
[0689] Skill profile p.sub.skill(S.sup.k) describes how different
skill elements combine to create the subject's overall performance.
This information can for example be determined by adding up the
composite scores for each skill element across the repertoire:
p.sub.skill(S.sup.k)={p.sub.skill,d(e.sub.i), d=1 . . . N.sub.p,
e.sub.i.di-elect cons.S.sup.k} [5]
where N.sub.p is the dimension of the skill profile and
p.sub.dkill,d(e.sub.i) is a composite score of the skill element
e.sub.i.
[0690] Each skill element e.sub.i can be selected and combined with
a training goal g.sub.i to form a training element
.gamma..sub.i=(e.sub.i, g.sub.i). Simultaneously, analysis of the
training element can determine feedback augmentations that are
appropriate for achieving particular training goals. The
augmentations include instructions, real-time cueing, as well as
interaction modes mediated by an apparatus.
[0691] Training goals take into account specific skill
characteristics. For example, statistical analysis of skill metrics
associated with a skill element can be used to predict the expected
progress along the skill metrics. When population data is
available, additional statistics from the subject's sub-group can
be used to provide reference values and goals for the various skill
attributes a.sub.i. Training goals are expressed as the desired
changes in skill attributes:
g.sup.k+1.sub.i=a.sup.k+1.sub.i-a.sup.k.sub.i=.DELTA.a.sup.k.sub.i.
[0692] The training goals for an epoch k are arranged in an active
training list
.GAMMA..sup.k=.gamma..sub.1.fwdarw..gamma..sub.2.fwdarw. . . .
.fwdarw..gamma..sub.Nb where N.sub.b is the length of the training
list. This can be used to plan or schedule the training session.
Since human information processing is limited, it can be helpful to
focus training on a limited set of skill elements. The primary
purpose of the training list is to designate which skill elements
to focus on, and also to configure the augmentation system. The
active training list describes an order of importance, with the
top-listed training goal representing the most significant training
goal. These elements have priority on using system resources such
as notifications or real-time feedback.
[0693] As described earlier in skill status, skill elements are
organized hierarchically to describe their acquisition stages,
which reflects their relative importance to the activity
performance. The active training list can be generated
automatically from the skill status taking into account the
relevance of skill elements, or selected by the user taking into
account information such as their preference, available time, and
conditions.
[0694] Training goals can be explicitly pursued, e.g., during a
dedicated training set. Alternatively, performance related to the
training goals can be tracked during the "free" performance of the
activity. Relevant information about these goals can be used to
notify the subject. Such notifications can, for example, highlight
when significant progress toward a goal has been achieved.
[0695] The longitudinal analysis coupled with the population data
provides both the microscopic and macroscopic information to
support training planning. Specifically, the population sub-group
and its association with the subjects' individual characteristics
(physical, training history, skill status, etc.) provide the
information needed to manage the skill development: at the
microscopic level, by providing references of realistic and
preferred skill and performance characteristics relative to the
group at a given level; and at the macroscopic level, by providing
directions on movement architecture, and other attributes such as
movement functional characteristics, to adopt for efficient and
safe performance.
[0696] Another population analysis is performer profiling. Specific
characteristics of a subject groups can be captured by their skill
profiles, which can be described by composite metrics that
emphasize different attributes. These profiles make it possible for
the assessment and diagnostics that drive a subject's behavior in
the direction of that subgroups' style.
[0697] In conclusion, the motion model, skill status, skill
elements, and training elements provide the quantities needed to
implement training as a data-driven, iterative process. For each
performance set, the training goals in the active training list are
tracked to provide progress reviews or notifications. As the
performance of several training elements has improved substantially
(e.g., when one or more training goals have been met), the motion
model and skill status can be re-assessed, leading to an update in
skill status. At that point, the user may continue with the
remaining elements in the training list or re-assess which aspects
of skill to emphasize.
Training Modes
[0698] One disclosed capability is the management of comprehensive
information relevant to a user's movement performance and its
application to drive and manage training. The disclosure also
addresses the problem of how this information is communicated to
the performer. The system can support several modes of
interactions. These modes distinguish themselves by the levels of
augmentation (types of feedback) and how the training elements are
used to direct training.
[0699] The following training modes are considered for illustrative
purposes: [0700] Fully guided training: training agent selects the
training elements and provides a training plan that specifies which
training elements are exercised and when to switch training
elements. This mode also includes drills. [0701] Partially guided
training: training agent selects the training elements and user
determines the order of training elements to exercise and when to
switch training elements. [0702] Interactive, augmented play:
training elements are selected by user, user determines the order
of training elements to exercise and when to switch training
elements. In augmented play, training elements can be integrated
within regular playing sessions. Tracking takes place in the
background and the training agent provides notifications on various
milestones for the selected elements. [0703] Free augmented play:
users can take advantage of feedback augmentations during regular
play.
[0704] The technology can also be used by a coach as a tool or
training assistant. In this scenario, the coach will essentially
become an element in the feedback training loop. The "augmented"
coach within this system can play several functions, including
interpreting the results of the skill assessment, planning the
session, and providing verbal and other instructions such as
demonstrating the movement.
Quantities and Variables for Implementation of the Training
System
[0705] The following describes the primary system components and
their system-wide integration in terms of logic diagrams shown in
FIGS. 49-58. FIG. 49 shows the top-level logic diagram for the
overall system and its primary processes, depicted in FIG. 21. The
main blocks in the diagram are as follows.
[0706] Data Acquisition 110 represents the process of capturing
performance data, which includes movement measurements from the
activity, and other relevant activity data. The movement
measurement covers the motion of the agent and his or her segments.
The activity data covers the quantities useful to assess outcomes
related to the performance and goals of the task or activity.
[0707] Modeling 120 represents the processes of modeling the
subject or agent's movement and the relevant interactions with the
task and environment elements. It includes extracting and
processing primary movement units (PMUs), which are subsequently
decomposed into movement components (described earlier), such as
movement phases and muscle synergies that are relevant to the
functional understanding of the movement patterns.
[0708] Skill Assessment and Diagnosis 130 represents the processes
used to determine the parameters that characterize a subject's
skill elements, which subsequently determine a subject's skill
profile and skill status. These processes can also take population
data as inputs, denoted here as reference data. This additional
data enables the determination of the player's profile.
[0709] Training Goal and Feedback Synthesis 140 represents the
processes involved in designing the various feedback augmentations
(instruction and notification, cueing laws for real-time feedback,
interaction laws for the apparatus).
[0710] Planning 150 represents the process of selecting the
training elements, as well as planning and possibly scheduling of
the training goal sequence, that will be used to manage the
training or activity session.
[0711] Finally, Activity Management 160 represents the actual
performance of the activity, including processes through which the
feedback acts on the users during the various interactions. It also
includes the process of tracking the training, managing the
session, and configuring the overall system. The configuration
determines the feedback profiles and how the training goals are
tracked during the performance.
[0712] Additional details associated with these processes are
provided in the following sections.
I. Data Acquisition
[0713] Data Acquisition 110 is the collection of all relevant
performance data from a given performance. It is achieved through a
variety of motion capture technologies, including IMUs that are
worn by the subject or embedded into equipment or clothing, and
optical or vision-based motion tracking systems. Data Acquisition
also encompasses measurement of other activity or task relevant
quantities such as outcomes and performers' behavioral data (e.g.,
visual gaze data). In addition, estimation techniques can be
applied to estimate unmeasured quantities from the available
measurements.
Motion Activity Measurements
[0714] Consider y.sub.t=y(t*.DELTA.T), 0<=t<=N.sub.t, where
.DELTA.T is the measurement time interval and N.sub.t is the number
of measurement samples. The measurements y encompass all relevant
data for the desired level of analysis. They can include other
behavioral measurements, such as gaze or muscle activity, as well
as contextual data (information about set, session, player/subject,
task, and environment conditions, etc.). Some quantities can be
estimated. In the following, the notation, y encompasses any type
of data, measured or estimated.
[0715] The performance data set for an activity set n:
Y.sub.n={y.sub.t, t=1 . . . N.sub.t,n} where N.sub.t,n is the
number of measurement samples in set n. The data set for the entire
performance epoch k is: .PSI..sup.k={Y.sub.n, n=1 . . .
N.sup.k.sub.n}, where N.sup.k.sub.n is the number of sets in epoch
k.
II. Modeling
[0716] Movement modeling 120 (see FIG. 50) uses the captured
performance data and possibly previous movement models 260 to form
the subject's up-to-date movement model. Movement modeling is an
ongoing process that evolves in parallel with the skill
acquisition. Therefore, it typically accounts for previous model
information, in an iterative process.
[0717] The modeling process described in FIG. 50 includes the
following steps: [0718] Extraction 210 [0719] Classification 220
[0720] Phase segmentation 230 [0721] Synergy decomposition 240
[0722] Motion model formation 250
Movement Patterns
[0723] The first step in movement modeling is the identification
and extraction 210 of the movement patterns associated with the
primary movement units (PMU). The PMUs are described in terms of
movement unit profiles, which can be represented as time histories
or trajectories in the state space. These profiles are then
classified 220 to determine the membership information needed to
determine the movement repertoire.
[0724] The measured movement data is segmented to extract the set
of primary Motion Units {s} for the activity and their associated
actions, events, and outcomes.
[0725] The set of extracted motion units for an activity set is
.XI..sub.n={s.sub.j, j=1 . . . N.sub.s,n}, where N.sub.s,n is the
number of motion units in set n. Similarly, the set of extracted
motion units for an epoch k is .XI..sup.k={s.sub.j, j=1 . . .
N.sup.k.sub.s}=.orgate..XI..sub.n, n=1 . . . N.sup.k.sub.n, where
N.sup.k.sub.s is the number of Motion Units in an epoch k (e.g.,
one or more sessions).
[0726] The repertoire is obtained through classification of the set
of primary movement units .XI. over the period of activity, for
example the set of movement units .XI..sup.k for an epoch k.
[0727] The movement repertoire for an epoch k: R.sup.k={P.sub.i,
i=1 . . . N.sub.c}, where N.sub.c is the number of primary classes
or clusters. The classification can be hierarchical, where one
primary pattern class P.sub.i can be decomposed into pattern
subclasses: P.sub.i={P.sub.i,A, A=1, . . . N.sub.c,i}, where
N.sub.c,i is the number of subclasses under P.sub.i, and
P.sub.i,A={P.sub.i,A,B, B=1 . . . N.sub.c,(i,A)}, where
N.sub.c,(i,A) is the number of sub sub-classes.
Movement Model
[0728] The classified movement units can then be further analyzed
to determine additional information relevant to the selected level
of analysis. For example, PMUs can be further segmented into
movement phases 230 associated with the muscle synergies, or other
forms of segments relevant to the execution and functional analysis
of movement. The logic to select the level of analysis is shown in
the inset in FIG. 50. The phase segmentation 230 generates
finite-state motion models, used in the functional analyses of
movement, as well as a finite-state estimator, which is used in the
cueing system. Finally, the synergy decomposition 240 generates
muscle synergies that can be used for physical and musculoskeletal
analysis.
[0729] The result of the motion modeling is a set of motion models
M.sup.k={.delta..sub.i, i=1 . . . N.sub.c} for an epoch k. This set
describes the entire repertoire R.sup.k, with each motion model
.delta..sub.i describing a primary movement unit P.sub.i. For
example, a motion model can be a finite-state statistical model
(HMM, etc.) such as:
.delta..sub.i:X.times.U->X, [5]
where .delta..sub.i is the state-transition function for the
pattern P.sub.i, U is the input alphabet, and X is the set of
states. Each model typically accounts for relevant functional
details, such as the movement phases and associated actions or
events for a specific movement pattern class (muscle activation,
environment or task state, sensory or perceptual state, etc.).
[0730] The synergy decomposition 240 uses the movement phase
profiles to determine components of muscle activation patterns that
are combined to produce the resulting movement throughout a phase.
Typically, adequate determination of movement synergies requires
measurement or estimation of the individual body segments movement,
and possibly other relevant quantities, including physiological
quantities such as electro-myographic measurements of electrical
muscle activation. Modeling the relationship between movement
components and the musculoskeletal system provides information that
can be used to estimate the biomechanical load and in turn help
prevent excessive wear and injury.
[0731] The various modeled quantities are combined to form the
motion model 250.
III. Skill Assessment and Diagnostics
[0732] Skill assessment and diagnostics 130 (see FIG. 51) and the
underlying skill modeling, as described elsewhere, builds on the
elements of the motion model (repertoire, movement phases, etc.)
and the skill and performance attributes that can be generated
through various metrics.
[0733] Movement pattern classes P.sub.i, and associated motion
models 8, provide the structure to perform skill modeling, various
forms of assessments, and diagnostics. The assessment is primarily
a descriptive process of various skill characteristics relevant to
the movement activity. As shown in FIG. 52A, the overall movement
skill assessment metrics encompasses several levels: physical
performance 312, pattern performance 313, task performance 314, and
competitive performance 315. Each level, if selected 311, will be
assessed across several components described earlier: outcomes,
functional, perceptual and memory and learning (see FIG. 10).
[0734] Skill modeling uses the attributes generated in the
assessment process and integrates them 316 to enable movement
diagnosis. The assessment step includes determining relevant
quantities from the movement data, elements of the movement model,
and movement activity performance. Reference values 317 from
population analysis or individuals can also be incorporated in the
assessment of the skill elements.
[0735] The diagnostic step includes interpreting these quantities
to identify which aspects of the movement technique or other
physical attributes need to be changed to improve the outcomes and
other behavioral characteristics critical for movement activity
performance. This process is achieved by determining the
relationship between outcomes and the various skill attributes. The
movement functional analysis plays a critical role in movement
technique diagnostics since it describes the mechanics of how
movement accomplishes its outcomes.
[0736] This information is then used in a subsequent step to
formulate training goals and to synthesize the augmentation that
can be used to drive training (FIG. 53).
Physical Performance Assessment
[0737] At the physical performance level 312, the assessment
evaluates skill in terms of the physical effort required to achieve
the outcomes and in terms of characteristics associated with the
biomechanical constraints, such as the strain on the muscles or
torques and forces on the skeleton, ligaments and joints. The
movement physical performance assessment is based on metrics such
as energy or jerk. These quantities can then be related to the
outcomes, or used to determine movement efficiency.
[0738] This assessment level also evaluates the relationship
between movement patterns, specific movement phases, and wear and
strain on the associated musculoskeletal structures. The features
extracted from this assessment can then be used to generate
feedback to help modify aspects of the associated movement
execution and thereby help mitigate injury.
[0739] The output of the physical performance assessment 312 are
metrics p.sub.i=h(P.sub.i, .delta..sub.i), such as peak power,
energy use, and joint torques associated with either select
movement segments or the overall movement pattern.
Pattern Performance Assessment
[0740] At the pattern performance level 313, the assessment
evaluates movement technique, as well as all other supporting
functions, such as perception, that a subject uses to achieve
outcomes under changing and uncertain conditions. The pattern
performance assessment provides critical information for the
movement diagnostics.
[0741] Basic movement skill assessment includes the analysis of
movement technique by extracting attributes of the movement
trajectories within a given movement class. Typical movement skill
attributes include: [0742] Smoothness: Many skilled movements are
obtained as a sequence of movement phases. Phases typically
represent individual synergies (described elsewhere). Skill
acquisition involves the consolidation of the phases into units of
movement behavior. Proficient subjects, therefore, are able to
execute the sequence seamlessly, while execution for beginners are
more disconnected and discrete. [0743] Consistency: The movement
profiles in a class represent general motor programs (described
elsewhere). Therefore, proficient subjects are expected to display
consistent trajectory characteristics within a class. [0744]
Timing: The successful execution of movements and their associated
outcomes depends on accurate spatial and temporal coordination.
Critical timing characteristics can be evaluated and used as skill
metrics.
[0745] More advanced movement skill assessment builds on the
movement structure (e.g., phase decomposition) and is based on
derivatives computed using sensitivity analysis. The primary
functional metrics are derivatives that capture how different
features describing the movement technique, participate in outcomes
and adaptation to task conditions.
[0746] The output of the pattern performance assessment 313 are
features f.sub.i=g(P.sub.i, .delta..sub.i) that capture relevant
characteristics of the movement technique. These can be determined
for relevant skill and performance characteristics, and can be
expressed as features of the finite states X, such as timing
characteristics, movement and body configuration at state
transitions, or movement phase profiles during phases.
[0747] Comprehensive functional movement skill assessment can also
include the perceptual functions or events that are relevant for
the coordination of the movement phase with task and environment
elements. As stated elsewhere, the scope and depth of the skill
assessment, and therefore also the scope of diagnostics and
feedback augmentations, depends on the available measurements.
Task Performance Assessment
[0748] At the task performance level 314, assessments evaluate a
subject's skills in terms of the range of movement patterns in the
repertoire developed to tackle the movement activity requirements
and handle the range of conditions prevailing during
performance.
[0749] As described elsewhere, in open skills, a range of different
movement behaviors have to be acquired to successfully deal with
task and environmental conditions. Diverse movement patterns are
needed to achieve different outcomes, and to attain those outcomes
under a variety of conditions.
[0750] Therefore, the primary skill metrics at the task performance
level focus on the range and quality of outcomes associated with
the actions or movement patterns in the repertoire. The output of
the task performance assessment 314 are task and outcome metrics
m.sub.i=f(P.sub.i, .delta..sub.i) which typically represent
descriptive quantities determined from the movement model and
outcome measurements and/or estimations. They can include: success
rate, movement outcome/result, variability, precision, as well as
statistical characteristics, for a specific session and/or relative
to historical data.
Competitive Performance Assessment
[0751] At the competitive performance level 315, assessments
evaluate a subject's skills in terms of overall strategies
developed to tackle the task and handle the range of conditions
prevailing during performance.
[0752] The motion patterns can be used as the state of the agent to
describe the agent-environment interaction dynamics at the task and
competitive level. For example, a player's sequence of motion
patterns in a game or set can be described by an HMM model,
P.sub.k+1=.PSI.(P.sub.k, c.sub.k), where .PSI. is the conditional
probability of transitioning from motion unit P.sub.k to P.sub.k+1
given subject observation of cues c.sub.k at time k, which includes
perception of environment conditions such as opponent shot and
incoming ball trajectory, and a subject's perception of their own
position on the court. The function .PSI. can capture a player's
strategy, as well as their ability to perceive the game status and
opponent actions and intention. Therefore, the function .PSI.
contains information that can be used to assess player's
competitive performance. Such an HMM model can be extended to
include any relevant state information such as the position of the
subjects or the position of the ball.
Integration of Skill Attributes and Definition of Skill Element
[0753] The Skill Attributes for a particular movement pattern
provide information for the overall assessment of the movement
skill, performance, as well as other relevant considerations such
as injury risk and the learning process itself. The skill and
performance attributes extracted for a movement pattern P.sub.i are
combined
a.sub.i=c.sub.i.quadrature.m.sub.i.quadrature.f.sub.i.quadrature.p.sub.i
to define the set of so-called Skill Elements 318.
[0754] A Skill Element provides the formal definition of the
concept of a unit of skills. The skill element e.sub.i combines the
pattern class P.sub.i, its movement functional structure MFS (e.g.,
specified by the motion model .delta..sub.i), and various relevant
attributes a.sub.i:
e.sub.i=(P.sub.i,.delta..sub.i,a.sub.i) [6]
[0755] The collection of attribute a.sub.i, in particular the
outcomes, the attributes relevant to technique and performance, and
the range of operating conditions, provide a comprehensive
description of each skill element. This information can be used to
score the skill elements, which helps determine which ones a
subject can perform more proficiently. An example of such a score
is the use of a composite cost function.
[0756] For example, the cost function Q can be defined as the
weighted sum of attributes, with the weights indicating the
relative importance of each attribute:
Q(a.sub.i)=.SIGMA..sub.e.sup.Naw.sub.ea.sub.i,e/.SIGMA..sub.e.sup.Naw.su-
b.e. [7]
Determination of Skill Status
[0757] A critical aspect of skill assessment is the acquisition
stage (e.g., formation, consolidation, optimization). This
information is described by the concept of skill status (FIG. 52B),
which provides information about the acquisition stage of each
skill element. This information is useful for the determination of
training or rehabilitation intervention.
[0758] The skill status can be determined by applying criteria
derived from a variety of skill attributes a.sub.i and their
associated statistics 321. TABLES 1 and 2 describes examples of
criteria that can be used to determine the acquisition stage of
movement patterns from the repertoire.
[0759] The Skill Status S.sup.k for an epoch k can be represented
as a partition on the set of skill elements that covers the
movement repertoire:
S.sup.k=S.sup.k.sub.form.orgate.S.sup.k.sub.con.orgate.S.sup.k.sub.opt,
[8]
where e.g., S.sup.k.sub.form is the subset of skill elements that
contain motion patterns satisfying the criteria discussed earlier
for the formation stage.
Determination of Skill Profile
[0760] As described earlier, the repertoire combines the collection
of movement patterns that have been acquired by an individual to
deal with the task requirements and environment conditions. The
motion model encompasses the movement repertoire, its movement
classes, and movement phases and synergies. The extracted
attributes from the various skill metrics provides additional
information to determine other quantities relevant to learning and
training.
[0761] The comprehensive description of an individual's skill can
be determined based on the set of skill elements associated with
all the classes (and potentially sub-classes) of movement patterns
in the repertoire:
S.sup.k={e.sub.i,i=1 . . . N.sup.k.sub.c}. [9]
This set is shown here for a particular epoch k.
[0762] The skill elements e.sub.i combined with the skill status
provide comprehensive information about the subject's movement
technique and performance. This information can be used to generate
a so-called skill profile 330, which describes the overall skill
and performance of a subject.
[0763] Skill Profile p.sub.skill(S.sup.k) describes how the full
set of skill elements combine to create the subject's overall
performance. This information can for example be determined by
adding up the composite scores for each skill element across the
repertoire:
p.sub.skill(S.sup.k)={p.sub.skill,d(e.sub.i),d=1 . . .
N.sub.p,e.sub.i.di-elect cons.S.sup.k}, [10]
where N.sub.p is the dimension of the skill profile and
p.sub.skill,d(e.sub.i) is a composite score of the skill element
e.sub.i. In some conditions p.sub.skill,d(e.sub.i) can be
simplified to be Q.sub.d(a.sub.i).
[0764] The skill profile can be illustrated graphically for example
by displaying the skill composite of each skill elements (see FIG.
17). One potential output of this assessment process is to generate
a list of the movement patterns sorted by skill level based on
composite score and development stage 326. This list provides the
basis to define the training elements.
[0765] As already described, higher-level assessment such as task
performance and competitive performance can be determined by how
elements are deployed in a task or game.
Determination of Other Forms of Statuses and Profiles
[0766] Movement classes can be arranged relative to physical and
biomechanical criteria. Typically, skill and physical attributes
evolve in parallel during learning; however, subjects can adopt
techniques that may be effective in achieving outcomes but
detrimental to their musculoskeletal health. Possible physical
development stages include, "physical build up," i.e., patterns
where the technique is affected primarily by a lack of adequate
strength, "endurance," i.e., patterns that exhibit premature wear,
and "excessive load," i.e., patterns that are executed with a level
of force that produces excessive wear and strain on the body. This
information can for example be used to determine an injury index
for each skill element. This index can then be added across the
repertoire to determine the injury profile.
[0767] Similar ideas of profiles, which are based on some composite
assessments, can be generated for other characteristics besides
movement skills. For example, the motion and skill model and
attributes can be combined to compute quantities such as
Physical/Fitness Profile p.sub.fitness(e.sub.i) or Injury Profiles
p.sub.health(e.sub.i).
Determination of Player Profile
[0768] Note that it is possible to add more importance to some
outcomes or actions or skill elements in the task by setting
different weights in the skill profile, e.g., giving more
importance to a forehand top-spin high compared to a forehand slice
low. Therefore, the skill profile can be tuned to particular task
requirements or styles of performance. For example, some of the
strokes and outcomes are more fundamental to player performance.
These can be characterized as core strokes. Different tiers of
skill elements or strokes can be defined, and the skill profile
therefore can be decomposed into different profile components to
capture different characteristics.
[0769] The relative weights assigned to the skill elements in the
skill profile enable characterizations of performer or player
types, which can be used to define a player profile 340. For
example, in tennis, strokes that are used in defensive play as
opposed to offensive play provide the information to characterize
the player type. This information is further developed under
population analysis.
Reference Ranges and Population Analysis
[0770] One aspect of assessment is the definition of reference
ranges that make it possible to more objectively assess a subject
performance or skills (see 317 in FIG. 52A). Reference values can
be used to provide absolute references, for example to measure how
the various extracted attributes compare to a representative group
of players. For example, in tennis, this allows a subject to
understand if their topspin amount produced for a particular stroke
class is high or low.
[0771] Reference values can be determined by extracting statistical
distributions across attributes for a group of subjects with
similar movement technique. The statistical data can then be used
to generate percentile ranks for any relevant attribute, and using
those for example to discretize the reference ranges into tiers
(such as low, medium, high, very high).
[0772] These various forms of performance, skill, health, or injury
profiles provide information that can be used for high-level
feedback on various aspects of performance including strategy,
physical fitness, as well as injury prevention.
[0773] Population information can be used to determine leaderboards
that can be helpful for a coach or a physical therapist. It can
also help motivate subjects to understand how they compare with
other individuals, e.g., in an absolute ranking, as well as to
understand the specific aspect of their skills or performance that
is responsible for their ranking and which aspects of their
movement technique or performance is the most actionable to help
them progress within their group.
IV. Training Goals and Feedback Synthesis
[0774] Training Goals and Feedback Synthesis 150 represents the
determination and specification of training goals, discussed
earlier, and associated augmentations that can be used to drive
training. These are selected across the different feedback
modalities.
[0775] The specification of a training goal 410 can be viewed as a
dual problem to the determination of an augmentation that will help
drive the training process toward the goal. Ideally, goal and
feedback synthesis is performed simultaneously.
[0776] Feedbacks target the skill elements, as well as other
aspects of skill and performance such as insights from the skill
profile, identified in the assessment. It can also use information
from diagnostic data such as from the skill status (S.sup.k)
130.
[0777] The synthesized feedback (instructions, real-time cueing
laws, etc.) determine the "augmentation space" available to skill
goals (FIG. 21). These augmentations define the scope of user
interactions within which the user can then choose to operate. FIG.
23 describes how the augmentation environment is enabled and
operated during performance.
[0778] The acquisition stage in the skill status computation
provides information that allows to determine the appropriate
feedback forms. For example, the formation of new patterns requires
different augmentations than the refinement or optimization of an
existing movement pattern. In general, several feedback modalities
can be combined (e.g., instructions, feedback cues, and apparatus
interactions). The feedback configuration 426 describes how
feedback modalities are combined to produce the user
augmentation.
Training Goals and Elements
[0779] Training goals help make training actionable, and enable
subjects to focus their attention during performance. The
quantitative specification of the training goal also means that it
can be measured or estimated, which allows to objectively track the
training progress for the particular skill element.
[0780] The computation of training goals 410 (FIG. 54A), as
previously discussed, is based on the skill elements, the larger
system in FIGS. 10, 30, 31, and can also account for the skill
status to help specify a meaningful training goal.
[0781] Training goals can be derived from the statistical analyses
of a subject's skill at the various assessment levels such as task
performance level, for example, based on the attributes within a
skill composite score, taking into account population reference
data (see FIG. 19). Or, for example at the physical performance
level based on the functional analysis (see FIG. 20 and FIG. 37).
The training goal can for example be derived based on an increment
(or a fraction increment) in a percentile tier for a skill level or
outcome level, respectively.
[0782] A training goal at the performance level can be determined
from the global score ranking shown in FIG. 34. One can proceed,
for example, by identifying the skill element in the repertoire
that has the largest impact on ranking (critical skill element).
And from there determine the skill attribute in the skill profile
composite (FIG. 38) that has the highest impact on skill profile.
It is possible to determine the target skill attribute a*
associated with a target increment in composite score for the
critical skill element e.sub.i using the statistical distribution
shown in FIG. 19 as follows:
g.sup.ki=a*.sub.i-a.sup.k.sub.i=Aa.sub.i,k [11]
where a* is the goal value for the feature that would result in the
target skill profile level, and where k stands for the epoch.
[0783] For a training goal at the physical performance level, one
can proceed following an outcome optimization based on functional
analysis, as given in the example of the forward swing analysis
shown in FIG. 37. The target skill feature f* associated with a
target increment in outcome level (spin) for a skill element
e.sub.i can be determined from:
g.sup.k.sub.i=f*.sub.i-f.sup.k.sub.i=.DELTA.f.sub.i,k [12]
where f* is the goal value for the feature that would result in the
target outcome level, and where k stands for the epoch.
[0784] Since skill deficiencies often manifest across multiple
attributes, one or more component, or even some combination of
components of attribute a.sub.i, can be selected as the critical
attribute to drive a particular goal g.sub.i. Furthermore, the
attributes can require targeted movement technique optimization.
Therefore, the attribute goal can be combined with the functional
analysis.
[0785] Note that the relationship between attributes and target
increment in skill level was described at the diagnostic level 130
(see distribution and tiers in FIG. 19). Dimensionality reduction
or embedding techniques can be used to determine the functional
relationship. This level of analysis is typically conducted during
functional movement modeling.
[0786] One question for the specification of training goals is the
determination of how actionable an attribute or feature is.
Functional analysis usually provides enough information to
determine causal relationship and identify the critical driving
attribute for training.
[0787] Which attribute to select to drive training 411 can also
depend on the acquisition stage (Skill Status). Training goals can
have different formats, depending on the level in the hierarchy
(outcome, or functional characteristic), and also the acquisition
stage of the targeted training element.
[0788] Training Element .gamma..sub.i,b=(e.sub.i,g.sub.1,b)
describes a Skill Element e.sub.i combined with a Training Goal
g.sub.1,b.
Levels and Types of Training Goals
[0789] For the formation patterns 412, the specific goals include
the spatial definition of the movement configuration. This
corresponds to the cognitive stage of skill acquisition where the
subject forms an understanding and representation of the movement
primarily focusing on its spatial configuration. The knowledge for
example includes the movement phases, including the configuration
of the body segments, and end effector and equipment (system
state), at phase transitions. Also relevant at the formation stage
is the understanding of the relationship between the movement
phases and their synchronization with the environment and task
elements and objects.
[0790] For the consolidation patterns 413, training focuses on
consolidating the sequence of movement phases into a smooth
trajectory. This stage corresponds to the consolidation of the
procedural memory where the movement knowledge is translated into
an automatic pattern that can be performed dynamically under
various conditions. This stage is mostly unconscious and relies on
feedback to validate the correct technique.
[0791] For the optimization patterns 414, the specific goals
include the refinement of the movement patterns and associated
functions to achieve best outcomes within the subject's
bio-mechanical constraints. The quantities that are optimized
include the functional characteristics (features associated with
movement outcome) and physical performance characteristics
(musculoskeletal loads). At this stage, the subject can form mental
representations that enable them to focus on features in the
technique that influence the outcome, or gain an understanding of
which elements of the task convey information that helps with the
movement modulation or timing.
[0792] These training goals can be codified based on the parameters
associated with the acquisition stages that are relevant to the
movement activity. These parameters include statistical
characteristics of the relevant parameters such as consistency,
smoothness, and timing (described earlier). Augmentations are
selected to target the aspects that are critical for the particular
acquisition stage.
Feedback Synthesis
[0793] The feedback laws are synthesized 420 using the training
elements (combining skill elements and training goals), including
information from skill profile and status (FIG. 54B). The
terminology of feedback is used in the larger sense, with the
following two primary feedback types: instructions 421, and
feedback cues 422. In addition, an apparatus 423 (see e.g., ball
machine in FIG. 2) can be used to provide additional interactions
for movement performance and training (see later discussion, see
FIG. 23).
[0794] Instructions are synthesized from the skill model parameters
and assessment 424, in particular the skill profile and player
profiles.
[0795] For instructions (see FIG. 55A), the communication
modalities include visual 431, verbal 432, and text 433.
Instructions 434 represent feedback that operate at the "knowledge"
level. They include aspects such as descriptions of the training
elements for the next training set, or details about the movement
features that will be augmented through feedback cues. Instructions
can also include visual descriptions or simulations of the spatial
configuration of formation patterns.
[0796] Cueing mechanisms 439 are synthesized from the motion model
and in particular the functional movement model 425. For cues (see
FIG. 55B), the cueing mechanisms include validation cues 435,
outcome optimization cues 436, alerts 437, and pattern formation
cues 438. These cues are used as feedback augmentations. The cueing
laws for the real-time feedback cues are determined from the
functional movement modeling and analysis.
[0797] If an apparatus is available, such as a ball machine, 427,
apparatus interaction modes are synthesized 423.
[0798] The instructions, cues, and apparatus can be combined to
create different augmentation profiles that lead to different
interaction forms. The synthesized instructions and cueing
mechanisms are first integrated to determine best combinations. The
goal is to combine these feedback modes to achieve synergy. The
settings and parameters define the available feedback
configurations 426 (FIG. 54B). These combinations are then used to
determine configuration parameters for the communication, cueing,
and apparatus systems.
General Augmentation Levels
[0799] Augmentations can operate at the symbolic/cognitive level,
cue level, and signal level. The augmentation laws and programs for
an epoch k are denoted as a collection of feedback laws
K.sup.k={.kappa..sub.i, i=1 . . . N.sub.c} and programs.
[0800] At the cognitive level, feedback is in the form of
instructions prior to performance, reports following the
performance, and notifications during the performance. Instructions
can be used to help subjects form mental representations of the
movement pattern focusing on aspects that are relevant to current
training elements.
[0801] Also relevant at the high-level are feedback related to
population analysis such as leaderboard. This type of feedback
plays at the psychological level.
[0802] Notifications can be used to provide feedback on training
progress, e.g., on a specific training goal. Reports provide a
summative overview of the subject's skills and training activity.
The generation of textual and other symbolic or graphical
information is performed via a communication system with an
instruction generator such as an expert system. Notification can be
implemented in the form of a state machine, or even using a
conversational agent which can output either text or natural
language.
[0803] At the cue and signal level, feedback is provided by the
cueing system (described elsewhere). Feedback cues target the
movement characteristics associated with the training goal (through
outcome validation, feature validation, etc.), as well as
associated sensory and perceptual processes. Cueing and signal
level feedback can operate as reinforcement or deterrents.
[0804] The cueing system can also provide visual cues to help form
visual attention needed to support a particular interaction for the
task or activity. The cueing system combines a cueing law specific
to the training element .gamma..sub.i that computes cueing signals
and a cue generator that translates and encodes these signals into
understandable signal forms (audible, visual, haptic, etc.). The
cueing laws are implemented for example by a state machine which
uses the movement measurement data y.sub.t, states x.sub.t, and/or
movement features f.sub.i to compute the cue signal.
[0805] The cue and signal level also encompass the interaction laws
for a possible apparatus. The primary role of the apparatus
interactions is to expand the operating range, for example to help
form new patterns. The apparatus can also provide physical
interactions such as those provided by an assistive robotic device
or exoskeleton. The apparatus action is driven, similarly to the
feedback cueing, by a feedback law and/or program.
V. Planning
[0806] Training planning addresses the question of which aspects of
movement performance are to be emphasized during training. The plan
or schedule describes the organization of a session in terms of the
training elements and associated training goals. The plan also
provides the structure to schedule and manage the session during
the performance of the activity 160 (see FIG. 56). Planning
typically takes into account the overall training goals, available
time, and other resources.
[0807] The prioritization can be determined from the stage of skill
acquisition of the skill elements and for the significance of the
skill elements to the task objective. The training elements can be
prioritized based on the skill status (S.sup.k) of each skill
element 415.
[0808] To facilitate the planning and management of a session,
several training elements can be selected. These selected training
elements produce a so-called training list. By selecting the active
training element, it indicates to the augmentation and tracking
system which aspects of movement performance have to be monitored
and actively cued.
[0809] The training process is formalized as an iterative learning
process. This model makes it possible to determine how the data is
managed. For example, epochs can be defined to coincide with major
developmental changes during a training cycle over which a new
skill level is reached with significant changes in attributes to
result in a new profile. This epoch has an associated data set with
motion model, skill model and various skill attributes, and
statistical descriptions. For each epoch, there are associated
training elements and goals that when completed will lead to a new
skill level. The delineation between epochs can be arbitrary. More
objective criteria can be used to determine training epochs, for
example, the validity of the motion model used for motion pattern
classification. As an individual's movement technique changes
sufficiently to compromise the motion processing, the training
system can prompt the user and the assessment cycle can be
re-initialized, which provides a new baseline for training. The
motion model enables the analysis of the skill acquisition process
for an individual and also across the larger population. Therefore,
patterns in skill acquisition can also be used to manage the
training process and determine the larger-scale training goals.
Training List
[0810] Training list for a current epoch k, .GAMMA..sup.k can be
represented as an indexed set (list) of training elements
.GAMMA..sup.k=.gamma..sub.1.fwdarw..gamma..sub.2.fwdarw. . . .
.fwdarw..gamma..sub.Nb, where N.sub.b is number of training
elements in epoch k.
[0811] The training list provides a way to emphasize a group of
training elements. The goals at the top of the list have the
highest priority. Training priority can be determined from the
skill status parameters and criteria (see TABLE 2), the development
stage, and information about the relevance of particular movement
patterns (skill elements) and associated outcome for the task (see
FIG. 13). The designation of the priority of a training element in
the training list can be performed either manually by the user, or
automatically based on assignment of the skill elements (see
primary, secondary, tertiary in FIG. 13).
Training Schedules
[0812] A training schedule for an epoch k, .SIGMA..sup.k can be
represented as a sequence of subsets of training elements
.SIGMA..sup.k=.SIGMA..sup.k.sub.1.fwdarw..SIGMA..sup.k.sub.2.fwdarw.
. . . .SIGMA..sup.k.sub.n.fwdarw. . . .
.fwdarw..SIGMA..sup.k.sub.Nn where N.sub.n is the number of
activity sets in epoch k and .SIGMA..sup.k.sub.n is a subset of the
training list .GAMMA..sup.k,
.SIGMA..sup.k.sub.n=.gamma..sub.n,1.fwdarw..gamma..sub.n,2.fwdarw.
. . . .fwdarw..gamma..sub.n,Nbn.
[0813] Each training element may include a stopping criterion to
signal when to transition to the next training goal. Stopping
criteria could be the number of movements to repeat in that
particular class, performance over a time duration, given progress
toward the goal (given fraction), or the accomplishment of the
goal, which can be determined statistically such as in clinical
significance tests.
[0814] The ensemble of training elements and goals can be used to
systematically plan and manage training or playing sessions. For
example, a training schedule composed of sets can be generated
before the session (see FIG. 45B). Each set can emphasize one or
more training goals.
VI. Activity Management
[0815] As discussed earlier, a training or play session is
typically divided into time periods. These periods are designated
as sets. Each set can have one or more training goals. These
elements are arranged across several sets to form a training
schedule. This structure makes it possible to decompose longer-term
training goals into intermediate goals.
[0816] The implementation of the training process takes place
through the augmented human system (see FIGS. 22 and 23).
[0817] Different feedback modalities call for different frequencies
of user interactions. Instructions for example are presented
intermittently typically following the selection of a training
element. Real-time feedback cues, on the other hand, are applied
concurrently with the movement performance. Real-time feedback can
also be communicated continuously or at discrete time periods
during the execution, or just following the movement outcome.
[0818] In some activities, an apparatus is used as part of the
performance. A typical apparatus in tennis is the ball machine. The
apparatus can be programmed to work conjointly with the feedback
cues and instructions.
System Configuration
[0819] The main parameters for the augmentation system
configuration 620 are designating the targets (e.g., subject,
coach, etc.), and specifying the type of instructions (e.g.,
verbal, audio, etc.) and the type of feedback cues. The primary
systems that mediate interactions are shown in FIG. 22. They
include the communication systems (e.g., tablet or smart watch),
the cueing systems (e.g., wearable device), and the apparatus
system (e.g., ball machine).
[0820] For the instructions, different targets can be selected
based on the training format. For example, in one scenario, a coach
interprets and communicates the instructions to the subject. In
this case, the coach would receive information about the subject's
performance during a particular set and use this information to
coach the subject before the next set. In another scenario, the
subject uses instructions to assess the progress on a given
training element.
[0821] The feedback forms under instructions include visual,
verbal, or text. These forms provide different modes of
interactions. For example, they can invite the user to browse the
movement repertoire. Or, they can invite the user to learn about
technique for a particular movement pattern.
[0822] A typical scenario includes the refinement or optimization
of a movement pattern. In this scenario, the cue profile combines
phase transition cues with outcome validation cues. Yet, in another
scenario, the subject could use cueing during the performance to
assist the formation of a new movement pattern or to optimize an
existing pattern.
[0823] Once the system is configured, the subject can start with
the activity performance 630. During performance, movement and
system behaviors are monitored 640 and data acquisition continues.
However, the emphasis of the assessment is characterizing the
movement skills with the augmentation and the performance relative
to the training goals. The activity can be paused at any instant
690.
[0824] Planning may take place ahead of the session or proceed
incrementally. Initial training elements and schedule are defined
based on current skill status. The training goals and elements for
the subsequent set are defined as a function of the subject's
completion of the training goals and overall performance, as well
as other factors such as wear, fatigue, or motivation. To support
possible changes in goals and configurations, the training system
enables interactive management during the performance of the
activity.
Session Management
[0825] Managing the training activity is an interactive process.
The management of the session 610 includes specifying which
training goals are pursued at a given time period in the activity
and updating the configuration of the augmentation system
(instructions, feedback cues, apparatus mode of action). The
training goals are typically provided as part of a training
schedule specifying training elements and associated goals. The
goals are pursued through the interaction of the subjects with the
augmentations. As discussed in the section on feedback synthesis,
training goals provide a quantitative description of the change in
a training element and can take into account the augmentation
profile available for the element. The augmentation systems are
configured based on the goals of the next period of activity. The
system configuration 620 (FIG. 57A) determines how the different
feedback modalities 621, 622, 623 are combined to create
performance interactions that are most effective for the pursuit of
the training goals.
[0826] FIG. 58 describes a session temporal structure delineating
the different periods of activity, shown as sets #1 to #4. A set
can be followed by a pause in movement activity. During activity
periods, the performer receives cues, and or notifications. During
pauses, the performer can review the performance data, and if
needed modify goals and system configurations.
Activity Monitoring
[0827] Once the training activity is initiated, the progress
towards the training goals can be tracked during the training
activity 640. The change in training elements provides the basis to
provide feedback on progress. The monitoring system 640 (FIG. 57B)
provides notifications 644 to the performer (or coach).
[0828] Notification criteria 643 can be used such as the number of
repetitions of a training element 641, the achievement of a certain
fraction of the training goal, or the time elapsed. Notifications
644 indicate if a training goal has been achieved 642, which can be
determined using a form of clinical significance test. The
significance test determines when the subject's technique has
progressed sufficiently for the skill attribute to have stabilized
near the target level.
[0829] Depending on the specific system implementation, a subject's
movement skill profile and skill status can be assessed at various
time intervals to accommodate for the different rates at which
various aspects of movement skills evolve. Therefore, the
assessment loop is closed (updated) at different rates for
different system configurations and different assessment
levels.
[0830] While movement technique can be modified through
instructions, demonstration, or feedback cues, the changes that
result from these inputs first need to be assimilated. For example,
the movement repertoire does not change rapidly since it requires
consolidation of movement into procedural memory. Therefore, the
assessment at the task performance (repertoire) level is typically
made at the interval of the epoch, spanning sessions to days or
months. Epochs may be linked to changes in a subject's movement
repertoire, but as described earlier the associated time periods
are defined based on the creation and maintenance of sets of
movement data and models (see FIG. 25).
[0831] The notification 644 can be issued using a range of
communication devices and signals. For example, the subject can be
prompted 645 through an audible signal and a message can be
displayed on a smart watch. Alternatively, notifications can be
translated by a natural language processor and via voice
communication. The message can indicate the progress toward a
specific training goal, or attainment of a particular outcome
threshold. The system can also prompt the user for an input 645.
For example, this allows the performer to make a note or comment
646, or to simply tag a particular movement pattern, for example to
indicate an issue or an outstanding result. At any time, the user
can also prompt the system, e.g., via a smartwatch to tag an
event.
Activity Interruptions
[0832] Depending on the notification and the status of the training
or activity, performance may be paused 690. Interruption in the
activity can be prompted by the subject, the systems, or the coach.
Typical scenarios include the following: [0833] The subject briefly
interrupts the session to gather more detailed information about a
particular movement pattern that was just performed. [0834] In
another situation, the subject wants to review his or her
performance over the last set. [0835] In another scenario, the
cueing system detects a decrease in effectiveness in one of the
active cueing mechanisms. [0836] In another scenario, the cueing
agent notices that the movement performance has achieved the target
level of the training goal. The user receives instruction to pause
to select the next training goal(s). [0837] In another situation,
the coach monitors the performance via the communication system and
decides to interrupt the performance to change the configuration of
the augmentation system. [0838] In yet another scenario, the system
detects changes in the outcomes or attributes that may be due to
the onset of fatigue or wear, or even injury.
[0839] The user can receive instructions to pause, for example
through a smartwatch, and subsequently pauses the session. Once the
activity is paused, depending on the reason for the interruption,
the activity can be resumed immediately 690, or suspended for a
longer period to allow for data review and changes in plans and
configuration. At this point, the performance data can be reviewed
in greater detail 650, and then depending on the required
attention, the performance is resumed or the session can be
suspended.
[0840] Before the session is resumed, the augmentation profile 670
and training goals 680 can be updated. The change in performance
associated with an active training element may require updating the
training priorities within the existing skills status and thus can
prompt a review of the training goals 680 in the training list.
Large changes in skill status may require an update of the motion
and skill models (iteration of the assessment loop leading to a new
skill status S.sup.k+1).
Activity Suspension
[0841] Once the activity is suspended, a more detailed review can
be initiated 650. The review is typically mediated by the
communication system, i.e., tablet or smart phone. The purpose of
the review is for the user or coach to go over the progress for the
current training or to address issues that have been brought up by
the training agent.
[0842] After the review, the user has two options: to end the
activity or to resume it 660. If the user decides to end the
activity, it closes the session. If the user resumes the activity,
it can be done under the same training list and augmentation
profile 670, or a new augmentation profile can be selected which
leads to the system re-configuration 620. Alternatively, a new
activity or training plan can be selected before resuming
performance 610.
[0843] In the event of a system induced interruption, the activity
review provides details on the cause of the interruption. The user
will then typically be prompted to return to the system
configuration 620 or activity planning 610.
Examples
[0844] In the data-driven movement skill training systems disclosed
herein, the systems may use movement skill assessment and
diagnostics at distinct levels of the human movement system
hierarchy to specify training goals. The systems may provide
different forms of augmentations synthesized to help pursue the
training goals. The system may also include a system to track
and/or manage the learning process.
[0845] Efficient movement training may require a systematic way of
organizing the training process. Training may be most effective if
it targets specific areas of weakness of an individual's movement
skill, accounts for individual health and fitness, and/or unfolds
according to plans that are compatible with the structure and
principles of natural skill development. Precisely assessing skills
before planning a training activity, and/or providing adequate
forms of feedback before, during, and after movement performance
may benefit a training process. Movement skills depend on a broad
array of functions and capabilities, which may make skill
assessment and/or modeling difficult.
[0846] The systems disclosed herein may employ a movement skill
model that may help identify skill deficiencies quantitatively. The
model may also analyze the relationship of the skill deficiencies
to the skill development process. This information may be used to
synthesize feedback augmentations and/or determine training goals,
which may be designed to induce changes in movement technique and
guide training during performance. The components of this system
may form a framework that allows planning and organizing training
activity in a data-driven manner. This system may include a
systematic and individualized approach to movement training suited
to subjects' physical characteristics and health.
[0847] In one embodiment, a system for processing a variety of
movement and performance data from an activity is provided. The
system may extract movement elements that support task
interactions. The system may classify movement elements according
to type and outcomes. The system may decompose movement elements
into segments associated with biomechanical and functional
characteristics of movement.
[0848] In one embodiment, a system for assessing and diagnosing
movement techniques of a subject is provided. The system may assess
movement technique and outcome for movement classes. The system may
identify development or learning stage of classes based on skill
and outcome attributes.
[0849] In one embodiment, a system for synthesizing feedback
appropriate for a subject's skill development stage is provided.
The system may determine training goals based on performance
criteria and learning stage of the subject. The system may
synthesize feedback augmentation specific to a development stage to
assist training towards training goals.
[0850] In one embodiment, a system for operationalizing or
augmenting training of a subject is provided. The system may
schedule training elements or training goals based on development
stage, which may include intervals for memory consolidation. The
system may track a subject's performance relative to training
goals. The system may provide feedback on one or more of training
elements, skill development, injury, physical wear, and fatigue.
The system may track overall skill development. The system may
update a training goal and/or a training schedule for the subject.
The system may determine augmentation from a combination of
feedback modalities that improve training effectiveness. The
feedback modalities may include one or more of instructions, cues,
and signals.
[0851] Additional examples and embodiments include an apparatus for
movement skill training, the apparatus comprising: a sensor system
comprising one or more sensors configured to obtain movement data
for a subject performing an activity; a processor system in
communication with the one or more sensors, the processor system
having a microprocessor and memory configured to: receive the
movement data from the one or more sensors, wherein the subject
performs a primary movement unit associated with the activity;
identify one or more movement patterns from the movement data,
wherein the movement patterns are associated with the subject
performing the primary movement unit; analyze the movement patterns
to identify one or more skill attributes descriptive of the subject
performing the primary movement unit; and assess the one or more
skill attributes to specify one or more training goals for the
subject, wherein the training goals are selected to address
deficiencies in the skill attributes.
[0852] The apparatus, wherein the one or more sensors comprise one
or more inertial sensors, accelerometers, gyroscopes, or inertial
measurement units, and wherein the movement data comprise one or
more of velocity, rotational velocity, acceleration or rotational
acceleration descriptive of movement patterns. The apparatus,
wherein the one or more sensors comprise a magnetometer configured
to acquire direction or orientation data descriptive of the
movement patterns, a transducer configured to acquire one or more
of position, velocity, pressure, strain, or torque data descriptive
of the movement patterns, an acoustic sensor configured to acquire
acoustic wave data descripting of the movement patterns, a visual
sensor or camera configured to acquire image data descriptive of
the movement patterns, and a video sensor configured to acquire
video data descriptive of the movement patterns.
[0853] The apparatus, where the one or more sensors are configured
to obtain the movement data from one or both of the subject and an
associated object used by the subject to perform the primary
movement unit, the movement data selected from one or more of
angle, angular velocity, direction, distance, force, linear
acceleration, position, pressure, rotation, rotational speed,
speed, strain, and torque. The apparatus, where the one or more
sensors are further configured to obtain activity data descriptive
of the subject performing the activity over a number of sessions
distributed over a calendar period, the processor system being
further configured to assess outcomes related to performance of the
training goals over the calendar period, based on the activity data
and the skill attributes. The apparatus, where the activity
performed by the subject is selected from badminton, baseball,
cricket, golf, rehabilitative exercises, running, skiing,
snowboarding, surfing, surgery or other medical procedure,
swimming, table tennis, tennis, and volleyball.
[0854] The apparatus, where the processing system is configures to
extract the one or more skill attributes from the one or more
movement patterns to define one or more skill elements, the skill
elements characterizing movement patterns for the subject to form,
movement patterns for the subject to consolidate, and movement
patterns for the subject to optimize. The apparatus, where the
processing system is configured to determine a skill status by
applying criteria derived from the skill attributes, the skill
status defining the movement patterns for the subject to form,
movement patterns for the subject to consolidate, and movement
patterns for the subject to optimize. The apparatus, where the
processing system is configured to combine the skill elements with
the skill status to generate a skill profile describing an overall
skill and performance of the subject. The apparatus, where the
processing system is configured to analyze the skill attributes
taking into account the skill status to produce the training goals.
The apparatus, where the processing system is configured for a user
to select one or more of the training goals on the skill status and
further configured to track and update the one or more training
goals based on changes in the one or more skill elements. The
apparatus, where the processing system is configured to derive one
or more training elements from the skill elements, wherein a skill
attribute associated with one or more of the one skill elements is
assigned to one of the training goals. The apparatus, where the
processing system is configured to generate a training schedule for
the subject that comprises the training element and associated
training goal.
[0855] The apparatus, where the processing system is configured to
configure one or more of a communication system, a cueing system,
and an apparatus system. The apparatus, where the communication
system is configured to provide symbolic, verbal, or visual
information. The apparatus, where the cueing system is configured
to provide audible, visual, or haptic feedback.
[0856] The apparatus, wherein one of the training goals comprises a
pattern to form, wherein the pattern is absent from the movement
patterns in the collected data. The apparatus, where the training
goal comprises developing the pattern to form from scratch or
through modification of an existing movement of the subject.
[0857] The apparatus, where a one of the training goals comprises a
pattern to consolidate, wherein the pattern in the collected data
is not sufficiently defined in the collected data to allow reliable
execution under dynamic conditions. The apparatus, where the
training goal comprises refining the pattern or creating procedural
memory to enable automatic or repeatable execution of the refined
pattern by the subject.
[0858] The apparatus, where one of the training goals comprises a
pattern to improve, wherein the pattern in the collected data does
not achieve a desired outcome. The apparatus, where the training
goal comprises refining a movement technique to use the least
energy or to produce the least strain on a musculoskeletal system
of the subject. The apparatus, where the collected data comprises
population data.
[0859] Additional embodiments and examples include a method of
training comprising: assessing movement skills of a subject;
identifying deficiencies in the movement skills; specifying
training goals for the subject; providing augmentation to the
subject; and tracking a training process of the subject; wherein
identifying deficiencies comprises relating the movement skills of
the subject to population data; and wherein specifying training
goals comprises using the population data to determine which
movement skills can be improved by the subject to improve skill
level and to produce long-term development.
[0860] The method, where producing long-term development comprises
identifying which aspects of the movement skills can be improved
and in what order. The method, where the training goals are
associated with training elements, and a training list comprises a
plurality of training elements. The method, where selecting the
training element indicates to a tracking system which aspects of
movement performance are to be monitored. The method, where
selecting the training element indicates to an augmentation system
which aspects of movement performance are to be actively cued.
[0861] The method, further comprising developing a training plan,
wherein the plan describes the organization of a training session
in terms of training elements and the training goals. The method,
where the training elements are compiled in a training list
arranged as a training schedule. The method, where the training
schedule comprises at least one session, each session is divided
into a plurality of sets, and each set is assigned at least one
training goal.
[0862] Further embodiments and examples include a closed-loop
system for data-driven training, the system comprising: an
assessment loop configured to collect data from a movement
performance by a user; a training loop configured to track progress
in at least one skill of the movement performance; and an
augmentation loop configured by the training loop to provide
information to the user during the movement performance.
[0863] The system, where the collected data comprises one or more
of movement data from a body segment of the user, movement data
from equipment used by the user, physiological data of the user,
outcome of the movement performance, and effect of the movement
performance. The system, where the physiological data comprise
electrical muscle activity collected from a surface or an
implantable electrode. The system, where the system is configured
to track at least one movement performance from a plurality of
users. The system, where the system is configured to track
interactions between the movement performances of the users.
[0864] The system, where the assessment loop comprises an extractor
configured to extract motion elements from a target motion of the
movement performance. The system, where the augmentation loop
collects movement information from the user and provides motion
elements to the extractor. The system, where a motion model is
produced from an output from the extractor. The system, where skill
assessment and diagnostics are performed on the motion model to
produce a skill model. The system, where the skill model further
comprises reference skill data.
[0865] The system, where session data is provided to the extractor
and the motion model further comprises the session data. The
system, where the augmentation loop comprises a movement process, a
cueing system, and a feedback loop between the movement process and
cueing system. The system, where an instruction module is
configured to receive a set of target skills from the user. The
system, where the instruction module processes the target skills
and provides the processed target skills to the training loop.
[0866] The system, where the cueing system comprises a cue
processor configured to translate movement data into a cue signal.
The system, where the cue processor implements a finite state
estimator comprising an approximation of a movement model of the
user. The system, where the cue processor implements a cueing law
calculator and the calculator operates on the finite state estimate
and the collected data to calculate if a cue will be delivered. The
system, where the cueing law calculator determines what the cue
should communicate. The system, where a feedback synthesis model
determines operation of the cueing law calculator.
[0867] The system, where the cueing system comprises a cue
generator configured to translate cue signals into physical
stimuli. The system, where the cue generator translates the cue
signal into a feedback stimulus generated by a transducer. The
system, where the feedback stimulus is selected from audio, visual,
haptic, and symbolic. The system, where the cueing system operates
in real time to provide feedback to the user during the movement
performance. The system, where the augmentation loop provides
feedback to a user that mimics human information processing
hierarchy. The system, where the feedback comprises one or more of
an instruction, a notification, a feedback cue, and a feedback cue
signal.
[0868] The system, where the instruction is generated from at least
one of a motion model, a skill model, and a diagnostic assessment.
The system, where the assessment loop comprises an extractor
configured to extract motion elements from a target motion of the
movement performance and the motion model is produced from an
output from the extractor. The system, where the skill model is
produced from assessing the motion model. The system, where the
diagnostic assessment comprises identifying deficiencies in the
movement performance of the user. The system, where the instruction
provides information about a training element and an associated
training goal. The system, where the instruction organizes a
training process. The system, where the instruction synthesizes one
or more cueing laws that govern the augmentation loop.
[0869] The system, where the instruction is generated during a
training session at an interval or after the session. The system,
where the interval is upon completion of a training set. The
system, where the instruction is presented verbally, symbolically,
or graphically. The system, where the cue is provided in real time
to the user. The system, where the cue targets specific movement
characteristics to directly impact movement outcome or performance.
The system, where the cue comprises a discrete audible, tactile, or
visual signal.
[0870] The system, where the feedback cue signal is provided in
real time to the user. The system, where the feedback cue signal
guides a user's movement and enhances a movement feature. The
system, where the feedback cue signal comprises a continuous or
semi-continuous audible, tactile, or visual signal or stimulation
of the user's muscles or nerves. The system, where the notification
provides information about a user's progress towards a training
goal. The system, where the notification is presented verbally,
symbolically, or graphically.
[0871] The system, where the feedback further comprises an activity
interaction provided by an apparatus. The system, where the
apparatus comprises a ball machine or an assistive robotic device.
The system, where the skill assessment loop is further configured
to update information about the user's skills. The system, where
information about the user's skills includes a motion model and a
skill model. The system, where information about the user's skills
includes a diagnostic tool for identifying deficiencies in a
movement technique. The system, where the identified deficiencies
are synthesized into training goals. The system, where the training
loop is managed by a training agent, and the training agent is
configured to identify training elements that can be activated as
training goals. The system, wherein the training agent suggests
training goals for a user and manages a user's training
schedule.
[0872] This application is described with respect to certain
embodiments. Equivalents can be substituted and changes can be made
to adapt these systems and methods to other problems and
applications, without departing from the scope of the invention as
defined by the claims.
* * * * *