U.S. patent application number 15/249122 was filed with the patent office on 2017-03-02 for automated motion of interest recognition, detection and self-learning.
The applicant listed for this patent is Focus Ventures, Inc.. Invention is credited to James Cavan Canavan, Shuo Feng, Grant Hughes, Steven Merel.
Application Number | 20170055918 15/249122 |
Document ID | / |
Family ID | 58096175 |
Filed Date | 2017-03-02 |
United States Patent
Application |
20170055918 |
Kind Code |
A1 |
Hughes; Grant ; et
al. |
March 2, 2017 |
AUTOMATED MOTION OF INTEREST RECOGNITION, DETECTION AND
SELF-LEARNING
Abstract
A motion monitoring system can detect and identify motions of
interest in motion data. The motion monitoring system can identify
instances of the motion of interest in the motion data and can
train a machine-learned model for identifying or characterizing
motions of interest in motion data based on extracted features from
the motions of interest. The motion monitoring system can also
monitor motion data from a user and can help the user alter the
performance of the motions of interest. For example, the motion
monitoring system can generate an improvement strategy that helps
the user improve their performance of the motion of interest. The
motion monitoring system's altering of a user's performance of a
motion of interest can be applied to physical therapy, gunfire
detection, workplace motion improvement, sports, and driving.
Inventors: |
Hughes; Grant; (Los Angeles,
CA) ; Canavan; James Cavan; (Lexington, KY) ;
Feng; Shuo; (Fullerton, CA) ; Merel; Steven;
(Santa Monica, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Focus Ventures, Inc. |
Santa Monica |
CA |
US |
|
|
Family ID: |
58096175 |
Appl. No.: |
15/249122 |
Filed: |
August 26, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62211513 |
Aug 28, 2015 |
|
|
|
62213003 |
Sep 1, 2015 |
|
|
|
62293658 |
Feb 10, 2016 |
|
|
|
62338016 |
May 18, 2016 |
|
|
|
62355528 |
Jun 28, 2016 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A61B 5/7225 20130101;
A63B 24/0003 20130101; A61B 5/7278 20130101; H04L 43/028 20130101;
A61B 5/112 20130101; A61B 5/725 20130101; A61B 5/7282 20130101;
A61B 5/7239 20130101; A61B 5/7267 20130101; A63B 24/0062 20130101;
A61B 5/1123 20130101 |
International
Class: |
A61B 5/00 20060101
A61B005/00; A61B 5/11 20060101 A61B005/11 |
Claims
1. A method comprising: receiving motion data comprising a
plurality of sensor signals describing a plurality of instances of
a motion of interest and a count of the instances of the motion of
interest; generating a set of time labels for the plurality of
sensor signals based on the one or more time labeling parameters,
each time label identifying a portion of the sensor signals that
can represent an instance of the motion of interest of the
plurality of instances of the motion of interest; extracting
features of the motion of interest based on the set of time labels;
and detecting a type of the motion of interest based on the
extracted features of the instances of the motion of interest.
Description
CROSS REFERENCE TO RELATED APPLICATION
[0001] This application claims the benefit of U.S. Provisional
Application No. 62/211,513, filed on Aug. 28, 2015, U.S.
Provisional Application No. 62/213,003, filed Sep. 1, 2015, U.S.
Provisional Application No. 62/293,658, filed Feb. 10, 2016, U.S.
Provisional Application No. 62/338,016, filed May 18, 2016, and
U.S. Provisional Application No. 62/355,528, filed Jun. 28, 2016,
each of which is herein incorporated by reference in its
entirety.
[0002] This application also is related to and incorporates by
reference in its entirety U.S. patent application Ser. No. ______,
filed 26 Aug. 2016, titled "System and Method for Automatically
Time Labelling Repetitive Data".
BACKGROUND
[0003] Field of the Invention
[0004] This disclosure relates generally to movement identification
and more specifically to identifying motions of interest in signal
data.
[0005] Description of Art
[0006] Sensors, such as accelerometers, can be used to generate
signals of physical motion, and these signals can describe the
physical motion of a sensor coupled to a person. For example, an
accelerometer attached to the wrist of a user can detect the
acceleration and motion of the user's arm. Conventionally, it is
difficult to identify a motion of interest (MOI) in sensor signals.
A motion of interest is one or more particular actions performed by
a user. For example, a motion of interest can be a single
repetition of a set of exercises (e.g., a single push-up or a
single bicep curl), an action for work (e.g., lifting a box or
reaching for a document), or an action associated with the usage of
a firearm (e.g., drawing or firing the weapon). Motions of interest
can be difficult to detect because the sensor signals can often
contain noise that masks the portions of the signal that represent
the motion of interest.
[0007] Some systems attempt to identify motions of interest in
sensor signals using machine-learned models. However, these models
often require humans to hand label sensor signals, which is time
and resource intensive. Thus, these models either require
significant resources to develop and continue training, or are
trained on small training data sets, which reduces the accuracy of
the models.
SUMMARY
[0008] A motion monitoring system can detect and identify motions
of interest in motion data. The motion monitoring system can
receive motion data from movement measurement devices. A movement
measurement device can be a wearable device worn by a user that can
transmit motion data describing the user's movement to the motion
monitoring system. The motion data can include sensor signals from
one or more sensors in the movement measurement device. The motion
data can also include a count of the instances of the motion of
interest and a motion of interest type.
[0009] The motion monitoring system can determine time labeling
parameters that are used to time label the instances of the motion
of interest in the sensor signals. Example time labeling parameters
can include a motion of interest duration estimate, a sensor signal
combination, and a motion of interest center. The motion monitoring
system can generate time labels that define the start and end of
candidate for motions of interest within the sensor signals. Each
time label can include a starting time stamp and an ending time
stamp. The motion monitoring system can generate the time labels
based on the time labeling parameters. For example, the motion of
interest duration estimate may be used to select a cutoff frequency
for a low pass filter that filters out high frequency noise, the
determined signal combination may be used to emphasize the motion
of interest within the sensor signals, and the motion of interest
center may be used to identify peaks or valleys associated with the
motion of interest.
[0010] The motion monitoring system can identify time labels that
represent true instances of motions of interest and time labels
that are just noise. The motion monitoring system may use a
reference signal and a count of instances of the motion of interest
in the motion data to identify the time labels that represent
motions of interest. The motion monitoring system can extract
features describing the motion of interest from the motion data.
Example features of a motion of interest include the speed, angle,
consistency, fatigue, high frequency resonance from muscle
twitching, range of motion, force, work performed, device
orientation, body part orientation, and form. The motion monitoring
system can then train a machine-learned model for identifying or
characterizing motions of interest in motion data based on the
extracted features.
[0011] The motion monitoring system can additionally help a user
alter the performance of motions of interest. For example, the
motion monitoring system can help a user improve the performance of
a motion of interest, or may encourage/discourage the performance
of the motion of interest. The motion monitoring system can
identify instances of motions of interest in initial motion data
and can generate a user motion model based on the identified
motions of interest. A user motion model describes the motive
abilities and behavior of a user. The user motion model can be
compared with a reference model to determine an improvement
strategy for the user.
[0012] The motion monitoring system can monitor additional motion
data that can describe motions of interest performed by the user
recommended by the improvement strategy. The motion monitoring
system can identify instances of motions of interest in the
additional motion data and can update the user motion model based
on the identified instances of the motion of interest. The motion
monitoring system can compare the updated user motion model to a
reference model and, based on whether the user has met success
criteria, can update the reference model or can adjust the
improvement strategy. The motion monitoring system's altering of a
user's performance of a motion of interest can be applied to
physical therapy, gunfire detection, workplace motion improvement,
sports, and driving, for example.
[0013] In some embodiments, the motion monitoring system
automatically generates motion of interest models based on motion
data without hand labeling by a human, unlike conventional
techniques. Thus, the motion data used to train motion of interest
models can include data from multiple users performing the same
motion of interest in order to quickly obtain a large training data
set. In addition, the motion monitoring system can identify
portions of the motion data that are noise and can generate the
motion of interest models based on the portions of the motion data
that actually represent the instances of the motion of interest.
Thus, the motion of interest models generated by the motion
monitoring system can be more accurate than conventional methods
can provide.
BRIEF DESCRIPTION OF DRAWINGS
[0014] The figures depict various embodiments of the present
invention for purposes of illustration only. One skilled in the art
will readily recognize from the following discussion that
alternative embodiments of the structures and methods illustrated
herein may be employed without departing from the principles of the
invention described herein.
[0015] Figure (FIG. 1 illustrates an example system environment for
a motion monitoring system, in accordance with some
embodiments.
[0016] FIG. 2 is a flowchart illustrating a method for detecting
and identifying motions of interest in motion data, in accordance
with some embodiments.
[0017] FIG. 3 shows sensor signals from a three dimensional
accelerometer, including an x-axis acceleration signal, a y-axis
acceleration signal, and a z-axis acceleration signal, in
accordance with some embodiments
[0018] FIG. 4A illustrates local maxima detected in sensor signals
and time intervals between the detected local maxima, in accordance
with some embodiments.
[0019] FIG. 4B illustrates local minima detected in sensor signals
and time intervals between the detected local minima, in accordance
with some embodiments.
[0020] FIG. 5 illustrates a reference signal used to determine the
motion of interest center, in accordance with some embodiments.
[0021] FIG. 6 illustrates time labels for sensor signals, in
accordance with some embodiments.
[0022] FIG. 7 illustrates time labels that have been identified as
motions of interest and time labels that have been identified as
noise, in accordance with some embodiments.
[0023] FIG. 8 is a flowchart illustrating a method for improving
the performance of a motion of interest, in accordance with some
embodiments.
[0024] FIG. 9 illustrates time labels that have been generated for
a reference signal, in accordance with some embodiments.
[0025] FIG. 10 illustrates time labels on the reference signal that
have been identified as motions of interest and time labels that
have been identified as noise, in accordance with some
embodiments.
[0026] FIG. 11 illustrates a sliding window that is used to extract
features from portions of the sensor signals that are identified as
representing a motion of interest and ignoring portions of the
sensor signals that are identified as noise, in accordance with
some embodiments.
[0027] FIG. 12 is a flowchart for monitoring patient improvement
from an injury and adjusting an recovery strategy based on the
patient's improvement, in accordance with some embodiments.
DETAILED DESCRIPTION
[0028] The features and advantages described in the specification
are not all inclusive and, in particular, many additional features
and advantages will be apparent to one of ordinary skill in the art
in view of the drawings and specification. Moreover, it should be
noted that the language used in the specification has been
principally selected for readability and instructional purposes,
and may not have been selected to delineate or circumscribe the
inventive subject matter.
Example System Environment and Architecture
[0029] FIG. 1 illustrates an example system environment for a
motion monitoring system, in accordance with some embodiments. FIG.
1 includes a movement measurement device 100, a network 110, and a
motion monitoring system 120. Alternate embodiments can include
additional, fewer, or different components than those illustrated
in FIG. 1 and the functionality of the components may be divided up
differently from the description below. For example, while only one
movement measurement device 100 is illustrated in FIG. 1, alternate
embodiments can include a plurality of movement measurement devices
100 in communication with the motion monitoring system 120 through
the network 110.
[0030] The movement measurement device 100 collects motion data
describing a user's movement and transmits the motion data to the
motion monitoring system 120. The movement measurement device 100
can be a wearable device, such as a smart watch, a fitness
bracelet/anklet, or a headset. In some embodiments, the movement
measurement device 100 communicates with a personal computing
device (e.g., a smart phone, a tablet, a personal computer) to
transmit the motion data to the motion monitoring system 120. The
movement measurement device 100 includes one or more sensors to
generate the motion data, such as an inertial measurement unit, an
accelerometer, a gyroscope, a GPS module, a magnetometer, an
electromyograph, or an electronic compass. The one or more sensors
of the movement measurement device 100 generate one or more sensor
signals to be included in the motion data. For example, an
accelerometer in the motion measurement device 100 may generate a
sensor signal that describes the acceleration of the movement
measurement device 100 over time. In some embodiments, the movement
measurement device 100 processes the motion data before
transmitting the motion data to the motion monitoring system 120.
For example, the movement measurement device 100 may encrypt,
compress, or reformat the motion data. In some embodiments,
multiple movement measurement devices 100 can be used
simultaneously to capture motion data. These movement measurement
devices 100 may communicate with the motion monitoring system or
with each other through the network 110.
[0031] The movement measurement device 100 can communicate with the
motion monitoring system 120 via the network 110, which may
comprise any combination of local area and wide area networks
employing wired or wireless communication links. In one embodiment,
the network 110 uses standard communications technologies and
protocols. For example, the network 110 includes communication
links using technologies such as Ethernet, 802.11, worldwide
interoperability for microwave access (WiMAX), 3G, 4G, code
division multiple access (CDMA), digital subscriber line (DSL),
Bluetooth, etc. Examples of networking protocols used for
communicating via the network 110 include multiprotocol label
switching (MPLS), transmission control protocol/Internet protocol
(TCP/IP), hypertext transport protocol (HTTP), simple mail transfer
protocol (SMTP), and file transfer protocol (FTP). Data exchanged
over the network 110 may be represented using any format, such as
hypertext markup language (HTML) or extensible markup language
(XML). In some embodiments, all or some of the communication links
of the network 110 may be encrypted.
[0032] The motion monitoring system 120 identifies and analyzes
motions of interest in motion data received from the movement
measurement device 100. The identification and analysis information
about a motion of interest may be provided by the motion monitoring
system 120 to the user through the movement measurement device 100
or through a personal computing device, such as a computer, a
laptop, a tablet, or a phone. The motion monitoring system 120 can
identify where a motion of interest is represented in motion data
from the movement measurement device and can identify the motion of
interest based on features of the motion data. Additionally, the
motion monitoring system 120 provides an analysis of the identified
motions of interest to a user. For example, the motion monitoring
system 120 may compare features of exercise motions of interest to
stored reference models of the exercise to determine whether the
user is performing the exercises correctly and whether the user is
improving in physical therapy at an appropriate rate. Methods that
can be used by the motion monitoring system 120 are described
below. The motion monitoring system 120 may store motion data from
the motion measurement device 100. The stored motion data may
include motion data collected from the user of the motion
measurement device 100, other users of the motion monitoring system
120, and motion data from third party systems. In some embodiments,
the motion monitoring system 120 stores historical motion data.
Detecting and Identifying Motions of Interest
[0033] FIG. 2 is a flowchart illustrating a method for detecting
and identifying motions of interest in motion data, in accordance
with some embodiments. Alternate embodiments may include more,
fewer, or different steps from the steps illustrated in FIG. 2, and
the steps may be performed in a different order from the order
described herein.
[0034] The motion monitoring system 120 receives 200 motion data
from a movement measurement device 100 that includes one or more
sensor signals from one or more sensors in the motion measurement
device 100. The sensor signals describe motions of interest
performed by the user, such as exercise repetitions in an exercise
regimen. FIG. 3 illustrates example sensor signals from the motion
measurement device 100, in accordance with some embodiments. FIG. 3
shows sensor signals from a three dimensional accelerometer,
including an x-axis acceleration signal 300A, a y-axis acceleration
signal 300B, and a z-axis acceleration signal 300C, in accordance
with some embodiments. In addition to the sensor signals, the
motion data can include a motion of interest count, which describes
the number of motions of interest that are included in the motion
data. In some embodiments, the motion of interest count is input by
the user to specify the number of motions of interest that are
represented by the motion data. In other embodiments, the motion of
interest count is pre-defined by the motion monitoring system 120
and the motion data must include the motion of interest count of
instances of the motion of interest. The motion data can also
include a name or other identifier of the type of the motion of
interest. In some embodiments, the motion data includes motion data
from multiple movement measurement devices 100 associated with one
or more users.
[0035] The motion monitoring system 120 determines 210 time
labeling parameters that are used to time label the motions of
interest in the sensor signals. In the embodiment illustrated in
FIG. 2, three time labeling parameters are estimated: a MOI
duration estimate, a sensor signal combination, and an MOI center,
which are described in further detail below. In other embodiments,
more, fewer, or different time labeling parameters can be
determined by the motion monitoring system 120.
[0036] The motion monitoring system 120 determines 220 an MOI
duration estimate by determining a raw estimate of the MOI
duration. A raw estimate is obtained by dividing the total length
of time covered by the sensor signals by the MOI count. In other
words, the raw estimate is an estimate that assumes that the
motions of interest are evenly distributed throughout the sensor
signals, and provides an upper limit of the motion of interest
duration. The raw period can be used as the window length of a
local extremum detector. The local extremum detector may be used to
detect local extrema in the one or more sensor signals and may
detect local minima, local maxima, or both. In some embodiments,
the local extremum detector detects a local extremum if it is a
local extremum of its type within a neighborhood with a width that
is substantially equal to the raw estimate. For example, a local
maximum may only be detected as a local maximum if it is the local
maximum within the neighborhood of the local maximum. In some
embodiments, the local extremum detector detects a local extremum
if it is a local extremum of its type within a symmetric
neighborhood around the local extremum.
[0037] The motion monitoring system 120 applies the local extremum
detector to each of the one or more sensor signals in the motion
data to detect local extrema within each of the sensor signals. The
motion monitoring system 120 can determine time intervals between
adjacent local extrema in the one or more sensor signals. If the
local extrema include local minima and local maxima, the motion
monitoring system 120 may determine time intervals between local
extrema of the same type (i.e., time intervals between local maxima
and time intervals between local minima) and can determine the MOI
duration estimate based on lengths of the time intervals. In some
embodiments, the MOI duration estimate is the median of the lengths
of the determined time intervals.
[0038] FIG. 4A illustrates local maxima 400 detected in sensor
signals 300 and time intervals 410 between the detected local
maxima 400, in accordance with some embodiments. FIG. 4B
illustrates local minima 420 detected in sensor signals 300 and
time intervals 430 between the detected local minima 420, in
accordance with some embodiments. To determine a MOI duration
estimate, the motion monitoring system can select the time interval
with the median length as the MOI duration estimate.
[0039] The motion monitoring system 120 determines 230 a signal
combination that emphasizes the motion of interest. The signal
combination can be a combination of one or more sensor signals that
accentuates the motion of interest in the sensor signals. For
example, for sensor signals describing push-ups, the combination
signal may accentuate movement or acceleration along an axis
perpendicular to the ground more than an axis parallel to the
ground. In some embodiments, the motion monitoring system 120
selects a combination from a set of combinations, and selects the
combination that provides a combination signal with the greatest
variance. Possible combinations, using accelerometer sensor signals
of FIG. 3 as an example, include:
.+-. A x .+-. A y .+-. A z .+-. ( A x + A y ) .+-. ( A x + A z )
.+-. ( A y + A z ) .+-. ( A x - A y ) .+-. ( A x - A z ) .+-. ( A y
- A z ) .+-. A x 2 + A y 2 + A z 2 ##EQU00001##
[0040] where A.sub.x, A.sub.y and A.sub.z denote the raw data from
accelerometer the x-axis acceleration signal 300A, the y-axis
acceleration signal 300B, and the z-axis acceleration signal 300C
respectively. Similar combinations apply to other sensor signals.
The potential combinations are not limited to the lists above. For
example, Euclidean sum, Eigenvector decomposition, and principle
component analysis can also be used. In some embodiments, the
combination of sensor signals may be selected based on the
computational intensity of the combination. For example, if a
complex combination allows for better time labeling of repetitions
but at a great computational cost, the motion monitoring system 120
may select a combination that provides for worse labeling accuracy,
but that is simpler to compute.
[0041] The motion monitoring system 120 determines 240 a center of
the motion of interest in a reference signal. FIG. 5 illustrates a
reference signal 500 used to determine the motion of interest
center, in accordance with some embodiments. The motion of interest
can be considered to be one of a series of repetitive motions that
transfer between two states A and B, such that a typical sequence
is A, B, A, B, . . . . Thus, a single instance of a motion of
interest in this case may be either A to B to A or B to A to B. The
starting states of these two cases are A and B, respectively, and
the motion of interest centers of these two cases are B and A,
respectively. In some embodiments, the repetitive motions that
transfer between the two states A and B periodically and an
instance of a motion of interest can be a portion of a single cycle
of the periodic transitions between the two states. For example, if
the motion of interest is a push-up, the two states are the
straight arm high body position and the bent arm low body position,
and the transitions between the two states can occur regularly
within the motion data as the user performs a set of push-ups. In
some embodiments, the transitions between the two states that
correspond to an instance of a motion of interest can occur
irregularly within the motion data, and an instance of the motion
of interest is a transition from one state to the other state and
back again to the first state. For example, if the motion of
interest is the user lifting a box, the two states can be when the
user is standing and when the user is squatting, and an instance of
the motion of interest can be when the user goes from standing to
squatting and back to standing.
[0042] The reference signal is a single signal that is used to
represent the motion data. In some embodiments, the reference
signal is generated based on the combination of the one or more
sensor signals determined by the motion monitoring system 120. The
reference signal may also be filtered using a low-pass filter with
a cutoff frequency that is selected based on the MOI duration
estimate. In some embodiments, the reference signal is the labeling
signal described in U.S. patent application Ser. No. ______, titled
"System and Method for Automatically Time Labelling Repetitive
Data".
[0043] To determine the motion of interest center, a portion at the
beginning of the reference signal may be considered the starting
state of the motion of interest. In some embodiments, the motion
monitoring system 120 can require that the beginning portion of the
reference signal be the starting state for a designated amount of
time. Local maxima 505 and local minima 510 can be detected over
the reference signal, and the median local maximum value 515 and
the median local minimum value 520 can be determined. The mean
value 525 of the reference signal can also be determined. The
motion monitoring system 120 calculates the mean-maximum difference
530 and the mean-minimum difference 535. The mean-maximum
difference 530 is the difference between the median local maximum
value 515 and the reference signal mean value 525. The mean-minimum
difference 535 is the difference between the median local minimum
value 520 and the reference signal mean value 525. The motion
monitoring system 120 selects, as the motion of interest center,
the local extremum whose corresponding difference is greater than
the other difference. For example, if the mean-minimum difference
535 is greater than the mean-maximum difference 530, then the
reference signal mean value 525 is closer to the median local
maximum value 515 than the median local minimum value 520. Thus,
the local minima 510 represent the motion of interest centers and
the local maxima 505 represent the starting state.
[0044] The motion monitoring system 120 generates 250 time labels
that define the start and end of candidates for motions of
interest. Each time label can include a starting time stamp and an
ending time stamp, which are the times within the one or more
sensor signals the candidate motion of interest starts and ends,
respectively. The motion monitoring system 120 can generate the
time labels based on the time labeling parameters. For example, the
MOI duration estimate may be used to select a cutoff frequency for
a low pass filter that filters out high frequency noise. The
determined signal combination may be used to emphasize the motion
of interest within the sensor signals, and the motion of interest
center may be used to identify peaks or valleys associated with the
motion of interest. The time labeling of sensor signals is
described in further detail in U.S. patent application Ser. No.
______ titled "System and Method for Automatically Time Labelling
Repetitive Data". FIG. 6 illustrates time labels 600 for sensor
signals, in accordance with some embodiments.
[0045] Some of the generated time labels may label noise in the
sensor signals that appears to be, but are not actually, motions of
interest. Conventional techniques typically collect motion data
when the subject is performing an assigned motion of interest so
that the motion data is labeled. Features are generated for machine
learning algorithms of a machine learning tool. The machine
learning algorithms train a classification model. The trained model
is used to classify feature values from sensor data collected from
an unknown activity/exercise. Additionally, conventional techniques
generate a set of feature values from a fixed duration portion of
the raw sensor data using a sliding window. The sliding window goes
through the entire motion data from the beginning to the end with a
certain step size. In conventional methods, all of the features
obtained in this step are considered to describe the motion of
interest, indiscriminately across all of the windows. Even though
some windows within the motion data set may be noise such as
walking or other activity instead of the motion of interest.
Therefore, conventional methods result in inaccurate training data
and therefore inaccurate models.
[0046] Instead, the motion monitoring system 120 identifies 260
time labels that are actually associated with motions of interest
rather than noise. The motion monitoring system 120 may use the
reference signal described above and the motion of interest count
to identify the motion of interest time labels. For example, the
motion monitoring system 120 may identify the local extrema of the
reference signal within the generated time labels, and then may
select, as true motions of interest, the set of the N time labels
with the most extreme local extrema, where N is the motion of
interest count. For example, if the motion of interest count is 10
and local minima of the reference signal represent the motions of
interest, the motion monitoring system 120 may identify the time
labels with the 10 lowest local minima as the true motions of
interest and the remaining time labels as noise. The motion
monitoring system 120 may also use machine-learned classifiers to
determine which time labels represent motions of interest based on
features of the portions of the one or more sensor signals within
the time labels. Examples of machine-learning algorithms that could
be used by the motion monitoring system 120 includes random forest,
k-means, linear regression, logistic regression, decision tree,
support vector machine, Naive Bayes, or k-nearest neighbors.
[0047] FIG. 9 illustrates time labels 900 that have been generated
for a reference signal 500, in accordance with some embodiments.
FIG. 10 illustrates time labels on the reference signal 500 that
have been identified as motions of interest 1000 and time labels
that have been identified as noise 1010, in accordance with some
embodiments. In the embodiments illustrated in FIG. 10, the time
labels containing the 10 lowest local minima have been identified
as instances of a motion of interest.
[0048] FIG. 7 illustrates time labels on the sensor signals that
have been identified as motions of interest 700 and time labels
that have been identified as noise 710, in accordance with some
embodiments.
[0049] The motion monitoring system 120 extracts 270 features
describing the motions of interest from the one or more sensor
signals. Example features of a motion of interest include the
speed, angle, consistency, fatigue, high frequency resonance from
muscle twitching, range of motion, force, work performed, device
orientation, body part orientation, and form. To extract the
features from the one or more sensor signals, the motion monitoring
system 120 can perform an angle analysis, a time analysis, or a
form analysis of the sensor signals. The angle analysis can
determine the angle between the direction of an axis of a sensor in
the movement measurement device 100 and the direction of gravity.
The angle analysis can also determine the change in that angle. The
time analysis can analyze the timing of features of an instance of
a motion of interest (e.g., the start and end timestamp of the
motion of interest) and can identify a pace or frequency of the
motions of interest, for example. A form analysis analyzes the
features of two instances of a motion of interest and can determine
the similarity or differences between those two instances. In some
embodiments, the motion monitoring system 120 only extracts
features from time-labeled portions of the sensor signals that have
been identified as representing motions of interest. For example,
the motion monitoring system 120 may use a sliding window to
extract features from the sensor signals, and may ignore portions
of the sensor signals that are time labeled and have not been
identified as representing a motion of interest. In this way the
content within the motion data is further refined when compared to
conventional techniques in order to discriminate motion data of the
true motion of interest from the noise. The performance in terms of
classification accuracy is improved since the features can be
extracted from portions of the motion data that is more
accurate.
[0050] FIG. 11 illustrates a sliding window 1100 that is used to
extract features from portions of the sensor signals that are
identified as representing a motion of interest 1110 and ignoring
portions of the sensor signals that are identified as noise 1120,
in accordance with some embodiments.
[0051] The motion monitoring system 120 trains 280 a
machine-learned MOI model for identifying the motion of interest
represented in the one or more sensor signals. The MOI model may be
a classifier model, such as a random forest, a support vector
machine, or a neural network, and can be trained based on the
extracted features from the identified MOI. The MOI model can be
used to identify instances of a motion of interest of a particular
type. For example, a push-up MOI model can be used to identify
motions of interest within the one or more sensor signals that
represent push-ups. Additionally, the MOI model can be used to
describe the features of a motion of interest. In some embodiments,
the motion monitoring system 120 trains an existing machine-learned
MOI model based on the extracted features to improve the accuracy
of the existing MOI model. In some embodiments, this step is
optional.
[0052] The motion monitoring system 120 detects 290 instances of
the motion of interest using the MOI model. The motion monitoring
system 120 can extract features from the additional motion data
and, based on the features, can use the MOI model to identify the
motion of interest in the additional motion data. For example, if
the motion monitoring system 120 receives motion data that
describes a user's performance of push-ups, the motion monitoring
system 120 can detect the push-ups in the motion data. Detecting
the instances of the motion of interest in the motion data can
include identifying portions of the motion data that represent the
motion of interest, determining a count of instances of the motion
of interest, and determining features of the instances of the
motion of interest.
Training Customized MOI Models
[0053] A user in communication with the motion monitoring system
120 may be able to use the methods described here to define their
own motion of interest and build a customized MOI model for the
particular MOI. To do so, the motion monitoring system 120 may make
application programming interfaces (API's) available to users
(e.g., software developers) to allow a user to generate a custom
MOI model.
[0054] To generate a MOI model for a custom motion of interest, the
motion monitoring system 120 can receive motion data describing the
motions of interest. The motion monitoring system 120 can also
receive a name for the type of the motion of interest and a count
of the instances of motion of interest in the motion data. In some
embodiments, the motion monitoring system 120 requires that a
particular number of instances of the motion of interest be
provided in the motion data. For example, the motion monitoring
system 120 may require that the user provide at least 10 instances
of the motion of interest in the motion data. The motion monitoring
system 120 can store the motion data and tag the motion data with
the motion of interest name and, in some embodiments, a user
identifier, such as a developer API key. The user identifier is
unique to the user, so custom motions of interest generated by
different users can be distinguished even if the different users
choose the same motion of interest name.
[0055] An MOI model can be generated based on the provided motion
data in accordance with the method described with FIG. 2. The user
can then perform a cross-validation test to estimate the accuracy
of the MOI model. In some embodiments, cross-validating the MOI
model includes partitioning the received motion data into a set of
groups, each containing an approximately equal portion of the
motion data. A MOI model is generated using the motion data from a
subset of the groups, and the motion data in the remaining groups
is used to test the resulting model by using the algorithm to
estimate the number of instances of the motion of interest in the
motion data in those groups. An accuracy value is obtained by
comparing the estimated number of MOI instances to the number
reported by the user when the motion data was originally received.
In some embodiments, this accuracy estimate is used as the accuracy
of the MOI model. However, in other embodiments, this process is
may be repeated on multiple different subsets of the set of groups,
and the accuracy values obtained are averaged together to obtain an
overall accuracy estimate. In some embodiments, if a subset of the
set of groups results in an accuracy value that is significantly
lower than accuracy values for other subsets of the set of groups,
the subset may be identified as a noisy subset, and the motion data
in the subset may be disregarded in training the MOI model and in
determining the accuracy of the MOI model.
[0056] In some embodiments, the MOI model is generated and tested
by the motion monitoring system 120. In other embodiments, the
motion monitoring system 120 provides the motion data to the user
to generate or test the MOI model locally.
Improving Motion of Interest Performance
[0057] FIG. 8 is a flowchart illustrating a method for improving
the performance of a motion of interest, in accordance with some
embodiments. Alternate embodiments may include more, fewer, or
different steps from the steps illustrated in FIG. 8, and the steps
may be performed in a different order from the order described
herein.
[0058] The motion monitoring system 120 receives 800 initial motion
data from a user's movement measurement device 100. The initial
motion data describes motions of interest performed by a user. In
some embodiments, the motion data is specifically generated for the
purpose of training an MOI model. For example, the user may take
additional steps to ensure that the motion data is less noisy or
may perform a specific number or type of motion of interest.
Additionally, the motion data may only contain motion data
generated by a single user, or may include motion data from
multiple users, where the motion data from each user may be
designated with a user identifier.
[0059] The motion monitoring system 120 identifies 805 the
instances of the motion of interest represented by the motion data
in accordance with the methods described by FIG. 2 and generates
810 a user motion model based on the identified motions. The user
motion model is a model of the user's motive ability and behavior.
The user motion model can include one or more MOI models that are
trained based on a single user's motion data and thereby describe a
motion of interest as performed by the user. The user motion model
can be used to identify motions of interest performed by the user
within motion data and can be used to describe features of motions
of interest performed by the user. For example, the user motion
model may describe the speed at which the user typically performs a
motion of interest. In some embodiments, the user motion model can
be used to perform the angle analysis, time analysis, or form
analysis to describe the features of the user's performance of a
motion of interest. The user motion model may use MOI models of a
user's performance of motions of interest to describe features of
the user's motive abilities. For example, the user motion model may
use MOI models to determine the user's strength or range of motion.
In some embodiments, the user motion model can describe the motive
abilities of different parts of a user's body. For example, the
user motion model may describe the strength or range of motion of
the user's elbow, shoulder, or knee.
[0060] The motion monitoring system 120 compares 815 the user
motion model to a reference model. The reference model can describe
the motive abilities and behaviors of multiple users. The reference
model may include MOI models generated based on motion data from
multiple users and can be used to identify or characterize motions
of interest performed by other users. In some embodiments, the
reference model is generated with motion data from users that are
similar to the user. For example, the reference model may be
generated based on motion data from users of the same age, sex,
physiology, movement behavior, or personality as the user.
Additionally, the motion monitoring system 120 may compare the user
motion model and the reference model based on historical behavior
of the users. For example, the motion monitoring system 120 may
compare the user's change in performance of a motion of interest
with the change of other users' performances of the motion of
interest. In some embodiments, the reference model can also be
generated from motion data such that the reference model can
characterize how a motion of interest can be performed ideally.
[0061] By comparing the user motion model to the reference model,
the motion monitoring system 120 can determine differences between
how the user performs a motion of interest to how the motion of
interest would be performed ideally. In some embodiments, the
motion monitoring system 120 determines the differences by
comparing features of the motion of interest performed by the user
to features of the ideally performed motion of interest described
by the reference model. For example, the motion monitoring system
120 can determine the differences in form, frequency, amplitude,
consistency, speed, acceleration, or velocity of motions of
interest.
[0062] The motion monitoring system 120 generates 820 an
improvement strategy for the user based on the comparison of the
user motion model to the reference model. The improvement strategy
can describe motions of interest the user can perform to improve
their performance of a motion of interest. For example, if the user
is not able to perform push-ups quickly or is not able to hold the
push-up in the lower body position for a long enough time, the
improvement strategy may recommend that the user perform more
push-ups or may recommend that the user perform other motions of
interest that can help improve the user's push-up performance
(e.g., bicep curls or bench presses). In some embodiments, the
improvement strategy includes recommendations for how the user can
adjust or practice the motion of interest. For example, if the
motion of interest is a push-up, the improvement strategy may
recommend that the user hold the lower body position for a longer
period of time or may designate a recommended number of push-ups to
perform. Similarly, if a user is bending their back while
performing a push-up, the improvement strategy may recommend that
the user adjust their performance of a push-up to straighten their
back.
[0063] The improvement strategy may also automatically change the
recommendations provided to the user over time based on a predicted
improvement by the user. For example, the improvement strategy may
increase the push-up sets or number of push-ups per set over time
as the user improves their push-up performance. In some
embodiments, the improvement strategy may include a recommendation
for the user to use additional or improved movement measurement
devices 100 to gather additional motion data to be used to
determine the improvement strategy. For example, if the user is
performing push-ups and currently is only wearing a movement
measurement device on their wrist, the improvement strategy may
recommend that the user wear a movement measurement device on their
chest or waist.
[0064] In some embodiments, the motion monitoring system 120
automatically generates the improvement strategy. In other
embodiments, the motion monitoring system 120 sends information
about the user's performance of the motion of interest (e.g., the
user motion model, the reference model, the motion data, the
identified motions of interest, features about the motion of
interest) to another user, and the other user provides the motion
monitoring system 120 with the improvement strategy. For example,
the motion monitoring system 120 may provide a personal trainer
with the information about the user's push-up performance and the
personal trainer may provide the motion monitoring system 120 with
an improvement strategy for the user.
[0065] The motion monitoring system 120 monitors the user's
progress in implementing the improvement strategy. The motion
monitoring system 120 receives 825 additional motion data from the
movement measurement device 100. The additional motion data can
include motion data describing motions of interest recommended by
the improvement strategy, and can also include other motions of
interest. The motion monitoring system 120 identifies 830 the
instances of the motion of interest in the additional motion data
and updates 835 the user motion model based on the additional
motion data. The motion monitoring system 120 may update the user
motion model to describe the user's historic progress as well as
the user's current performance of the motion of interest. In some
embodiments, the motion monitoring system 120 updates the reference
model such that the user motion model is being compared to a
reference model that is generated based on motion data from users
similar to the user. These users similar to the user may be
identified based on the additional motion data or the updated user
motion model. For example, if the user motion model was originally
compared to a reference model based on users that improve quickly
and the additional motion data shows that the user is improving
more slowly than originally predicted, the reference model may be
adjusted to compare the updated user motion model with users that
improve more similarly to the user.
[0066] The motion monitoring system 120 compares 840 the updated
user motion model to the reference model and determines 845 if
success criteria have been met. The success criteria describe the
final motive ability the user seeks to accomplish through the
improvement strategy. The success criteria can include a threshold
value for one or more features of a motion of interest, and may be
determined based on features of an ideally performed motion of
interest. If the user's improvement meets the success criteria, the
motion monitoring system may update 850 the reference model based
on the user's motion data. The update to the reference model may
include updating the reference model to describe how the user's
performance changed over time based on the improvement
strategy.
[0067] If the user's improvement does not meet the success
criteria, the motion monitoring system 120 may adjust 855 the
improvement strategy based on the comparison of the user motion
model and the reference model. For example, if the user's
performance of the motion of interest is significantly deficient
compared to the reference model and the rate at which the user is
improving is low, the motion monitoring system 120 may adjust the
improvement strategy to recommend a different way to perform the
motion of interest or a different motion of interest to perform. In
some embodiments, the motion monitoring system 120 only adjusts the
improvement strategy if the user's improvement is significantly
below the success criteria.
[0068] While the description above uses improving a user's
performance of a motion of interest as an example, the method
described by FIG. 8 is not limited to improving a user's
performance of a motion of interest. It is possible to adjust the
methods described herein to generally alter a user's performance of
a motion of interest. For example, the method above can be adjusted
to encourage or discourage a user from performing a motion of
interest. In these examples, the motion monitoring system 120 may
monitor the additional motion data and identify if the user does
not perform a motion of interest they are supposed to perform or if
the user performs a motion of interest they are not supposed to
perform.
Example Applications of MOI Model
[0069] The following are examples of how the system and methods
herein can be applied to particular contexts. These examples are
not meant to be limiting and are not meant to be an exhaustive list
of all possible applications of the described systems and methods.
One of skill in the art can appreciate additional applications of
the systems and methods herein.
Injury Detection and Recovery
[0070] The systems and methods herein can be used to help a patient
recover from an injury through physical therapy. The motion
monitoring system 120 may instruct a patient to perform diagnostic
movements and motion data from the diagnostic movements can be used
to generate a user motion model. The user motion model can be used
to determine a recovery improvement strategy for the user, which
can include exercise motions of interest the user is recommended to
perform regularly. The motion monitoring system 120 can monitor the
patient's progress and, if the patient is not improving at a
predicted rate, the motion monitoring system 120 can adjust the
recovery improvement strategy to better the patient's improvement
and avoid further injuring the patient. The success criteria can be
set for the user to recover to some or all of their pre-injury
ability. In some embodiments, the motion monitoring system 120
generates and adjusts the user's recovery improvement strategy. In
other embodiments, a doctor or physical therapist is provided with
information about the user's motions of interest and improvement,
and provides the motion monitoring system with the recovery
improvement strategy.
[0071] Examples for the use of embodiments include activity
detection in everyday life or physical therapy such as
classification of sitting, walking, running, etc. In addition
embodiments can be used to discriminate exercise in training, such
as pushups, crunches, weight training, etc.
[0072] FIG. 12 is a flowchart illustrating a methodology that can
be used in accordance with some embodiments. In embodiments, a user
of a movement measurement device 100, such as a fitness tracking
device, wears the device or otherwise uses 1202 the device so that
motion data of the user, e.g., a patient, is automatically tracked.
This information is sent to the motion monitoring system, which can
be one or more servers. In one embodiment the motion monitoring
system 120 is remote and data is sent over a wide area network 110,
e.g., the Internet. Data transmission can be based on any
transmission technique, such as wired, wireless including
Bluetooth, WiFi, near field communication techniques, etc. A user
may be instructed to perform 1204 prescribed movements and the data
from the prescribed movements is analyzed 1206. The analysis can be
done on the movement measurement device 100, using a processor
within close proximity to the user and/or using a processor at a
remote location. In one example, a computer based algorithm
analyzes the prescribed movement data and provides information
about the analyzed data to a second person, for example, a
physician or physical therapist, who further analyzes the
information and can prescribe a recovery strategy 2108 that
includes a variety of movements.
[0073] The patient then performs 1210 recovery strategy movements
and the data from the recovery strategy movements is transmitted
1212 to a remote server, for example. The data may first be
transmitted from one or more movement measurement devices to a
computing device, e.g., an application being executed on a desktop
computer, a laptop computer, a mobile phone running an application,
a computer tablet, another fitness tracking device, etc. In other
embodiments the movement measurement device (s) transmits data to
the server directly.
[0074] The data from the recovery strategy movements (RSM) is
analyzed 1214 and if the analysis indicates that the user/patient's
recover is not complete 1216 then the recovery strategy can be
modified 1218 and the process can continue with step 1210.
Determining whether the recovery is complete can be accomplished
with an algorithm executed on a computer and/or by analysis by a
person, e.g., a physician/physical therapist/medical personnel or
anyone else trained to interpret the data. The recovery strategy
movement data (e.g., motion data) can be stored and used 1220 to
assist in generating recovery strategies for future patients.
[0075] One benefit of the embodiments is the ability to collect
long term motion data about an individual's movement behaviors in
order to establish signatures that can later be used to identify
specific changes in behavior, potential illnesses/diseases,
conditions, and injuries/potential injuries. The long term motion
data can be stored by the motion monitoring system 120 and can be
used in generating/updating the user motion model, or reference
models for identifying potential illnesses, diseases, conditions,
or injuries.
[0076] The motion monitoring system 120 can identify asymmetrical
user movements that might indicate a loss of motor skill, a loss of
range of motion, or other injury. The motion monitoring system 120
can compare the patient's user motion model with the reference
model to determine if an injury has occurred or if an injury is
likely to occur. For instance, a user can begin with a baseline
range of motion on a joint, such as an elbow for a baseball
pitcher. Over time not only could the range of motion for the elbow
be monitored via a movement measurement device 100, but the motion
monitoring system 120 can continually monitor a range of motion on
that elbow and shoulder as the patient performs motions of interest
either while exercising or from a recovery improvement strategy. In
other examples, a range of motion can also be determined for bicep
curls, shoulder lateral raises, or rubber band rotation exercises.
If the user's range of motion reaches a limit, the motion
monitoring system 120 can determine that an injury is more likely
and can notify the user of the potential for injury to ensure the
user is less likely to be injured. Additionally, the motion
monitoring system 120 can adjust a recovery improvement strategy to
reduce the likelihood that further injury will occur. This type of
assessment and monitoring can be performed on any limb of the user
with any variation of one or multiple sensors. Additionally, the
motion monitoring system 120 may determine a likelihood of injury
or potential injury based on reductions in tempo or load of motions
of interest in a workout.
[0077] The user motion model can be continually compared to the
reference model to track deviations in the user's performance of a
motion of interest over time. Deviations are a change in a feature
of a motion of interest, such as pace, form, consistency, or range
of motion. Deviations can be identified and sent to one or more
users, including the patient, a doctor, a physical therapist, or an
electronic medical record, for example. For example, motions of
interest from a patient that is recovering from a knee replacement
surgery can be monitored for speed of movement, range of motion
progression, and overall knee-movement form during the recovery
period. Based on the motion data from the patient, the motion
monitoring system 120 may determine that the patient is not
performing the exercises as quickly as they did previously during
the recovery period. The motion monitoring system 120 may then
modify the recovery improvement strategy to ensure recovery is safe
and to reduce the likelihood that the patient will become reinjured
or discouraged. Similarly, the motion monitoring system 120 may
adjust the recovery improvement strategy for patients who are
recovering more quickly than similar patients.
[0078] The motion monitoring system 120 may compare a user's MOI
model to a reference model that has been generated from motion data
from users with a similar injury history or recovery history as the
user. In some embodiments, the motion monitoring system 120
determines users that have similar injury or recovery history to
the patient based on pre-surgical, post-surgical, or post-injury
motion data.
[0079] The motion monitoring system 120 can identify injuries in a
user by determining if certain motion of interest features indicate
that the user is compensating for an injury. For example, one
common result of having weak hips is excess strain on the knee,
which can be caused by tight hip flexors or weak gluteus medius.
This can make the thigh rotate inward, which can cause the user to
perform motions of interest with features that show that the user
is putting undue strain on the patella. The motion monitoring
system 120 can identify that the user may have weak hips, and thus
may generate an improvement strategy that can improve the user's
hip strength. In some embodiments, the motion monitoring system 120
determines if the user is compensating for an injury based on the
comparison of the user motion model and the reference model.
[0080] The motion monitoring system may initially receive motion
data from a single movement measurement device 100 for initial
motion data. The motion monitoring systems 120 may generate an
improvement strategy that recommends additional movement
measurement devices 100. For example, the motion movement system
120 may recommend a post-operation patient perform a series of
motions of interest so that the motion monitoring system 120 can
monitor the patient's progress in real time. As the patient
progresses, the motion monitoring system 120 can compare the user's
MOI model to reference models and can detect motion of interest
features that are associated with a specific form of post operation
movement impairment. The motion monitoring system 120 can generate
or adjust an improvement strategy that recommends that more or
better sensors be used to more accurately monitor this patient's
progress/regress. For example, an ACL injury might have a poor
range of motion recovery, and thus the motion monitoring system may
recommend that more or better sensors be used to understand this
range of motion degradation.
[0081] In some embodiments, the motion monitoring system 120 can
generate a user motion model based on motion data from one side of
the user's body and use motion data from the other side of the
user's body to generate a reference model. For example, if the
motion monitoring system 120 determines that a patient has some
potential loss of motor skill or range of motion in the left leg,
the physician can ask the patient to wear a movement measurement
device 100 on both their left leg and right leg. The user motion
model may be generated based on motion data from the left leg, and
a reference model may be generated based on motion data from the
right leg. The comparison of the user motion model and the
reference model can show if the user has experienced a loss of
speed, rotational velocity, acceleration, or range of motion, or
has different fatigue points. For example, a user may have recently
had their left shoulder replaced, and during recovery, they may
have a tempo on a motion of interest with their left arm that is
slightly slower than their right arm. Using motion data from other
users, the motion monitoring system 120 may identify that the left
arm's tempo disparity from the right is correlated with a shoulder
injury or a poor shoulder replacement surgery.
[0082] The motion monitoring system 120 can identify which motions
of interest were performed, which were performed incorrectly, and
whether certain motions of interest were not performed at all or
were performed with fatigue. The motion monitoring system 120 can
adjust an improvement strategy to reflect the user's current state.
The motion monitoring system 120 can use past motion data to
identify incorrect motions of interest and correlate them with
fatigue points or as potentially inducing an injury.
[0083] In some embodiments, the motion monitoring system 120
adjusts the improvement strategy based on whether the user performs
motions of interest recommended by the improvement strategy. For
example, the user may deviate from the improvement strategy by
changing the number of repetitions, number of sets, weight used,
the time of day, or types of exercises. The motion monitoring
system 120 may notify the user of the deviation from the
improvement strategy and may provide the user with an adjusted
improvement strategy based on the deviation. In some embodiments,
the motion monitoring system 120 notifies another user if the user
deviates from the improvement strategy, such as a doctor or a
therapist. The motion monitoring system 120 can identify the
deviation in the improvement strategy and can notify the other user
via an application notification, email, text, or phone call. The
motion monitoring system 120 may also allow the other user to
contact the user directly to intervene with the user's performance
of the improvement strategy. For example, a therapist may be
allowed to call the user to inform them of the deviation from the
improvement strategy or to discuss the user's performance of the
improvement strategy.
[0084] The motion monitoring system 120 may dynamically adjust an
improvement strategy based on the user's real time performance of
the motions of interest. For example, the motion monitoring system
120 may generate an improvement strategy that recommends a user do
regular arm curls, hammer curls, and pushups for therapy. If user A
performs regular curls poorly, the motion monitoring system 120 can
determine that the motion of interest was performed poorly and can
adjust the improvement strategy in real time, recommending that
user A should skip the hammer curls or maybe replace the hammer
curls with a less strenuous exercise as any more curls may risk
injuring the user.
[0085] In some embodiments, the motion monitoring system 120 uses
electromyography (EMG) motion data to determine features about a
user's performance of a motion of interest, such as the quality of
the user's muscle firing. The motion monitoring system 120 can
determine if the user has an injury or the type of the injury using
the EMG data. For example, if the EMG data shows that the user's
muscle is firing weakly or atypically, the motion monitoring system
120 may determine that the user has an injury, and may determine
the type of the injury based on features of the EMG motion data. In
some embodiments, the motion monitoring system 120 can determine
whether the user has an in jury by comparing the user motion model
with a reference model generated based on motion data from other
users.
[0086] The systems and methods described herein can also be used
for insurance companies to assist in identifying risk profiles.
Examples include: (1) automated premium adjustments based on daily,
weekly, monthly motion of interest routines and regimens; (2)
remote observational testing and monitoring through a movement
measurement device of an individual for premium and health care
price adjustment; (3) altering a price charged to a consumer based
on motion data; (4) confirming adherence to rules/laws; for
example, if a user does X when they should have done Y, then data
is transmitted to the insurer who may increase the user's premium
or risk profile; similarly if the user does what they are supposed
to do, such as adhering to an exercise program, then that
information may be transmitted to an insurer who may decrease the
user's premium or positively adjust the user's risk profile.
Gunfire Detection and Training
[0087] The motion monitoring system 120 can be used to detect
gunfire and to train a user to handle and fire a firearm properly
and effectively. The motion monitoring system 120 can receive
motion data from movement measurement devices 100 on one or more
law enforcement officers. The motion monitoring system 120 can
identify motions of interest related to firearms, including
drawing, loading, firing, holstering, or dropping the firearm. The
motion monitoring system can also identify motions of interest that
relate to where the user is positioning the firearm, including
whether the firearm is in the user's holster, whether the firearm
is being held at the user's hip in a defensive posture, if the
firearm is being held in front of the user in an assertive posture,
or if the firearm is being aimed in an aggressive posture.
[0088] Additionally, the motion monitoring system 120 can extract
features of the motions of interest from the motion data, including
the type of firearm being fired, the type of ammunition being used,
the rate of fire, the consistency of fire, the angle of the user's
hand while the firearm is in use, and the aiming of the firearm. In
some embodiments, these features can be determined by the motion
monitoring system 120 based on an analysis of the motion data or a
comparison of a user MOI model to a reference model.
[0089] The motion monitoring system 120 can determine if a user is
aiming the firearm correctly. The motion monitoring system 120 can
use motion data of the user aiming and firing the firearm to
generate a user motion model that can describe the user's aim. The
motion monitoring system 120 can compare the user motion model to a
reference model that describes ideal aim for the firearm and can
generate an improvement strategy for how the user can improve their
aim. For example, if the user is firing the firearm too rapidly,
the motion monitoring system 120 can recommend the user slow their
fire rate to improve aim.
[0090] Additionally, the motion monitoring system 120 can use
motion data to determine if a law enforcement officer is following
a proper procedure for use of a firearm. A law enforcement officer
may be expected to perform certain actions before resorting to
presenting or using a firearm. For example, the law enforcement
officer may be expected to provide a verbal warning first or may be
expected to present a more defensive posture before using the
firearm. The motion monitoring system 120 can generate a user
motion model for the law enforcement officer that describes the law
enforcement officer's actions before presenting/using a firearm and
can compare the user motion model to a reference model that
describes a user performing the proper procedure for
presentation/use of a firearm. The motion monitoring system 120 can
generate an improvement strategy for the law enforcement officer to
improve their adherence to proper firearm use procedure and can
intervene directly when the law enforcement officer is not
following the proper procedure. In some embodiments, the motion
monitoring system 120 notifies other users, such as the law
enforcement officer's supervisor or other individuals in the law
enforcement office, if the law enforcement officer is not adhering
to proper firearm use procedure so that those users can intervene
directly to train or discipline the law enforcement officer.
Additionally, the motion monitoring system 120 can recreate the law
enforcement officer's actions in a situation where a firearm was
used so that the events of the situation can be more thoroughly
evaluated by other users afterwards.
Workplace Activity Monitoring and Improvement
[0091] The motion monitoring system 120 can monitor motion data
from a worker to ensure that the worker is performing workplace
tasks effectively and safely. The motion monitoring system 120 can
receive motion data from movement measurement devices 100 on one or
more workers at a business. The motion monitoring system 120 can
identify motions of interest that relate to the worker's
performance of their job, such as lifting a box, using a tool, or
walking/running. The motion monitoring system 120 can also identify
important motions of interest that are not directly related to the
worker's tasks, such as if the worker falls or if the worker is
asleep/unconscious. The motion monitoring system 120 can generate a
user motion model for the worker's performance of the workplace
motions of interest and can compare the user motion model to a
reference model for ideal performance of the workplace motions of
interest. The reference model may be generated based on motion data
from one or more users performing the workplace motion of interest
correctly. The motion monitoring system 120 can determine if the
user is performing the motion of interest ineffectively or in a way
that could lead to an injury. For example, the motion monitoring
system can determine if a worker is lifting boxes in a way that
could injure their back. The motion monitoring system 120 can
generate an improvement strategy for how the worker can improve
their performance of the motion of interest and can monitor the
worker's improvement over time, adjusting the improvement strategy
if necessary. In some embodiments, the motion monitoring system 120
notifies other users, such as the worker's supervisor or a trainer,
that the user is performing a motion of interest incorrectly,
allowing the other user can intervene to train the worker directly
or whether to allow the worker to continue performing the motion of
interest.
Sports Performance Monitoring and Improvement
[0092] The motion monitoring system 120 can monitor motion data
from an athlete to improve the athlete's performance in a sport.
The motion monitoring system 120 can receive motion data from
movement measurement devices 100 on one or more athletes. The
motion monitoring system 120 can identify motions of interest that
relate to the sport the athlete is participating in, such as
running, jumping, walking, throwing, catching, punching, kicking,
swimming, biking, or hitting. The motion monitoring system 120 can
generate a user motion model for the athlete's performance of the
motions of interest and can compare the user motion model to a
reference model for ideal performance of the motions of interest.
For example, the motion monitoring system 120 may compare the
user's form in throwing a ball to an ideal throw. The reference
model may be generated based on motion data from one or more other
athletes performing the motion of interest correctly. The motion
monitoring system 120 can determine if the athlete is performing
the motion of interest ineffectively or in a way that could lead to
an injury. For example, the motion monitoring system can determine
if the athlete's running form is ineffective and can be improved to
allow the athlete to run more quickly and with a lower likelihood
of injury. The motion monitoring system 120 can generate an
improvement strategy for how the athlete can improve their
performance of the motion of interest and can monitor the athlete's
improvement over time, adjusting the improvement strategy if
necessary. The improvement strategy can include exercises the
athlete should perform or recommended adjustments to the motion of
interest. In some embodiments, the motion monitoring system 120
notifies other users, such as the athlete's coach or trainer, that
the athlete is performing a motion of interest incorrectly,
allowing the other user to intervene.
Driving Monitoring and Improvement
[0093] The motion monitoring system 120 can monitor motion data
from a driver to improve the driver's driving abilities and reduce
the likelihood of a car accident. The motion monitoring system 120
can receive motion data from movement measurement devices 100 on
one or more drivers performing motions of interest related to
driving a car. The motion monitoring system 120 can identify
motions of interest that relate to the driver's behavior when
driving the car, such as pressing a gas or brake pedal, pressing a
clutch, shifting gears, turning the steering wheel, looking at
cross-traffic, checking a mirror, opening/closing the door/window,
adjusting the side-view or rear-view mirrors, putting on a seat
belt, adjusting the seat, or interacting with an audio system. The
motion monitoring system 120 can also identify motions of interest
performed by the driver that are not related to driving the car,
but may impact the driver's driving ability, such as using a phone
or putting on make-up. The motion monitoring system 120 can use the
identified motions of interest to other aspects of the driver's
driving behavior, such as their hand positioning on the steering
wheel, their speed, or frequency of lane change.
[0094] The motion monitoring system 120 can generate a user motion
model for the driver's driving performance and can compare the user
motion model to a reference model for ideal driving performance.
The reference model may be generated based on motion data from one
or more other drivers driving cars ideally. The motion monitoring
system 120 can determine if the driver is driving a car safely. For
example, the motion monitoring system 120 can determine if the
driver is speeding, checking their phone, driving too aggressively,
changing lanes too quickly, or does not check cross traffic.
Additionally, the motion monitoring system 120 can determine if the
driver's performance of driving motions of interest suggests that
the driver is not in a state to safely drive the car, such as if
the driver is tired or intoxicated. The motion monitoring system
120 can generate an improvement strategy for how the driver can
improve their driving and can monitor the driver's improvement over
time, adjusting the improvement strategy if necessary. The
improvement strategy can include recommendations of additions,
removals, or adjustments to motions of interest related to driving
for the user drive more safely. In some embodiments, the motion
monitoring system 120 notifies other users, such as a relative of
the driver or law enforcement, that the driver is performing unsafe
motions of interest while driving the car, allowing the other user
can intervene to train the athlete directly. For example, if the
driver performs a motion of interest that suggests that the driver
is breaking the law or driving intoxicated, the motion monitoring
system 120 may automatically notify law enforcement.
Additional Considerations
[0095] Reference in the specification to "one embodiment" or to "an
embodiment" means that a particular feature, structure, or
characteristic described in connection with the embodiments is
included in at least one embodiment. The appearances of the phrase
"in one embodiment" or "an embodiment" in various places in the
specification are not necessarily all referring to the same
embodiment.
[0096] Some portions of the detailed description are presented in
terms of algorithms and symbolic representations of operations on
data bits within a computer memory. These algorithmic descriptions
and representations are the means used by those skilled in the data
processing arts to most effectively convey the substance of their
work to others skilled in the art. An algorithm is here, and
generally, conceived to be a self-consistent sequence of steps
(instructions) leading to a desired result. The steps are those
requiring physical manipulations of physical quantities. Usually,
though not necessarily, these quantities take the form of
electrical, magnetic or optical signals capable of being stored,
transferred, combined, compared and otherwise manipulated. It is
convenient at times, principally for reasons of common usage, to
refer to these signals as bits, values, elements, symbols,
characters, terms, numbers, or the like. Furthermore, it is also
convenient at times, to refer to certain arrangements of steps
requiring physical manipulations or transformation of physical
quantities or representations of physical quantities as modules or
code devices, without loss of generality.
[0097] However, all of these and similar terms are to be associated
with the appropriate physical quantities and are merely convenient
labels applied to these quantities. Unless specifically stated
otherwise as apparent from the following discussion, it is
appreciated that throughout the description, discussions utilizing
terms such as "processing" or "computing" or "calculating" or
"determining" or "displaying" or "determining" or the like, refer
to the action and processes of a computer system, or similar
electronic computing device (such as a specific computing machine),
that manipulates and transforms data represented as physical
(electronic) quantities within the computer system memories or
registers or other such information storage, transmission or
display devices.
[0098] Certain aspects of the embodiments include process steps and
instructions described herein in the form of an algorithm. It
should be noted that the process steps and instructions of the
embodiments can be embodied in software, firmware, or hardware, and
when embodied in software, could be downloaded to reside on and be
operated from different platforms used by a variety of operating
systems. The embodiments can also be in a computer program product
which can be executed on a computing system.
[0099] The embodiments also relate to an apparatus for performing
the operations herein. This apparatus may be specially constructed
for the purposes, e.g., a specific computer, or it may comprise a
computer selectively activated or reconfigured by a computer
program stored in the computer. Such a computer program may be
stored in a computer readable storage medium, such as, but is not
limited to, any type of disk including floppy disks, optical disks,
CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random
access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards,
application specific integrated circuits (ASICs), or any type of
media suitable for storing electronic instructions, and each
coupled to a computer system bus. Memory can include any of the
above and/or other devices that can store information/data/programs
and can be transient or non-transient medium, where a non-transient
or non-transitory medium can include memory/storage that stores
information for more than a minimal duration. Furthermore, the
computers referred to in the specification may include a single
processor or may be architectures employing multiple processor
designs for increased computing capability.
[0100] The algorithms and displays presented herein are not
inherently related to any particular computer or other apparatus.
Various systems may also be used with programs in accordance with
the teachings herein, or it may prove convenient to construct more
specialized apparatus to perform the method steps. The structure
for a variety of these systems will appear from the description
herein. In addition, the embodiments are not described with
reference to any particular programming language. It will be
appreciated that a variety of programming languages may be used to
implement the teachings of the embodiments as described herein, and
any references herein to specific languages are provided for
disclosure of enablement and best mode.
[0101] In addition, the language used in the specification has been
principally selected for readability and instructional purposes,
and may not have been selected to delineate or circumscribe the
inventive subject matter. Accordingly, the disclosure of the
embodiments is intended to be illustrative, but not limiting, of
the scope of the embodiments
[0102] While particular embodiments and applications have been
illustrated and described herein, it is to be understood that the
embodiments are not limited to the precise construction and
components disclosed herein and that various modifications,
changes, and variations may be made in the arrangement, operation,
and details of the methods and apparatuses of the embodiments
without departing from the spirit and scope of the embodiments.
* * * * *