U.S. patent application number 14/506971 was filed with the patent office on 2015-04-09 for head worn sensor device and system for exercise tracking and scoring.
This patent application is currently assigned to Zinc Software Limited. The applicant listed for this patent is Darran John Hughes. Invention is credited to Darran John Hughes.
Application Number | 20150100141 14/506971 |
Document ID | / |
Family ID | 51663185 |
Filed Date | 2015-04-09 |
United States Patent
Application |
20150100141 |
Kind Code |
A1 |
Hughes; Darran John |
April 9, 2015 |
Head Worn Sensor Device and System for Exercise Tracking and
Scoring
Abstract
The subject invention is directed to a computer-implemented
method, a device and a system for detecting and scoring exercises,
including maintaining, by a mobile device, a library of motion
signatures, in which a motion signature for an exercise is a
sequence of characteristic features and a characteristic feature is
a movement in the exercise; receiving, by a mobile device, a time
series of data from a sensor device, the sensor device attached to
the head or torso of a user, the data comprising accelerometer data
from an accelerometer included in the sensor device, detecting a
single repetition of a designated exercise performed by the user
and calculating a motion score for the single repetition of the
detected exercise.
Inventors: |
Hughes; Darran John;
(Dublin, IE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Hughes; Darran John |
Dublin |
|
IE |
|
|
Assignee: |
Zinc Software Limited
|
Family ID: |
51663185 |
Appl. No.: |
14/506971 |
Filed: |
October 6, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61887967 |
Oct 7, 2013 |
|
|
|
Current U.S.
Class: |
700/92 |
Current CPC
Class: |
A61B 5/1118 20130101;
G06K 9/00335 20130101; G16H 20/30 20180101; A61B 5/1123 20130101;
G06Q 10/0639 20130101; A61B 5/6814 20130101; G16H 40/67 20180101;
A61B 2562/0219 20130101; A61B 2503/10 20130101; A61B 5/7282
20130101; A61B 5/02416 20130101; A61B 2505/09 20130101; A61B 5/6823
20130101 |
Class at
Publication: |
700/92 |
International
Class: |
A63B 71/06 20060101
A63B071/06 |
Claims
1. A computer-implemented method for detecting and scoring
exercises, comprising: maintaining, by a mobile device, a library
of motion signatures, wherein a motion signature for an exercise
comprises a sequence of characteristic features and wherein a
characteristic feature is a movement in the exercise; receiving, by
a mobile device, a time series of data from a sensor device, the
sensor device attached to the head or torso of a user, the time
series of data comprising accelerometer data from an accelerometer
included in the sensor device; detecting a single repetition of a
designated exercise performed by the user, wherein detecting
comprises: segmenting the time series of accelerometer data into a
sequence of characteristic features as specified by the motion
signature that corresponds to the designated exercise; and
calculating a motion score for the single repetition of the
detected exercise.
2. The method of claim 1 wherein detecting further comprises
validating that the range of values in each segment of
accelerometer data is within a specified range of values.
3. The method of claim 1 where the received time series of data
further comprises data from a photoplethysmograph included in the
sensor device, the method further comprising: calculating a heart
rate for the user; and calculating a heart rate score for the
designated exercise.
4. The method of claim 3 wherein the heart rate score is based on
the increase in the user's heart rate while performing the
designated exercise.
5. The method of claim 3 further comprising: generating an exercise
score for the designated exercise based on both the motion score
and the heart rate score.
6. The method of claim 5 further comprising: generating a set score
by summing the individual exercise scores for a set of repetitions
of the designated exercise performed by the user.
7. The method of claim 1 wherein at least one motion signature in
the library of motion signatures corresponds to an exercise
selected from the group consisting of squats, lungs, press ups,
sit-ups, jumping jacks, running on the spot, side lunge, squat
jumps, rotating lunge, high jumps, crunches, standups, and
burpees.
8. The method of claim 1 wherein a motion signature corresponds to
a single repetition of an exercise selected from the group
consisting of a squat, a lunge, a press up, a sit-up, and a
burpee.
9. The method of claim 1 wherein calculating a motion score is
based on the magnitude of the motion performed by the user.
10. The method of claim 1 further comprising: displaying, by the
mobile device, a progress indicator selected from the group
consisting of the number of repetitions of the designated exercise
already performed in the set, the number of repetitions of the
designated exercise remaining to be performed in the set and the
time elapsed since the beginning of the set.
11. The method of claim 1 wherein the sensor device attaches to an
ear of the user.
12. The method of claim 1 wherein the time series of sensor data
further comprises data from at least one sensor selected from the
group consisting of an accelerometer, a magnetometer and a
gyroscope and the at least one sensor is also included in the
sensor device.
13. A mobile device, comprising: a processor; a wireless
transceiver in communication with a sensor device, the sensor
device attached to the head or torso of a user; and a
non-transitory memory in communication with the processor for
storing (1) a library of motion signatures, wherein a motion
signature for an exercise comprises a sequence of characteristic
features and wherein a characteristic feature is a movement in the
exercise, and (2) instructions, which when executed by the
processor, cause the mobile device: to receive a time series of
data from the sensor device, said data comprising accelerometer
data from an accelerometer included in the sensor device; to detect
a single repetition of a designated exercise performed by the user,
wherein detecting comprises: segmenting the accelerometer data into
a sequence of characteristic features as specified by the motion
signature that corresponds to the designated exercise; and to
calculate a motion score for the single repetition of the detected
exercise.
14. The device of claim 13 wherein detecting further comprises
validating that the range of values in each segment of
accelerometer data is within a specified range of values.
15. The device of claim 13 where the received time series of data
further comprises data from a photoplethysmograph included in the
sensor device, wherein the instructions, when executed by the
processor, further cause the mobile device: to calculate a heart
rate for the user; and to calculate a heart rate score for the
designated exercise.
16. The device of claim 15 wherein the heart rate score is based on
the increase in the user's heart rate while performing the
designated exercise.
17. The device of claim 16 wherein the instructions, when executed
by the processor, further cause the mobile device: to generate an
exercise score for the designated exercise based on both the motion
score and the heart rate score.
18. The device of claim 17 wherein the instructions, when executed
by the processor, further cause the mobile device: to generate a
set score by summing the exercise scores for each repetition of a
set of repetitions of the designated exercise performed by the
user.
19. The device of claim 13 wherein at least one motion signature in
the library of motion signatures corresponds to an exercise
selected from the group consisting of squats, lunges, press ups,
sit-ups, jumping jacks, running on the spot, side lunge, squat
jumps, rotating lunge, high jumps, crunches, standups, and
burpees.
20. The device of claim 13 wherein a motion signature corresponds
to a single repetition of a discrete exercise selected from the
group consisting of a squat, a lunge, a press up, a sit-up, and a
burpee.
21. The device of claim 22 wherein calculating a motion score is
based on the magnitude of the motion performed by the user.
22. The device of claim 13 wherein the instructions, when executed
by the processor, further cause the mobile device: to display, by
the mobile device, a progress indicator selected from the group
consisting of the number of repetitions of the designated exercise
already performed in the set, the number of repetitions of the
designated exercise remaining to be performed in the set and the
time elapsed since the beginning of the set.
23. The device of claim 13 wherein the sensor device worn by the
user is attached to the head or torso of the user.
24. The device of claim 13 wherein the time series of sensor data
further comprises data from at least one sensor selected from the
group consisting of an accelerometer, a magnetometer and a
gyroscope and the at least one sensor is also included in the
sensor device.
25. A system for detecting and scoring exercises, comprising: a
sensor device that attaches to the head of a user comprising: an
accelerometer that generates a time series of tri-axial data that
measures the acceleration of the user's head or torso; and a
wireless transmitter that transmits the time series of tri-axial
accelerometer data; a mobile device, comprising: a processor; a
wireless transceiver in communication with the sensor device that
receives the time series of tri-axial accelerometer data from the
sensor device; and a non-transitory memory in communication with
the processor for storing (1) the time series of tri-axial
accelerometer data, (2) a library of motion signatures, wherein a
motion signature for an exercise comprises a sequence of
characteristic features and wherein a characteristic feature is a
movement in the exercise, and (3) instructions, which when executed
by the processor, cause the mobile device: to detect a single
repetition of a designated exercise performed by the user, wherein
detecting comprises: segmenting the accelerometer data into a
sequence of characteristic features as specified by the motion
signature that corresponds to the designated exercise; and to
calculate a motion score for the single repetition of the detected
exercise.
Description
TECHNICAL FIELD
[0001] Various embodiments generally relate to a head worn sensor
device that provides a stream of sensor data to a computing device
system for tracking and scoring of exercises.
BACKGROUND
[0002] Using the head or torso as a location for tracking body
motion during exercises has two key advantages over the feet or
hands. First, the motion of the head or torso is more strongly
correlated with core body motion. The hands or feet can move
through much bigger motions without much trunk motion. This is not
possible for the head or torso. Secondly, an advantage of using the
head and more specifically the ear as a location for tracking body
motion is that high resolution heart activity can be measured very
accurately at the ear lobe using an ear clip or in ear
photoplethysmograph (PPG). The combination of both heart activity
and head motion may be used to calculate easily understandable
scores for a wide variety of physical exercises that provide a
measure of performance.
[0003] Motion sensing by a sensor attached to the head or torso,
but not attached at the wrist or hand or foot, combined with
analysis methods to detect and score exercises such as sit ups,
squats, lunges, press-ups, burpees, yoga positions, running,
cycling and rowing based on head motion trajectory does not exist
in the prior art. Further, the combination of motion sensing and
heart sensing at the head or torso in combination with analysis
methods to detect and score exercises based on head motion
trajectory combined with heart rate is equally novel.
[0004] It is well known in the state of the art that simple
accelerometer based readings at the wrist can easily be inaccurate
or "cheated" by waving the arms around without also moving body
trunk. Thus, a sensor device that is attached to the torso or head,
and more specifically to the ear, ensures that head motion and by
extension core trunk movement is actually occurring and thus avoids
such cheating.
[0005] Thus, it is with respect to these considerations and others
that the present invention has been made.
SUMMARY OF THE DESCRIPTION
[0006] The subject invention concerns a sensor device attached to
the head or torso of a user, referred to herein as a sensor device
or the device, which includes an accelerometer for motion tracking
and a heart rate sensor. The invention detects, analyzes and scores
specific exercises performed by a user based on the corresponding
head motion trajectories during the exercise and the increased
heart rate the exercise causes in the user. The sensor can be
applied to a wide range of exercises, including individual
exercises such as press-ups and sit-ups and continuous motion
exercises such as running, rowing or cycling. Exercises may be
performed either outdoors or indoors and may be performed with or
without the use of gym machines.
[0007] The sensor device transmits data to a connected computing
device, also referred to herein as a mobile device, such as a
computer or smartphone that in turn displays real-time or historic
exercise data to the user. The connected computing device may be
any computer-based device with an audiovisual display, a CPU and
wireless communication link. This includes but is not limited to a
PC, a laptop computer, a tablet, a mobile phone such as a smart
phone, a smart watch or gym equipment with audiovisual capabilities
such as treadmill or rowing machine. Applications running on the
connected computing device manage the display of information to the
user during or after an exercise session.
[0008] The subject invention includes methods for analyzing
discrete exercises such as sit-ups and continuous exercises such as
running. It includes a method performed by a connected device such
as a mobile device that transmits wirelessly for analyzing discrete
exercises by segmenting an incoming data stream from a sensor
device into a sequence of characteristic features of a discrete
exercise and detecting or confirming which exercise is being
performed by a user. The sequence of characteristic features when
taken in sequence form a motion signature that corresponds to an
exercise such as a squat, press-up or sit-up. The subject invention
further provides methods for scoring how well an exercise is
performed by a user.
[0009] The mobile device can operate in two usage modes. One is an
interactive mode where the mobile device gives the user real-time
feedback that they have completed a specific exercise such as a
squat with the context of an interactive workout session. This
real-time feedback is displayed via the application running on the
connected computing device. The other mode is automatic mode where
the sensor device sends data to the computing device to
automatically record and analyze the motion of an exercise session
such as a rowing or running session so that it can be presented as
a data visualisation by an application after the session. For
example, a user display may provide in depth information on
performance and identify points where fatigue changed overall form
of running style. Automatic mode operates automatically without
requirement for user interaction.
[0010] Various embodiments of the subject invention are directed
towards a computer-implemented method and system for detecting and
scoring exercises, including maintaining, by a mobile device, a
library of motion signatures, in which a motion signature for an
exercise is a sequence of characteristic features and a
characteristic feature is a movement in the exercise; receiving, by
a mobile device, a time series of data from a sensor device, the
sensor device attached to the head or torso of a user, the data
comprising accelerometer data from an accelerometer included in the
sensor device; detecting a single repetition of a designated
exercise performed by the user and calculating a motion score for
the single repetition of the detected exercise.
[0011] Other embodiments of the subject invention are directed
toward a mobile device, that includes a processor, a wireless
transceiver in communication with a sensor device, the sensor
device attached to the head or torso of a user, and a
non-transitory memory in communication with the processor for
storing (1) a library of motion signatures, in which a motion
signature for an exercise includes a sequence of characteristic
features and where a characteristic feature is a movement in the
exercise, and (2) instructions, which when executed by the
processor, cause the mobile device to receive a time series of data
from the sensor device, said data comprising accelerometer data
from an accelerometer included in the sensor device, to detect a
single repetition of a designated exercise performed by the user,
in which detecting includes segmenting the accelerometer data into
a sequence of characteristic features as specified by the motion
signature that corresponds to the designated exercise, and to
calculate a motion score for the single repetition of the detected
exercise.
[0012] Yet further embodiments of the subject invention are
directed towards a computer-implemented method and a device for
detecting and scoring an exercise, including receiving, by a mobile
device, a time series data for a time interval from a sensor device
worn by a user, said data including tri-axial accelerometer data
from an accelerometer included in the sensor device, maintaining,
by a mobile device, a library of continuous motion signatures, each
motion signature corresponding to a different exercise that is
performed by a user and each continuous motion signature includes a
threshold value for at least one axis of motion and one or more
ratios that relate spectral peak values in different axes of
motion, calculating, by a mobile device, a frequency spectrum for
the received time series data for the time interval, and for a
designated exercise calculating the spectral peaks of each of the
three axes of motion, determining that the spectral peak exceeds
the threshold value for the at least one axis of motion specified
by the continuous motion signature that corresponds to the
designated exercise, determining that the ratio of the calculated
spectral peaks exceeds the ratio specified by the continuous motion
signature that corresponds to the designated exercise, and
calculating an interval motion score for the designated
exercise.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Non-limiting and non-exhaustive embodiments of the present
invention are described with reference to the following drawings.
In the drawings, like reference numerals refer to like parts
throughout the various figures unless otherwise specified.
[0014] For a better understanding of the present invention,
reference will be made to the following Detailed Description of the
Preferred Embodiment, which is to be read in association with the
accompanying drawings, wherein:
[0015] FIG. 1A is an illustration of one embodiment of an ear worn
sensor device that includes a main body worn behind the ear
connected to an ear clip.
[0016] FIG. 1B illustrates an embodiment of a system that includes
a sensor device, attached to a user's ear, in communication with a
mobile device.
[0017] FIG. 2 illustrates one embodiment of a hardware architecture
of a sensor device, in accordance with an embodiment of the subject
invention.
[0018] FIG. 3 illustrates one embodiment of an architecture of a
mobile device, in accordance with an embodiment of the subject
invention.
[0019] FIG. 4A illustrates a starting position and an intermediate
position for a squat.
[0020] FIG. 4B illustrates a starting position and an intermediate
position for a sit-up.
[0021] FIG. 5 is a flow diagram that illustrates one embodiment of
a method implemented by a discrete exercise component for detecting
and scoring discrete exercises.
[0022] FIG. 6 illustrates a typical time series of accelerometer
measurements in the Y axis that correspond to a squat exercise
performed by a user.
[0023] FIG. 7 illustrates an embodiment of a segmentation method in
which a finite state machine (FSM) is used to identify the motion
signature of a squat.
[0024] FIG. 8 shows an example embodiment of a discrete exercise
user interface for an exercise application that is used for
discrete exercises.
[0025] FIG. 9 illustrates an exemplary time series of measurements
taken by an accelerometer in the sensor device, attached to the
head or torso of a user, while the user is performing a sit-up
exercise.
[0026] FIG. 10 illustrates a finite state machine (FSM) used to
segment accelerometer data into a sequence of characteristic
features that correspond to a sit-up.
[0027] FIG. 11 illustrates the general structure of a finite state
machine (FSM) used to segment discrete exercises.
[0028] FIG. 12 is a flow diagram that illustrates one embodiment of
a method implemented by the continuous exercise component for
analyzing gym sessions and automatically identifying which gym
machines were used.
[0029] FIG. 13A shows recorded gym session y axis accelerometer
data.
[0030] FIG. 13B shows the percentage energy in the largest spectral
peak.
[0031] FIG. 13C shows the segmented regions which were assigned as
periodic and therefore segments in which exercises are
performed.
[0032] FIG. 14 shows an example embodiment of a gym session user
interface that is presented by the exercise application.
[0033] FIG. 15 illustrates an example of the spectrum output for
accelerometer data from sensor device attached to a user running on
a treadmill.
[0034] FIG. 16 is a flow diagram that illustrates one embodiment of
a method implemented by the continuous exercise component for
detecting and scoring continuous exercises.
[0035] FIG. 17 illustrates an example of Y axis accelerometer data
received from the sensor device attached to the head of a user
running on a treadmill.
[0036] FIG. 18 illustrates an exemplary time series of measurements
taken by an accelerometer during a burpee exercise for both the X
axis and Y axis accelerometer values.
DETAILED DESCRIPTION
[0037] The invention now will be described more fully hereinafter
with reference to the accompanying drawings, which form a part
hereof, and which show, by way of illustration, specific exemplary
embodiments by which the invention may be practiced. This invention
may, however, be embodied in many different forms and should not be
construed as limited to the embodiments set forth herein; rather,
these embodiments are provided so that this disclosure will be
thorough and complete, and will fully convey the scope of the
invention to those skilled in the art. Among other things, the
invention may be embodied as methods, processes, systems, business
methods or devices. Accordingly, the present invention may take the
form of an entirely hardware embodiment, an entirely software
embodiment or an embodiment combining software and hardware
aspects. The following detailed description is, therefore, not to
be taken in a limiting sense.
[0038] As used herein the following terms have the meanings given
below:
[0039] Discrete exercise--An exercise with a beginning and an end
that is formed of a series of movements and which is typically
performed repetitively by a person or user. Examples include
squats, sit-ups and burpees.
[0040] Continuous exercise--An exercise that does not have a clear
beginning or end and is performed over a time period by a user.
Examples include running, and cross training on an elliptical gym
machine.
[0041] Motion signature--A series of movements by a user that taken
together correspond uniquely to an exercise typically performed by
a user as part of an exercise session such as a squat, a sit-up or
a burpee. As used herein, motion signatures are analyzed in the
acceleration domain and are detected using a finite state machine;
but the term is not limited to this particular technical
approach.
[0042] Characteristic feature--A movement that must be performed as
part of an exercise. These are typically peaks, troughs and flats
in an accelerometer data stream. A motion signature is formed from
a sequence of characteristic features.
[0043] Segmentation--As used herein, segmentation refers to the
process of identifying a specific time segment of accelerometer
data and mapping it to a characteristic feature of an exercise; it
further refers to the process of identifying a sequence of
segments, and hence characteristic features, that taken together
form a motion signature and correspond to an exercise.
[0044] Detection--means the overall process of confirming, based on
a time series of sensor data, that a motion or sequence of motions
performed by a user corresponds to a designated exercise. In
certain embodiments, detection can also include identifying which
of a predefined set of exercises a motion or sequence of motions
corresponds to.
Generalized Operation
[0045] The operation of certain aspects of the invention is
described below with respect to FIGS. 1-8.
[0046] FIG. 1A is an illustration of one embodiment of an ear worn
sensor device 1 that includes a main body 2 worn behind the ear
connected to an ear clip 3. In one embodiment, main body 2 includes
one or more positional sensors such as a tri-axis accelerometer.
Main body 2 also includes a BLUETOOTH or other wireless
communication method that allows it to send real time heart and
motion data to a computing device such as a PC, tablet computer or
smartphone. In one embodiment, accelerometer data is sampled at 55
Hz.
[0047] Ear clip 3 performs PPG measurement at the ear lobe via an
infrared LED and a photodiode. It connects to main body 2 via a
shielded electrical cable. PPG technology is well known in the
state of the art utilising changes in absorption of infrared light
during the cardiac cycle to transduce overall heart activity and so
allow calculation of heart rate. In one embodiment PPG data is
sampled at 110 Hz.
[0048] In one embodiment, Main body 2 houses the key electronics
components including a rechargeable battery and a wireless
communication module. In other embodiments, the main body 1 and ear
clip 3 may be a single device. In yet other embodiments, sensor
device 1 may be worn on other parts of the head than the ear, for
example it may be integrated into a headband or earphones. In other
embodiments, sensor device 1 is attached to the torso rather than
the head. Generally, sensor device 1 is not intended for use on the
arms, legs, hands or wrist. In yet other embodiments, sensor device
1 may include other sensors such as a magnetometer, or a
gyroscope.
[0049] FIG. 1B illustrates a system 10 that includes sensor device
1, attached to a user's ear, in communication with a mobile device
12. In one embodiment, a wireless link 11, such as BLUETOOTH or
near field communications is used. In other embodiments, a physical
connection such as a USB or ETHERNET cable may be used to connect
to mobile device 12. While a mobile device is illustrated in FIG.
1B, any computing device or service may be used which is capable of
receiving and processing or storing sensor data. In certain
embodiments, sensor device 1 and mobile device 12 are integrated
into a single device; for example, a CPU may be included in such an
integrated device and some of the processing may be performed
inside the integrated device while other parts of the processing
are performed by a mobile device or connected computing device. In
yet other embodiments, sensor device 1 and aspects of mobile device
12, such as the CPU and memory, may be integrated into a single
device and the integrated device itself may be integrated into
another device worn by a user such as headphones, a helmet, a hat,
or glasses.
[0050] FIG. 2 illustrates one embodiment of a hardware architecture
of sensor device 1. Sensor device 1 includes a positional sensor
205 such as a 3 axis accelerometer, a 3 axis magnetometer, or a 3
axis gyroscope that acquires a time series of position or motion
data for each of the X, Y, and Z axes. A photoplethysmograph (PPG)
sensor 210 connects to an analog to digital converter (ADC) 215 and
to a main CPU 225. CPU 225 stores incoming sensor data in a memory
220 or streams the sensor data in realtime to a wireless i/o
controller 235 where data packets are formatted and transmitted to
mobile device 12. Sensor device 1 further includes a power source
230 such as a battery. While an accelerometer, which provides
motion information, is assumed to be the sole positional sensor 205
hereinbelow, the subject invention can be adapted to one or more
other positional sensors without departing from its scope and
spirit.
[0051] FIG. 3 illustrates one embodiment of mobile device 12. In
one embodiment, mobile device 12 is a commercially available
smartphone such as an IPHONE by Apple Computer, or a GALAXY by
Samsung that includes an operating system 330 capable of running
applications, an input device 340 such as a keyboard, mouse, other
pointing device or touch screen, a wireless transceiver 350 capable
of receiving data from sensor device 1 and a display 360 capable of
displaying information. Mobile device 12 also includes non
transitory memory for storing programs and data such as a library
of motion signatures, a rechargeable batter and a power connection
to recharge the battery. In addition to a smartphone, mobile device
12 may be any computer capable of connecting to sensor device 1
including a tablet computer, a personal computer with a wireless
transceiver, or a gym machine such as a treadmill with a wireless
transceiver. In certain embodiments, mobile device is also capable
of communicating across wireless phone and data networks including
the Internet.
[0052] Included in mobile device 12 is a signal processing engine
(SPE) 310 that receives sensor data from sensor device 1 and inter
alia detects an exercise being performed by a user and calculates
exercise scores. SPE 310 combines a variety of motion and heart
analysis methods and provides exercises scores that can be used by
applications running on mobile device 12 such as an exercise
application 320 and one or more other applications 322.
Applications 320-322 may utilize data from SPE 310 to provide a
user with realtime or historic, ie post session, information on
their performance during an exercise session.
[0053] SPE 310 includes a discrete exercise detection and scoring
component 312 (henceforth referred to as discrete exercise
component 312) and a continuous exercise detection and scoring
component 314 (henceforth referred to as continuous exercise
component 314). Discrete exercise component 312 includes methods
for detecting and scoring discrete exercises such as squats and
sit-ups. Continuous exercise component 314 includes methods for
detecting and scoring continuous exercises such as running. SPE 310
access a library of motion signatures stored on, or available from,
mobile device 12 that enable it to detect and score exercises.
Overview of Detecting and Scoring Discrete Exercises
[0054] In one embodiment, exercise application 320 gives the user
realtime feedback on a bootcamp workout session. A bootcamp workout
session, or simply session, includes a number of exercises, capable
of being analyzed and scored by SPE 310, that are performed by the
user during a session. As output, SPE 310 provides an indication of
which exercise is being performed and provides one or more
corresponding scores to applications 320-322.
[0055] A session is divided into a number of sets, each set
representing a number of repetitions of an exercise included in the
bootcamp workout. The application may display a countdown of the
repetitions. SPE 310 scores each repetition of an exercise and each
time the score is above a minimum threshold the countdown
decrements one until the number of repetitions are complete and the
set is complete and the next set will start. Once a session is
complete the user may be shown a summary of their performance
including overall workout score. Other applications can utilise SPE
310 to create scoring-based feedback for activities such as gym
session tracking and rowing performance. In certain embodiments, a
bootcamp workout session may include exercises that SPE 310 is not
capable of analyzing.
[0056] Discrete exercise component 312 detects exercises based on a
data stream of sensor data from sensor device 1. Sensor data from
sensor device 1 may include accelerometer data, magnetometer,
gyroscope and PPG data. In one embodiment, only the data stream
consists only of tri-axial orthogonal (x, y, z) accelerometer data
values.
[0057] Discrete exercise component 312 identifies or detects an
exercise by analyzing the data stream to find a motion signature,
as previously defined, and to determine if the motion signature
corresponds to the motion signature in a library of motion
signatures. A motion signature is typically comprised of a series
of movements that together form a discrete exercise. For example,
each exercise has a characteristic starting orientation which must
be held by the user for a short amount of time (e.g. around 0.5
seconds) before exercise tracking will start. For example, for
press-ups the starting orientation is the user's head pointing
downwards, for sit-ups it is the user's head pointing upwards.
[0058] FIG. 4A illustrates a starting position 410 and an
intermediate position 420 for a squat. The up and down arrows
indicate that the motion of the exercise is predominantly up and
down.
[0059] FIG. 4B illustrates a starting position 430 and an
intermediate position 440 for a sit-up. The arrows indicate that
the motion of the exercise is predominantly an arc.
[0060] Typically, each exercise has a characteristic set of peaks,
troughs and flat regions in its stream of tri-axial, orthogonal,
acceleration data. FIG. 6, which is described in further detail
hereinbelow, shows an example of the sensor stream data for a
typical squat exercise. The Y axis represents up and down and the Z
axis represents forward and backward directions with respect to the
head. The squat exercise has a characteristic head trajectory that
includes a trough, followed by a peak, followed by a trough.
[0061] Discrete exercise component 312 segments the data stream for
an exercise, i.e. it determines a start point and an end point
within the data stream, and then validates that the characteristic
features of the head trajectory for a given exercise, i.e. the
peaks, troughs and flats, are included in the segment. In one
embodiment, discrete exercise component 312 receives an indication
from one of applications 320-322 of which exercise the user is
performing, i.e. which exercise corresponds to the incoming sensor
data stream. In other embodiments, discrete exercise component 312
performs an identification step to determine which of a set of
exercises is being performed and thus corresponds to the incoming
data stream.
[0062] The received stream of time series data is analyzed using a
finite state machine (FSM) approach. In other embodiments other
methods of pattern detection such as neural networks, HMMS,
Bayesian networks could also be employed. A distinct FSM is
associated with each exercise that is designed to match the
sequence of characteristic features, i.e. the trajectory of the
head or torso, of the exercise. Each characteristic feature, i.e.
separately identifiable movement, is represented by a single state
in the FSM. For example, the FSM for the squat exercise has a state
A for the initial trough, a state D for the peak and a state E for
the second trough. Each state has a maximum or minimum threshold
value associated with it. States can also have a minimum and
maximum time associated with them.
[0063] A motion signature thus has a corresponding data structure
stored by mobile device 12 in its memory that includes information
for each state in its FSM. As previously mentioned the state
information includes a maximum and minimum threshold value and may
include a minimum or maximum time.
[0064] At the outset of the FSM detection process, an exercise is
designated. An incoming data segment is passed to FSM detection
method. The FSM starts in a reset state. If the incoming data
corresponds to the sequence of the designated exercise's
characteristic features then the FSM will pass through all the
states required. If the motion is not similar and an unexpected
motion is encountered the FSM is reset and the motion is considered
to not match the designated exercise.
[0065] If a given segment of data is validated as corresponding to
the designated exercise by the FSM method then the designated
exercise is scored based on aspects of its motion. Each exercise
has slightly different scoring features.
[0066] For some exercises it suffices to define FSM states with
respect to a single axis. For example a squat only involves direct
up and down motion and so only motion in the Y axis needs to be
tested. Head rotation pitch and roll angles are tracked throughout
motion to confirm the overall orientation of the head is as
expected. If either absolute values of these angles or their range
over the period of the exercise is above a specific threshold then
these exercises will be scored zero. An example would be moving
head too much during a squat exercise where head should be held
relatively straight throughout the motion.
[0067] The main output of discrete exercise component 312 is an
exercise score that ranges from 0 to 100 for each exercise. Any
movement that does not match the expected sequence of movements, as
defined by the motion signature for the designated exercise, will
score zero. Also any motion which does not reach the minimum
threshold of total motion (e.g. a squat where the person only drops
a few inches) will also score zero. Exercises performed well will
have motion score closer to 100, e.g. press-ups that are faster or
push higher will score closer to 100.
[0068] In one embodiment, discrete exercise component 312 only uses
accelerometer data, as including only an accelerometer in sensor
device 1 is a cost effective solution and produces robust
results.
[0069] Discrete exercise component 312 may also measure heart rate
activity in tandem with motion tracking. A heart rate score is
derived in parallel with the motion score based on the increase in
heart rate above a baseline level. Higher increases in heart rate
above baseline will give higher heart rate scores. Each exercise
has an expected heart rate increase for an average person and this
value is used to weight the heart rate score. If the heart sensor
is not attached properly to the ear and so the system does not get
a valid PPG signal then the heart rate score will be zero.
[0070] Discrete exercise component 312 calculates both a heart rate
score and a motion score. Each can be presented independently or
combined in an overall exercise score. A workout session consisting
of a series of potentially different exercises may be scored as the
summation of the scores of each individual exercise. This overall
workout score can be used to give the user feedback on their
achievement and to show the user's performance over time.
[0071] Further, discrete exercise component 312 can be extended to
process additional types of sensor data including data from a
gyroscope or magnetometer to allow even higher levels of accuracy
of the scoring metrics.
Detailed Description of the Operation of Discrete Exercise
Component
[0072] FIG. 5 is a flow diagram that illustrates one embodiment of
a method 500 implemented by discrete exercise component 312 for
detecting and scoring discrete exercises.
[0073] At step 505 mobile device 12 receives a stream of data from
sensor device 1 and provides the data stream to discrete exercise
component 312 for processing. At step 510 discrete exercise
component 312 parses the incoming stream of data. In certain
embodiments, data is transmitted by sensor device 1 in packets that
contain data from each sensor in sensor device 1. In one
embodiment, the packets include accelerometer data, PPG data and
sensor status data such as battery level. Step 510 parses, or
separates, the data in these packets to extract individual data
streams for each channel or sensor, typically accelerometer data
and PPG heart rate data, if present.
[0074] At step 515 a spectrum analysis is applied to the PPG heart
rate data, if present, for a time interval, to calculate heart
rate. In one embodiment, a Fast Fourier Transform (FFT) is applied
to 8 second windows of data and the strongest spectral peak is
identified as the heart rate frequency. In one embodiment, this
step is repeated every second, with the FFT based on PPG data from
the the previous 8. Motion artifacts peaks are identified by
correlating spurious peaks with spectral peaks of accelerometer
data. The state of the art includes many methods for robustly
identifying heart rate even with motion artifacts.
[0075] After computing the user's heart rate, a heart rate score is
calculated at step 520 that reflects the increase in the user's
heart rate as a result of performing the exercise. The computation
of the heart rate score is described in further detail with
reference to Equation 2 hereinbelow.
[0076] The accelerometer data stream is passed to a detection
component 525 that has two steps: first a segmentation method 530
is performed and then a validation method 535 is performed.
[0077] At the start of a set of exercises, segmentation method 530
determines that the person is in the correct starting orientation
for a designated exercise. To accomplish this, segmentation method
530 attempts to identify a time period where the head orientation
is stable in the target starting position. For example, for squats
target position is head facing forward. A 0.5 second circular
buffer for each of the three axes is used to identify this
initialization period. The mean and range of the circular buffer
values is calculated every 0.1 second. If the range is below a
threshold and the overall mean direction vector is within a
threshold angle of the target direction then the starting
orientation is identified and subsequent data is analyzed for
individual exercise movement during the subsequent validation step
535.
[0078] The exact processing performed by segmentation method 530
depends on the specific exercise. As discussed hereinbelow with
reference to FIG. 7, in one embodiment a finite state machine (FSM)
is defined that includes one state for each of the characteristic
features of the user's head trajectory while performing the
exercise.
[0079] Validation method 535 is performed to determine that the
sequence of movements performed by the user during the exercise
movements are within a normal range, as discussed further
hereinbelow with reference to FIGS. 6-7.
[0080] Next, at step 540 a motion score is calculated for each
exercise performed by the user. The motion score is discussed in
further detail below with reference to Equation 1. Generally, the
motion score reflects the size or magnitude of the motion performed
by the user when performing a single repetition of an exercise.
Thus, for example, a deeper squat yields a higher motion score.
[0081] Finally, an overall exercise score is generated at step 545
that combines the heart score and the motion score. If heart rate
data isn't available then the motion score can be used as the
exercise score.
[0082] FIG. 6 illustrates a typical time series of accelerometer
measurements in the Y axis that correspond to a squat exercise
performed by a user. The characteristic head trajectory of this
exercise is in the up-down (Y) axis and so the analysis is carried
out in this axis. The shape of the accelerometer waveform is
valley, peak, valley and represents the motion signature for a
squat.
[0083] Indicator 6A refers to the pre-initialized state in which
the user is getting into the starting position. The user reaches
the starting position, at 6B, i.e. head facing forward for a squat.
A brief period, typically 0.5 sec is required in the starting
position with no motion, and acceleration at g (9.8 m/sec.sup.2).
Then the user starts to descend. This creates an acceleration trace
that first dips as the person moves down and total upward force
falls below g. The data values fall below threshold T1 and
eventually the decrease in acceleration reaches its lowest state at
6C. As the user slows their descent and starts to use their leg
muscles to push them back upward towards the bottom of the squat
the data output rises above threshold T2 and reaches a peak at 6D.
During this period there is a peak in the output data as overall
upward acceleration exceeds g. As the user rises again the Y axis
accelerometer data falls below T1 again to a trough at 6E. As the
user returns to the original position the data values rise above T1
again to reach approximately the starting value at 6F.
[0084] FIG. 7 illustrates the process of segmentation in which a
finite state machine (FSM) is used to identify the motion signature
of a squat. The transitions between the indicated locations in the
graph of FIG. 6 are characteristic features in a motion signature
of a squat and are represented as states in an FSM. As a user
executes a squat the Y axis accelerometer data is analyzed for
values that invoke successive state transitions in the FSM. The
characteristic features from the accelerometer data illustrated in
FIG. 6 that are identified and which trigger transitions from one
state to the next in an FSM that corresponds to a squat are: a
transition from the starting state 7A to 7B when the user moves
into the correct starting position and remains there for a brief
period of time, 0.5 seconds in one embodiment, a transition to
state 7C when the Y axis accelerometer data goes below threshold
T1, transition to state 7D when data goes above threshold T2,
transition to 7E at second trough wen the accelerometer data goes
below T1 again and a return to the starting position at state 7B
when second trough is exited and the data raises above T1 again. In
this way the FSM segments an exercise into a series of
characteristic features which taken in sequence form a motion
signature that is detected as a particular exercise such as a
squat.
[0085] Similarly for other exercises, characteristic features of a
motion signature for an exercise, which are typically peaks,
troughs and flats in all three axes are identified and each
characteristic feature is associated with a state within an FSM.
Press-ups are implemented in essentially the same way to squats
except in Z axis, ie forward axis. Jumping jacks also can be
tracked using a single axis approach. Lunges are similarly
implemented except motion must be tracked via FSM for both up/down
and forward backward motion. Sit-ups involves motion in both the Y
and Z axis. Details of the specific FSM used for sit-ups
segmentation is given below. Generally, the same methodology may be
applied to a broad range of exercises, with distinct movements
within a motion signature being mapped to FSM states. This range
includes but is not limited to sit-ups, press-ups, squats, front
lunges, jumping jacks, burpees, pull ups, side lunges, rotating
lunges, crunches, oblique crunches, squat thrusts, high jumps,
standups and tricep dips.
[0086] As the segmentation system processes the incoming data, a
circular buffer saves the previous 4 seconds of data. Time markers
are set for when the first state is entered and the last state
exited in the FSM. These markers are used to identify the segment
of data that represents the full exercise. This segment is copied
to a data buffer and provided to the validation and scoring
systems. Providing a segment to validation means that the detection
subsystem thinks a motion similar to the designated exercise has
just occurred and is represented by data in the provided
segment.
[0087] Validation method 535 validates that the head orientation of
each characteristic position in the motion signature for the
detected exercise occurs within pre-defined limits. Validation
method 535 will filter out segments if the exercise does meet the
limits; for example, a squat where the user doesn't descend low
enough or a burpee where the user didn't jump high enough. In
parallel with the operation of segmentation method 530 overall head
orientation angles are calculated. Head orientation is calculated
by lowpass filtering the accelerometer data. A moving average
filter of 0.5 seconds is applied to each of the x, y z tri-axial
accelerometer data streams to obtain trend direction. This filters
out any fast motion based component of the signal and leaves the
gravity vector direction. The resulting gravity vector direction
then gives the direction of the head as the orientation of sensor
device 1 is known relative to the head.
[0088] In the case of a squat, the pitch angle, roll angle, and
zenith angle (.phi.) (the angle made between the Up vector in the
head's reference frame and the Up vector in the earth's reference
frame) can be calculated from this gravity vector ascertained from
the accelerometer data using trigonometry methods that are well
known in the art. When the users head is vertical and straight then
roll, pitch and zenith angle will all be zero.
[0089] Circular buffers of pitch, roll and zenith angles are saved.
On segmentation start these buffers are cleared. At the end of a
segmentation method, the angle buffers are also sent to validation
method 535.
[0090] Validation method 535 takes the accelerometer buffer as well
as the pitch, roll and zenith buffers as inputs once segmentation
completes. For squats, validation method 535, calculates an
approximate measure of the depth of the squat and thresholds this
against a minimum level required to validate as an acceptable
squat. In one embodiment, this depth value can be approximated by
measuring D, the range of Y axis accelerometer values over the
exercise time, i.e. the vertical height between the peak, 6D, and
lowest trough, 6C. In another embodiment, this depth value is
approximated by calculating A the area of the triangle formed
between the points 6C, 6D and 6E.
[0091] The head pitch, roll and zenith angles during the exercise
are examined both in terms of mean head rotation as well as the
head pitch and roll range (i.e. max-min) during the exercise. If
the head angles are outside the target range or the head angles
change too much during the duration of the exercise the exercise is
considered invalid. In one embodiment, both the mean and range of
the head roll and pitch angles are compared over the time of the
exercise. In this embodiment, mean angles need to be within 20
degrees of horizontal. And the angle range (or change in angle)
over the exercise needs to less than 15 degrees.
[0092] As part of validation method 535, the duration of the
exercise is also tested. If the duration is below a minimum time or
above a maximum time the exercise is considered invalid.
[0093] The motion score computed for each repetition of an exercise
is dependent on the extent and size, or magnitude, of the motion
performed by the user when performing the repetition. For example,
a deeper squat gives a higher motion score. Similarly a press-up
that goes through a large up and down motion is given a higher
score. For sit-ups, rotating the body through a larger angle gives
a bigger score.
[0094] For squats, the motion score (S.sub.M) is calculated by
Equation 1 below:
S.sub.M=A-g*Sin(.parallel..PHI..sub.range|) (Equation 1)
[0095] The value A is the estimate of depth of the squat that is
calculated by getting the area of the triangle formed by the peak
and adjacent troughs, such as the points 6C, 6D and 6E in FIG. 6.
The second term includes .phi..sub.range, the zenith angle range
during the exercise, this is subtracted to remove the potential
effect of head motion during the exercise thus improving the motion
score. Typically this score ranges from 7 m/s.sup.2 for a shallow
squat to 16 m/s.sup.2 for a deeper squat. In another embodiment,
the motion score can be calculated based on the squat depth
estimate made by calculating the height of the peak above the first
trough.
[0096] In one embodiment, the heart rate is calculated repeatedly
while the exercise is being performed. For each exercise the heart
rate score (S.sub.H) is given by Equation 2 as:
S.sub.H=K*(HR-HR.sub.BASE) (Equation 2)
[0097] The heart rate baseline, HR.sub.BASE, is measured during a
calibration stage when the user first uses the system. Each
exercise is given its own context specific weighting to allow the
system to avoid being skewed by cardio intensive exercises. For
example jumping jacks will increase heart rate more than press ups
but aren't as strong for building arm muscles.
[0098] The heart and motion scores are combined to give an overall
exercise score (S.sub.E) as given below by Equation 3.
S.sub.E=S.sub.M*S.sub.H (Equation 3)
[0099] A set consists of a specific number of repetitions of a
single exercise. The set score (S.sub.TOTAL) is the sum of exercise
scores over a set, as given below in Equation 4.
S TOTAL = i = 1 reps S E ( Equation 4 ) ##EQU00001##
[0100] A total workout score may be computed as the overall sum of
the set scores across all sets of exercises performed during a
workout. In this way the sensor and exercise detection system can
be used as part of an interactive workout system. Workouts can be
created by the user or selected from pre defined sets of exercises.
They can be relatively light for beginners or more intensive for
more experienced or fitter users.
[0101] FIG. 8 shows an example embodiment of a discrete exercise
user interface for exercise application 320 that is used for
discrete exercises. An exercise list panel 810 at the bottom of
user interface 800 shows a list of exercises. The current exercise,
being performed by the user, is given at the top of the list, in
this case "Forward Lunges." The next exercise to be performed,
"Squats", is shown in the next position. At the bottom, an exercise
that has been performed, "Crunches", is shown. A status panel 820
shows the elapsed workout time and the user's current heart rate. A
progress panel 830 in the upper portion of user interface 800
provides information about the exercise in progress, forward lunge.
The circular area in the center with the number "8" shows the
number of lunges left to perform in the current set.
Sit-Up Segmentation
[0102] FIG. 9 illustrates an exemplary time series of measurements
taken by an accelerometer in sensor device 1, attached to the head
or torso or a user, while the user is performing a sit-up exercise.
Measurements for the X axis and Y axis are illustrated. The basic
motion of this exercise is that the head moves through a roughly 90
degree arc of motion and returns to where it started, as
illustrated in FIG. 4B.
[0103] Initially, the user starts from a standing position,
indicated as 9A. The accelerometer output reflects that gravity
vector is aligned with the Y axis, i.e. standing and looking
forward. The X axis at 9A is near zero and on the Y axis (9A) is
near to g, i.e. 9.8 ms.sup.-2). When the user gets on the floor to
start a sit-up the accelerometer data changes to reflect the fact
that the gravity vector is aligned with the X axis (in its negative
direction as the gravity vector is pointing backwards from the
head). The point 9B is near -9.8 ms.sup.-2. As the user starts to
lift their head off the ground the X Axis accelerometer data dips
below -g (-9.8 gms) at point 9C. The head then continues to move
through an arc of approximately 90 degrees. During this motion the
X axis accelerometer data increases from below -g to around
approximately g in value. The peak X axis acceleration data occurs
at point 9D. As the user then descends back to the original
position the x axis returns to around -g at point 9E and 9E'. This
completes one complete cycle, or repetition, of the sit-up.
[0104] FIG. 10 illustrates a finite state machine (FSM) used to
segment accelerometer data into a sequence of characteristic
features that correspond to a sit-up. Generally, segmentation
method 530 includes variations on a general method to segment data
for various discrete exercises including squats, sit-ups, etc. As
discussed with reference to the squat, this approach maps
individual states of an FSM to characteristic features of the
exercise being analyzed, in this case, a sit-up. As illustrated in
FIG. 10 a 5 state FSM is applied with the first state representing
standing up before starting the sit-ups and 4 states representing
different stages in the sit-up. Each of the states in the FSM of
FIG. 10 refer to the states or locations in the accelerometer data
stream shown in FIG. 9.
[0105] State 9A represents the stage when a user is not yet lying
on the floor ready to start sit-ups (i.e. 9A in the X axis
accelerometer data graph). When the user gets on the floor the
state transitions to 9B. The orientation of the head of the user is
tested using the vectors defined by the X and Y axis data and the
transition from 9A to 9B is triggered when the X, Y axis
accelerometer data approaches (0, -g).
[0106] The transition from 9B to 9C occurs when the head raises off
the ground. This causes X axis data to dip below -g. This event is
triggered by comparing the x-axis value against a threshold.
[0107] The transition from 9C to 9D occurs when the head reaches
the apex of the sit-up motion. Again, the transition is triggered
by thresholding the X axis value.
[0108] If the threshold value is not reached within a specific time
frame the FSM will transition from 9C to 9B (9B being the "reset"
position where the user's head is on the ground).
[0109] As the user descends, his/her head returns to the starting
position which triggers the transition from state 9D to 9E. At this
stage the X axis dips a little under -g. As data returns towards -g
the FSM transitions back to the 9B and is ready to begin the cycle
again.
Burpee Segmentation
[0110] FIG. 18 illustrates an exemplary time series of measurements
by an accelerometer during a burpee exercise for both the X axis
and Y axis accelerometer values. The basic motion of this exercise
is that the person starts from a standing position and then touches
the ground and kicks out their legs so as to be in a pushup
position. The person then pulls their legs back in and jumps
upwards to land back in the starting position.
[0111] Initially a user starts from a standing position, indicated
as 18A in FIG. 18. The accelerometer output reflects that gravity
vector is aligned with the Y axis, i.e. standing and looking
forward. The X axis at 18A is near zero and on the Y axis 18A' is
near to g, i.e. 9.8 ms.sup.-2). When the user begins the burpee by
descending to the floor the accelerometer data changes to reflect
the fact that the gravity vector is aligned with the X axis (in its
positive direction as the gravity vector is pointing forwards from
the head). The Y axis at point 19B is near 0 ms.sup.-2. As the
person kicks their legs out a spike in the y axis output is seen at
point 19C. As the person then jumps back upwards a trough is seen
in the y axis output at point 19D. When the person lands again in
the standing position, 19E, they are ready to begin another
repetition of the exercise.
[0112] The characteristic movements within the burpee can be
captured via an FSM where state transitions occur when thresholds
are reached in the accelerometer data. In a similar manner to the
sit-up segmentation each state in the FSM will represent a
different stage of the motion. If all the specific movements are
performed correctly all states of the FSM will be passed through
and the motion will be assigned as a burpee exercise and further
analyzed in the validation stage to insure factors such as the
height of the jump are large enough to confirm the motion as a
valid burpee.
General Segmentation Method
[0113] In one embodiment all exercises utilize the segmentation
approach described with reference to the squat (FIG. 7) and the
sit-up (FIG. 10) to map states of an FSM to characteristic features
of the exercise.
[0114] FIG. 11 illustrates the general structure of a finite state
machine (FSM) used to segment discrete exercises. While FIG. 11
illustrates in FSM with 7 states, the general FSM is an N+1 state
machine. With 1 state representing an uninitialized state (i.e. a
user has not yet reached the starting position for the exercise)
and N states connected in a cyclical fashion representing different
characteristic stages of the motion.
[0115] Thresholding values of accelerometer data (X, Y or Z axis)
are used to trigger transitions from one state to the next. If
these thresholds are not achieved within a specific time frame, or
if a transition takes too long, the FSM transitions back to the
reset state (state 2 in FIG. 11). In this way discrete exercise
motions can be detected and validated based on accelerometer data
measurements taken at the head or torso of a user.
Tracking Continuous Exercises
[0116] The invention can be used to track details of a gym-based
exercise session. The most common cardio gym equipment used to
perform continuous exercises are detected and scored by the
invention. These machines include treadmill, rowing machine and
elliptical trainer (or cross trainer), exercise bike and stepper.
When a person exercises on each of these machines the patterns of
forces acting on their heads are strongly correlated with each
machine type. This enables sensor device 1 in combination with
signal processing engine 310 to be used to identify and track gym
based workout sessions that use these common items of gym
equipment.
[0117] FIG. 17 illustrates an example of Y axis accelerometer data
received from sensor device 1 attached to the head of a user
running on a treadmill.
[0118] As with discrete exercises the invention can be used in two
modes of operation for monitoring user performance on cardio gym
equipment: automatic and interactive. In automatic mode, the
invention automatically detects which machines were used in a gym
session, after the session terminates, based solely on the recorded
accelerometer data.
[0119] In interactive mode, an application, such as exercise
application 320, prescribes a gym session to be performed by a
user. An example may be 20 minutes on the treadmill, 10 minutes on
the rowing machine and 10 minutes on the cross trainer. In this
mode the user is presented real-time feedback about their session
in progress. An example of such real-time feedback on progress is a
countdown timer showing how much time is left on the current
machine. This timer will pause and restart as users stops and
starts on a gym machine such starting and stopping pedaling on an
exercise bike.
[0120] The detection of which gym machine is being used is based on
the trajectory of head motion. For example, a rowing machine
creates mainly horizontal head motion backwards and forwards. On an
exercise bike the repetitive motion is mainly small head motions
side to side. A treadmill and an elliptical trainer both create
mainly vertical up and down head trajectories. A key difference
between these two is that the strength of the impact force on a
treadmill is stronger for a given pace than for an elliptical
trainer and so can be distinguished by analyzing the shape of the
acceleration data over the course of a single cycle of motion. This
can also be distinguished by the first differential of the y axis
accelerometer values which for a treadmill are usually greater than
twice as large as the elliptical machine values due to this larger
impact force.
Tracking Continuous Exercises: Automatic Mode
[0121] The subject invention employs spectral methods of analysis
to detect that a gym machine is being used. In automatic mode the
session acceleration data is recorded for analysis after the
session.
[0122] The first goal of the automatic mode analysis is to
ascertain at which times during the gym session repetitive motion
is occurring, i.e. if the user is performing a repetitive exercise
of some type such as running or rowing. Once this initial
segmentation of the total session into regions of periodic motion
is performed each segment is then classified as a specific exercise
type based on parameters calculated from the accelerometer x, y, z
values between the start and end of that segment.
[0123] FIG. 12 is a flow diagram that illustrates one embodiment of
a method 1200 implemented by continuous exercise component 314 for
analyzing gym sessions and automatically identifying which gym
machines were used. In this embodiment, sensor data is received
from sensor device 1 and recorded by mobile device 12 at step 1205.
At step 1210 the accelerometer x,y,z values are detrended using a
moving average filter and then an FFT, or other spectral analysis
technique, is applied to 5 second windows of each of the
accelerometer axes. In one embodiment, there is a 2 second overlap
between consecutive windows.
[0124] At step 1215 the FFT spectrum for each window is then
analyzed using a peak detection method to find the largest spectral
peak in each accelerometer axis.
[0125] At step 1220 the energy in the largest spectral peak is
compared to the total FFT energy. Regions of highly periodic motion
will have a high percentage of total energy in the largest spectral
peak. In one embodiment the total percentage energy in the peak is
taken as the energy in 3 FFT bins given a sampling rate of 55 Hz
and an FFT window size of 256. The three bins include the central
bin of the peak together with the bin on either side. Peak energy
percentages above 55% will robustly identify regions of periodic
motion during a gym session for all exercise machines. This value
of 55% was used as the threshold to assign segments as either
periodic or non-periodic regions.
[0126] FIGS. 13A-C show a visualization of steps 1210, 1215 and
1220 applied to an example gym session data set. FIG. 13A shows the
recorded gym session y axis accelerometer data. FIG. 13B shows the
percentage energy in the largest spectral peak. A line 1310 shows
the 55% threshold line. FIG. 13C shows the segmented regions which
were assigned as periodic and therefore exercising segments.
[0127] At step 1225 in FIG. 12 the periodic segments are classified
as one of 5 types of gym machine. These types are TREADMILL,
ELLIPTICAL, BICYCLE, STEPPER and ROWING. For each periodic region a
number of parameters are calculated and used to compare against a
set of known threshold values to help define that region as one of
the exercise types.
[0128] The parameters measured are FFT peak amplitude, FFT peak
frequency, Maximum first differential of accelerometer values in
the 10 second window and the segment motion vector. Parameters are
measured for each 10 second window over the time of the periodic
segment. The parameters are averaged over the segment time for each
periodic section.
[0129] The segment motion vector is the main direction of head
motion during the periodic segment. For a treadmill or elliptical
machine this is mainly up and down (i.e. strongly aligned the Y
axis), for a rowing machine this is mainly forward and backwards
(i.e. strongly aligned with the X axis), for an exercise bike this
is mainly side to side (i.e. strongly aligned with the Z axis).
This segment motion vector is created from of values in the X axis,
Y Axis and Z axis FFT values at the spectral peak frequency. This
is then normalized to unit length.
[0130] Table 1 below provides one embodiment of the parameter
thresholds used to classify periodic segments as exercise
types.
TABLE-US-00001 TABLE 1 Periodic Region Classification Table
Exercise Alignment Spectral Peak Spectral First Type Axis Frequency
Peak Size Differential TREADMILL Y Axis High High High ELLIPTICAL Y
Axis High High Low STEPPER Y Axis Low Low Low ROWING X Axis Low Low
Low BICYCLE Z Axis High Low Low
[0131] As can be seen from Table 1, above, a segment which has
mostly Y axis (up and down) motion would be Y Axis aligned would be
assigned as either TREADMILL, ELLIPTICAL or STEPPER.
[0132] The peak frequency threshold is chosen as 1.7 Hz, above this
is considered High and below is Low. If a segment is Y Axis aligned
but has a low spectral peak frequency under 1.7 Hz this segment is
classified as a STEPPER machine as it is too slow to represent
treadmill or elliptical machine head motion. The spectral peak size
(height) threshold is taken as twice the peak size measured in the
Y axis when walking at normal pace. Above this is considered High
and below this Low. The first differential of accelerometer values
uses a threshold of 4 ms.sup.-3, above this is High and below this
is low. Stage 1225 uses these thresholds and the above table and
applies this methodology of threshold comparison to each periodic
segment to assign it uniquely to a specific machine.
[0133] At step 1230, the output of the method 1200 is a gym session
summary that includes list of time segments together with their
related exercise class. An example being for a 45 minute gym
session the person was on the treadmill from minute 3 to minute 22
then on the rowing machine from minute 24 until minute 34 and on
the elliptical from minute 35 until minute 45. In other embodiments
further information such as average pace and average heart rate per
exercise machine can be provided.
[0134] FIG. 14 shows an example embodiment of a gym session user
interface 1400 that is presented by exercise application 320. It
includes a panel 1410 that graphs the heart rate during the
session. A panel 1420 shows the session time and the average heart
rate, in beats per minute (BPM), of the session. A panel 1430 shows
the minutes spent on a treadmill, on a rowing machine and on an
exercise bike.
Tracking Continuous Exercises: Interactive Mode
[0135] In interactive mode, a specific gym workout is prescribed to
the user via an application, such as exercise application 320,
running in mobile device 12. Spectral analysis methods are used to
determine if the user has started or stopped performing a specific
exercise for a sequence of exercises. For each exercise a specific
amount of time is set. In one embodiment of the invention a user is
shown a countdown clock showing how much more time is left for that
specific exercise. If they stop performing the exercise the
countdown clock pauses; once they restart performing the exercise
the countdown clock begins again.
[0136] Each exercise can also be scored both on motion and heart
activity. FIG. 15 illustrates an example of the spectrum output for
accelerometer data from sensor device 1 attached to a user running
on a treadmill. As illustrated the spectrum has a main peak at the
running frequency, just under 3 Hz, i.e. the user's running motion
repeats nearly three times a second.
[0137] FIG. 16 is a flow diagram that illustrates one embodiment of
a method 1600 implemented by continuous exercise component 314 for
detecting and scoring continuous exercises. As with discrete
exercise component 312, the method starts by receiving sensor data
at step from sensor device 1. The sensor data includes
accelerometer data and may include PPG heart rate data. At step
1610 the incoming stream of time series sensor data is parsed into
PPG heart rate data and accelerometer motion data streams.
[0138] If heart rate data is present then at step 1615 the user's
heart rate over an interval is calculated. Then, at step 1620, the
heart rate of the user may be used to score the run based on
increase above baseline heart rate (as measured during a
calibration stage) using Equation 5 below:
S.sub.H=K*(HR-HR.sub.BAsE) (Equation 5)
[0139] At step 1625 an FFT is used to obtain the spectrum for an
interval of motion data. Then, at step 1630 the power in the
spectral peak (P.sub.peak) as well as the frequency of the peak
(F.sub.peak) is used to score the running activity. The motion
score for a time interval (S.sub.M) is calculated using the
Equation 6 below:
S.sub.M=P.sub.PEAK*F.sub.PEAK (Equation 6)
[0140] At step 1635, the motion and heart rate scores can be
combined to generate an overall interval score (S.sub.i) for a 10
second interval of running, as given by Equation 7 below:
S.sub.i=S.sub.M*S.sub.H (Equation 7)
[0141] A full running session on the treadmill is scored through
the summation of the interval scores, S.sub.i, over the duration of
the run, as given in Equation 8 below. N represents the total
number of 10 s intervals in the treadmill run session.
S TOTAL = i = 1 N S i ( Equation 8 ) ##EQU00002##
[0142] The same methodology is used to score performance for
treadmill, cross trainer, rowing machine, stepper machine and
cycling machine. Gym sessions can then be given an overall score
(summing each machine score) and an overall gym performance score
given. In this way gym sessions can be tracked over time for
performance.
Tracking Head Bob and Gait Analysis During Running
[0143] The above methods for gym machines can be applied in a
similar way for outdoor running, rowing, cycling and cross country
skiing with performance tracked and scored over time based on the
heart and motion data from head worn sensor device 1.
[0144] In addition to raw scoring of overall performance more in
depth data on style and form can be ascertained from the sensor
data.
[0145] In terms of running motion, the data from the accelerometer
can be analyzed to yield an overview of the gait cycle of a runner.
The gait cycle during a running is split into contact time
(T.sub.C) and flight time (T.sub.f). The output from commercially
available accelerometers typically provide high enough resolution
output to ascertain both T.sub.C and T.sub.f.
[0146] FIG. 16 graphs an example of the vertical forces measured by
an accelerometer in a sensor device attached to the head of a user
while running. The time where the force is less than G occurs when
the runner is in the air in free-fall. The time between take off
point A to landing point B in the graph is T.sub.f and the time
between B and next take off point C is T.sub.C.
[0147] The ratio of T.sub.C and T.sub.f can be used to score
running style for the person. More expert runners will have higher
ratios where novice runners will have lower ratios. The average of
this ratio for each 60 second time interval of running is
calculated below in Equation 9.
S Flight = i = 1 N T f ( i ) / T c ( i ) N ( Equation 9 )
##EQU00003##
[0148] Asymmetry in running style can also be calculated through
comparing alternate step cycles. By comparing peak force for both
left and right stride over intervals of the run can give levels of
asymmetry. Typically, expert runners will have an asymmetry score
(S.sub.Asymmetry) close to 1 whereas novice runners can have
asymmetry scores greater than 1.05 (stronger force on right foot
stride) or less than 0.95 (stronger on left foot stride), as
calculated in equation 10 below.
S Asymmetry = F PEAK ( RIGHT ) F PEAK ( LEFT ) ( Equation 10 )
##EQU00004##
[0149] Head bob can also give indication of running style. The head
worn sensor can give indication of how much the head "bobs" both
forward and backward through the run as well as side to side
motion.
[0150] The rotation of the head back and forward though the gait
cycle can be measured either though a gyro or based on
accelerometer output. Side to side motion in the axis perpendicular
to motion is also a key metric of head bob. A head bob score for an
interval is given below by Equation 11.
S.sub.HEADBOB=P.sub.range.times.F.sub.range (Equation 11)
[0151] Where P.sub.range is the average head pitch angle range over
the gait cycle, ie forward and backwards rotation, and F.sub.range
is the average range of acceleration values in the x direction over
a gait, i.e. side to side head bob motion.
[0152] The above scores for gait flight, asymmetry and head bob for
each 60 second interval of a run are calculated and can be
visualised in graphs post run giving the runner a lot of extra
information about their running style and information on how to
improve.
[0153] In combination with the interval averages as outlined above,
the standard deviations of all the above metrics are also
calculated. This gives a measure of how variable the running style
is during the run.
[0154] The standard deviation of gait flight time over the interval
is also calculated using Equation 12, below.
.sigma..sub.Flight=.sigma.(T.sub.f(i)/T.sub.C(i)) (Equation 12)
[0155] The standard deviation of asymmetry time over the interval
is also calculated using Equation 13, below.
.sigma..sub.Asymmetry=.sigma.(F.sub.Peak(Right)/F.sub.Peak(Left))
(Equation 13)
[0156] The standard deviation of head bob over the interval is also
calculated, using Equation 14 below.
.sigma..sub.HEADBOB=.sigma.(P.sub.range.times.F.sub.range)
(Equation 14)
[0157] The above three metrics are calculated repeatedly over 60
second intervals of a run. An expert runner would expect to see low
values for these metrics until they reach a fatigue point within
their run. This point will be seen within the data output as the
point at which these values standard deviation values start to
increase as running style becomes more erratic and noisy. By
providing this data to runners gives them rich data set to
understand their running style and so allow them to adapt and
improve over time.
[0158] It should be clear that the above methodology can be equally
be applied to other activities such as rowing or cycling to give
athletes more in depth data about their performance and style and
so allow them to improve their technique over time.
[0159] It may be further understood that different equations to
calculate these measures may be used, and different intervals may
be used without departing from the scope or spirit of the subject
invention.
Position and Trajectory Tracking in Yoga
[0160] The sensor can also be applied to head motion tracking
during yoga positions. Yoga moves have specific orientations that
need to be held for specific time periods. These can be tracked as
head orientation can be calculated (as in equation 3 above
above).
[0161] Overall stillness can be calculated by the standard
deviation of change of accelerometer output, as given in Equation
15 below.
S.sub.Stillness=.sigma.(.DELTA.Acc) (Equation 15)
[0162] Yoga positions can be scored based on the person getting
into the correct position (as measured at the head) and keeping
stillness core under a specific threshold for the required amount
of time.
Position and Trajectory tracking in Golf and Tennis
[0163] The sensor can also be applied to other sports where
specific motion of the head has bearing on overall performance.
[0164] In a golf swing it is important that the head is kept
relatively still and oriented towards the ball. Once contact with
the ball is made the head should move smoothly around to allow eyes
to track the ball trajectory.
[0165] The golf swing data can be segmented into two sections. One
is pre ball impact and the other post ball impact. The time at
which the ball is hit can be clearly seen in the output data as a
short sharp impulse sent though the body.
[0166] During the backswing the head will usually translate and
rotate back in the direction of the backswing. Some motion is
needed to achieve ideal body stance but too much motion here is bad
for overall efficiency of swing.
[0167] The amount of head rotation and head translation are
measured using the sensor output. The golf swing can be scored by a
combination of total head motion and smoothness of motion after
contact. A common issue with golf swing is moving the head on the
backswing. This is the critical time and so is weighted higher in
overall score.
[0168] Once the data is segmented the stillness of the head is
measured for the pre impact stage using Equation 16 below.
S.sub.stillness=.sigma.(.differential.Acc) (Equation 16)
[0169] The movement of the head post impact should be smooth and to
the left. The score can be presented to the user for each gold
swing and give indication of performance as well as indicators of
where swing motion can be improved.
[0170] Using a similar method to that described hereinabove with
respect to golf, the motion of the head during a tennis serve can
be analyzed and scored based on the output of the ear worn
sensor.
[0171] The tennis serve goes through 5 distinct stages with respect
to head motion.
[0172] Stage 1: Initial preparation stage looking down the line
where the ball should be sent.
[0173] Stage 2: Ball release stage where the player throws the ball
the air and the head tracks the upward motion of the ball.
[0174] Stage 3: The backswing stage as the head moves back a little
as the arm swings back prior to forward swing.
[0175] Stage 4: Forward swing and contact. As the player swings the
racket foreward to hit the ball the head start to rotate forward to
look down the line again.
[0176] Stage 5: Post contact swing: After contact the player gets
head looking fully forward in preparation for return of ball.
[0177] The output of the head worn sensor is used to segment the
serve motion into these 5 stages and give timing and scoring
information for each stage.
[0178] The relative timing of each stage and the smoothness of
motion between each the main steps are used to score the overall
serve.
Other Embodiments of the Invention
[0179] The subject invention involves tracking motion and heart
data at the head, and preferably the ear, and has been initially
implemented as an ear worn sensor device 1 attached to the ear
using a clip as illustrated in FIG. 1.
[0180] Another embodiment of the current invention allows heart
rate be measured without the requirement of the ear clip by using
reflectance PPG measured at the skull behind the ear. This in
effect would create a device that would incorporate element 2 from
FIG. 1 but not element 3 the ear clip. The PPG signal measured
behind the ear here is not as high quality as at the earlobe but
still includes a strong enough heart signal to allow heart rate
calculation.
[0181] Another embodiment of the subject invention is the
incorporation of the principal components of sensor device 1 into
headphones, a helmet or hat, or a pair of glasses. In these
embodiments, the sensor electronics are built into the respective
device. Heart data can be measured through a connected PPG sensor
(at ear lobe or behind ear). The above system again wirelessly
sends data to a connected client device such as smart phone.
[0182] In yet other embodiments of the subject invention, the above
headphones, glasses or helmet, include on-board display abilities
so feedback is given directly to the user via this display. These
embodiments might integrate certain features and components of
mobile device 12 such as a processor and memory. For example,
glasses exist in the state of the heart with HUD (heads up display)
abilities where information can be presented to the person via
information projected onto the glasses.
[0183] Another embodiment of the subject invention is the
incorporation of the device into earphones. Since PPG data can be
measured via reflective PPG within the ear canal a sensor device
may be integrated within an earbud-style earphone. The additional
placement of a 3 axis accelerometer within the earbud would allow
motion and heart data to be measured at the head and so allow
tracking and scoring as defined above.
[0184] Another embodiment of the current invention is a system that
only uses motion data without heart data. This removes the
requirement for a PPG sensor. The output scores to the applications
are motion scores only.
[0185] Another embodiment of the current invention is the
incorporation of the device into an ear ring at the earlobe. This
embodiment consists of an ear ring that includes the components of
the exemplary architecture of FIG. 2. This results in a device with
single component rather than two parts as indicated in FIGS.
1A-B.
[0186] The above specification, examples, and data provide a
complete description of the manufacture and use of the composition
of the invention. Since many embodiments of the invention can be
made without departing from the spirit and scope of the invention,
the invention resides in the claims hereinafter appended.
* * * * *