U.S. patent application number 13/027157 was filed with the patent office on 2012-03-29 for 3d monocular visual tracking therapy system for the rehabilitation of human upper limbs.
This patent application is currently assigned to INSTITUTO NACIONAL DE ASTROFISICA OPTICA Y ELECTRONICA. Invention is credited to Gildardo Azcarate-Hernandez, Roger Luis-Velasquez, David Reinkensmeyer, Ronald Sturat-Leder, Luis Enrique Sucar Succar.
Application Number | 20120077163 13/027157 |
Document ID | / |
Family ID | 44366947 |
Filed Date | 2012-03-29 |
United States Patent
Application |
20120077163 |
Kind Code |
A1 |
Sucar Succar; Luis Enrique ;
et al. |
March 29, 2012 |
3D MONOCULAR VISUAL TRACKING THERAPY SYSTEM FOR THE REHABILITATION
OF HUMAN UPPER LIMBS
Abstract
It is described a 3D monocular tracking system, being robust,
having low cost, easy to install and use, useful for the upper
limbs rehabilitation in a patient in need thereof, as well as a
home self-directed therapy method for patients having upper limbs'
movement disability. The system comprising a) a handle or gripper;
b) a computational vision system comprising a video camera; c)
software comprising a set of games; d) a processor; and, e) a
display apparatus.
Inventors: |
Sucar Succar; Luis Enrique;
(Puebla, MX) ; Luis-Velasquez; Roger; (Puebla,
MX) ; Azcarate-Hernandez; Gildardo; (Acapulco,
MX) ; Sturat-Leder; Ronald; (Delg. Coyacan, MX)
; Reinkensmeyer; David; (Irving, CA) |
Assignee: |
INSTITUTO NACIONAL DE ASTROFISICA
OPTICA Y ELECTRONICA
|
Family ID: |
44366947 |
Appl. No.: |
13/027157 |
Filed: |
February 14, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61337965 |
Feb 12, 2010 |
|
|
|
Current U.S.
Class: |
434/247 |
Current CPC
Class: |
A61H 1/00 20130101; A61B
5/1114 20130101; A63B 23/16 20130101; A63B 71/0622 20130101; G16H
20/30 20180101; A61H 1/0274 20130101; A61H 2201/5092 20130101; A63B
21/0004 20130101; A63B 21/4035 20151001 |
Class at
Publication: |
434/247 |
International
Class: |
A63B 69/00 20060101
A63B069/00 |
Claims
1. A 3D monocular tracking system for the rehabilitation of a
patient's upper limbs, characterized by comprising: a) a handle or
gripper comprising a cylindrical gripper and two colored spheres,
one at each end of the gripper (FIG. 1); b) a computational vision
system comprising a video camera to determine, through the tracking
of the handle balls, the position and 3D movement of the patient's
arm and/or, as well as its rotation around 3 orthogonal axis, c) a
Gesture Therapy software comprising a program for 3D tracking of
the two colored spheres in the handle, and a set of games oriented
to daily life activities, which include different difficulty levels
to evaluate the patient's progress at every game; d) a processor
configured for: selecting between the set of games, an appropriate
game to the patient; calling the game; providing the required
algorithms to determine the position and 3D movement of the
patient's hand; recognizing the position and 3D movement of the
patient's arm and/or hand; controlling the game, based on the
position and 3D movement of the patient's arm and/or hand; adapting
to the patient according to his (her) condition in the therapy
progress; automatically alternating, for example, the working space
for the limb's movement; detecting the pressure when the patient
tights/loses the joystick by a pressure sensor incorporated
therein, thereby promoting the rehabilitation of the hand's
movements. e) a display apparatus configured to show 3D virtual
environments that the patient can interact.
2. A system, according to claim 1, further characterized in that
the handle or gripper comprises one or more pressure or strength
sensors allowing to determine when the patient tightens the handle
and the force with which is performed.
3. A system, according to claim 1, further characterized in that
the video camera is selected from a web camera, a processor
integrated camera (computer), an infrared camera.
4. A system, according to claim 3, further characterized in that
the video camera is a web camera.
5. A system, according to claim 1, further characterized in that
the display apparatus is selected from a computer screen, a
television monitor, a digital personal assistant screen (PDA, by
its English initials), a cellular phone screen.
6. A system, according to claim 5, further characterized in that
the display apparatus is a computer screen.
7. A system, according to claim 1, further characterized in that
the patient has suffered from stroke.
8. A home self-directed therapy method for patients having upper
limbs' movement disability, preferably one having suffered from
stroke, characterized by comprising: offering a patient a monocular
3D tracking system for the rehabilitation of a patient's upper
limbs; selecting, from the set of games, an appropriate game for
the rehabilitation of the patient; calling the game; recognizing
the position and 3D movement of the patient's arm and/or hand;
showing 3D virtual environments, visible for the patient, wherein
the 3D virtual environments respond to the 3D movement of said
patient's arm and/or hand; adapting to the patient according to his
(her) condition and therapy progress; automatically alternating,
for example, the working space for the limb's movement; repeating
the above steps during the necessary period of time.
9. A method, according to claim 8, further characterized in that
the system comprises: a) a handle or gripper comprising a
cylindrical gripper and two colored spheres, one at each end of the
gripper (FIG. 3); b) a computational vision system comprising a
video camera to determine, through the tracking of the handle
balls, the position and 3D movement of the patient's arm and/or
hand, as well as its rotation around 3 orthogonal axis (FIG. 1 and
FIG. 4); c) a Gesture Therapy software comprising a set of games
oriented to daily life activities, which include different
difficulty levels to evaluate the patient's progress at every game;
d) a processor configured for: selecting, from the set of games, an
appropriate game to the patient; calling the game; providing the
required algorithms to determine the position and 3D movement of
the patient's hand; recognizing the position and 3D movement of the
patient's arm and/or hand; controlling the game, based on the
position and 3D movement of the patient's arm and/or hand; e) a
display apparatus configured to show 3D virtual environments that
the patient can interact.
10. A method, according to claim 9, further characterized in that
the handle or gripper comprises one or more pressure or strength
sensors allowing to determine when the patient tightens the handle
and the force with which is performed.
11. A method, according to claim 9, further characterized in that
the video camera is selected from a web camera, a processor
integrated camera (computer), an infrared camera.
12. A method, according to claim 11, further characterized in that
the video camera is a web camera.
13. A method, according to claim 9, further characterized in that
the display apparatus is selected from a computer screen,
television monitor, digital personal assistant screen (PDA, by its
English initials), cellular phone screen.
14. A method, according to claim 13, further characterized in that
the display apparatus is a computer screen.
15. A method, according to claim 8, further characterized in that
the patient has suffered from stroke.
Description
TECHNICAL FIELD
[0001] The present invention relates to equipment and systems used
in rehabilitation medicine to recover limb motility for patients
suffering from cardiovascular diseases or other kinds of disease,
and more particularly, it is related to a 3D monocular visual
tracking therapy system for the rehabilitation of the upper limbs
of patients having suffered from any kind of injury such as a
stroke, as well as to the method of carrying out such
rehabilitation.
BACKGROUND OF THE INVENTION
[0002] Every year millions of people worldwide suffer accidents or
diseases which cause the loss of their motor abilities.
Cerebrovascular diseases, commonly known as strokes, are clear
examples thereof. About 80% of the people which survive to a stroke
lose their movement ability in an arm and hand.
[0003] After having suffered from a stroke, an intensive activity
therapy for several weeks is the most common treatment to recover
the movement abilities. However, due to the increased budgetary
pressures in the hospitalization system, more often the
rehabilitation treatments are reduced and the patients are sent
home early without having reached a correct and vital
rehabilitation. On the other hand, hiring a professional
physiotherapeutic is not an option for most of the patients due to
the high cost of the therapy session.
[0004] In view of the above, patients try to rehabilitate
themselves by making the same exercises assigned at the hospital;
however, as they do not have the guides neither have they the
knowledge thereto, usually they exercise in an inappropriate
manner, resulting in a low or null progress in their
rehabilitation. In other cases, due to laziness or lack of
motivation, the patients do not exercise at all.
[0005] Accordingly, a cost-effective solution for a self-directed
therapy system for the home is necessary so a higher number of
patients can have it.
[0006] In order to help patients to recover their higher limbs'
movement, either of the arm or the hand, several systems have been
developed. Most of these systems consist in complex robotic arms or
gloves having sensors to detect the patient's arm movement, making
them expensive and not accessible for most of the patients.
[0007] Due to the great advances in computer technology,
computational vision has now a wide variety of applications, within
we can mention human-machine interface applications, virtual
reality, animation and movement capture, including applications of
increased reality or of tracking. In this context, rehabilitation
systems based on the visual tracking of the human movement have
become another alternative for the patients, which may be based on
marks, wherein the image is captured by cameras, tagging the body's
joints, or alternatively, free of tags using conventional video
cameras to capture the movement.
[0008] The combination of several factors is required for the use
of the systems based on visual tracking for the rehabilitation of
patients, i.e., they may have a low cost and, at the same time,
have a high accuracy and ability to be executed in real time.
[0009] Visual tracking systems can meet only part of these
requirements, since its design presents various difficulties, such
as depth inconsistencies, feature deformities, complexity in the
kinematics' models and occlusions. To simplify these problems, most
of the algorithms to carry out the tracking employ tri-dimensional
models of the person's shape or multiple cameras to enhance the
robustness.
[0010] According to the above, several visual tracking systems are
found in the prior art, which are focused on different
applications, such as the International Publication No. WO
2008/134745 focused in the therapy of patients having some physical
or cognitive disability. Said application discloses a portable
therapy apparatus (a car) which captures images of the patient
using 2 or more cameras (stereo) or depth cameras (depth camera) to
estimate the 3D position, and to select and control one of the
therapeutic applications based on the gestures recognition of the
user, which are automatically detected from the images. The system
contemplates the selection of the different activities by the user
or by the attendant; however, the activities are not automatically
adapted according to the progress of the patient. Another drawback
is that the use of several cameras makes the system more expensive
and complex.
[0011] On the other hand, in the North American Patent Application
No. 2008/0085048, a computer controlled system is disclosed, which
allows a human to control a robotic apparatus using gestures and
movements being recognized by the system, and causing the robotic
apparatus to react thereto. The system includes, among other
components, a video camera recording the image and a software
implemented in a computer allowing the recognition of dynamic
gestures and static poses of a user. However, this system is not
used in therapeutic applications.
[0012] The tracking device disclosed in the US Patent Application
No. US 2006/0274032 is used to get information to control the
execution of a game program, this device comprises a body mountable
on a game control or on the user's body, and an inertial sensor
(accelerometer, mechanic gyroscope or laser gyroscope) operating to
produce 3D information to quantify a body's movement through the
space. The system may further comprise a camera and the controller
may include LEDs to facilitate the tracking by video analysis. This
device, in addition to being focused for use in video games, can
only estimate the movement and not the 3D position since it uses
inertial sensors to detect the 3D movement.
[0013] Likewise, US Patent Application No. US 2006/0209021, related
to an apparatus and a method to move a virtual mouse using a video
camera, which traces an input gesture, extracts from the image of
the input gesture the region corresponding to the right or left
hand, recognizes each hand's gesture, as well as the command
corresponding to said gesture, and executes the command. However,
this apparatus has as object providing the user with a more
convenient interface to devices such as a computer, and not its
application in the therapy of patients requiring upper limbs
rehabilitation. In addition, the follow-up of the movement is not
carried out in 3D.
[0014] On the other hand, US Patent Application No. 2005/0255434
discloses a training interactive system comprising computational
vision and including, among other components, at least one video
camera to get images of the trained person, as well as pattern
recognition algorithms and image analysis to recognize features in
the images and thereby detecting gestures of the trained person.
The system requires of additional elements like LEDs to be able to
carry out the follow-up of the people in training. This system is
used in training applications but not in therapeutic
applications.
[0015] The U.S. Pat. No. 7,262,760 claims the use of a 3D pointer
apparatus which transforms sensed movement data from a first
reference frame into a second reference frame. The system includes
at least one sensor to detect the rotation of the pointer
apparatus, an accelerometer to detect its acceleration and a
processor to receive the sensor outlet and that of the
accelerometer, all of which increases the systems costs.
[0016] Finally, U.S. Pat. No. 6,256,033 describes a computer
implemented method to recognize a person's gestures within an image
sequence and executing an operation based on the semantic meaning
of the gesture, wherein the subject enters in the vision field of a
camera connected to a computer and makes a gesture. The gesture is
examined by the system by means of a program, one image at a time,
thereby deriving position data and comparing them with previously
derived data representing gestures already known by the system. The
comparisons are made in real time and the system may be trained to
recognize new gestures. The main drawback of this method is that
the recognition is made examining one image at a time and comparing
the data with previously derived data and already known by the
system. On the other hand, this method is not used in therapeutic
applications.
[0017] As it may be seen from the above, most of the visual
tracking systems found in the prior art have the great
inconvenience that they were designed and developed for training or
to video games, but no so for the rehabilitation of patients who
have suffered a stroke. In addition to the above, some systems
found in the state of the art used in the rehabilitation of
patients include apparatus or devices highly expensive, or require
the implementation of various components such as the use of several
cameras, among other, representing a big problem for the hospital
system due to the decrease in the budget, and therefore, this means
that patients are sent to home before completing their
rehabilitation. And if this situation moved into the home of the
user, wherein said user has to use own resources, the
rehabilitation practically becomes impossible.
BRIEF DESCRIPTION OF THE INVENTION
[0018] As it was mentioned above, therapy systems found in prior
art used in upper limbs, arm and hand, of patients having suffered
from stroke use very expensive devices, such as the case of the
system using a robotic arm, thereby limiting its accessibility.
Moreover, when it is used in a fixed manner on the arm of the
patient, impeding the execution of natural movements.
[0019] In addition to the above, as well as budget problems of the
hospitals that sending patients to their home before completing his
rehabilitation, which requires that self-directed therapy systems
in home reach a greater number of patients, and it is required that
said systems have an affordable cost.
[0020] In view of the aforementioned, a low cost visual tracking
therapy system has been developed, with suitable accuracy and
executable in real time, wherein said tracking is such that allows
recovering the three-dimensional (3D) position of an object, thus
allowing the rehabilitation of patients having suffered from stroke
for restoring their upper limbs movement.
[0021] The therapy system of the present invention generally
comprises: a conventional digital camera allowing the capture of
images, which does not require additional calibration or standards;
a display apparatus configured to show patient interacting 3D
virtual environments; a handling element in a grip-shaped that
includes a colored sphere located at each one of its ends; a
processor or CPU allowing the processing of numerical information;
and, a software including two computer programs. One program
carries out a 3D visual tracking of a patients' hand, by estimating
its 3D spatial position every instant, and sends it to the games
system. The second program includes a set of games oriented to
daily life activities, which include different difficulty levels to
evaluate the patient's progress at every game. Whole system is
called "Gesture Therapy".
[0022] The therapy system of the present invention is a low cost
system, easy to install and operate, using a conventional digital
camera which does not require additional supports to determine,
through the tracking of the handle spheres, the position and 3D
movement of the patient's arm and/or hand, as well as its rotation
around 3 orthogonal axis.
[0023] Said system is capable of working in various lightning
conditions, and to eventually overcome harsh movements. Moreover,
it allows to overcome acceptably being out of frame, i.e., when the
tracked object gets out of the visibility field of the camera and
then comes back, even if the getting out and re-entry points
differ.
[0024] Additionally, the therapy system of the present invention is
a recognizing and tracking system for a specific point in the
patient's limb, i.e., it is capable of knowing at any time where
the hand is, overcoming the problems regarding depth detection,
such that the use of a single digital camera is enough to carry out
the detection and the 3D tracking.
[0025] The processor is configured for: [0026] selecting from a set
of games, an appropriate game for the patient; [0027] calling the
game; [0028] providing the required algorithms to determine the
position and 3D movement of the patient's hand; [0029] recognizing
the position and 3D movement of the patient's arm and/or hand;
[0030] controlling the game based on the position and 3D movement
of the patient's arm and/or hand; [0031] adapting to the patient
according to his (her) condition and therapy progress;
automatically alternating, for example, the working space for the
limb's movement; [0032] detecting the pressure when the patient
tights/loses the joystick by a pressure sensor incorporated
therein, thereby promoting the rehabilitation of the hand's
movements.
OBJECTS OF THE INVENTION
[0033] Considering the defects of the prior art, it is an object of
the present invention to provide a 3D monocular visual tracking
therapy system, easy to install and operate, but highly efficient
for the rehabilitation of patients having suffered from stroke.
[0034] Another object of the present invention is to provide a
monocular visual tracking therapy system being robust, having low
cost, easy to install and use, useful for the upper limbs
rehabilitation in a patient in need thereof.
[0035] Moreover, is another object of the present invention to
provide a home self-directed therapy method for patients having
upper limbs' movement disability.
BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The novel aspects considered characteristic of the present
invention are established particularly in the appended claims.
However, the invention itself, both in its configuration and in its
operation method, together with other objects and advantages
thereof, will be better understood in the following detailed
description of a specific embodiment, when read along with the
appended drawings, in which:
[0037] FIG. 1 is a schematic representation wherein the monocular
visual tracking therapy system is shown, constructed according to a
specific embodiment of the present invention.
[0038] FIG. 2 represents the use of the 3D monocular tracking
system for the rehabilitation of a patient's upper limbs. In left
upper box a possible configuration of the system is illustrated,
with a screen and a camera, through which the 3D tracking of the
ball in the patient's hand is carried out. In the other 3 boxes are
illustrated different game examples at the screen, oriented to
rehabilitation, for the patient to interact with the movement of
his (her) hand.
DETAILED DESCRIPTION OF THE INVENTION
[0039] The present invention is related to a 3D monocular tracking
system for the rehabilitation of a patient's upper limbs.
[0040] During the development of the present invention it was found
that this system determines the exact 3D position of the arm and/or
hand, based on a single video camera, therefore not requiring
additional sensors and its cost is lower than the traditional
systems.
[0041] Accordingly, in a one aspect of the present invention, a 3D
monocular tracking system for the rehabilitation of a patient's
upper limbs is described, preferably of a patient having suffered
from stroke, which comprises:
a) a handle, joystick or grip-shaped comprising a cylindrical
handle and two colored spheres, one at each end of the gripper
(FIG. 1); b) a computational vision system comprising a video
camera to determine, through the tracking the handle balls, the
position and 3D movement of the patient's arm and/or hand position,
as well as its rotation around 3 orthogonal axis; c) a gesture
therapy software comprising a program for 3D tracking of the two
colored spheres in the handle, a set of games oriented to daily
life activities, which include different difficulty levels to
evaluate the patient's progress at every game; d) a processor
configured for: [0042] selecting between the set of games, an
appropriate game to the patient; [0043] calling the game; [0044]
providing the required algorithms to determine the position and 3D
movement of the patient's hand; [0045] recognizing the position and
3D movement of the patient's arm and/or hand; [0046] controlling
the game based on the position and 3D movement of the patient's arm
and/or hand; [0047] adapting to the patient according to his (her)
condition in the therapy progress; automatically alternating, for
example, the working space for the limb's movement; [0048]
detecting the pressure when the patient tights/loses the joystick
by a pressure sensor incorporated therein, thereby promoting the
rehabilitation of the hand's movements. e) a display apparatus
configured to show 3D virtual environments that the patient can
interact.
[0049] In a preferred embodiment of the present invention, the
handle or gripper further comprises one or more pressure or
strength sensors, for determining when the patient tightens the
joystick and the force with which is performed, thus measuring the
force in the whole hand or in each finger. This measurement may be
sent to the processor and incorporated in the virtual environments.
Thereby, the patient's hand and fingers may also be exercised, and
his (her) ability to grip and lose, which are important
rehabilitation elements.
[0050] In other preferred embodiment of the present invention, the
video camera is selected from a web camera, a camera integrated to
the processor (computer), or an infrared camera; with the web
camera being particularly preferred.
[0051] Likewise, in an additional embodiment of the present
invention, the display apparatus is selected from a computer
screen, television monitor, digital personal assistant screen (PDA,
by its English initials), cellular phone screen; with the computer
screen being particularly preferred.
[0052] The use of the 3D monocular tracking system of the present
invention is shown in FIG. 2. FIG. 2a represents a computer with
the Gesture Therapy software and a web camera to carry out the 3D
monocular tracking, as well as the patient's hand holding the
handle with one of the colored spheres. Likewise, some useful games
for the rehabilitation of the patient's arm and/or hand are shown.
FIG. 2b shows a car racing game allowing the patient to train
lateral arm movements to control the car without getting out of the
track. FIG. 2c shows a game consisting in cleaning a stove, being
useful to train movements allowing stretching and shrinking the
arm. Finally, the game in FIG. 2d consists in painting specific
squares in a cube, thereby training the shoulder and elbow
movements in different directions. In this last example the
pressure sensor may be incorporated to "take" the paint, thereby
training the hand's movements (opening and closing).
[0053] In FIG. 1, a therapy system (1000), comprised of a handle or
gripper (100), a web camera (200) and a display apparatus
(300).
[0054] In FIG. 3: a handle or gripper (100), comprised by a gripper
(110) and two colored spheres (120).
[0055] Further aspects of the present invention consider a home
self-directed therapy method for patients having upper limbs'
movement disabilities, preferably one having suffered from stroke,
characterized by comprising: [0056] offering a patient a monocular
3D tracking system for the rehabilitation of the arm and/or hand;
[0057] selecting between a set of games, an appropriate game for
the rehabilitation of the patient; [0058] calling the game; [0059]
recognizing the position and 3D movement of the patient's arm
and/or hand; [0060] detecting when the patient tightens or loses
the handle; [0061] showing 3D virtual environments, visible for the
patient, wherein the 3D virtual environments respond to the 3D
movement of said patient's arm and/or hand; [0062] repeating the
above steps during the necessary period of time. The repetitions
number depends on the state of the patient and the severity of the
injury; in clinical studies it has been observed that after 10
sessions of one hour with the system, most patients show a
significant improvement according to the clinical indexes.
[0063] According to the above-described, it may be seen that the
system and method of the present invention have been envisioned to
provide a robust system, having a low cost, easy to install and
use, useful for the rehabilitation of the arm and hand's movement
in people in need thereof, and it will be obvious for those skilled
in the art that the embodiments of the 3D monocular tracking system
for the rehabilitation of a patient's upper limbs, as well as for
the home self-directed therapy method for patients having upper
limbs' movement disability, as described above and shown in the
drawings, should be considered as illustrative and non-limitative
of the present invention, since several detail changes are possible
without departing from the scope of the invention.
[0064] The present invention will be better understood form the
following examples, which shall be construed only as illustrative
to permit a better understanding of the preferred embodiments of
the present invention, without implicating that there are not
further embodiments does not illustrated therein capable of being
practiced based on the above detailed description of the
invention.
EXAMPLES
[0065] The system was used in a group of 22 patients having
suffered from stroke in a hospital (Rehabilitation Unit of the
Neurology and Neurosurgery Institute in Mexico City). The patients
used a prototype of the system in the hospital, interacting with
different games, guided by a therapist. The therapy lasted 7 weeks,
with 3 sessions per week; before and after the therapy each patient
was assessed by a therapist using two different clinical ranges
(Fugl-Meyer index and motricity index). The results show a
statistically significant improvement in both ranges after the
sessions with the "Gesture Therapy" system. In addition, a
motivation survey was made to all patients after having used the
system, the survey results showing a much higher motivation
compared to the "traditional" therapy.
[0066] Therefore, the present invention shall not be considered as
restricted, except for the prior art demands and by the scope of
the appended claims.
* * * * *