U.S. patent application number 14/155087 was filed with the patent office on 2014-07-17 for muscle interface device and method for interacting with content displayed on wearable head mounted displays.
This patent application is currently assigned to Thalmic Labs Inc.. The applicant listed for this patent is Thalmic Labs Inc.. Invention is credited to Matthew Bailey, Aaron Grant, Stephen Lake.
Application Number | 20140198034 14/155087 |
Document ID | / |
Family ID | 51164759 |
Filed Date | 2014-07-17 |
United States Patent
Application |
20140198034 |
Kind Code |
A1 |
Bailey; Matthew ; et
al. |
July 17, 2014 |
MUSCLE INTERFACE DEVICE AND METHOD FOR INTERACTING WITH CONTENT
DISPLAYED ON WEARABLE HEAD MOUNTED DISPLAYS
Abstract
There is disclosed a muscle interface device and method for
interacting with content displayed on wearable head mounted
displays. In an embodiment, the muscle interface device comprises a
sensor worn on the forearm of a user, and the sensor is adapted to
recognize a plurality of gestures made by a user's hand and or
wrist to interact with content displayed on the wearable head
mounted display. The muscle interface device utilizes a plurality
of sensors, including one or more of capacitive EMG, MMG, and
accelerometer sensors, to detect gestures made by a user. The
detected user gestures from the sensors are processed into a
control signal for allowing the user to interact with content
displayed on the wearable head mounted display in a discreet
manner.
Inventors: |
Bailey; Matthew; (Kitchener,
CA) ; Grant; Aaron; (Kitchener, CA) ; Lake;
Stephen; (Kitchener, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Thalmic Labs Inc. |
Kitchener |
|
CA |
|
|
Assignee: |
Thalmic Labs Inc.
Kitchener
CA
|
Family ID: |
51164759 |
Appl. No.: |
14/155087 |
Filed: |
January 14, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61752226 |
Jan 14, 2013 |
|
|
|
Current U.S.
Class: |
345/156 |
Current CPC
Class: |
G02B 27/017 20130101;
G06F 3/015 20130101; G06F 3/017 20130101; G06F 3/014 20130101; G06F
3/016 20130101; H04W 4/80 20180201; G06F 3/0484 20130101; G06F
3/011 20130101 |
Class at
Publication: |
345/156 |
International
Class: |
G06F 3/01 20060101
G06F003/01 |
Claims
1. A muscle interface device for interacting with content displayed
on a wearable head mounted display, comprising: a plurality of
sensors that in use are worn on a portion of an arm of a user to
detect signals generated by at least one muscle of the arm of the
user; and a transmitter communicatively coupled to the plurality of
sensors and which in use transmits communication signals indicative
of muscle activity detected by the plurality of sensors to a
receiver on the wearable head mounted display; wherein the
communication signals transmitted to the receiver on the wearable
head mounted display cause one or more interactions with content
displayed on the wearable head mounted display.
2. The muscle interface device of claim 1, further comprising: a
processor communicatively coupled to the plurality of sensors and
the transmitter, and which interprets the signals detected by the
plurality of sensors as a gesture and translates the signals
detected by the plurality of sensors to a control signal to
interact with content displayed on the wearable head mounted
display, wherein the control signal is transmitted by the
transmitter at least as part of the communications signals.
3. The muscle interface device of claim 1 wherein the detected
signals are transmitted to the receiver on the wearable head
mounted display to be interpreted as a gesture by the wearable head
mounted display.
4. The muscle interface device of claim 1, further comprising a
haptic feedback module for confirming the interpretation of a
gesture.
5. The muscle interface device of claim 1 wherein the plurality of
sensors includes at least one sensor selected from the group
consisting of: an electromyographic (EMG) sensor and a
mechanomyographic (MMG) sensor.
6. The muscle interface device of claim 1 wherein the transmitter
includes a wireless transmitter to wirelessly transmit wireless
communications signals to the receiver on the wearable head mounted
display.
7. The muscle interface device of claim 1, further comprising: an
accelerometer to detect an acceleration of the arm of the user and
to provide a signal representing the acceleration.
8. A method for interacting with content displayed on a wearable
head mounted display, the method comprising: displaying content on
the wearable head mounted display; detecting a user gesture by a
muscle interface device, wherein the muscle interface device
includes a plurality of sensors responsive to signals generated by
muscles of a user and a transmitter, and wherein detecting the user
gesture includes detecting signals generated by muscles of the user
by the plurality of sensors of the muscle interface device;
identifying the user gesture by at least one circuit of the muscle
interface device; transmitting the identified gesture to the
wearable head mounted display by the transmitter of the muscle
interface device; receiving the identified gesture by the wearable
head mounted display; and translating the identified gesture to a
control signal by the wearable head mounted display to interact
with content displayed thereon.
9. The method of claim 8 wherein the transmitter of the muscle
interface device is a wireless transmitter, and wherein:
transmitting the identified gesture to the wearable head mounted
display by the transmitter of the muscle interface device includes
wirelessly transmitting the identified gesture to the wearable head
mounted display by the wireless transmitter of the muscle interface
device; and receiving the identified gesture by the wearable head
mounted display includes wirelessly receiving the identified
gesture by the wearable head mounted display.
10. The method of claim 8, further comprising: pairing the muscle
interface device with the wearable head mounted display.
11. The method of claim 8 wherein the plurality of sensors includes
one or more electromyography (EMG) sensors, and wherein detecting a
user gesture by a muscle interface device includes detecting an
electrical signal produced by a muscle of the user by the one or
more EMG sensors of the muscle interface device.
12. The method of claim 8 wherein the plurality of sensors includes
one or more mechanomyography (MMG) sensors, and wherein detecting a
user gesture by a muscle interface device includes detecting a
vibration produced by a muscle of the user by the one or more MMG
sensors of the muscle interface device.
13. The method of claim 8 wherein the muscle interface device
further comprises an accelerometer, and wherein detecting a user
gesture by the muscle interface device includes detecting
acceleration by the accelerometer.
Description
FIELD OF THE INVENTION
[0001] The present disclosure relates generally to muscle interface
devices, and more specifically to a muscle interface device and
method for interacting with content displayed on wearable head
mounted displays.
BACKGROUND OF THE INVENTION
[0002] In recent years, wearable head mounted displays have begun
to gain wider acceptance, with a number of recently introduced
wearable head mounted display devices having the potential for
widespread adoption by consumers.
[0003] One such device disclosed in U.S. Pat. No. 8,203,502 issued
to Chi et al. utilizes a finger operable input device such as a
touch pad built into the wearable head mounted display (e.g. built
into a side-arm of a pair of glasses, with one of the lenses
functioning as a display screen) such that a user can interact with
and control content appearing on the display screen with
positioning and movement of a finger along a planar direction
relative to a surface of the input device. A potential drawback of
this approach is that a user is required to conspicuously raise his
or her hand to touch the input device each time the user wants to
interact with content displayed on the screen.
[0004] Another such device is disclosed in US 2012/0293548 (Perez
et al.) in which a head mounted display provides users with
supplemental information on a display screen provided in at least
one of the lenses of a pair of glasses. A processing unit may be
connected to the head mounted display to provide the computing
power necessary for its operation. However, the method of user
interaction with the display is not specified.
[0005] Yet another example of such a device is disclosed in U.S.
Pat. No. 8,212,159 issued to Tang et al. in which a source image is
projected onto screens built into head mounted displays worn by a
user. Tang et al. focuses on the method and system for projection,
and does not specify the manner of user interaction with the
head-mounted display device.
[0006] Therefore, what is needed is an effective user interface
device for wearable head mounted displays which can be utilized in
an inconspicuous manner.
SUMMARY OF THE INVENTION
[0007] The present disclosure relates to a muscle interface device
and method for interacting with content displayed on wearable head
mounted displays.
[0008] More generally, the muscle interface device comprises a
sensor worn on the forearm of a user, and the sensor is adapted to
recognize a plurality of gestures made by a user's hand and or
wrist to interact with content displayed on the wearable head
mounted display.
[0009] In an embodiment, the muscle interface device utilizes a
plurality of electromyographic (EMG) sensors to detect electrical
activity produced by muscles during contraction, and convert the
electrical signals for processing. The electrical signals detected
from the muscles are interpreted as gestures (e.g. a combination of
hand, wrist and arm movements) made by a user which provide a
control input to a wearable head mounted display. The control input
is preferably provided wirelessly via a near field communication
protocol, such as Bluetooth.TM., for example.
[0010] In another embodiment, various types of sensors may be used
alone or in lieu of or in combination with EMG sensors to detect
gestures made by a user, for processing as a control input for
interacting with a wearable head mounted display. This may be one
or more mechanomyographic (MMG) sensors to detect vibrations made
by muscles during contraction, or one or more accelerometer sensors
to detect larger movements.
[0011] In another embodiment, the muscle interface device includes
a calibration module with a routine for calibrating the muscle
interface device for use with the wearable head mounted
display.
[0012] Other features and advantages of the present invention will
become apparent from the following detailed description and
accompanying drawings. It should be understood, however, that the
detailed description and specific examples are given by way of
illustration and not limitation. Many modifications and changes
within the scope of the present invention may be made without
departing from the spirit thereof, and the invention includes all
such modifications.
DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 illustrates a user wearing a head mounted display and
a muscle interface device in accordance with an embodiment.
[0014] FIG. 2A illustrates a detailed view of a muscle interface
device in accordance with an embodiment.
[0015] FIG. 2B illustrates a data graph corresponding to an
electrical signal detected by an EMG sensor.
[0016] FIG. 3 illustrates wireless near field communication between
a head mounted display and a muscle interface device in accordance
with an embodiment of the invention.
[0017] FIG. 4 illustrates a user's hand and wrist gesture processed
as a control signal by the muscle interface device for interacting
with content displayed on the head mounted display.
[0018] FIG. 5 illustrates a schematic system architecture of a
muscle interface device in accordance with an embodiment.
[0019] FIG. 6 illustrates a schematic flow chart of a method in
accordance with an embodiment.
[0020] In the drawings, embodiments of the invention are
illustrated by way of example. It is to be expressly understood
that the description and drawings are only for the purpose of
illustration and as an aid to understanding, and are not intended
as a definition of the limits of the invention.
DETAILED DESCRIPTION
[0021] The present disclosure relates to a muscle interface device
and method for interacting with content displayed on wearable head
mounted displays.
[0022] In an aspect, the muscle interface device comprises a sensor
worn on the forearm of a user, and the sensor is adapted to
recognize gestures made by a user to interact with content
displayed on the wearable head mounted display. In another aspect,
the user interface method comprises processing a control signal
from gestures made by a user's hand, wrist and arm movements to
interact with content displayed on the wearable head mounted
display.
[0023] In a preferred embodiment, the muscle interface device
utilizes a plurality of sensors to detect electrical signals and/or
vibrations produced by muscles during contraction, and movement of
the arm, and converts the electrical signals or vibrations to a
digital signal for processing. The electrical signals and/or
vibrations detected from the muscles are interpreted as gestures
made by a user which provide a control input to a wearable head
mounted display.
[0024] In an embodiment, the control input is preferably provided
wirelessly via a near field communication protocol, such as
Bluetooth.TM., for example. However, it will be appreciated that
other types of near field communications may be used, including any
NFC protocol developed for smart phones and similar devices.
[0025] In addition to capacitive EMG, MMG, and accelerometer
sensors, various other types of sensors may be used to detect
gestures made by a user. However, some of these sensors may have
drawbacks if used alone.
[0026] For example, surface electromyographic (sEMG) sensors may be
used to measure forearm muscle activity. An sEMG sensor typically
requires direct contact with the skin of the user in order to
measure the electrical activity conducted from the underlying
muscles, through the fat and skin. There are some inherent
limitations with sEMG, as the quality of the acquired signal is
directly related to the skin impedance, which varies according to
the user's skin perspiration, amount of arm hair, fat content, and
a number of other attributes. This may necessitate the use of
moisturizing and conductive gels, shaving the skin, or other skin
preparation practices to get a reliable and repeatable signal from
this type of sensor. In contrast, the capacitive EMG sensors of the
present invention do not require direct surface contact or skin
preparations.
[0027] As another example, one or more accelerometer sensors may be
used to measure larger gestures made by a user, for example
involving the elbow or even the shoulders of a user. When used
together with EMG and/or MMG sensors for detecting more limited
gestures (e.g. made by the hand and/or wrist for example), the
accelerometer sensors may be utilized to increase the range of
control inputs that may be generated for interaction with a
wearable head mounted display.
[0028] An illustrative embodiment will now be described with
reference to the drawings.
[0029] Shown in FIG. 1 is an illustrative user 100 wearing a head
mounted display 310 with display control 300, and a muscle
interface device 200 in accordance with an embodiment. In this
illustrative example, muscle interface device 200 is a flexible,
stretchable band that may be worn on the forearm of user 100 as
shown.
[0030] FIG. 2A illustrates a detailed view of the muscle interface
device 200 of FIG. 1 in accordance with an embodiment. As shown,
muscle interface device 200 may comprise a central processing unit
210, and one or more batteries 220, which may be rechargeable, and
which may be utilized concurrently or sequentially in conventional
manner. As shown, muscle interface device 200 includes a plurality
of sensors 230 which may be positioned radially around the
circumference of the band, such that the sensors 230 can detect
gestures made by user 100. Muscle interface device 200 may further
include a feedback mechanism, such as a vibratory motor 240 to
provide haptic feedback as described further below.
[0031] In an embodiment, the muscle interface device 200 is
calibrated when first worn, prior to operation, such that the
positioning of the sensors 230 does not need to depend on the
location of particular muscles in the forearm.
[0032] In an embodiment, sensors 230 include at least one or more
capacitive EMG sensor adapted to detect electrical signals in the
forearm of user 100 for generating a control signal. Capacitive EMG
does not require direct contact with the skin as with sEMG sensors
described earlier. Rather, capacitive EMG sensors are capacitively
coupled to the electrical signals generated by contracting muscles,
and may operate at a distance of up to 3 mm from the skin. By way
of example, the capacitive EMG signal can be an oscillating
waveform that varies in both frequency and amplitude, and the
majority of signal information is contained within the 5 Hz to 250
Hz frequency band. An illustrative example of an EMG signal is
shown in FIG. 2B.
[0033] In another embodiment, one or more MMG sensors comprising
piezoelectric sensors may be used to measure the vibrations at the
surface of the skin produced by the underlying muscles when
contracted. By way of example, the MMG signal generated may be an
oscillating waveform that varies in both frequency and amplitude,
and a majority of signal information is contained within the 5 Hz
to 250 Hz frequency band. Because the MMG signal is acquired via
mechanical means, electrical variations like skin impedance do not
have an effect on the signal. The MMG signal is very similar to the
illustrative example of the EMG signal shown in FIG. 2B.
[0034] Thus, capacitive EMG or MMG both provide a reliable control
signal that can be obtained over the duration of a full day, as
skin perspiration and moisturization changes do not affect the
signal.
[0035] In another embodiment, sensors 230 may include one or more
accelerometer sensors for detecting additional aspects of gestures
made by user 100 in three degrees of freedom. The accelerometer
signal consists of a three digital channels of data, each
representing the acceleration in either the x, y, or z direction.
The signal is subject to all of the accelerations that the users
arm is subject to, and may further incorporate motion of the body
as a whole.
[0036] Now referring to FIG. 3, shown is an illustration of
wireless near field communication (NFC) between muscle interface
device 200 and the head mounted display 310 and display control
300. This wireless NFC is utilized to transmit the control signal
from muscle interface device 200 to display control 300.
[0037] This is illustrated by way of example in FIG. 4, in which
user's hand and wrist gesture is detected and processed as a
control signal by the muscle interface device 200 for interacting
with content displayed on the head mounted display 310.
[0038] In this particular example, a gesture 410 made by the user
extending an index finger, and making a wrist flexion motion 420 is
detected by the sensors 230 of muscle interface device 200, and
processed by CPU 210 (FIG. 2) as a control signal for causing a
menu appearing on display 310 to scroll downwards.
[0039] As another example, a similar gesture in which user 100
extends the index finger and makes a wrist extension motion is
detected by sensors 230 of muscle interface device 200 and
processed by CPU 210 (FIG. 2) as a control signal for causing a
menu appearing on display 310 to scroll upwards.
[0040] As yet another example, a gesture in which user 100 extends
the index finger and makes a poking motion involving a slight
movement of the elbow and shoulder may be detected by sensors 230
of muscle interface device 200 and processed by CPU 210 (FIG. 2) as
a control signal for causing a highlighted menu item appearing on
display 310 to be selected.
[0041] If the user extends a different finger other than the index
finger, sensors 230 will detect this and may cause a different
control signal to be generated. For example, extending the little
finger or "pinky" finger instead of the index finger may cause
muscle interface device 200 to interpret the user's gestures with
functions analogous to clicking a right mouse button rather than a
left mouse button in a conventional mouse user interface. Extending
both the index and pinky fingers at the same time may cause muscle
interface device 200 to interpret the user's gestures with yet
other functions analogous to clicking a third mouse button in a
conventional mouse user interface.
[0042] Thus, muscle interface device 200 may be adapted to be
calibrated to recognize a wide range of user gestures made by a
user, based on measurements from a plurality of sensors in the
muscle interface device 200.
[0043] In an embodiment, muscle interface device 200 may itself be
adapted to interpret the gestures from the detected signals as
described.
[0044] However, in an alternative embodiment, the detected signals
may be transmitted to the head mounted display 310 and display
control 300 to be interpreted as gestures at the display control
300. Whether the detected signals are interpreted at the device 200
or at the display control 300, the detected signal is first
interpreted as a recognized gesture in order to interact with
content displayed on the display 310.
[0045] In another embodiment, upon interpretation of a gesture,
muscle interface device 200 may include a haptic feedback module to
provide feedback that a gesture has been recognized. This haptic
feedback provides a user with confirmation that the user's gesture
has been recognized, and successfully converted to a control signal
to interact with content displayed on display 310. The haptic
feedback module may comprise, for example, a vibrating mechanism
such as a vibratory motor 240 built into the muscle interface
device.
[0046] Alternatively, rather than haptic feedback provided by the
muscle interface device 200, confirmation of recognition of a
gesture may be provided by auditory feedback, either generated by a
speaker on the muscle interface device, or operatively connected to
the head mounted display 310.
[0047] In still another embodiment, confirmation of recognition of
a gesture may also be provided visually on the display 310 itself.
If there is more than one possible gesture that may be interpreted
from the detected signals, rather than providing resulting in an
error, the muscle interface device 200 and/or the display control
300 may provide a selection of two or more possible gestures as
possible interpretation, and the user may be prompted to select
from one of them to confirm the intended gesture and corresponding
control.
[0048] Now referring to FIG. 5, shown is an illustrative schematic
system architecture 500 of a muscle interface device in accordance
with an embodiment. As shown, system architecture 500 includes a
CPU 502, memory 504, system clock 506, an NFC module 508 (e.g.
Bluetooth.TM.), and a direct memory access (DMA) controller 510. As
shown, DMA controller 510 is adapted to receive inputs from various
sensors including one or more EMG sensors 520, MMG sensors 530 and
accelerometer sensors 540.
[0049] In an embodiment, detected signals from one or more EMG
sensors 520 are processed through signal filter 532 and converted
from analog to digital signals by ADC 534. If one or more MMG
sensors 530 are also used, then the detected signals from the sEMG
sensors 520 are processed through signal filter 522 and converted
from analog to digital signals by ADC 524. Digital signals from one
or more accelerometer sensors 540 may also be processed through
signal filter 542 and received by DMA controller 510.
[0050] In an illustrative embodiment the data from the various
types of sensors 520, 530, 540 is acquired through an analog
filtering chain. The data is bandpassed through filters 522, 532
between 10 Hz to 500 Hz, and amplified (e.g. by a total of 1000
times). This filtering and amplification can be altered to whatever
is required to be within software parameters. A notch filter at 60
Hz, or at any other relevant frequency, may also be used to remove
powerline noise.
[0051] In an embodiment, data from the sensors is converted to 12
bit digital data by ADCs 524, 534, and then clocked into onboard
memory using clock 506 by the DMA controller 510 to be processed by
the CPU 502.
[0052] Now referring to FIG. 6, shown is a schematic flow chart of
a method in accordance with an embodiment. As shown, method 600
begins at block 602, where method 600 pairs a muscle interface
device with a wearable head mounted display. Method 600 then
proceeds to block 604, where content and/or user interface (UI) is
displayed on the wearable head mounted display.
[0053] Method 600 then proceeds to block 606, where method 600
determines if the displayed content and/or UI is navigable. If no,
method 600 returns to block 604. If yes, method 600 proceeds to
block 608, where method 600 detects user gestures utilizing the
muscle interface device 200, and wirelessly transmits the
identified gesture to the display control 300 (FIG. 1).
[0054] Method 600 then proceeds to block 610, where display control
400 receives the detected gestures from the muscle interface device
200, and translates the detected gestures to a control signal to
interact with navigable content displayed on the wearable head
mounted display.
Use Cases
[0055] As will be appreciated, the muscle interface device and
method of the present disclosure may be used for interaction with a
wearable head mounted display in a wide range of applications, in
virtually any application in which wearable head mounted displays
are contemplated. By providing a discrete method of interacting
with a wearable head mounted display, a user is able to interact
with such a display in any operating environment, including
situations where overt gesturing (e.g. raising the hand to touch an
input device provided on the wearable head mounted display itself)
is not desirable.
[0056] While various embodiments and illustrative examples have
been described above, it will be appreciated that these embodiments
and illustrative examples are not limiting, and the scope of the
invention is defined by the following claims.
* * * * *