U.S. patent application number 14/229681 was filed with the patent office on 2014-07-31 for sensor array touchscreen recognizing finger flick gesture from spatial distribution profiles.
The applicant listed for this patent is Lester F. Ludwig. Invention is credited to Lester F. Ludwig.
Application Number | 20140210786 14/229681 |
Document ID | / |
Family ID | 26773010 |
Filed Date | 2014-07-31 |
United States Patent
Application |
20140210786 |
Kind Code |
A1 |
Ludwig; Lester F. |
July 31, 2014 |
SENSOR ARRAY TOUCHSCREEN RECOGNIZING FINGER FLICK GESTURE FROM
SPATIAL DISTRIBUTION PROFILES
Abstract
Touch screen user interfaces configured to detect a touch
gesture for controlling software applications, computers, devices,
machinery, and process environments. Such user interfaces can be
manipulated by users and provide a wide range of uses with computer
applications, assistance to the disabled, and control of electronic
devices, machines, and processes. Enhancements can include velocity
and pressure sensing capabilities. The touch screen can be realized
with a transparent touch sensor array positioned over a visual
display. The touch screen can be configured to measure a spatial
distribution profile from a touch on the touch screen and recognize
gestures based on dynamics among spatial pressure distribution
profiles. In an example implementation, the gesture recognition may
rely solely on dynamics of shape differences between hand contact
patterns and omit the use of pressure, as the shape of the spatial
distribution profile does not vary much by pressure.
Inventors: |
Ludwig; Lester F.; (San
Antonio, TX) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Ludwig; Lester F. |
San Antonio |
TX |
US |
|
|
Family ID: |
26773010 |
Appl. No.: |
14/229681 |
Filed: |
March 28, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14160511 |
Jan 21, 2014 |
8743076 |
|
|
14229681 |
|
|
|
|
11761978 |
Jun 12, 2007 |
8717303 |
|
|
14160511 |
|
|
|
|
09812400 |
Mar 19, 2001 |
7786370 |
|
|
11761978 |
|
|
|
|
09313533 |
May 15, 1999 |
6610917 |
|
|
09812400 |
|
|
|
|
60085713 |
May 15, 1998 |
|
|
|
Current U.S.
Class: |
345/174 |
Current CPC
Class: |
G10H 3/26 20130101; G10H
1/0058 20130101; G10H 2220/525 20130101; G06F 3/0412 20130101; G10H
2220/521 20130101; G06F 3/041 20130101; G10H 1/00 20130101; G10H
2230/101 20130101; G06F 3/0488 20130101; G06F 3/04883 20130101;
G10H 1/0066 20130101; G10H 3/18 20130101; G10H 2230/271 20130101;
G10H 2230/095 20130101; G10H 2230/115 20130101; Y10S 84/10
20130101; G10H 1/46 20130101; G10H 2230/145 20130101; G10H 1/348
20130101; G06F 3/04847 20130101; G06F 3/005 20130101; G06F 3/017
20130101 |
Class at
Publication: |
345/174 |
International
Class: |
G06F 3/0488 20060101
G06F003/0488 |
Claims
1. An apparatus comprising: a transparent touchpad positioned over
a visual display, the transparent touchpad configured to sense a
contiguous region of contact from at least a portion of a human
hand on a surface of the transparent touchpad, the transparent
touch pad comprising a sensor array having a plurality of sensors,
each sensor having a unique spatial location and an associated
unique address, wherein the transparent touch pad is configured to
sense the contiguous region of contact by a generation of sensor
measurement values associated with each of the plurality of
sensors, wherein a plurality of sensor spatial locations are
configured to provide associated sensor measurement values
responsive to the contiguous region of contact; a value of at least
one control parameter for the contiguous region of contact
responsive to a measured change in at least one of the associated
sensor measurement values, the value obtained by performing a
calculation on the associated sensor measurement values and
interpreting the results as signifying a finger flick touch
gesture, wherein the at least one control parameter is associated
with the finger flick touch gesture; and at least one derived
control parameter assigned to a specific control signal; wherein
the finger flick touch gesture is recognized from dynamics among
postures derived from spatial distribution profiles.
2. The apparatus of claim 1 wherein the transparent touch pad is
configured to sense the contiguous region of contact by determining
measurement values and coordinates for selected pixels associated
with the portion of the human hand on the surface of the
transparent touchpad.
3. The apparatus of claim 1, wherein the transparent touch pad is
configured to determine the sequential dynamics among postures
based on a pattern for the contiguous region of contact.
4. The apparatus of claim 1, wherein the transparent touch pad is
configured to sense the contiguous region of contact by a
generation of the associated sensor measurement values based on
identifying a point of contact for the sensed region of contact
relative to the surface of the transparent touchpad.
5. The apparatus of claim 1, wherein the transparent touchpad is
configured to sense the contiguous region of contact by a
generation of the associated sensor measurement values based on
identifying measurement values that exceed a threshold.
6. The apparatus of claim 1, wherein the transparent touch pad is
further configured to track a trajectory of movement of the portion
of the human hand contacting the surface of the transparent
touchpad.
7. The apparatus of claim 1, wherein the visual display is further
configured to provide dynamically assigned labels.
8. The apparatus of claim 1, wherein at least one control parameter
is responsive to a geometric center of the contiguous region of
measurement values.
9. The apparatus of claim 1, wherein at least one control parameter
is responsive to a center of pressure of the contiguous region of
measurement values.
10. The apparatus of claim 1, wherein at least one control
parameter is responsive to an average pressure of the contiguous
region of measurement values.
11. The apparatus of claim 1, wherein at least one control
parameter is responsive to a total size of the contiguous region of
measurement values.
12. The apparatus of claim 1, wherein the transparent touch pad is
further configured to sense at least a second contiguous region of
contact from another portion of a human hand on a surface of the
transparent touchpad, and the apparatus is configured to derive a
value of at least another control parameter responsive the second
contiguous region of contact.
13. An apparatus comprising: a transparent touch sensor array
comprising a plurality of transparent sensors positioned over a
display associated with the apparatus to form a transparent
touchpad, wherein the transparent touch sensor array is configured
to sense contact with one or more fingers on a corresponding
contiguous region on the transparent touch pad, wherein each
transparent sensor of the plurality of transparent sensors has a
corresponding spatial location, associated address, and is
configured to provide associated sensor measurement values; at
least one control parameter associated with a finger flick touch
gesture, the finger flick touch gesture recognized from dynamics
among postures derived from spatial distribution profiles and
associated with a measured change to at least one of the associated
sensor measurement values; and at least one control signal
associated with the at least one control parameter; and wherein the
finger flick touch gesture corresponds to one or more interactions
with displayed visual content of an application operating on the
apparatus.
14. The apparatus of claim 13, wherein the postures are associated
with a single continuous touching of the transparent touch sensor
array.
15. The apparatus of claim 13, wherein the display is configured to
dynamically provide at least one label at a corresponding spatial
location of at least one of the plurality for transparent sensors
in response to the control signals.
16. The apparatus of claim 13, wherein the display is configured to
control displayed visual content based on controlling at least one
of a selection, a motion, and a sizing of the displayed visual
content.
17. The apparatus of claim 13, wherein the spatial distribution
profiles consist of one or more shapes of hand contact
patterns.
18. The apparatus of claim 13, further comprising at least one
control parameter associated with another touch gesture, the
another touch gesture recognized from dynamics among postures
derived from spatial distribution profiles and associated with a
measured change to at least one of the associated sensor
measurement values.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. application Ser.
No. 14/160,511, filed Jan. 21, 2014, which is a continuation of
U.S. application Ser. No. 11/761,978, filed Jun. 12, 2007, which is
a continuation of U.S. application Ser. No. 09/812,400, filed Mar.
19, 2001, now U.S. Pat. No. 7,786,370, issued Aug. 31, 2010, which
is a division of U.S. application Ser. No. 09/313,533, filed May
15, 1999, now U.S. Pat. No. 6,610,917, issued Aug. 26, 2003, which
claims the benefit of priority of U.S. Provisional Application No.
60/085,713, filed May 15, 1998.
FIELD OF INVENTION
[0002] The present invention relates generally to a control system,
and in particular, to a tactile input controller for controlling an
associated system.
SUMMARY OF THE INVENTION
[0003] Touchpad user interfaces for controlling external systems
such as computers, machinery, and process environments via at least
three independent control signals. The touchpad may be operated by
hand, other parts of the body, or inanimate objects. Such an
interface affords a wide range of uses in computer applications,
machine and process control, and assistance to the disabled. In one
embodiment simple contact position-sensing touchpads, producing
control signals responsive to a contact region, are enhanced to
provide several independent control signals. Enhancements may
include velocity sensors, pressure sensors, and electronic
configurations measuring contact region widths. Touch-screens
positioned over visual displays may be adapted. According to other
aspects pressure-sensor array touchpads are combined with image
processing to responsively calculate parameters from contact
regions. Six independent control parameters can be derived from
each region of contact. These may be easily manipulated by a user.
In one implementation, smaller pressure-sensor arrays are combined
with data acquisition and processing into a chip that can be tiled
in an array.
DESCRIPTION OF THE DRAWINGS
[0004] The above and other aspects, features and advantages of the
present invention will become more apparent upon consideration of
the following description of preferred embodiments taken in
conjunction with the accompanying drawing figures, wherein:
[0005] FIG. 1 shows an example of how two independent contact
points can be independently discerned, or the dimensional-width of
a single contact point can be discerned, for a resistance
null/contact controller with a single conductive contact plate or
wire and one or more resistive elements whose resistance per unit
length is a fixed constant through each resistive element;
[0006] FIG. 2 shows how a pressure-sensor array touch-pad can be
combined with image processing to assign parameterized
interpretations to measured pressure gradients and output those
parameters as control signals;
[0007] FIG. 3 illustrates the positioning and networking of
pressure sensing and processing "mini-array" chips in larger
contiguous structures;
[0008] FIG. 4 illustrates the pressure profiles for a number of
example hand contacts with a pressure-sensor array;
[0009] FIG. 5 illustrates how six degrees of freedom can be
recovered from the contact of a single finger; and
[0010] FIG. 6 illustrates examples of single, double, and quadruple
touch-pad instruments with pads of various sizes and supplemental
instrument elements.
[0011] FIG. 7 illustrates an example implementation involving
dynamically assigned labels.
DETAILED DESCRIPTION
Overview
[0012] Described herein are two kinds of novel touch-pads.
Null/contact touchpads are contact-position sensing devices that
normally are in a null state unless touched and produce a control
signal when touched whose signal value corresponds to typically one
unique position on the touch-pad. A first enhancement is the
addition of velocity and/or pressure sensing. A second enhancement
is the ability to either discern each dimensional-width of a single
contact area or, alternatively, independently discern two
independent contact points in certain types of null/contact
controllers. A third possible enhancement is that of employing a
touch-screen instance of null/contact touch pad and positioning it
over a video display.
[0013] The invention also provides for a pressure-sensor array
touch-pad. A pressure-sensor array touch-pad of appropriate
sensitivity range, appropriate "pixel" resolution, and appropriate
physical size is capable of measuring pressure gradients of many
parts of the human hand or foot simultaneously. A pressure-sensor
array touch-pad can be combined with image processing to assign
parameterized interpretations to measured pressure gradients and
output those parameters as control signals. The pressure-sensor
"pixels" of a pressure-sensor array are interfaced to a data
acquisition stage; the data acquisition state looks for sensor
pixel pressure measurement values that exceed a low-level
noise-rejection/deformity-reject threshold; contiguous regions of
sufficiently high pressure values are defined; the full collection
of region boundaries are subjected to classification tests; various
parameters are derived from each independent region; and these
parameters are assigned to the role of specific control signals
which are then output to a signal routing, processing, and
synthesis entity.
[0014] It is possible to derive a very large number of independent
control parameters which are easily manipulated by the operating
user. For example, six degrees of freedom can be recovered from the
contact of a single finger. A whole hand posture can yield 17
instantaneously and simultaneously measurable parameters which are
independently adjustable per hand. The recognized existence and/or
derived parameters from postures and gestures may be assigned to
specific outgoing control signal formats and ranges. The hand is
used throughout as an example, but it is understood that the foot
or even other body regions, animal regions, objects, or physical
phenomena can replace the role of the hand.
[0015] It will be evident to one of ordinary skill in the art that
it is advantageous to have large numbers of instantaneously and
simultaneously measurable parameters which are independently
adjustable. For instance, a symbol in a 2-D CAD drawing can be
richly interactively selected and installed or edited in moments as
opposed to tens to hundreds of seconds as is required by mouse
manipulation of parameters one or two at a time and the necessary
mode-changes needed to change the mouse action interpretation. As a
result, said touch-pad has applications in computer workstation
control, general real-time machine control, computer data entry,
and computer simulation environments.
[0016] Various hardware implementations are possible. A
particularly advantageous implementation would be to implement a
small pressure-sensor array together with data acquisition and a
small processor into a single chip package that can be laid as
tiles in a larger array.
Null/Contact Touch-Pads
[0017] Distinguished from panel controls and sensors are what will
be termed null/contact touch-pads. This is a class of
contact-position sensing devices that normally are in a null state
unless touched and produce a control signal when touched whose
signal value corresponds to typically one unique position on the
touch-pad. Internal position sensing mechanisms may be resistive,
capacitive, optical, standing wave, etc. Examples of these devices
include one-dimensional-sensing ribbon controllers found on early
music synthesizers, two-dimensional-sensing pads such as the early
Kawala pad and more modern mini-pads found on some lap-top
computers, and two-dimensional-sensing see-through touch-screens
often employed in public computer kiosks.
[0018] The null condition, when the pad is untouched, requires
and/or provides the opportunity for special handling. Some example
ways to handle the untouched condition include: [0019] sample-hold
(hold values issued last time sensor was touched, as does a
joystick) [0020] bias 1107a, 1107b (issue maximal-range value,
minimal-range value, mid-range value, or other value) [0021]
touch-detect on another channel (i.e., a separate out-of-band
"gate" channel).
[0022] Additional enhancements can be added to the adaptation of
null/contact touch-pad controllers as instrument elements. A first
enhancement is the addition of velocity and/or pressure sensing.
This can be done via global impact and/or pressure-sensors. An
extreme of this is implementation of the null/contact touch-pad
controller as a pressure-sensor array; this special case and its
many possibilities are described later.
[0023] A second enhancement is the ability to either discern each
dimensional-width of a single contact area or, alternatively,
independently discern two independent contact points in certain
types of null/contact controllers. FIG. 1 shows an example of how
two independent contact points can be independently discerned, or
the dimensional-width of a single contact point can be discerned,
for a resistance null/contact controller with a single conductive
contact plate (as with the Kawala pad product) or wire (as in a
some types of ribbon controller products) and one or more resistive
elements 1100 whose resistance per unit length is a fixed constant
through each resistive element. It is understood that a
one-dimensional null/contact touch-pad typically has one such
resistive element while a two-dimensional null/contact touch-pad
typically has two such resistive elements that operate
independently in each direction.
[0024] Referring to FIG. 1, a constant current source 1101 can be
applied to the resistive element as a whole 1102a-1102b, developing
a fixed voltage across the entire resistive element 1100. When any
portion of the resistive element is contacted by either a
non-trivial contiguous width and/or multiple points of contact
1104-1105, part of the resistive element is shorted out 1100a, thus
reducing the overall width-to-end resistance of the resistance
element. Because of the constant current source 1101, the voltage
developed across the entire resistive element 1102a-1102b drops by
an amount equal to the portion of the resistance that is shorted
out.
[0025] The value of the voltage drop then equals a value in
proportion to the distance separating the extremes of the wide
and/or multiple contact points 1104-1105. By subtracting 1111,
1112, 1113 the actual voltage across the entire resistive element
from the value this voltage is normally 1110, a control voltage
proportional to distance separating the extremes of the wide and/or
multiple contact points 1104-1105 is generated. Simultaneously, the
voltage difference between that of the contact plate/wire 1103 and
that of the end of the resistive element closest to an external
contact point 1102a or 1102b is still proportional to the distance
from said end to said external contact point. Using at most simple
op-amp summing and/or differential amplifiers 1108a, 1108b, 1112, a
number of potential control voltages can be derived; for example
one or more of these continuously-valued signals: [0026] value of
distance difference between external contact points (or "width"; as
described above via constant current source, nominal reference
voltage, and differential amplifier 1113 [0027] center of a
non-trivial-width region (obtained by simple averaging, i.e., sum
with gain of 1/2) [0028] value of distance difference 1109a between
one end of the resistive element and the closest external contact
point (simple differential amplifier) [0029] value of distance
difference between the other end of the resistive element and the
other external contact point (sum above voltage with "width"
voltage with appropriate sign) 1109b.
[0030] Further, through use of simple threshold comparators,
specific thresholds of shorted resistive element can be deemed to
be, for example, any of a single point contact, a recognized
contact region width, two points of contact, etc., producing
corresponding discrete-valued control signals. The detection of a
width can be treated as a contact event for a second parameter
analogous to the single contact detection event described at the
beginning. Some example usages of these various continuous and
discrete signals are: [0031] existence of widths or multiple
contact points may be used to trigger events or timbre changes
[0032] degree of widths may be used to control degrees of
modulation or timbre changes [0033] independent measurement of each
external contact point from the same end of the resistive element
can be used to independently control two parameters. In the
simplest form, one parameter is always larger than another; in more
complex implementations, the trajectories of each contact point can
be tracked (using a differentiator and controlled parameter
assignment switch); as long as they never simultaneously touch,
either parameter can vary and be larger or smaller than the
other.
[0034] It is understood that analogous approaches may be applied to
other null/contact touchpad technologies such as capacitive or
optical.
[0035] A third possible enhancement is that of employing a
touch-screen instance of null/contact touch-pad and positioning it
over a video display. The video display could for example provide
dynamically assigned labels, abstract spatial cues, spatial
gradients, line-of-site cues for fixed or motor-controlled
lighting, etc. which would be valuable for use in conjunction with
the adapted null/contact touch-pad controller.
[0036] These various methods of adapted null/contact touch-pad
elements can be used stand-alone or arranged in arrays. In
addition, they can be used as a component or addendum to
instruments featuring other types of instrument elements.
Pressure-Sensor Array Touch-Pads
[0037] The invention provides for use of a pressure-sensor array
arranged as a touch-pad together with associated image processing.
As with the null/contact controller, these pressure-sensor array
touch-pads may be used stand-alone or organized into an array of
such pads.
[0038] It is noted that the inventor's original vision of the below
described pressure-sensor array touch-pad was for applications not
only in music but also for computer data entry, computer simulation
environments, and real-time machine control, applications to which
the below described pressure-sensor array touch-pad clearly can
also apply.
[0039] A pressure-sensor array touch-pad of appropriate sensitivity
range, appropriate "pixel" resolution, and appropriate physical
size is capable of measuring pressure gradients of many parts of
the flexibly-rich human hand or foot simultaneously. FIG. 2 shows
how a pressure sensor array touch-pad can be combined with image
processing to assign parameterized interpretations to measured
pressure gradients and output those parameters as control
signals.
[0040] The pressure-sensor "pixels" of a pressure-sensor array
touch-pad 1300 are interfaced to a data acquisition stage 1301. The
interfacing method may be fully parallel but in practice may be
advantageously scanned at a sufficiently high rate to give good
dynamic response to rapidly changing human touch gestures. To avoid
the need for a buffer amplifier for each pressure-sensor pixel,
electrical design may carefully balance parasitic capacitance of
the scanned array with the electrical characteristics of the
sensors and the scan rates; electrical scanning frequencies can be
reduced by partitioning the entire array into distinct parts that
are scanned in parallel so as to increase the tolerance for address
settling times and other limiting processes.
[0041] Alternatively, the pressure-sensor array 1300 may be
fabricated in such a way that buffer amplifier arrays can be
inexpensively attached to the sensor array 1300, or the sensors may
be such that each contains its own buffer amplifier; under these
conditions, design restrictions on scanning can be relaxed and
operate at higher speeds. Although the pressure sensors may be
likely analog in nature, a further enhancement would be to use
digital-output pressure-sensor elements or sub-arrays.
[0042] The data acquisition stage 1301 looks for sensor pixel
pressure measurement values that exceed a low-level
noise-rejection/deformity-rejection threshold. The sufficiently
high pressure value of each such sensor pixel is noted along with
the relative physical location of that pixel (known via the pixel
address). This noted information may be stored "raw" for later
processing and/or may be subjected to simple boundary tests and
then folded into appropriate running calculations as will be
described below. In general, the pressure values and addresses of
sufficiently high pressure value pixels are presented to a sequence
of processing functions which may be performed on the noted
information: [0043] contiguous regions of sufficiently high
pressure values are defined (a number of simple run-time adjacency
tests can be used; many are known--see for example [Ronse; Viberg;
Shapiro; Hara]) [0044] the full collection of region boundaries are
subjected to classification tests; in cases a given contiguous
region may be split into a plurality of tangent or co-bordered
independently recognized regions [0045] various parameters are
derived from each independent region, for example geometric center,
center of pressure, average pressure, total size,
angle-of-rotation-from reference for non-round regions,
second-order and higher-order geometric moments, second-order and
higher-order pressure moments, etc. [0046] assignment of these
parameters to the role of specific control signals (note events,
control parameters, etc.) which are then output to a signal
routing, processing, and synthesis entity; for example, this may be
done in the form of MIDI messages.
[0047] Because of the number of processes involved in such a
pipeline, it is advantageous to follow a data acquisition stage
1301 with one or more additional processing stages 1303, 1305,
1309, and 1311. Of the four example processing functions just
listed, the first three fall in the character of image processing.
It is also possible to do a considerable amount of the image
processing steps actually within the data acquisition step, namely
any of simple adjacency tests and folding selected address and
pressure measurement information into running sums or other running
pre-calculations later used to derive aforementioned parameters.
The latter method can be greatly advantageous as it can
significantly collapses the amount of data to be stored.
[0048] Regardless of whether portions of the image processing are
done within or beyond the data acquisition stage, there are various
hardware implementations possible. One hardware approach would
involve very simple front-end scanned data acquisition hardware and
a single high-throughput microprocessor/signal-processor chip.
Alternatively, an expanded data acquisition stage may be
implemented in high-performance dedicated function hardware and
this would be connected to a lower performance processor chip. A
third, particularly advantageous implementation would be to
implement a small pressure-sensor array together with data
acquisition and a small processor into a single low-profile chip
package that can be laid as tiles in a nearly seamless larger
array. In such an implementation all image processing could in fact
be done via straightforward partitions into message-passing
distributed algorithms.
[0049] One or more individual chips could direct output parameter
streams to an output processor which would organize and/or assign
parameters to output control channels, perhaps in a programmable
manner under selectable stored program control. A tiled macro array
of such "sensor mini-array" chips could be networked by a tapped
passive bus, one- or two-dimensional mode active bus daisy-chain, a
potentially expandable star-wired centralized message passing chip
or subsystem, or other means.
[0050] Creating a large surface from such "tile chips" will aid in
the serviceability of the surface. Since these chips can be used as
tiles to build a variety of shapes, it is therefore possible to
leverage a significant manufacturing economy-of-scale so as to
minimize cost and justify more extensive feature development.
Advanced seating and connector technologies, as used in laptops and
other high-performance miniature consumer electronics, can be used
to minimize the separation between adjacent chip "tiles" and
resultant irregularities in the tiled-surface smoothness. A tiled
implementation may also include a thin rugged flexible protective
film that separates the sensor chips from the outside world. FIG. 3
illustrates the positioning and networking of pressure sensing and
processing "mini-array" chips 1400 in larger contiguous structures
1410.
[0051] With the perfection of a translucent pressure-sensor array,
it further becomes possible for translucent pressure-sensor arrays
to be laid atop aligned visual displays such as LCDs, florescent,
plasma, CRTs, etc. as was discussed above for null/contact
touch-pads. The displays can be used to label areas of the sensor
array, illustrate gradients, etc. FIG. 7 illustrates an example
implementation involving dynamically assigned labels on a video
display 700 for use in conjunction with a transparent touch-screen
701. Note that in the "tile chip" implementation, monochrome or
color display areas may indeed be built into each chip.
[0052] Returning now to the concept of a pressure-sensor array
touch-pad large enough for hand-operation: examples of hand contact
that may be recognized, example methods for how these may be
translated into control parameters, and examples of how these all
may be used are now described. In the below the hand is used
throughout as an example, but it is understood that the foot or
even other body regions, animal regions, objects, or physical
phenomena can replace the role of the hand in these illustrative
examples.
[0053] FIG. 4 illustrates the pressure profiles for a number of
example hand contacts with a pressure-sensor array. In the case
1500 of a finger's end, pressure on the touch-pad pressure-sensor
array can be limited to the finger tip, resulting in a spatial
pressure distribution profile 1501; this shape does not change much
as a function of pressure. Alternatively, the finger can contact
the pad with its flat region, resulting in light pressure profiles
1502 which are smaller in size than heavier pressure profiles 1503.
In the case 1504 where the entire finger touches the pad, a
three-segment pattern (1504a, 1504b, 1504c) will result under many
conditions; under light pressure a two segment pattern (1504b or
1504c missing) could result. In all but the lightest pressures the
thumb makes a somewhat discernible shape 1505 as do the wrist 1506,
cuff 1507, and palm 1508; at light pressures these patterns thin
and can also break into disconnected regions. Whole hand patterns
such as the fist 1511 and flat hand 1512 have more complex shapes.
In the case of the fist 1511, a degree of curl can be discerned
from the relative geometry and separation of sub-regions (here
depicted, as an example, as 1511a, 1511b, and 1511c). In the case
of the whole flat hand 1512, there can be two or more sub-regions
which may be in fact joined (as within 1512a) and/or disconnected
(as an example, as 1512a and 1512b are); the whole hand also
affords individual measurement of separation "angles" among the
digits and thumb (1513a, 1513b, 1513c, 1513d) which can easily be
varied by the user.
[0054] Relatively simple pattern recognition software can be used
to discern these and other hand contact patterns which will be
termed "postures." The pattern recognition working together with
simple image processing may, further, derive a very large number of
independent control parameters which are easily manipulated by the
operating user. In many cases it may be advantageous to train a
system to the particulars of a specific person's hand(s) and/or
specific postures. In other situations the system may be designed
to be fully adaptive and adjust to a person's hand automatically.
In practice, for the widest range of control and accuracy, both
training and ongoing adaptation may be useful. Further, the
recognized postures described thus far may be combined in sequence
with specific dynamic variations among them (such as a finger
flick, double-tap, etc.) and as such may be also recognized and
thus treated as an additional type of recognized pattern; such
sequential dynamics among postures will be termed "gestures."
[0055] The admission of gestures further allows for the derivation
of additional patterns such as the degree or rate of variation
within one or more of the gesture dynamics. Finally, the recognized
existence and/or derived parameters from postures and gestures may
be assigned to specific outgoing control signal formats and ranges.
Any training information and/or control signal assignment
information may be stored and recalled for one or more players via
stored program control.
[0056] For each recognized pattern, the amount of information that
can be derived as parameters is in general very high. For the human
hand or foot, there are, typically, artifacts such as shape
variation due to elastic tissue deformation that permit recovery of
up to all six degrees of freedom allowed in an object's orientation
in 3-space.
[0057] FIG. 5 illustrates how six degrees of freedom can be
recovered from the contact of a single finger. In the drawing, the
finger 1600 makes contact with the touch-pad 1601 with its end
segment at a point on the touch-pad surface determined by
coordinates 1611 and 1612 (these would be, for example, left/right
for 1611 and forward/backward for 1612). Fixing this point of
contact, the finger 1600 is also capable of rotational twisting
along its length 1613 as well as rocking back and forth 1614. The
entire finger can also be pivoted with motion 1615 about the
contact point defined by coordinates 1611 and 1612. These are all
clearly independently controlled actions, and yet it is still
possible in any configuration of these thus far five degrees of
freedom, to vary the overall pressure 1616 applied to the contact
point. Simple practice, if it is even needed, allows the latter
overall pressure 1616 to be independently fixed or varied by the
human operator as other parameters are adjusted.
[0058] In general other and more complex hand contacts, such as use
of two fingers, the whole hand, etc. forfeit some of these example
degrees of freedom but often introduce others. For example, in the
quite constrained case of a whole hand posture, the fingers and
thumb can exert pressure independently (5 parameters), the finger
and thumb separation angles can be varied (4 parameters), the
finger ends 1504a can exert pressure independently from the middle
1504b and inner 1504c segments (4 parameters), the palm can
independently vary its applied pressure (1 parameter) while
independently tilting/rocking in two directions (2 parameters) and
the thumb can curl (1 parameter), yielding 17 instantaneously and
simultaneously measurable parameters which are independently
adjustable per hand. Complex contact postures may also be viewed
as, or decomposed into, component sub-postures (for example here,
as flat-finger contact, palm contact, and thumb contact) which
would then derive parameters from each posture independently. For
such complex contact postures, recognition as a larger compound
posture which may then be decomposed allows for the opportunity to
decouple and/or renormalize the parameter extraction in recognition
of the special affairs associated with and constraints imposed by
specific complex contact postures.
[0059] It is noted that the derived parameters may be pre-processed
for specific uses. One example of this would be the quantization of
a parameter into two or more discrete steps; these could for
example be sequentially interpreted as sequential notes of a scale
or melody. Another example would be that of warping a parameter
range as measured to one with a more musically expressive
layout.
[0060] Next examples of the rich metaphorical aspects of
interacting with the pressure sensor array touch-pad are
illustrated. In many cases there may be one or more natural
geometric metaphor(s) applicable, such as associating left-right
position, left-right twisting, or left-right rotation with stereo
panning, or in associating overall pressure with volume or spectral
complexity. In more abstract cases, there may be pairs of
parameters that go together--here, for example with a finger end,
it may be natural to associate one parameter pair with (left/right
and forward/backward) contact position and another parameter pair
with (left/right and forward/backward) twisting/rocking. In this
latter example there is available potential added structure in the
metaphor by viewing the twisting/rocking plane as being
superimposed over the position plane. The superposition aspect of
the metaphor can be viewed as an index, or as an
input-plane/output-plane distinction for a two-input/two-output
transformation, or as two separated processes which may be caused
to converge or morph according to additional overall pressure, or
in conjunction with a dihedral angle of intersection between two
independent processes, etc.
[0061] Next, examples of the rich syntactical aspects of
interacting with the pressure-sensor array touch-pad are
illustrated. Some instruments have particular hand postures
naturally associated with their playing. It is natural then to
recognize these classical hand-contact postures and derive control
parameters that match and/or transcend how a classical player would
use these hand positions to evoke and control sound from the
instrument. Further, some postures could be recognized either in
isolation or in gestural-context as being ones associated with (or
assigned to) percussion effects while remaining postures may be
associated with accompanying melodies or sound textures.
[0062] As an additional syntactic aspect, specific hand postures
and/or gestures may be mapped to specific selected assignments of
control signals in ways affiliated with specific purposes. For
example, finger ends may be used for one collection of sound
synthesis parameters, thumb for a second potentially partially
overlapping collection of sound synthesis parameters, flat fingers
for a third partially-overlapping collection, wrist for a fourth,
and cuff for a fifth, and fist for a sixth. In this case it may be
natural to move the hand through certain connected sequences of
motions; for example: little finger end, still in contact, dropping
to flat-finger contact, then dropping to either palm directly or
first to cuff and then to palm, then moving to wrist, all never
breaking contact with the touch-pad. Such permissible sequences of
postures that can be executed sequentially without breaking contact
with the touch-pad will be termed "continuous grammars."
[0063] Under these circumstances it is useful to set up parameter
assignments, and potentially associated context-sensitive parameter
renormalizations, that work in the context of selected (or all
available) continuous grammars. For example, as the hand contact
evolves as being recognized as one posture and then another,
parameters may be smoothly handed-over in interpretation from one
posture to another without abrupt changes, while abandoned
parameters either hold their last value or return to a default
value (instantly or via a controlled envelope).
[0064] Now a number of example applications of the pressure-sensor
array touchpad are provided. It is known to be possible and
valuable to use the aforementioned pressure-sensor array touch-pad,
implicitly containing its associated data acquisition, processing,
and assignment elements, for many, many applications such as
general machine control and computer workstation control. One
example of machine control is in robotics: here a finger might be
used to control a hazardous material robot hand as follows: [0065]
left/right position: left/right hand position [0066] in/out
position: in/out hand position [0067] in/out rock: up/down hand
position [0068] rotation: hand grip approach angle [0069] overall
pressure: grip strength [0070] left/right twist: gesture to lock or
release current grip from pressure control
[0071] A computer workstation example may involve a graphical
Computer-Aided Design application currently requiring intensive
mouse manipulation of parameters one or two at a time: [0072]
left/right position: left/right position of a selected symbol in a
2-D CAD drawing [0073] in/out position: up/down position of a
selected symbol in 2-D CAD drawing [0074] left/right twist: symbol
selection--left/right motion through 2-D palette [0075] in/out
rock: symbol selection--up/down motion through 2-D palette [0076]
rotation: rotation of selected symbol in the drawing [0077] overall
pressure: sizing by steps [0078] tap of additional finger: lock
selection into drawing or unlock for changes [0079] tap of thumb:
undo [0080] palm: toggle between add new object and select existing
object
[0081] Clearly a symbol can be richly interactively selected and
installed or edited in moments as opposed to tens to hundreds of
seconds as is required by mouse manipulation of parameters one or
two at a time and the necessary mode-changes needed to change the
mouse action interpretation.
Touch-Pad Array
[0082] Touch-pad instrument elements, such as null/contact types
and pressure-sensor array types described earlier, can be used in
isolation or arrays to create electronic controller instruments.
The touch-pad(s) may be advantageously supplemented with panel
controls such as push buttons, sliders, knobs as well as impact
sensors for velocity-controlled triggering of percussion or pitched
note events. If one or more of the touch-pads is transparent (as in
the case of a null/contact touch screen overlay) one or more video,
graphics, or alphanumeric displays 2711 may placed under a given
pad or group of pads.
[0083] FIG. 6 illustrates examples of single 2710, double 2720, and
quadruple 2730 touchpad instruments with pads of various sizes. A
single touch-pad could serve as the central element of such an
instrument, potentially supplemented with panel controls such as
push buttons 2714, sliders 2715, knobs 2716 as well as impact
sensors. In FIG. 6, a transparent pad superimposed over a video,
graphics, or one or more alphanumeric displays is assumed, and
specifically shown is a case of underlay graphics cues being
displayed for the player. Two large sensors can be put side by side
to serves as a general purpose left-hand/right-hand multi-parameter
controller.
[0084] All publications and patent applications mentioned in this
specification are herein incorporated by reference to the same
extent as if each individual publication or patent application was
specifically and individually indicated to be incorporated by
reference. The invention now being fully described, it will be
apparent to one of ordinary skill in the art that many changes and
modifications can be made thereto without departing from its spirit
or scope.
REFERENCES CITED
[0085] The following references are cited in this patent
application using the format of the first one or two authors last
name(s) within square brackets "[ ]", multiple references within a
pair of square brackets separated by semicolons ";" [0086] [Ronse]
Ronse, Christian and Devijver, Pierre A., Connected Components in
Binary Images: the Detection Problem, John Wiley & Sons Inc.
New York, 1984; [0087] [Viberg] Viberg, Mats, Subspace Fitting
Concepts in Sensor Array Processing, Linkoping Studies in Science
and Technology. Dissertations No. 27 Linkoping, Sweden 1989; [0088]
[Shapiro] Shapiro, Larry S, Affine Analysis of Image Sequences,
Cambridge University Press, 1995; [0089] [Hara] Hara, Yoshiko
"Matsushita demos multilayer MPEG-4 compression", Electronic
Engineering Times, Apr. 19, 1999.
* * * * *