U.S. patent application number 13/024569 was filed with the patent office on 2011-06-09 for user interface device, such as a mouse, with a plurality of scroll wheels.
Invention is credited to Lester F. LUDWIG.
Application Number | 20110134039 13/024569 |
Document ID | / |
Family ID | 34838368 |
Filed Date | 2011-06-09 |
United States Patent
Application |
20110134039 |
Kind Code |
A1 |
LUDWIG; Lester F. |
June 9, 2011 |
USER INTERFACE DEVICE, SUCH AS A MOUSE, WITH A PLURALITY OF SCROLL
WHEELS
Abstract
A computer mouse having a housing is provided with at least two
independent scroll wheels and a user interface sensor arrangement
that provides signals responsive to movement of the housing
relative to two orthogonal axes. The independent scroll wheels
produce signals responsive to their respective operation. Signals
from the scroll wheels and signals from the user interface sensor
arrangement can be multiplexed before transmission to an external
device such as a computer. The two independent scroll wheels can be
physically oriented orthogonally to each other. Signals from the
two independent scroll wheels can be used to control the horizontal
and vertical position of a visual object displayed in a computer
window, adjust vertical and horizontal scroll bars, and for other
purposes. Implementations can also include a visual display and an
auditory output.
Inventors: |
LUDWIG; Lester F.;
(Belomont, CA) |
Family ID: |
34838368 |
Appl. No.: |
13/024569 |
Filed: |
February 10, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12618698 |
Nov 13, 2009 |
|
|
|
13024569 |
|
|
|
|
10779368 |
Feb 13, 2004 |
7620915 |
|
|
12618698 |
|
|
|
|
Current U.S.
Class: |
345/163 |
Current CPC
Class: |
G06F 3/038 20130101;
G06F 3/04883 20130101; G06F 3/04845 20130101; G06F 3/04847
20130101; G06F 2203/04104 20130101; G06F 2203/04106 20130101; G06F
3/04144 20190501; G06F 2203/0384 20130101; G06F 3/04815 20130101;
G06F 3/03543 20130101; G06F 3/03547 20130101; G06F 3/03549
20130101; G06F 3/0346 20130101; G06F 1/266 20130101; G06F 3/0383
20130101 |
Class at
Publication: |
345/163 |
International
Class: |
G06F 3/033 20060101
G06F003/033 |
Claims
1. A user interface device for controlling an external device,
comprising: a housing; a first scroll wheel disposed on the
housing, wherein the first scroll wheel is configured to generate a
first signal relative to user manipulation; a second scroll wheel
disposed on the housing at a different location than the first
scroll wheel, wherein the second scroll wheel is configured to
generate a second signal relative to user manipulation; a user
interface sensor configured with the housing, wherein the user
interface sensor is configured to generate a plurality of third
signals responsive to movement of the housing relative to two
orthogonal axes, wherein the first scroll wheel rotates about an
axis of rotation that is orthogonally orientated to an axis of
rotation of the second scroll wheel, wherein the first and the
second signals control a first function of the external device, and
wherein the plurality of third signals control a second function of
the external device.
2. The user interface device of claim 1, wherein the first function
is used to adjust the horizontal and vertical position of a visual
object displayed on a display screen associated with the external
device.
3. The user interface device of claim 1, wherein the first function
is used to adjust a position of horizontal and vertical scroll bars
displayed in a window displayed on a display screen associated with
the external device.
4. The user interface of device claim 1, wherein the second
function positions a cursor on a display screen associated with the
external device.
5. The user interface device of claim 1 further comprises a
multiplexer configured to multiplex the first signal, second
signal, and the plurality of third signals into a single
multiplexed output signal.
6. The user interface device of claim 5 wherein the multiplexed
output signal is made available to the external device.
7. The user interface device of claim 5 wherein the multiplexed
output signal is transmitted electrically to the external
device.
8. The user interface device of claim 5 wherein the multiplexed
output signal is transmitted optically to the external device.
9. The user interface device of claim 5 wherein the multiplexed
output signal is transmitted by a wireless link to the external
device.
10. The user interface device of claim 1, wherein the first signal
is used to adjust the position of a vertical scroll bar associated
with a window displayed on a display screen of the external
device.
11. The user interface device of claim 1, wherein the second signal
is used to position a horizontal scroll bar associated with a
window displayed on a display screen of the external device.
12. The user interface device of claim 1, wherein the first signal
is used to position a horizontal scroll bar associated with a
window displayed on a display screen of the external device.
13. The user interface device of claim 1, wherein the second signal
is used to position a vertical scroll bar associated with a window
displayed on a display screen of the external device.
14. The user interface device of claim 1, wherein the plurality of
third signals is used to position a cursor on a display screen of
the external device.
15. The user interface device of claim 1, the user interface device
further comprising a visual display attached to the housing.
16. The user interface device of claim 1, the user interface device
further provides auditory output.
17. The user interface device of claim 1, the user interface device
further comprising a preprocessor.
18. The user interface device of claim 1, wherein the first and
second scroll wheels are oriented orthogonally to each other.
19. The user interface device of claim 1, the user interface device
further comprising at least a third scroll wheel.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. application Ser.
No. 12/618,698 filed on Nov. 14, 2009, which is a continuation of
U.S. application Ser. No. 10/779,368 filed on Feb. 13, 2004, now
U.S. Pat. No. 7,620,915 issued on Nov. 17, 2009.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] This invention relates to user interface devices for use
with a computer, and in particular to computer mice, trackballs,
touchpads, multiple-parameter pointing and data entry devices, and
user interface metaphors.
[0004] 2. Description of the Related Art
[0005] User interface devices for data entry and graphical user
interface pointing have been known for many years. The most common
devices include the computer mouse (usually attributed to English,
Engelbart, and Bennan "Display-Selection Techniques for Text
Manipulation, IEEE Transactions on Human Factors in Electronics,
pp. 5-15, vol. HFE-8, No. 1, March 1967), the trackball, the
touchpad in both finger-operated (for example, the various
finger-operated devices produced by Symantec Corp., of Springfield,
Oreg.) and stylus-operated (for example, products used with desktop
workstation computers by Wacom Technology Corp., of Vancouver,
Wash.) versions, and display-overlay touchscreens. Other historical
and exotic devices include various types of light pens and the Data
Glove.TM. (produced by VPL Research, Inc., of Redwood City,
Calif.).
[0006] Most user interface devices for data entry and graphical
user interface pointing commonly used with computers or with
equipment providing computer-like user interfaces have two
wide-range parameter adjustment capabilities that are usually
assigned to the task of positioning a screen cursor within a
two-dimensional display. In many cases, one, two, or three
binary-valued "discrete-event" controls are provided, typically in
the form of spring-loaded push-buttons.
[0007] More recently, computer mice have emerged that provide an
additional "scroll" finger-wheel adjustment (for example, between
two control buttons) to provide a third wide-range parameter
adjustment capability (for example, various products developed by
Logitech Inc., of Fremont, Calif.). A mouse of this configuration
is often referred to as a "Scroll Mouse" since this third
wide-range parameter is typically assigned the task of positioning
a vertical scroll bar in an actively selected window. This
additional finger-wheel adjustment may also operate as a
spring-loaded push-button, thus providing an additional
binary-valued "discrete-event" control. Typically this additional
binary-valued "discrete-event" control is used to turn on and off
an automatic scrolling feature which controls the rate and
direction of automatic scrolling according to vertical displacement
of the displayed cursor.
SUMMARY OF THE INVENTION
[0008] In accordance with embodiments of the invention, two
independently adjustable and positionable cursors are employed in a
visual interface for editing an electronic document. The document
may comprise text symbols, text objects, and graphics objects,
among others. If both cursors are located in close proximity within
the document, then these cursors may be simultaneously displayed in
a single window. Otherwise, two areas of the document, each
comprising one of the cursors, may be simultaneously displayed
using a separate window for each cursor, or alternately selected
and displayed in a single window. Copy or cut operations may be
made with one cursor, while paste operations may be repeated using
the second cursor. Cursor locations may be left unchanged, or they
may be moved between or within editing operations. The two cursors
may be freely controlled by an enhanced pointing device, such as a
mouse with an added touchpad or trackball, or sequentially selected
and controlled by a conventional mouse.
[0009] In accordance with embodiments of the invention, a
traditional hand-movable computer mouse is configured with an
additional user interface sensor. For convenience, the term "user
interface sensor" will be used herein to collectively refer to
devices such as trackballs, touchpads, mouse devices,
scroll-wheels, joysticks, and other such devices.
[0010] In one aspect of the invention, the addition of a user
interface sensor provides alternative physical modalities for the
same pair of adjustable parameters so that a user may switch
between using the user interface device as a traditional
hand-movable computer mouse and using the user interface device as
a trackball or touchpad.
[0011] In another aspect of the invention, the addition of a user
interface sensor provides alternative resolution modalities for the
same pair of adjustable parameters so that a user may switch
between using the invention as a traditional hand-movable computer
mouse to obtain one level of parameter adjustment resolution, and
using the invention as a trackball or touchpad, for example, to
obtain a different level of parameter adjustment resolution.
[0012] In another aspect of the invention, the addition of a user
interface sensor provides alternative types of warping modalities
for the same pair of adjustable parameters so that a user may
switch between using the invention as a traditional hand-movable
computer mouse to obtain one type of parameter adjustment (for
example, linear) and using the invention as a trackball or
touchpad, for example, to obtain a different type of parameter
adjustment (for example, logarithmic, gamma-corrected, arccosine,
exponential, etc.).
[0013] In another aspect of the invention, the addition of a user
interface sensor provides alternative offset modalities for the
same pair of adjustable parameters so that a user may switch
between using the invention as a traditional hand-movable computer
mouse to obtain one type of centering of parameter adjustment and
using the invention as a trackball or touchpad, for example, to
obtain a different centering of parameter adjustment.
[0014] In another aspect of the invention, the addition of a user
interface sensor may be used to provide additional parameters that
may be simultaneously controlled.
[0015] In another aspect of the invention, the addition of a user
interface sensor may be used to provide additional parameters that
are of a different isolated context from those assigned to a
traditional hand-movable computer mouse.
[0016] In a further more detailed aspect of the invention, the
addition of a touchpad may be used to provide many additional
parameters that are of a different context than those of a
traditional hand-movable computer mouse.
[0017] In a further more detailed aspect of the invention, the
touchpad may be a null-contact touchpad adapted to measure at least
one maximum spatial span of contact in a given direction.
[0018] In a yet further detailed aspect of the invention, the
null-contact touchpad is adapted to measure at least one maximum
spatial span of contact in a given direction at a specifiable
angle.
[0019] In an additional further detailed aspect of the invention,
the null-contact touchpad is adapted to measure pressure applied to
the null-contact touchpad.
[0020] In a further more detailed aspect of the invention, the
touchpad may comprise a pressure sensor array touchpad adapted to
measure, among other things, one or more of the following: the
rocking position of a contacting finger in a given direction; the
rotational position of a contacting finger; the pressure of a
contacting finger; and parameters relating to a plurality of
contacting fingers.
[0021] In another aspect of the invention the addition of a user
interface sensor may be realized via a replaceable module accepted
by an adaptation of a traditional hand-movable computer mouse. In
this implementation, a user may initially obtain the invention in
one configuration and field-modify it to another configuration.
[0022] In another aspect of the invention, a traditional
hand-movable computer mouse may be implemented as a removable
module in a laptop computer or other affiliated equipment, and may
include a wireless link with the laptop computer or other
affiliated equipment.
[0023] In yet a further aspect of the invention, a traditional
hand-movable computer mouse is implemented as a removable module in
a laptop computer or other affiliated equipment, and the mouse
further comprises a user interface sensor.
[0024] In another aspect of the invention, a traditional
hand-movable computer mouse additionally comprises a trackball or
touchpad, for example. In this aspect, the mouse comprises a
wireless link to an associated computer or other affiliated
equipment.
[0025] In another aspect of the invention, a visual display is
provided.
[0026] In another aspect of the invention, an auditory output is
provided.
[0027] In another aspect of the invention, two or more individual
user interface sensors may be combined without incorporation of
such sensors with a traditional hand-movable computer mouse.
BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The above and other aspects, features and advantages of the
present invention will become more apparent upon consideration of
the following description of preferred embodiments taken in
conjunction with the accompanying drawing figures, wherein:
[0029] FIGS. 1a-1i illustrate various exemplary implementations
involving merging, selecting, multiplexing, and preprocessing
distributed in various ways between the body of the user interface
device and an associated piece of equipment;
[0030] FIGS. 2a-2c depict an embodiment of the invention comprising
a traditional mouse fitted with a trackball, illustrating three
exemplary button configurations;
[0031] FIGS. 3a-3c depict an embodiment of the invention comprising
a traditional mouse fitted with a touchpad, illustrating three
exemplary button configurations;
[0032] FIGS. 4a-4d depict various exemplary degrees of freedom that
may be measurably assigned to a trackball for interactively
controlling parameters in a user interface;
[0033] FIGS. 5a-5d depict various exemplary degrees of freedom that
may be measurably assigned to a touchpad for interactively
controlling parameters in a user interface;
[0034] FIG. 6 depicts an exemplary implementation of the invention
directed towards the control of both a traditional text cursor and
a dual-scrollbar in a typesetting application;
[0035] FIG. 7 depicts an exemplary implementation of the invention
directed towards the active selection from a clip-art or symbol
library and adjustment of positioning or other attributes of the
active selection in a drawing or layout application;
[0036] FIG. 8 is a flowchart showing exemplary operations and
overhead involved in selecting and adjusting a specific pair of
parameters from among a larger group of adjustable parameters;
[0037] FIGS. 9a-9b illustrate how the exemplary operations and
overhead depicted in FIG. 8 introduce excessive overhead in
situations where many parameters with a larger group of adjustable
parameters must be adjusted in pairs;
[0038] FIGS. 10a-10b illustrate one technique for adding an
additional scroll-wheel to a conventional scroll-wheel mouse;
[0039] FIGS. 11a-11b illustrate a simple example of open
adjustments being made within various levels of hierarchy of
graphical object groupings;
[0040] FIG. 12 illustrates aspects of the 3D orientation of an
object in 3-dimensional space, and in particular the three
coordinates of position and the three angles of rotation;
[0041] FIGS. 13a-13b illustrate one technique for using two cursors
in a text cut-and-paste operation; and
[0042] FIGS. 14a-14d illustrate exemplary embodiments of a mouse
where the traditional mouse buttons have been replaced by
trackballs or touchpads.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0043] In the following description, reference is made to the
accompanying drawing figures which form a part hereof, and which
show by way of illustration specific embodiments of the invention.
It is to be understood by those of ordinary skill in this
technological field that other embodiments may be utilized, and
structural, electrical, as well as procedural changes may be made
without departing from the scope of the present invention.
[0044] By way of overview, a number of different applications that
take advantage of the functionality of additional, wide-range
adjustment parameters will now be discussed. In one example, an
additional finger-wheel adjustment device providing a third,
wide-range parameter adjustment capability is typically assigned to
vertical scroll bar positioning. In accordance with the invention,
such a design may be supplemented with a fourth, wide-range
parameter adjustment capability so that a horizontal scroll bar
position control may be achieved. With the increasing popularity of
the web (with many web pages wide enough to require horizontal
scrolling) and publisher layout tools (typically involving pages
wide enough to require horizontal scrolling), as well as the need
for simultaneous interactive horizontal and vertical scrolling
actions that do not disturb a screen cursor location when using
"zoom" controls, a fourth wide-range parameter adjustment
capability in traditional user interface devices for data entry and
graphical user interface pointing is quite valuable.
[0045] There are many other potential uses for additional
wide-range adjustment parameters in traditional user interface
devices for data entry and graphical user interface pointing. Some
opportunities have wide-range applicability, such as in providing
interactive separate adjustment of the selections for "cut" or
"copy" operations from the interactive adjustment of insertion
location or selection for a "paste" operation. Other opportunities
are more specialized but still widely applicable, such as making an
active selection from a clip-art or symbol library and adjusting
the position or other attributes of said active selection in a
drawing or layout application. Yet other opportunities may be very
specialized, such as in 3D modeling, data visualization, advanced
color adjustment, lighting control, machine control, or audio and
image signal processing.
[0046] There are many opportunities for adjusting the same two
widely-varying parameters in more than one way. For example, one
user interface modality (such as normal mouse operation) may be
used for normal parameter adjustment, while a second user interface
modality may be used for adjustments involving a different
resolution, warping (i.e., logarithmic, gamma-corrected, arccosine,
exponential, etc.), centering offset, etc. Another important case
is where the same two widely-varying parameters are controlled with
the same resolution, warping, offset, etc., but in a different user
interface modality (e.g., a trackball or touchpad may have some
advantages in certain situations over use of a traditional mouse).
A more widely applicable example is that of responding to and
preventing hand/wrist/arm fatigue and injury. A traditional mouse
fitted with an additional user interface sensor allows a user to
interchangeably enter information with either the mouse body or
another user interface sensor, changing which user interface
modality is used (obtaining the same results with either) to
relieve fatigue or pain, or prevent injury.
[0047] More specifically, the addition of a user interface sensor
provides many opportunities for alternative means of adjustment of
a common pair of adjustable parameters. The user may benefit from
having both adjustment modalities available, changing modalities as
needed or desired. For example: [0048] A user may simply switch
between using the invention as a traditional hand-movable computer
mouse and using the invention as another kind of user interface
sensor. [0049] The user may benefit from having both modalities
available to avoid or in response to hand fatigue. [0050] The user
may also benefit from having both modalities available due to the
type of pointing or data entry interaction needed--depending on the
case, one type of modality may perform better than another. [0051]
The trackball, touchpad, or other user interface sensor apparatus
may be used to provide alternative resolution modalities so that a
user may switch between using the invention as a traditional
hand-movable computer mouse to obtain one level of parameter
adjustment resolution and using the invention as a user interface
sensor to obtain a different level of parameter adjustment
resolution. [0052] The trackball or touchpad may be used to provide
alternative warping modalities for the same pair of adjustable
parameters so that a user may switch between using the invention as
a traditional hand-movable computer mouse to obtain one type of
parameter adjustment (for example, linear) and using the invention
as another kind of user interface sensor to obtain a different type
of parameter adjustment resolution (for example, logarithmic,
gamma-corrected, arccosine, exponential, etc.). [0053] The user
interface sensor may be used to provide alternative offset
modalities for a common pair of adjustable parameters so that a
user may switch between using the invention as a traditional
hand-movable computer mouse to obtain one centering of parameter
adjustment and using the invention as another kind of user
interface sensor to obtain a different centering of parameter
adjustment. These modalities can provide one or more "location
bookmarks" for cursor location, each affiliated with a sub-context
within an interactive application.
[0054] Further, the addition of another user interface sensor
provides many opportunities for the simultaneous adjustment of
additional parameters that may or may not require simultaneous
interactive control. The traditional computer mouse may be used to
simultaneously adjust two parameters while the additional user
interface sensor may be configured to allow the fingers to
simultaneously adjust at least two additional parameters. In some
applications, these additional parameters may be closely related to
those assigned to the traditional computer mouse. For example, the
traditional computer mouse may be used to simultaneously adjust the
location within a window of a text, graphic, or other object, while
the additional user interface sensor allows the fingers to be used
to adjust the type or attributes of the text, graphic, or other
object. In other applications, these additional parameters may be
of a different isolated context from those assigned to the
traditional computer mouse. For example, the traditional computer
mouse may be used to simultaneously adjust two parameters dealing
with affairs within an active application window, while the
addition of another user interface sensor allows the fingers to be
used to adjust at least two additional parameters dealing with
broader window system affairs such as vertical and horizontal
scrollbars, window selection, window resizing, etc., or
intermediate-level affairs such as zoom control, help-window
navigation, clip-art selection, etc. Another application would be
to provide separate adjustment of selections for "cut" or "copy"
operations from the adjustment of insertion location or selection
for a "paste" operation.
[0055] In instances of the invention involving the addition of a
touchpad, the touchpad may be configured and/or enhanced to allow
the fingers to adjust three or more additional interactive measured
parameters. These additional interactive measured parameters may be
assigned to control more sophisticated interactive affairs such as
3-dimensional space position, 3-dimensional space orientation,
color model navigation, image or audio processing parameter
settings, etc.
[0056] The additional interactive measured parameters (above the
two typically associated with traditional touchpads) may be
provided in a number of ways. For example, the touchpad may be a
relatively low-cost null-contact touchpad that has been adapted to
measure at least one maximum spatial span of contact in a given
direction. The user may also control an additional parameter by
varying the width between the spatial extremes of a single point of
contact (i.e., how much finger flesh makes contact with the pad) or
multiple points of contact (i.e., the spread between two contacting
fingers). As there are two geometrically orthogonal sensing
directions on a touchpad, this provides the user with a method for
controlling four total parameters from a touchpad. Further,
rotational transformations or other methodologies may be used to
measure the angle of rotation of an oblong contact profile. The
measured angle may be used as a fifth interactive parameter, and/or
used to adapt the measurement of maximum spatial span of contact in
an arbitrary angle. The null-contact touchpad may be further
adapted to measure pressure applied to the null-contact touchpad
via, for example, use of an attached pressure sensor. The pressure
may be used as a sixth interactive parameter, and may or may not
have rapid pressure changes recognized as `tap` or `click`
events.
[0057] Another way to provide the additional interactive measured
parameters (above the two typically associated with traditional
touchpads) with a touchpad is to implement the touchpad with a
pressure sensor array. Through use of operations effectively
amounting to image processing, a pressure sensor array touchpad can
be adapted to measure the rocking position of a contacting finger
in two orthogonal directions, as well as the rotational position
and average pressure of a contacting finger. Thus a pressure sensor
array touchpad can be adapted to provide up to six widely variable
interactive adjustable parameters from the contact of a single
finger. A pressure sensor array touchpad can be further adapted to
measure parameters relating to a plurality of contacting
fingers.
[0058] All of these considerations and others demonstrate the
potential value in providing the addition of another user interface
sensor to a traditional hand-movable computer mouse. In the
descriptions to follow, various implementations and exemplary
applications of exemplary embodiments are considered and
explained.
1. Exemplary Signal Flow and Processing
[0059] The invention provides for a wide range of signal flow and
processing configurations and implementations. FIGS. 1a-1i
illustrate various exemplary implementations involving merging,
selecting, multiplexing, and preprocessing distributed in various
ways between the body of the user interface device and an
associated piece of equipment. FIGS. 1a-1d concern the aggregated
pair of user interface sensors in isolation, while FIGS. 1e-11
address arrangements where some functions of the invention are
performed in the associated external equipment. It is noted that
the invention further provides for any of these exemplary
functionalities, as well as other functionalities, to be combined
or made selectable. In any of the exemplary implementations
disclosed herein, power may be supplied to these implementations by
the associated external equipment or by other devices such as
batteries, storage capacitors, photoelectric devices, and the
like.
[0060] FIG. 1a shows an implementation 100a featuring two user
interface sensors 101, 102, each of which may be a particular type
of user interface sensor (which again may be a trackball, touchpad,
mouse, or other user interface device) that can be collocated
within the common physical enclosure 100 (demarcated by the
dotted-line boundary). First user interface sensor 101 produces
signal 108 and second user interface sensor 102 produces another
signal 109, which are directed to a merge or select function 103.
The merge or select function produces outgoing signal 110 which is
provided to associated external equipment. Here signals 108,109
(from first and second user interface sensors 101,102) would
typically lose their individual identities within the outgoing
signal 110 and as such may be used or processed interchangeably
(without individual attribution to either the first or second user
interface sensor) by the associated external equipment.
[0061] Merge or select function 103 may take several forms in
various implementations. For example, in one embodiment it may
simply be fixed to only perform a merge operation. In another
embodiment it may only provide a selection function; here the
selection function may be controlled by the user using a switch or
some sort of action, or the selection may be remotely controlled by
external equipment. As an alternative, merge or select function 103
may instead provide a user adjustable "merge" or "select"
function.
[0062] FIG. 1b shows an implementation that is similar to that of
FIG. 1a. The primary difference is that the FIG. 1b implementation
100b replaces merge or select function 103 of FIG. 1a with
multiplex function 104 to produce outgoing signal 110. Here signals
108, 109 (from first and second user interface sensors 101, 102)
retain their individual identities within outgoing signal 110 and
as such may be used or processed separately by the associated
external equipment.
[0063] FIG. 1c shows implementation 100c which is similar in many
respects to that of FIG. 1a. However, the FIG. 1c embodiment
utilizes preprocessor 105 applied to signal 109 to produce
processed signal 109a. The processed signal 109a, along with signal
108, is directed to merge or select function 103, resulting in
outgoing signal 110. Preprocessor 105 may thus introduce a
pre-processing step (such as resolution modification, warping
modification, offset modification, etc.) on signal 109 to produce a
signal of distinguished value from that of signal 108.
[0064] FIG. 1d illustrates another exemplary implementation 100d
which is similar to that of FIG. 1c but with an additional
preprocessor 106 applied to signal 108. Preprocessor 106 produces
processed signal 108a, which along with signal 109a, is directed to
merge or select function 103 to produce outgoing signal 110.
Preprocessor 106 may therefore introduce a pre-processing step
(such as resolution modification, warping modification, offset
modification, etc.) on signal 108 to produce a signal of either
equivalent or distinguished value from that of signal 109a.
[0065] FIG. 1e shows implementation 100b of FIG. 1b (which features
two user interface sensors 101,102 and a multiplex function 104
producing an outgoing signal 110) used in conjunction with
subsequent functions provided by associated external equipment
150a. These subsequent functions are shown within the functional
boundary 150 of the associated external equipment 150a.
[0066] Outgoing signal 110 from the common physical enclosure 100,
is presented to demultiplexer 117 within external equipment 150a.
Demultiplexer 117 produces signal 118 corresponding to or
associated with pre-multiplexed signal 108, and an additional
signal 119 corresponding to or associated with the pre-multiplexed
signal 109. Here, signals 118, 119 are presented to merge or select
function 113 producing merged or selected signal 120. This
implementation is functionally similar or equivalent to that of
FIG. 1 a except that the various types of merge or selection
functions 103 are provided within the associated external equipment
150a (for example in software, perhaps within an application where
it is customized for the needs of that application) rather than
being provided within physical unit 100b.
[0067] FIG. 1f shows implementation 100b of FIG. 1b used in
conjunction with subsequent functions provided by associated
external equipment 150b, and similar to that of FIG. 1e, except
signal 119 produced by demultiplexer 117 is directed to
preprocessor 115 to produce processed signal 119a before being sent
to merge or selection function 113. This implementation is thus
functionally similar or equivalent to that of FIG. 1c except that
the various types of merge or selection 103 and preprocessor 105
functions in FIG. 1c are provided within the associated external
equipment 150b (for example in software, perhaps within an
application where it is customized for the needs of that
application) rather than being provided within the physical unit
100b.
[0068] FIG. 1g shows implementation 100b of FIG. 1b used in
conjunction with subsequent functions provided by associated
external equipment 150c. This arrangement expands on that shown in
FIG. 1f in that signal 118 produced by demultiplexer 117 is
directed to preprocessor 116 to produce processed signal 118a. The
processed signal is then sent to merge or selection function 113.
This implementation is thus functionally similar or equivalent to
that of FIG. 1d except that the various types of merge or selection
103 and preprocessor 105, 106 functions in FIG. 1d are provided
within the associated external equipment 150c (for example in
software, perhaps within an application where it is--customized for
the needs of that application) rather than being provided within
the physical unit 100b.
[0069] FIG. 1h shows implementation 100b of FIG. 1b used in
conjunction with subsequent functions provided by associated
external equipment 150d. Here again, as in the arrangement of FIG.
1f, signal 119 produced by demultiplexer 117 is directed to
preprocessor 115 to produce processed signal 119a. In contrast to
other embodiments, processed signal 119a is not directed to merge
or selection 113 function and retains its identity for use by a
different destination from that of signal 118 within associated
external equipment 150d.
[0070] As a final illustrative example in this series, FIG. 1i
shows implementation 100b of FIG. 1b used in conjunction with
subsequent functions provided by associated external equipment
150e. Here, as in the arrangement of FIG. 1g, signals 118, 119
produced by demultiplexer 117 are directed to preprocessors 115,
116 to produce processed signals 118a, 119a. In contrast to other
embodiments, processed signals 118a, 119a are not directed to merge
or selection 113 function and thus retain their identity for use by
differing destinations within associated external equipment
150e.
[0071] Having presented various exemplary signal flows and
processing realizations provided for by the invention, attention is
now directed to exemplary implementations utilizing specific types
of user interface sensors. It is to be understood that the various
sensors, techniques and methods disclosed herein may be implemented
using computer software, hardware, and firmware, and combinations
thereof.
2. Implementations Utilizing Specific Types of Additional User
Interface Sensors
[0072] In this section, a number of exemplary implementations of
the invention utilizing various types of additional user interface
sensors added to an original user interface sensor or device. The
first three sections address cases where the original user
interface sensor is a movable mouse and the additional user
interface sensor is a trackball, touchpad, or other exemplary
technology, including additional scroll-wheels. Then exemplary
adaptations of trackballs and touchpads, each traditionally used to
provide simultaneous adjustment of two interactive widely-varying
parameters, are extended to provide simultaneous adjustment of as
many as six interactive widely-varying parameters and other forms
of control. This section continues by presenting exemplary
implementations where the original user interface sensor is not a
mouse, where there is a plurality of additional user interface
sensors, where there is a visual display or auditory output, where
the additional user interface sensor is a removable module, and
where the implementation itself is a removable module.
[0073] 2.1 Trackball Implementations of Additional User Interface
Sensors
[0074] FIGS. 2a-2c illustrate a number of implementations provided
for by the invention where a trackball controller is used as an
additional user interface sensor apparatus incorporated into a
traditional hand-movable computer mouse. In each of these
implementations it is understood that the trackball may be freely
operated without disturbing previous or currently-varying parameter
adjustments made by the mouse.
[0075] In one implementation, a trackball controller is added to
the top surface of a conventional computer mouse as depicted in
FIG. 2a. The conventional mouse buttons may be located in various
places in view of the presence of the trackball and in synergy with
it. In the configuration depicted in FIG. 2a, buttons 201 and 202
are located on the surface of mouse 200; button 201 being on the
left of the trackball and button 202 being on the right of the
trackball.
[0076] In a second configuration depicted in FIG. 2b, buttons 231
and 232 are now separated and located on the sides of mouse 230 as
is the case with many trackball interfaces; button 231 being on the
left side of mouse 230 and button 232 being on the right side of
the mouse.
[0077] In a third possible configuration depicted in FIG. 2c,
elongated buttons 261 and 262 are shown located on the surface of
the mouse 260; button 261 wraps around the left of the trackball
and button 262 wraps around the right of the trackball. The
elongated buttons 261 and 262 may be positioned so that a user can
readily and rapidly move fingers from trackball 265 to buttons 261
and 262, or even operate one of these buttons with one finger while
another finger contacts trackball 265.
[0078] It is noted that unlike the touchpad described below, the
trackball has an effectively unconfined range of contiguous data
entry.
[0079] 2.2 Touchpad Implementations of Additional User Interface
Sensor
[0080] FIGS. 3a-3c illustrate a number of exemplary implementations
provided for by the invention where a touchpad controller is-used
as an additional user interface sensor incorporated into a
traditional hand-movable computer mouse. In each of these
implementations it is understood that the touchpad may be freely
operated without disturbing previous or currently-varying parameter
adjustments made by the mouse.
[0081] In one implementation, trackball 205 in FIG. 2a can be
replaced with touchpad 305 as shown in FIG. 3a. Additionally, this
touchpad implementation can also support the alternative button
configurations of FIGS. 2b and 2c. By way of illustration, FIG. 3b
shows buttons 331, 332 positioned on either side of the mouse body
330, while FIG. 3c shows elongated buttons 361, 362 on the surface
of the mouse wrapping around either side of touchpad 365.
[0082] It is noted that unlike the trackball, the touchpad
typically has a confined maximum range of data entry by contiguous
operation of a finger, stylus, etc.
[0083] 2.3 Other Implementations of Additional User Interface
Sensors.
[0084] In accordance with other embodiments, the invention provides
for still other types of additional user interface sensors. In each
of these embodiments it is understood that any of these user
interface sensors may be freely operated without disturbing
previous or currently varying parameter adjustments made by the
mouse or other associated device.
[0085] As a first example, an X-Y joystick may be used in place of
the trackball or touchpad described above. The joystick may have a
spring-return or may retain the last position it was placed.
Similar to the touchpad and unlike the trackball, the X-Y joystick
typically has a confined maximum range of travel.
[0086] As another example, two or more scrolling finger wheels may
be used in place of the trackball or touchpad described above. The
scrolling finger wheels may be implemented with an unconfined
maximum range of travel similar to the trackball, or with a
confined range of travel like the touchpad and X-Y joystick. In
this embodiment, it may be advantageous to have one or more finger
scroll-wheels mounted with its adjustment direction perpendicular
to that of another finger scroll-wheel so that each wheel may be
appropriately associated with vertical and horizontal scroll bars
of a window, or other useful orthogonally-based user interface
metaphor. For example, looking ahead to FIGS. 10a and 10b,
embodiments 1000 are depicted comprising the usual components of a
scroll-wheel mouse (mouse body 1001, buttons 1011 and 1012, and the
usual scroll wheel 1021) complemented with an additional scroll
wheel 1022 with adjustment direction perpendicular to that of
finger scroll-wheel 1021.
[0087] As another example, two or more rotating control knobs may
be used in place of the trackball or touchpad described above. Like
the scrolling finger wheels, the rotating control knobs may be
implemented with an unconfined maximum range of travel like the
trackball or with a confined range of travel like the touchpad and
X-Y joystick.
[0088] The invention also provides for more exotic types of user
interface sensor technologies--for example, proximity detectors of
various types (RF, infrared, etc.), video cameras (using any of the
techniques provided in U.S. Pat. No. 6,570,078), and other emerging
and unforeseen technologies--to be used as the additional user
interface sensor aggregated in the same physical enclosure as the
first user interface sensor.
[0089] 2.4 Larger numbers of Interactively Widely-Varying
Adjustable Parameters from the Additional User Interface Sensor
[0090] In accordance with embodiments of the invention, additional
user interfaces may be used to capture larger numbers (i.e., more
than two) of widely-varying adjustable parameters from the
additional user interface sensor. Some examples are provided here,
but many others are possible as may be readily understood by one
skilled in the art.
[0091] FIGS. 4a-4d address the case of the trackball. FIG. 4a
illustrates a freely-rotating trackball 205 and the two principle
orthogonal adjustment directions 401,402 that are responsively
resolved and measured in traditional trackball user interface
devices. However, at least two other physical degrees of freedom
may be readily exploited, and at least six total parameters can be
interactively adjusted and measured. FIG. 4b shows the application
of downward pressure 403 on trackball 205. Such pressure 403 may be
applied without disturbing current values established in orthogonal
adjustment directions 401, 402. Further, the trackball may be
implemented so that downward pressure 403 may be applied while
simultaneously adjusting trackball 205 in orthogonal adjustment
directions 401, 402, particularly if the signal produced by the
measurement of downward pressure 403 incorporates a modest "grace"
zone of non-responsiveness for light pressure values. Downward
pressure impulses may alternatively be sensed and treated as
discrete event "taps," as commonly used in contemporary touchpad
interfaces found in laptop computers, for example.
[0092] FIG. 4c shows the application of "yaw" rotation 404 (i.e.,
rotation around the vertical axis) of trackball 205. This yaw
rotation 404 may be applied without disturbing current values
established in orthogonal adjustment directions 401,402 and can be
readily measured and adjusted as a widely-varying parameter.
Further, by grasping trackball 205 (or other operational methods),
the yaw rotation 404 and traditional orthogonal adjustment
directions 401,402 may be independently and simultaneously
adjusted. It is also noted that in principle up to six
widely-varying physical degrees of freedom can be simultaneously
measured from a properly configured trackball 205 by placing the
trackball 205 in a cradle that senses not only downward pressure
403 or displacement but also lateral pressure or displacement. As
shown in FIG. 4d, both forward-backward, non-rotational force 405
and left-right, non-rotational force 406 may be applied to the
trackball in a manner that the values of force or displacement in
each of these directions 405, 406 can be independently measured.
Thus, by grasping trackball 205 (or other operational methods),
three rotational directions of orientation 401, 402, 404 and three
non-rotational directions of force or displacement 403, 405, 406
may be independently and simultaneously adjusted by a user and
measured as six independent interactively adjustable user interface
parameters. These correspond, effectively, to measurable
adaptations of the six degrees of freedom of an orientable object
in 3-dimensional space as found in classical mechanics and
aeronautics--that is: [0093] "roll" rotation (adapted to 401)
[0094] "pitch" rotation (adapted to 402) [0095] "up-down:
displacements (adapted to 403) [0096] "yaw" rotation (adapted to
404) [0097] "forward-backward" displacements (adapted to 405)
[0098] "left-right" displacements (adapted to 406).
[0099] Most trackball sensing technologies use optically based
techniques for sensing the two traditional components of rotation
("roll" and "pitch") of the trackball. The trackball itself may be
configured with an optical pattern on it with spatially varying
reflectivity for a range of the light spectrum. The pattern may be
such that it can spatially vary light reflectively in these two
traditional components of trackball rotation. Alternatively, two
spatially varying reflectivity patterns, each active at different
ranges of the light spectrum or light polarization, may be
superimposed or integrated with the trackball.
[0100] A number of approaches may be used to obtain measurements
for all three directions of rotation. In one completely optical
approach, a second or third spatially varying reflectivity pattern
active at, respectively, a second or third portion of the light
spectrum (or light polarization if available) may be superimposed
or integrated with the patterns employed for traditional "roll" and
"pitch" rotation sensing, and an additional optical source and
sensor is used to obtain measurement of the added varying
reflectivity pattern. Depending on the pattern(s) used, sensor
signals may be directly usable or may require processing of the
three primitive signals measured by the sensors to obtain a clean
decomposition of the measurement signals into independent "roll,"
"pitch," and "yaw" signals independently responsive to the "roll,"
"pitch," and "yaw" components of trackball rotation.
[0101] As another alternative, the trackball may include
internally, or on its surface, or both, materials with spatially
varying patterns of magnetic properties, capacitive properties,
electromagnetic properties, ultrasonic acoustic properties,
resonance phenomena of any of these properties, polarization
phenomena of any of these properties, etc., individually or in
combination, each of which may be active at specific ranges of
polarization, frequencies, etc. These may be used together with or
in place of optical measurement approaches. Again, depending on the
pattern(s) used, sensor signals may be directly usable or may
require processing of the three primitive signals measured by the
sensors to obtain a clean decomposition of the measurement signals
into independent "roll," "pitch," and "yaw" signals independently
responsive to the "roll," "pitch," and "yaw" components of
trackball rotation as is clear to one skilled in the art. It is
also noted that the third component of rotation of the
freely-rotating trackball may be interpreted or even measured as a
discrete "click" event.
[0102] Similarly, a number of approaches may be used to obtain
measurements for one, two, or three directions of non-rotational
trackball displacement. For example, the trackball may be secured
in a saddle allowing free rotation of the trackball but causing any
displacement actions on the trackball to invoke displacements of
the saddle. The saddle displacement may be measured with a
displacement sensor which itself may comprise one or more pressure,
optical, resistive, capacitive, magnetic, electromagnetic,
continuous-range sensors, switches, etc. It is also noted that one
or more components of displacement of the freely-rotating trackball
may be interpreted or even measured as a discrete "click"
event.
[0103] FIGS. 5a-5d turn now to the case of the touchpad. FIG. 5a
illustrates touchpad 305 and the two principle orthogonal data
entry directions 501, 502 that are responsively resolved and
measured in traditional touchpad user interface devices. The
touchpad shown in FIGS. 5a-5d, which provides at least four other
physical degrees of freedom, may be implemented using the
techniques presented in U.S. Pat. No. 6,570,078, for example.
[0104] FIG. 5b illustrates the use of downward pressure 503 in the
context of a touchpad. In contemporary touchpad interfaces, such as
those found in laptop computers for example, such downward pressure
503 is sensed and utilized as discrete event "taps." However,
downward pressure 503 may also be measured and adjusted as an
independent and simultaneously interactive widely-varying
parameter. Further, as illustrated in FIG. 5c, the rotational angle
504 of a finger contacting a touchpad with rough-elliptical contact
boundary can also be measured as a widely-varying parameter. In
FIG. 5d, both forward-backward 505 and left-right 506 components of
the tilt of a contacting finger can additionally be measured as
independent and simultaneously interactive widely-varying
parameters.
[0105] The sensing of multiple fingers, the application of contact
syntaxes and grammars, and other user interface control expansions
of an adequately configured touchpad may also be achieved using,
for example, the techniques presented in U.S. Pat. No.
6,570,078.
[0106] The invention also provides for larger numbers (i.e., more
than two) of widely varying adjustable parameters from other types
of user interface sensor technologies. In the case of an X-Y
joystick, the joystick may be configured to rotate on its axis,
pulled in and out, fitted with a knob or trackball, etc., in a
measurable fashion to provide additional and simultaneous
interactively adjustable parameters. In the cases of finger
scroll-wheels and rotational knobs, three or more of these devices
may be provided. When implementing video cameras, known techniques
for the extraction of additional parameters may be used. Examples
of the various types of video extraction techniques that may be
used are presented in U.S. Pat. No. 6,570,078.
[0107] 2.5 Non-Mouse User Interface Sensors
[0108] In one of its most abstract forms, the invention involves
the incorporation of two conventional user interface sensors (such
as a mouse, trackball, touchpad, joystick, etc.) into an embodiment
where the user may freely use both of the user interface sensors
individually or simultaneously with the same hand. As such, the
invention provides for implementations that do not include a mouse
as one of the user interface sensors. For example, two or more
individual user interface sensors can be combined without need of a
traditional hand-movable computer mouse. Such an implementation may
be accomplished by implementing one of the possible user interface
sensors in place of the traditional hand-movable computer mouse
where taught in other descriptions of the invention. This
configuration may be useful when built into a laptop computer,
control console, musical instrument, and test instrument, among
others.
[0109] In one exemplary implementation of a non-mouse embodiment, a
trackball and touchpad may be arranged in the same physical
enclosure so that the front, middle, or back of the palm may freely
operate a conventional trackball while one or more selected
extended or arching fingers may simultaneously or alternatively
operate a touchpad. In this example, the touchpad may be a
conventional touchpad providing two widely-varying simultaneously
interactive parameters from a single finger contact, or the
touchpad may be a more enhanced version providing as many as six
widely-varying and simultaneously interactive parameters from a
single finger contact. The touchpad may also be configured to
accept multiple points of contact, recognize gestures, support
syntax and grammatical constructions using, for example, the
teachings provided by U.S. Pat. No. 6,570,078.
[0110] In another non-mouse implementation, two trackballs may be
arranged in the same physical enclosure. In one possible
arrangement, the two trackballs may be positioned so that they lie
parallel to the length of the hand, enabling the front, middle, or
back of the palm to freely operate a first trackball while one or
more extended or arching fingers may simultaneously or
alternatively operate the second trackball.
[0111] In another arrangement, the two trackballs may be positioned
so that they lie parallel to the width of the hand, so that the
fingers and/or thumb on the left side of the hand may operate a
leftmost trackball while the remaining fingers and/or thumb on the
right side of the hand may individually or simultaneously operate a
rightmost trackball. In each of these arrangements, either or both
of the trackballs may be a conventional trackball providing two
widely-varying and simultaneously interactive parameters, or it may
be a more enhanced trackball providing as many as six
widely-varying and simultaneously interactive parameters as
described earlier.
[0112] In addition to the just-described embodiments, alternative
arrangements, such as the combination of a palm-operated trackball
and a recessed joystick, and others, are also provided for by the
invention.
[0113] 2.6 Use of More than One Additional User Interface
Sensor
[0114] Typically the arrangements of two non-mouse user interface
sensors described above in Section 2.5 can also be applied to
embodiments of the invention where a mouse user interface sensor is
used. In such embodiments, a mouse user interface sensor is
supplemented with at least two additional user interface sensors
ergonomically arranged so that the two additional user interface
sensors may be simultaneously or alternatively operated by the same
hand. If these embodiments are further configured so the mouse body
is readily moved with adequate precision via the back of the
operating hand, then all three user interface sensors may be
simultaneously or alternatively operated by the same hand in an
ergonomically advantageous manner.
[0115] FIGS. 14a-14d illustrate some exemplary embodiments of the
just-described features. FIG. 14a illustrates a mouse where
traditional mouse buttons have been replaced by trackballs 1405a,
1405b. These trackballs 1405a, 1405b may accept a downward pressure
impulse and as such act as the traditional mouse buttons. However,
trackballs 1405a, 1405b are also adjustable and each readily
provides two or more additional widely-variable and simultaneously
adjustable parameters in addition to the two parameters adjusted by
moving the mouse body 1400.
[0116] FIG. 14b shows a similar arrangement where traditional mouse
buttons have been replaced by touchpads 1435a, 1435b. If desired,
these touchpads may accept a downward pressure impulse and as such
act as traditional mouse buttons, but are also adjustable as
touchpads and as such each readily provides two or more additional
widely-variable and simultaneously adjustable parameters. As
described in Section 2.5, a single hand may be positioned to
comfortably operate simultaneously or alternatively both trackballs
or both touchpads. If these embodiments are further configured so
the mouse body is readily moved with adequate precision via the
back of the operating hand, then either of these embodiments
readily provides six to twelve widely-variable and simultaneously
adjustable parameters.
[0117] Other configurations are of course possible. For example,
the configurations of FIGS. 14a and 14b may be blended as depicted
in FIG. 14c, or in its mirror image. As another example, FIG. 14d
illustrates a more extreme realization comprising a
left-fingers/thumb trackball 1465a, a right-fingers/thumb trackball
1465b, a palm trackball 1465c, and a traditional clickable
scroll-wheel 1468. Yet another alternative is to replace one or
more of the trackballs of the FIG. 14d embodiment with a touchpad
user interface sensor.
[0118] 2.7 Incorporation of Visual Display and Auditory Output
[0119] If desired, any of the mouse and non-mouse embodiments may
further include a visual display. The visual display may provide
details of adjustable parameter values, operation modalities, etc.
The visual display may be physically associated with a physical
enclosure (such as that of a traditional computer mouse), or may be
displayed on the computer screen or display of other associated
equipment.
[0120] Alternatively or additionally, any of the mouse and
non-mouse embodiments may further provide auditory output. The
auditory output may provide details of adjustable parameter values,
operation modalities, error conditions in usage of the invention, a
condition relating to elapsed time or other metric of consistent
use of a single usage modality, etc. The auditory output associated
with the invention may be physically associated with a physical
enclosure (such as that of a traditional computer mouse), or may be
produced by speakers or headsets affiliated with the computer or
other associated equipment.
[0121] 2.8 Provisions for Field Installation or Replacement of
Additional User Interface Sensor
[0122] The invention also provides for the user interface sensor to
be implemented using a replaceable module accepted by an adaptation
of a traditional computer mouse. In this implementation a user may
initially obtain the invention in one configuration and field
modify it to another configuration.
[0123] 2.9 Implementation as a Module Removable from Affiliated
Equipment
[0124] The invention also provides for a traditional computer mouse
to be implemented as a removable module in a laptop computer or
other affiliated equipment, and may include a wireless link with
such devices. In particular, this removable module may further
include one or more user interface sensors, with these sensors
operable as a traditional trackball or touchpad when the invention
is stowed in the laptop computer or other affiliated equipment in
such a way that the invention's traditional hand-movable computer
mouse modality is unmovable and hence unusable.
3. Exemplary Applications
[0125] Departing now from the range of extreme realization and
embodiments of the invention, attention is directed towards
particular applications of the invention. A number of examples of
various embodiments of the invention in a wide range of
applications will now be presented. Many of these applications are
viable with only the simplest physical embodiments of the invention
(for example, those suggested by FIGS. 2a-2c and 3a-3c). In the
discussion that follows, particular note is directed towards the
discussion in Section 3.3 involving FIGS. 8 and 9a-9b. Although the
discussion is motivated by a graphical layout application, the
principles of the discussion in Section 3.3 involving FIGS. 8 and
9a-9b are very general, and the discussion illustrates the power of
the invention for various applications in almost directly
quantifiable terms.
[0126] 3.1 Wrist/Hand/Arm-Fatigue Relief and Prevention
Application
[0127] The danger and damage stemming from extensive continuous or
mis-postured mouse usage to wrist, hand, and arms are sadly
misfortunate and increasingly well recognized. As the present
invention provides a plurality of different user interface sensors,
it is well suited for use in responding to and preventing
wrist/hand/arm fatigue due to overuse. In one exemplary
implementation, user interface parameters can be interchangeably
adjusted with either the movement of the mouse body or the use of
an integrated trackball or touchpad with identical task results.
Thus a user with a tiring hand can change at will the user
interface sensor employed according to how the hand feels or the
nature of a specific task. In addition, to prevent fatigue or
injury, the user can also switch back and forth between moving the
mouse body and using the trackball/touchpad either by free choice
or by following auditory or visual prompting from a time or usage
monitor.
[0128] 3.2 Double-Scrollbar Application
[0129] Contemporary mice often feature a small rotating wheel
between the buttons for use in operating the vertical scroll bar of
a window without changing the position of the mouse. In one
particular application of the invention, the left-right sensing
capability of the trackball or touchpad may be used to add a
similar capability for horizontal scroll bars of a window.
[0130] In a trackball implementation, a user can move the vertical
bar 611 of FIG. 6 up by rotating trackball 205 away from
him/herself, or one can move the vertical bar 621 down by rotating
trackball 205 towards him/herself. By rotating trackball 205 to the
left, the user can move the horizontal bar 621 left. Similarly, the
user can move the horizontal bar 621 right by rotating the
trackball 205 to the right. In a touchpad implementation, a user
can move scroll bar 611 up by sliding the finger away from
her/himself or move scroll bar 611 down by sliding the contacting
finger towards her/himself; similarly, the user can move the scroll
bar 621 left by sliding the finger to the left or move the scroll
bar 621 right by sliding the contacting finger to the right.
[0131] In another implementation, the vertical and horizontal
scroll bars may be adjusted with a conventional scroll-wheel mouse
that has been fitted with an additional scroll-wheel. FIGS. 10a and
10b depict exemplary embodiments 1000 of such an arrangement which
comprise the usual components of a scroll-wheel mouse including
mouse body 1001, buttons 1011 and 1012, and traditional scroll
wheel 1021, with these components further complemented by an
additional scroll wheel 1022 with adjustment direction
perpendicular to that of finger scroll-wheel 1021. FIG. 10a
illustrates an arrangement where the additional scroll-wheel is
located closer to the user while FIG. 10b illustrates an
arrangement where the additional scroll-wheel is located farther
away from the user. In each of these arrangements, the two
scroll-wheels are shown co-centered with respect to the mouse body;
for simultaneous adjustment it may be advantageous to locate the
additional scroll-wheel 1022 to one side or the other of the
centered positions shown in FIGS. 10a and 10b. One approach useful
for supporting both left-handed and right-handed users, which may
provide additional utility, would be to provide two off-centered
additional scroll-wheels, one on either side of the center line of
the mouse body 1001 and conventional scroll-wheel 1021.
[0132] 3.3 Traditional 2D Layout, CAD, and Graphics
Applications
[0133] In most contemporary 2-dimensional layout and graphics
applications, such as those commonly used for viewgraphs, page
layout, electronic CAD, etc., numerous mouse operations are
necessary for each of the many types of object attribute
modification, etc. Typically, these mouse operations are required
because the mouse only allows for the interactive adjustment of two
widely-varying parameters at a time, and the user must change
context several times as the parameters adjusted by the mouse are
chosen, adjusted, and then replaced with another pair of
parameters. The present invention is useful in many of these
circumstances because it allows for more than two parameters to be
adjusted at the same time.
[0134] FIG. 7 shows an example of a session involving the authoring
of a viewgraph. The viewgraph authoring task showcased in this
example includes the creation of a flowchart diagram (here
depicting a business workflow process) and as such also illustrates
related needs and attributes of a 2D CAD program involving layout
of a diagram (such as a circuit, algorithm, etc.) or physical
object (such as a PC board, control panel, semiconductor
photolithography mask, etc.). In that this example further involves
drawing, the example also illustrates the related needs and
attributes of a paint-box or electronic drafting application.
[0135] In this broadly representative application, application
window 700 is shown comprising menu area 700a and drawing area
700b. Within the drawing area, viewgraph title 701 and portion 702
of the flowchart to be drawn, comprising thus far a sequence of
flowgraph objects connected by arrows, have already been entered
and rendered. New flowgraph objects may be introduced in standard
fashion by selecting the type of new object desired from a palette,
initially putting an instance of the selected object type in a
convenient place in the drawing area, adjusting the size,
orientation, color, and/or other attributes, and putting into final
position. Often a number of the last few steps are interactively
cycled through multiple times before the newly introduced object is
adequately drawn and the user directs their attention to the next
task. In this example, the palette of available objects of a
specific high-level task is shown as an overlapping stack of three
sub-class palettes 703a, 703b, 703c, each providing a selection of
available objects within that sub-class. Here, for example,
sub-class palette 703a has been selected (as indicated by the heavy
line) from other available objects 705, within the sub-class
palette 703a. Upon selection, an initial highly adjustable
rendering of a specific instance 714 of the selected object 704
appears in a convenient location, which may be selected by the
user.
[0136] The specific instance 714, rendered in this highly
adjustable initial state, is typically surrounded by graphical
handles 716 which facilitate sizing, positioning within the drawing
area, and often at least rotational orientation (for example, using
the mouse with the ALT key simultaneously held down to
interactively adjust the angle of rotation of the object should
that be needed). Traditionally, the cursor controlled by the mouse
717 can be moved within object 714 to relocate it within drawing
area 700b or can, as shown in FIG. 7, be positioned atop one of the
graphical handles 716 to permit the mouse to adjust the horizontal
and vertical scale of object 714, i.e., adjust its size and
aspect-ratio. In some application packages, the latter adjustment
is permitted to collapse the object through to `zero` thickness in
one of the adjustment dimensions and continue through to re-render
the object in mirror image, thus additionally providing a form of
vertical and horizontal flip by using the size and aspect-ratio
resizing.
[0137] As familiar and widely accepted as these sorts of operations
are, there is considerable overhead involved in such sequences of
repeated selecting and adjusting (and in some cases additional
deselecting) pairs of parameters from a larger collection of
parameters. To see several aspects of the power of the present
invention, these operations are now examined in more detail in
generalized form.
[0138] FIG. 8 is a flowchart showing tasks involved in selecting
and adjusting one of a plurality of available pairs of adjustable
parameters by using a user interface device permitting the
adjustment of only one pair of parameters at a time. In FIG. 8, the
task goal is simply to adjust a pair of selected parameters 801
from a larger group of adjustable parameters. However, since a
larger number of parameters are available for adjustment than are
available at one time with the user interface sensor, the specific
pair of parameters must first be selected. In most known graphical
user interface systems and methods, this typically involves first
using the user interface device to control the movement of a cursor
to a selection area of the graphical interface in a first overhead
step 811 and then selecting the adjustment context (parameter pair)
in a second overhead step 821.
[0139] In some situations the selected pair of parameters may be
immediately adjusted in goal operation 801, but typically the
cursor must then at least be moved, in a third overhead operation
812, to a location outside of the selection area affiliated with
operations 811 and 821 to another location (such as a drawing or
typing area) affiliated with the context (parameter pair) that has
just been selected for adjustment. In some situations the selected
pair of parameters may be immediately adjusted in goal operation
801, but typically the context must be activated (for example, by
clicking in an open portion of a drawing area or selecting an
existing object) in a fourth overhead operation 822.
[0140] After the selected pair of parameters are adjusted (for
example, by sizing a rectangle, etc.) the cycle may then
immediately be repeated in some variant form for another pair of
parameters, but typically the parameters must be deselected (for
example, by another click to set the final value) in a fifth
overhead operation 823 before the cursor may be moved to the
selection area in another instance of operation 811. In summary, in
order to adjust one pair of parameters from a larger group of
parameters, as many as five overhead operations (as many as two
cursor movements 811, 812 and as many as three select/deselect
clicks 821, 822, 823) are commonly required.
[0141] FIGS. 9a-9b show broader implications of the overhead called
out in FIG. 8. FIG. 9a depicts the sequential adjustment of pairs
of parameters chosen from a larger group of pairs of parameters in
a scenario suggestive of no interactive iteration. One pair of
parameters is adjusted with up to five overhead operations in
action 901, then a second pair of parameters is adjusted with up to
five overhead operations in action 902, then a third pair of
parameters is adjusted with up to five overhead operations in
action 903, and so on. Here the overhead slows things down but may
not be a significant encumbrance to the broader goal of actions
901, 902, 903, etc.
[0142] In contrast, FIG. 9b depicts an interactive adjustment of
pairs of parameters from a larger group of parameters in a scenario
suggestive of one where interactive iteration is required, as the
setting of one pair of parameters is difficult to complete without
setting other parameters. Here the overhead is likely a significant
encumbrance to the higher goal involving the pair-wise adjustment
actions 901, 902, 903, etc. For example, consider the interactive
adjustment of six parameters, one pair at a time, through pair-wise
adjustment actions 901,902,903; not only are up to five operations
of overhead involved for each of the pair-wise adjustment actions
901,902,903, but a considerable extra number of passes must be made
through these pair-wise adjustment actions 901, 902,903 due to the
fact that the adjustment of some parameters depends on or interacts
heavily with the values of other parameters. The situation gets
even more cumbersome should additional pair-wise adjustment
operations be required. FIG. 9b further shows the potential for one
or more additional adjustment actions 950 which in principle may be
iterated as well as and combined with pair-wise adjustment actions
901, 902, 903 (as suggested by fully-connected iteration paths 921,
922, 923). In contrast to such sequences or iterative graphs of
pair-wise adjustment actions, the present invention readily offers,
for example, four, six, eight or even higher numbers of
simultaneously adjustable parameters controllable by the same hand,
which, when selected in a context, eliminate the many overhead
operations depicted in FIGS. 8 and 9a-9b, and the many additional
iteration steps depicted in FIG. 9b.
[0143] Returning now to the generalized graphical layout situation
described earlier and depicted in FIG. 7, the following operations
are routinely performed in 2D graphics, layout, and CAD
applications: [0144] A. Selection of palette containing object;
[0145] B. Selection of object from palette; [0146] C. Selection of
"layer" object is to be assigned to (common in CAD, but typically
not used in standard draw and paint packages); [0147] D. Adjustment
of object placement in drawing; [0148] E. Adjustment of object
sizing; [0149] F. Adjustment of object rotation; [0150] G.
Adjustment of object line thickness(es); [0151] H. Adjustment of
object line color(s); [0152] I. Adjustment of object fill color(s);
and [0153] J. Adjustment of object fill pattern(s);
[0154] Of these, operations B, D, and E are almost always utilized,
operations A, G, and I are frequently utilized, and operations C,
F, H, and J are rarely utilized.
[0155] Thus, in one exemplary application of the invention, it may
be advantageous to group specific collections of operations that
are commonly used together (this may be application specific) so
that the benefits of having four or more widely-adjustable
interactive parameters simultaneously available can be applied to
speed the execution of basic common operations. For example: [0156]
Employing a four-parameter version of the invention: [0157]
Operation 1: Mouse for operation B [0158] Operation 2: Mouse for
operation D, trackball or touchpad for operation E [0159] Employing
a six-parameter version of the invention comprising a 4-parameter
touchpad: [0160] Operation 1: Mouse for operation B, touchpad
finger-location for operation D, touchpad finger-tilt for operation
E. [0161] Employing a six-parameter version of the invention
comprising a mouse and two trackballs or touchpads: [0162]
Operation 1: Mouse for operation B, first trackball/touchpad for
operation D, second trackball/touchpad for operation E.
[0163] Other operations can be later applied in groupings and
operations appropriate for the application.
[0164] As a possible alternative to the preceding example, it may
be advantageous to assign a principal one of the user interface
sensors to the sequential adjustment of each of such universal (or
otherwise principal) operations and reserve the additional user
interface sensors for rapid "in-context" interactive access to less
frequently used operations. For example: [0165] Employing a
four-parameter version of the invention: [0166] Operation 1: Mouse
for operation B, trackball or touchpad for operation A and/or
operation C; [0167] Operation 2: Mouse for operation D, trackball
or touchpad for operation E and/or operation F; [0168] Operation 3:
Mouse for operation G, trackball or touchpad for operation H; and
[0169] Operation 4: Mouse for operation I, trackball or touchpad
for operation J. [0170] Employing a six-parameter version of the
invention comprising a 4-parameter touchpad: [0171] Operation 1:
Mouse for operation B, touchpad finger-location for operation A and
touchpad finger-tilt for operation C; [0172] Operation 2: Mouse for
operation D, touchpad finger-location for operation E and touchpad
finger-tilt for operation F; [0173] Operation 3: Mouse for
operation G, touchpad finger-location (and touchpad finger-tilt as
useful) for operation H; and [0174] Operation 4: Mouse for
operation I, touchpad finger-location (and touchpad finger-tilt as
useful) for operation J. [0175] Employing a six-parameter version
of the invention comprising a mouse and two trackballs or
touchpads: [0176] Operation 1: Mouse for operation B, first
trackball/touchpad for operation A and second trackball/touchpad
for operation C; [0177] Operation 2: Mouse for operation D, first
trackball/touchpad for operation E and second trackball/touchpad
for operation F; [0178] Operation 3: Mouse for operation G, first
trackball/touchpad (and second trackball/touchpad as useful) for
operation H; and [0179] Operation 4: Mouse for operation I, first
trackball/touchpad (and second trackball/touchpad as useful) for
operation J.
[0180] As another alternative example, the user may freely assign
user interface sensor parameters to operations A through J (and
others as may be useful) for each of a number of steps as may match
the task or tasks at hand. These assignments may be stored for
later retrieval and use, and may be named by the user. The stored
assignments may be saved along with specific files, specific
applications, or as a general template the user may apply to a
number of applications. It is noted that such variable assignments
may be particularly useful to users as their hands fatigue, to
prevent fatigue or injury, or as an adjustment for a temporary or
permanent disability.
[0181] 3.4 Multi-Resolution Mouse Application
[0182] In another exemplary family of applications, one user
interface sensor (for example, the mouse body) is used for course
adjustment or fine adjustment of user interface parameters, while
the additional user interface sensor (for example, a trackball or
touchpad) is used for the remaining level of parameter adjustment
resolution.
[0183] In most user-interface applications it is advantageous to
have multiple scales of graphical user interface pointing and data
entry. Many window systems provide an `acceleration` setting which
changes the pointing and data entry values on a more significant
scale used for user interface changes made less frequently. Many
applications further internally adjust the resolution as the
corresponding visual display is "zoomed" in and out.
[0184] In many user interface applications, additional levels of
resolution selection may be useful. For example, in pointing usage
in text work, multiple resolutions would be advantageous in
amending fine print or in making isolated changes in thumbnail
overviews of 40% actual size or less. Similarly, in graphics work,
fine resolution may be especially useful in making fine adjustments
to figures. In the fine adjustment of figures, it may be further
advantageous to employ each of the separate user interface sensors
in conjunction with corresponding snap-grids of differing grid
spacing, particularly if one of the grid spacings is a sub-multiple
of the other. A potentially useful extension of this would be to
impose locally-applicable grid spacing on individual graphic or
other objects, particularly objects which have been resized and
hence for which the standard snap-grid spacing is no longer
useful.
[0185] In a further application, the user interface may be directed
towards non-positional adjustments, such as the adjustment of a
rotation angle or of the color of a graphic object; here multiple
resolutions may be valuable to make careful adjustments and coarse
adjustments as needed. Similarly, scroll bars for long documents
may also benefit from rapid access to multiple resolution scales,
for example one user interface sensor may be used to navigate
within a page (using a fine-grained navigation scale) while a
second user interface sensor may be used to navigate across pages
(using a coarser-grained navigation scale).
[0186] 3.5 Provision of Both Absolute and Relative Positioning
[0187] As discussed earlier, some types of user interface sensors,
such as the touchpad and X-Y joystick for example, naturally have a
limited maximum range of operation while others such as a mouse,
trackball, and scroll-wheel have an essentially unlimited maximum
range of operation. Although most user interface sensors are
interpreted in relative terms (that is, the stimulus from the
sensor is interpreted as a command to move a cursor, scroll bar,
etc., incrementally in some direction relative to a current
position), stimulus signals from any of these types of user
interface signals may be interpreted in either a relative or
absolute manner with varying degrees of naturalness or problematic
qualities.
[0188] The present invention provides for one user interface sensor
to be used for absolute positioning of a cursor, scroll bar, etc.,
or other means of parameter adjustment while another user interface
sensor is used for traditional relative adjustment of such
parameters. For example, a scroll bar may be adjusted in the usual
fashion by a mouse body or trackball and in an absolute manner by a
touchpad wherein the extreme values of the adjusted parameter
correspond to the extreme positions at the edges of the touchpad.
In one embodiment or application setting these two user interface
sensors may control the same parameters--here it is often the
result that the two sensors adjust the same parameters with
different resolutions. Further, in this situation it is fairly
likely that at least one of the resolution scale factors will be
adjusted automatically. For example, in a document editor, as the
number of pages of the document varies, the resolution of the
absolute positioning sensor will correspondingly vary (so that the
extremities in range of, for example, a touchpad correspond to the
top of the first page and end of the last page) while the relative
positioning sensor may retain the same incrementing/decrementing
vertical scrolling resolution scale regardless of the number of
pages.
[0189] 3.6 Color-Selection Application
[0190] In color adjustment, three parameters are involved in the
full interactive span of any complete color space (RGB, HSB, YUV,
etc.). By adding additional parameters to the overall user
interface, all three parameters can be adjusted simultaneously
rather than simply two at a time. As the present invention provides
at least four simultaneous interactively adjustable parameters
overall, it is thus potentially useful for fully interactive color
adjustment within a complete color space model. Further, should the
additional user interface sensor be such that it alone provides
three simultaneously interactively adjustable parameters, the first
user interface sensor (for example, the mouse body) may be used as
a pointer to select objects and the additional user interface
sensor may be used to adjust attributes of the selected object such
as its color, border color, etc.
[0191] 3.7 Multi-level Graphic Object Grouping and Editing
Application
[0192] In many drawing applications, lower-level graphical or other
objects (such as lines, basic shapes, and text areas) may be
grouped to form an aggregated object. This aggregated or "grouped"
object (collectively referred to herein as an "aggregated object")
can then be moved, rotated, flipped, resized, etc. as if it were a
lower-level graphic or other object. Grouping can also typically be
done hierarchically and in mixed hierarchies, i.e., a plurality of
lower-level graphical or other objects may first be grouped, and
the resulting aggregated object may then itself be grouped with
other aggregated objects and/or lower-level graphical or other
objects.
[0193] Often one or more of the lower-level graphical or other
objects comprising the aggregated object may need modification. In
the case of text, most applications permit modifications to be made
to individual text objects within an aggregated object. However,
for any isolated adjustment to any other lower-level graphical or
other object the aggregated object must be first disaggregated or
"ungrouped" to completely free the involved lower-level graphical
or other object from any grouping it had been involved in. After
the modification, the grouping must be reconstructed. Often this
becomes a cumbersome situation, particularly where the adjustments
within the group are themselves an interactive response to other
adjustments made within a drawing.
[0194] The additional number of widely-adjustable simultaneously
interactive parameters made possible by the invention may be
advantageously applied to this problem. For example, one user
interface sensor may be used to navigate the levels of grouping and
another user interface sensor may be used to perform operations on
objects (lower-level or "grouped") within that level of grouping of
the overall aggregated object.
[0195] As an illustrative example, FIG. 11a shows a portion 1100 of
a larger drawing, the portion 1100 featuring box 1101, two arrowed
lines 1111, 1112, and grouped object 1102 (here itself comprising
two triangles connected by a line). In this example it is given
that grouped object 1102 is itself grouped with box 1101 to form a
second grouped object, and this second grouped object is itself
grouped with the two arrowed lines 1111, 1112 to form a third
grouped object. The user's task is to modify FIG. 11a so that it
becomes FIG. 11b. To do this, effectively the user must, in some
order of operation: [0196] Reposition grouped object 1102 from its
original position in FIG. 11a to a new position in FIG. 11b; [0197]
Copy or otherwise reproduce grouped object 1102 to create an
accompanying grouped object 1102a, and position it within box 1101;
[0198] Introduce a vertically distributed ellipsis 1103 and
position it within box 1101--typically, a vertically distributed
ellipsis 1103 is either rotated text or itself a fourth grouped
object created from three aligned text elements; and [0199] Ensure
elements 1102, 1102a, and 1103 are in the end grouped with box 1101
to form the second grouped object, and this second grouped object
is itself grouped with the two arrowed lines 1111, 1112 to form a
third grouped object.
[0200] Utilizing the invention, one user interface sensor is used
to select the second group level, and the second user interface
sensor is used to perform insert, copy, paste, and position
operations within this level of grouping without any form or type
of ungrouping operation. If the vertically distributed ellipsis
1103 itself is realized as a fourth grouped object created from
three aligned text elements, when it is pasted into the drawing via
this modality its 1103 grouping is subordinated appropriately
(i.e., structured as a peer to grouped objects 1102, 1102a) within
the second grouping level.
[0201] Although readily implemented using the novel user interface
sensors described herein that make it particularly easy to
simultaneously adjust a plurality of pairs of parameters, the
aspects of the invention illustrated here can also be implemented
with a conventional user interface sensor such as a traditional
mouse, trackball, or touchpad. In this case, the conventional user
interface sensor such as a traditional mouse, trackball, or
touchpad must first be used to select the level of grouping and
then be used to make the desired modifications within that level of
grouping; to make modifications at a different level of grouping,
the new level of grouping must be selected in a separate operation,
thus adding overhead, as depicted in FIGS. 8 and 9a-9b. Although
this novel ability to move and modify arbitrary graphic or other
objects within groupings may be implemented in this way, having an
additional number of widely-adjustable simultaneously interactive
parameters--made possible by the main themes of the present
invention--is clearly more efficient, as many or all of the
overhead operations depicted in FIGS. 8 and 9a-9b can be eliminated
via usage of the additional widely-adjustable simultaneously
interactive parameters.
[0202] 3.8 3D Graphic Object Placement and Orientation
Application
[0203] CAD and drawing packages that enable 3D object placement and
orientation within a 3D space typically extend the capabilities of
traditional 2D layout, CAD, and graphics applications as described
in Section 3.3 to serve additional geometric needs. As illustrated
in FIG. 12, the placement and orientation of 3D object 1200 within
a 3D space 1250 (oriented with respect to a reference point 1251)
requires that one additional position dimension and two additional
orientation angles be specified to complete the full collection of
three position dimensions 1201, 1202, 1203 and three orientation
angles 1211, 1212, 1213.
[0204] To interactively adjust these parameters pairwise (or
individually with a knob box as has been done historically in some
systems) involves complex repetitive passes among high overhead
operations as depicted in FIGS. 9a-9b, for example among steps 901,
902, 903. The necessity of making many high-overhead operations,
for example moving among steps 901, 902, 903, can be functionally
disruptive as well as slow and inefficient. The ability to
interactively freely adjust the full collection of three position
dimensions 1201, 1202, 1203 and three orientation angles 1211,
1212, 1213 is thus of extremely high value.
[0205] The invention provides for a wide range of mappings between
the six position and orientation parameters 1201, 1202, 1203, 1211,
1212, 1213 involved in the placement and orientation of 3D object
1200 within a 3D space, and the large numbers of widely-adjustable
and simultaneously interactive parameters facilitated by various
realizations of the invention. As one example, a mouse fitted with
two trackballs as in FIG. 14a may be used to specify these six
parameters in various ways. One technique is to use the position of
mouse body 1400 to control two of the position coordinates (for
example 1202, 1203), one of the trackballs (for example 1405a) to
control the orientation angles (1212, 1213) corresponding to these
two axes, and the remaining trackball (1405b) to control the
remaining axis (1201) and its corresponding orientation angle
(1211). In this example, trackballs 1405a, 1405b are configured or
used in 2-parameter modalities.
[0206] In another example, the mouse of FIG. 14a is fitted with two
trackballs, the trackballs may be configured in 3-parameter
modalities with one of the trackballs used for controlling three
position dimensions 1201, 1202, 1203 and the second trackball
configured to correspondingly control the three orientation angles
1211, 1212, 1213. Here the position of mouse body 1400 may be used
to control other aspects of drawing operations.
[0207] In another implementation, a touchpad configured for
4-parameter operation involving two parameters of finger position
and two parameters of finger tilt may be combined with a trackball
configured for 2-parameter operation. In this example, finger
position may be used to control two of the position coordinates
(for example 1202, 1203), finger tilt may be used to control the
orientation angles (1212, 1213) corresponding to these two axes,
and the trackball to control the remaining axis (1201) and its
corresponding orientation angle (1211). If the configuration
includes a mouse body, its position may be used to control other
aspects of drawing operations.
[0208] In another example, a configuration like that of FIG. 14d
may use left-fingers/thumb trackball 1465a to control a first
position coordinate 1201 and its corresponding orientation angle
1211, the right-fingers/thumb trackball 1465b to control a second
position coordinate 1202 and its corresponding orientation angle
1212, and palm trackball 1465c to control third position coordinate
1203 and its corresponding orientation angle 1213.
[0209] The invention further provides for a wide range of
additional mappings and geometric metaphors between the user
interface sensor geometry and the three position dimensions 1201,
1202, 1203 and three orientation angles 1211, 1212, 1213 of a 3D
object.
[0210] 3.9 Multiple Cursors and Cut and Paste Application
[0211] The invention additionally provides for a plurality of pairs
of user interface sensor parameters to be used to control the
respective positions of a corresponding plurality of individual
cursors, selections, and/or insertion points. Multiple cursors and
associated operations of multiple selection and insertion points
can have many applications. Below, a few of these possibilities
that would be apparent to one skilled in the art are showcased in a
cut-and-paste editing example.
[0212] Cut, copy, and paste operations using traditional user
interface devices usually involve multiple operations to switch
between contexts introducing considerable overhead as depicted in
FIGS. 9a-9b and 8. For instance, FIG. 13a illustrates a text
editing example with text display window 1300 involving the
selection of a clause 1320 (highlighted in this example) with the
intention of relocating it to a new position 1351. In such an
operation with a traditional 2-parameter mouse/trackball/touchpad
user interface, the cursor is first used to select clause 1320 and
then used to select the insertion position 1351.
[0213] When writing or editing it is often the case that material
needs to be fetched from elsewhere and put in the spot where one is
currently writing. Here, the cursor is initially in the spot where
the insertion is to occur and the user must then lose the cursor
position currently set in this spot to go searching and then to
select and cut or copy the material to be pasted; following this
the user must then search again, perhaps taking considerable time,
for said initial spot and re-establish the cursor location there.
Equally often there are other situations where material must be
split up and distributed in a number of far-flung places. Here, the
cursor is initially in the spot where the material to be
sequentially divided and relocated is originally aggregated; the
user must repeatedly select the portion of the remaining aggregated
material to be relocated and then lose that cursor position to go
searching for the new destination insertion spot, perform the
insertion, and then search again, perhaps taking considerable time,
for the initial spot and re-establish the cursor location there. In
both of these cases it would be advantageous if the user could
"bookmark" an initial cursor location, search and perform the
desired fetch or relocation operations, and readily return without
search to the "bookmarked" location. Although this novel and
advantageously valuable capability could be realized with a
conventional mouse through context redirection operations involving
the steps depicted in FIGS. 8 and 9a-9b, the present invention
provides for a wide range of readily realized and easy-to-use
implementations.
[0214] The invention may be used in a minimal configuration capable
of interactively specifying at least two pairs of widely adjustable
interactive parameters. Returning to the specific example
associated with FIG. 13a, one pair of parameters is used to set the
location of first cursor 1301 which is used in a selection
operation, while the second pair of parameters is used to set the
location of second cursor 1351 which is to be used to independently
set an insertion point. The user may then perform the cut and paste
operation with a single mouse click, resulting in the outcome
depicted in FIG. 13b. The relocated text clause 1320 has been
transferred to a position determined by the insertion cursor 1351
(here shown to the left of the cursor 1351; it could just as easily
be to the right of it), and cursor 1301 used to make the selection
remains in position. Either cursor 1301 or 1351 may now be moved
and/or used for other cut, copy, paste, or (via the keyboard) new
text insertion operations.
[0215] Although in this example the two cursor locations were close
enough to be displayed in the same window 1300, the value of this
application of the invention is significantly increased should the
two positions be separated by many pages, many tens of pages, or
even many hundreds of pages of text. Such situations may be handled
by any number of approaches as is clear to one skilled in the art.
In one approach involving a single display window, the area
comprising the cursor whose corresponding user interface sensor was
last manipulated is displayed in the single display window. In
another approach involving a single display window, a click event
or other user interface stimulus may be used to toggle among the
areas comprising the various cursor locations. In yet another
approach, at least two windows may be rendered, with one of the
cursors displayed and operable within one window and a second
cursor displayed and operable in a second window.
[0216] The invention also provides for these general principles to
be applied to other types of objects and applications, such as
spreadsheet cells (involving data, formula objects, and cell
formats), graphical objects, electronic CAD diagrams (where objects
may be connected with formulas, dynamic models, etc.), and others
as will be apparent to one skilled in the art.
[0217] 3.10 Simulation, Processing, and Analysis Applications
[0218] Simulation, processing, and analysis applications typically
involve a large number of parameters which are adjusted to model,
affect or investigate the resulting behaviors, end results, and/or
implications. Conventional 2-parameter user interface devices such
as a mouse/trackball/touchpad require the user to adjust these
parameters pairwise (or individually with a knob box as has been
done historically in some systems) involving complex repetitive
passes among high-overhead operations as depicted in FIGS. 9a-9b,
for example among steps 901, 902, 903. As in the case of 3D object
positioning and orientation, the division among high-overhead
operations, for example moving among steps 901, 902, 903, can be
functionally disruptive as well as slow and inefficient. The
ability to interactively and freely adjust larger collections of
parameters simultaneously is thus also of extremely high value.
[0219] 3.11 Live Signal Processing and Lighting Applications
[0220] In artistic performance, composition, and recording
applications, control of large numbers of parameters requiring
simultaneous interactive adjustment is common. Conventional
recording, mixing, video, and light control consoles typically have
large numbers of controls with carefully designed spatial layouts
to facilitate the rapid and precise adjustment of multiple
parameters via knobs, sliders, pushbuttons, toggle switches, etc.
The introduction of computer GUIs has added considerable value and
new capabilities, including "soft" reconfigurable consoles and
functional assignments, but in the bargain, typically encumber
users--accustomed to rapid and precise operation of multiple
parameters--with a 2-parameter mouse/trackball/touchpad having the
overhead of iterative context-switching operations depicted in
FIGS. 8 and 9a-9b. As in the case of 3D object positioning and
orientation, the division among high-overhead operations, for
example moving among steps 901, 902, 903, can be functionally
disruptive as well as slow and inefficient. The ability to
interactively freely adjust larger collections of parameters
simultaneously is thus also of extremely high value.
[0221] 3.12 Real-Time Machine Control and Plant Operations
[0222] Similarly, real-time machine control and plant
(manufacturing, chemical, energy, etc.) operations also
traditionally involve controlling a significant number of
parameters requiring simultaneous interactive adjustment.
Conventional real-time machine control and plant operation consoles
typically have large numbers of controls with carefully designed
spatial layouts to facilitate the rapid and precise adjustment of
multiple parameters via knobs, sliders, pushbuttons, toggle
switches, etc. The introduction of computer GUIs can add
considerable value and new capabilities, including "soft"
reconfigurable consoles and functional assignments, but in the
bargain typically significantly encumber users--accustomed to rapid
and precise operation of multiple parameters--with a 2-parameter
mouse/trackball/touchpad having the overhead of iterative
context-switching operations depicted in FIGS. 8 and 9a-9b. As in
the case of 3D object positioning/orientation and artistic
applications described earlier, the division among high-overhead
operations, for example moving among steps 901, 902, 903, can be
functionally disruptive as well as slow and inefficient. The
ability to interactively freely adjust larger collections of
parameters simultaneously is thus also of extremely high value.
[0223] A very few examples of this category of application where
the invention may be useful include many forms of robotics control,
computer-control manufacturing tools, industrial optical and
electron microscopy, camera control (pan, tilt, zoom, focus, and/or
iris), plant process elements (heaters, pumps, values, stirrers,
aerators, actuators, activators, etc.), and a wide range of other
related and divergent possibilities apparent to those skilled in
the art.
4. Concluding Remarks
[0224] The present invention at its core provides for a wide range
of systems and methods for realizing and applying user interfaces
providing, in many cases, at least four widely variable
simultaneously interactively adjustable parameters. In so doing,
the invention more broadly encompasses novel user interface
structures, metaphors, and applications readily suggested and
enabled by the core of the invention but which may be indeed
realized in ways not involving the core of the invention.
[0225] While the invention has been described in detail with
reference to disclosed embodiments, various modifications within
the scope of the invention will be apparent to those of ordinary
skill in this technological field. It is to be appreciated that
features described with respect to one embodiment typically may be
applied to other embodiments. Therefore, the invention properly is
to be construed with reference to the claims.
* * * * *