U.S. patent application number 09/783725 was filed with the patent office on 2002-01-24 for voice controlled computer interface.
This patent application is currently assigned to Lernout & Hauspie, Belgian corporation. Invention is credited to Firman, Thomas R..
Application Number | 20020010582 09/783725 |
Document ID | / |
Family ID | 23461140 |
Filed Date | 2002-01-24 |
United States Patent
Application |
20020010582 |
Kind Code |
A1 |
Firman, Thomas R. |
January 24, 2002 |
Voice controlled computer interface
Abstract
Voice utterances are substituted for manipulation of a pointing
device of the kind which is manipulated to control motion of a
cursor on a computer display and to indicate desired actions
associated with the position of the cursor on the display. Voiced
utterances are converted to commands, expressed in a predefined
command language, to be used by an operating system of a computer,
by converting some voiced utterances into commands corresponding to
actions to be taken by the operating system, and converting other
voiced utterances into commands which carry associated text strings
to be used as part of text being processed in an application
program running under the operating system. A table is generated
for aiding the conversion of voiced utterances to commands for use
in controlling an operating system of a computer to achieve desired
actions in an application program running under the operating
system.
Inventors: |
Firman, Thomas R.; (Oakland,
CA) |
Correspondence
Address: |
DAVID L. FEIGENBAUM
Fish & Richardson P.C.
225 Franklin Street
Boston
MA
02110-2804
US
|
Assignee: |
Lernout & Hauspie, Belgian
corporation
|
Family ID: |
23461140 |
Appl. No.: |
09/783725 |
Filed: |
February 14, 2001 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
09783725 |
Feb 14, 2001 |
|
|
|
08976908 |
Nov 24, 1997 |
|
|
|
08976908 |
Nov 24, 1997 |
|
|
|
08674341 |
Jul 2, 1996 |
|
|
|
08674341 |
Jul 2, 1996 |
|
|
|
08450776 |
May 25, 1995 |
|
|
|
08450776 |
May 25, 1995 |
|
|
|
08200886 |
Feb 23, 1994 |
|
|
|
08200886 |
Feb 23, 1994 |
|
|
|
08165014 |
Dec 9, 1993 |
|
|
|
5377303 |
|
|
|
|
08165014 |
Dec 9, 1993 |
|
|
|
07973435 |
Nov 9, 1992 |
|
|
|
07973435 |
Nov 9, 1992 |
|
|
|
07370779 |
Jun 23, 1989 |
|
|
|
Current U.S.
Class: |
704/246 ;
704/E15.045 |
Current CPC
Class: |
G06F 3/167 20130101;
G10L 15/26 20130101; G06F 3/038 20130101 |
Class at
Publication: |
704/246 |
International
Class: |
G10L 017/00 |
Claims
1. A system for enabling voiced utterances to be substituted for
manipulation of a pointing device, the pointing device being of the
kind which is manipulated to control motion of a cursor on a
computer display and to indicate desired actions associated with
the position of the cursor on the display, the cursor being moved
and the desired actions being aided by an operating system in the
computer in response to control signals received from the pointing
device, the computer also having an alphanumeric keyboard, the
operating system being separately responsive to control signals
received from the keyboard in accordance with a predetermined
format specific to the keyboard, the system comprising a voice
recognizer for recognizing a voiced utterance, and an interpreter
for converting the voiced utterance into control signals which will
directly create a desired action aided by the operating system in
the computer without first being converted into control signals
expressed in the predetermined format specific to the keyboard.
2. A method for converting voiced utterances to commands, expressed
in a predefined command language, to be used by an operating system
of a computer, comprising converting some voiced utterances into
commands corresponding to actions to be taken by said operating
system, and converting other voiced utterances into commands which
carry associated text strings to be used as part of text being
processed in an application program running under said operating
system.
3. A method of generating a table for aiding the conversion of
voiced utterances to commands for use in controlling an operating
system of a computer to achieve desired actions in an application
program running under the operating system, said application
program including menus and control buttons, said method comprising
parsing the instruction sequence of the application program to
identify menu entries and control buttons, and including in said
table an entry for each menu entry and control button found in said
application program, each said entry containing a control command
corresponding to said menu entry or control button.
4. A method of enabling a user to create an instance in a formal
language of the kind which has a strictly defined syntax,
comprising providing a graphically displayed list of entries which
are expressed in a natural language and which do not comply with
said syntax, permitting the user to point to an entry on said list,
and automatically generating said instance corresponding to the
identified entry in the list in response to said pointing.
Description
BACKGROUND OF THE INVENTION
[0001] This invention relates to voice controlled computer
interfaces.
[0002] Voice recognition systems can convert human speech into
computer information. Such voice recognition systems have been
used, for example, to control text-type user interfaces, e.g., the
text-type interface of the disk operating system (DOS) of the IBM
Personal Computer.
[0003] Voice control has also been applied to graphical user
interfaces, such as the one implemented by the Apple Macintosh
computer, which includes icons, pop-up windows, and a mouse. These
voice control systems use voiced commands to generate keyboard
keystrokes.
SUMMARY OF THE INVENTION
[0004] In general, in one aspect, the invention features enabling
voiced utterances to be substituted for manipulation of a pointing
device, the pointing device being of the kind which is manipulated
to control motion of a cursor on a computer display and to indicate
desired actions associated with the position of the cursor on the
display, the cursor being moved and the desired actions being aided
by an operating system in the computer in response to control
signals received from the pointing device, the computer also having
an alphanumeric keyboard, the operating system being separately
responsive to control signals received from the keyboard in
accordance with a predetermined format specific to the keyboard; a
voice recognizer recognizes the voiced utterance, and an
interpreter converts the voiced utterance into control signals
which will directly create a desired action aided by the operating
system without first being converted into control signals expressed
in the predetermined format specific to the keyboard.
[0005] In general, in another aspect of the invention, voiced
utterances are converted to commands, expressed in a predefined
command language, to be used by an operating system of a computer,
converting some voiced utterances into commands corresponding to
actions to be taken by said operating system, and converting other
voiced utterances into commands which carry associated text strings
to be used as part of text being processed in an application
program running under the operating system.
[0006] In general, in another aspect, the invention features
generating a table for aiding the conversion of voiced utterances
to commands for use in controlling an operating system of a
computer to achieve desired actions in an application program
running under the operating system, the application program
including menus and control buttons; the instruction sequence of
the application program is parsed to identify menu entries and
control buttons, and an entry is included in the table for each
menu entry and control button found in the application program,
each entry in the table containing a command corresponding to the
menu entry or control button.
[0007] In general, in another aspect, the invention features
enabling a user to create an instance in a formal language of the
kind which has a strictly defined syntax; a graphically displayed
list of entries are expressed in a natural language and do not
comply with the syntax, the user is permitted to point to an entry
on the list, and the instance corresponding to the identified entry
in the list is automatically generated in response to the
pointing.
[0008] The invention enables a user to easily control the graphical
interface of a computer. Any actions that the operating system can
be commanded to take can be commanded by voiced utterances. The
commands may include commands that are normally entered through the
keyboard as well as commands normally entered through a mouse or
any other input device. The user may switch back and forth between
voiced utterances that correspond to commands for actions to be
taken and voiced utterances that correspond to text strings to be
used in an application program without giving any indication that
the switch has been made. Any application may be made susceptible
to a voice interface by automatically parsing the application
instruction sequence for menus and control buttons that control the
application.
[0009] Other advantages and features will become apparent from the
following description of the preferred embodiment and from the
claims.
DESCRIPTION OF THE PREFERRED EMBODIMENT
[0010] We first briefly describe the drawings.
[0011] FIG. 1 is a functional block diagram of a Macintosh computer
served by a Voice Navigator voice controlled interface system.
[0012] FIG. 2A is a functional block diagram of a Language Maker
system for creating word lists for use with the Voice Navigator
interface of FIG. 1.
[0013] FIG. 2B depicts the format of the voice files and word lists
used with the Voice Navigator interface.
[0014] FIG. 3 is an organizational block diagram of the Voice
Navigator interface system.
[0015] FIG. 4 is a flow diagram of the Language Maker main event
loop.
[0016] FIG. 5 is a flow diagram of the Run Edit module.
[0017] FIG. 6 is a flow diagram of the Record Actions
submodule.
[0018] FIG. 7 is a flow diagram of the Run Modal module.
[0019] FIG. 8 is a flow diagram of the In Button? routine.
[0020] FIG. 9 is a flow diagram of the Event Handler module.
[0021] FIG. 10 is a flow diagram of the Do My Menu module.
[0022] FIGS. 11A through 11I are flow diagrams of the Language
Maker menu submodules.
[0023] FIG. 12 is a flow diagram of the Write Production
module.
[0024] FIG. 13 is a flow diagram of the Write Terminal
submodule.
[0025] FIG. 14 is a flow diagram of the Voice Control main driver
loop.
[0026] FIG. 15 is a flow diagram of the Process Input module.
[0027] FIG. 16 is a flow diagram of the Recognize submodule.
[0028] FIG. 17 is a flow diagram of the Process Voice Control
Commands routine.
[0029] FIG. 18 is a flow diagram of the ProcessQ module.
[0030] FIG. 19 is a flow diagram of the Get Next submodule.
[0031] FIG. 20 is a chart of the command handlers.
[0032] FIGS. 21A through 21G are flow diagrams of the command
handlers.
[0033] FIG. 22 is a flow diagram of the Post Mouse routine.
[0034] FIG. 23 is a flow diagram of the Set Mouse Down routine.
[0035] FIGS. 24 and 25 illustrate the screen displays of Voice
Control.
[0036] FIGS. 26 through 29 illustrate the screen displays of
Language Maker.
[0037] FIG. 30 is a listing of a language file.
SYSTEM OVERVIEW
[0038] Referring to FIG. 1, in an Apple Macintosh computer 100, a
Macintosh operating system 132 provides a graphical interactive
user interface by processing events received from a mouse 134 and a
keyboard 136 and by providing displays including icons, windows,
and menus on a display device 138. Operating system 132 provides an
environment in which application programs such as MacWrite 139,
desktop utilities such as Calculator 137, and a wide variety of
other programs can be run.
[0039] The operating system 132 also receives events from the Voice
Navigator voice controlled computer interface 102 to enable the
user to control the computer by voiced utterances. For this
purpose, the user speaks into a microphone 114 connected via a
Voice Navigator box 112 to the SCSI (Small Computer Systems
Interface) port of the computer 100. The Voice Navigator box 112
digitizes and processes analog audio signals received from a
microphone 114, and transmits processed digitized audio signals to
the Macintosh SCSI port. The Voice Navigator box includes an
analog-to-digital converter (A/D) for digitizing the audio signal,
a DSP (Digital Signal Processing) chip for compressing the
resulting digital samples, and protocol interface hardware which
configures the digital samples to obey the SCSI protocols.
[0040] Recognizer Software 120 (available from Dragon Systems,
Newton, Mass.) runs under the Macintosh operating system, and is
controlled by internal commands 123 received from Voice Control
driver 128 (which also operates under the Macintosh operating
system). One possible algorithm for implementing Recognizer
Software 120 is disclosed by Baker et al, in U.S. Pat. No.
4,783,803, incorporated by reference herein. Recognizer Software
120 processes the incoming compressed, digitized audio, and
compares each utterance of the user to prestored utterance macros.
If the user utterance matches a prestored utterance macro, the
utterance is recognized, and a command string 121 corresponding to
the recognized utterance is delivered to a text buffer 126. Command
strings 121 delivered from the Recognizer Software represent
commands to be issued to the Macintosh operating system (e.g., menu
selections to be made or text to be displayed), or internal
commands 123 to be issued by the Voice Control driver. During
recognition, the Recognizer Software 120 compares the incoming
samples of an utterance with macros in a voice file 122. (The
system requires the user to space apart his utterances briefly so
that the system can recognize when each utterance ends.) The voice
file macros are created by a "training" process, described below.
If a match is found (as judged by the recognition algorithm of the
Recognizer Software 120), a Voice Control command string from a
word list 124 (which has been directly associated with voice file
122) is fetched and sent to text buffer 126.
[0041] The command strings in text buffer 126 are relayed to Voice
Control driver 128, which drives a Voice Control interpreter 130 in
response to the strings.
[0042] A command string 121 may indicate an internal command 123,
such as a command to the Recognizer Software to "learn" new voice
file macros, or to adjust the sensitivity of the recognition
algorithm. In this case, Voice Control interpreter 130 sends the
appropriate internal command 123 to the Recognizer Software 120. In
other cases, the command string may represent an operating system
manipulation, such as a mouse movement. In this case, voice Control
interpreter 130 produces the appropriate action by interacting with
the Macintosh operating system 132.
[0043] Each application or desktop accessory is associated with a
word list 124 and a corresponding voice file 122; these are loaded
by the Recognition Software when the application or desktop
accessory is opened.
[0044] The voice files are generated by the Recognizer Software 120
in its "learn" mode, under the control of internal commands from
the Voice Control driver 128.
[0045] The word lists are generated by the Language Maker desktop
accessory 140, which creates "languages" of utterance names and
associated Voice Control command strings, and converts the
languages into the word lists. Voice Control command strings are
strings such as "ESC", "TEXT", "@MENU(font,2)", and belong to a
Voice Control command set, the syntax of which will be described
later and is set forth in Appendix A.
[0046] The Voice Control and Language Maker software includes about
30,000 lines of code, most of which is written in the C language,
the remainder being written in assembly language. A listing of the
Voice Control and Language Maker software is provided in microfiche
as appendix C. The Voice Control software will operate on a
Macintosh Plus or later models, configured with a minimum of 1
Mbyte RAM (2 Mbyte for HyperCard and other large applications), a
Hard Disk, and with Macintosh operating system version 6.01 or
later.
[0047] In order to understand the interaction of the Voice Control
interpreter 130 and the operating system, note that Macintosh
operating system 132 is "event driven". The operating system
maintains an event queue (not shown); input devices such as the
mouse 134 or the keyboard 136 "post" events to this queue to cause
the operating system to, for example, create the appropriate text
entry, or trigger a mouse movement. The operating system 132 then,
for example, passes messages to Macintosh applications (such as
MacWrite 139) or to desktop accessories (such as Calculator 137)
indicating events on the queues (if any). In one mode of operation,
Voice Control interpreter 130 likewise controls the operating
system (and hence the applications and desktop accessories which
are currently running) by posting events to the operating system
queues. The events posted by the Voice Control interpreter
typically correspond to mouse activity or to keyboard keystrokes,
or both, depending upon the voice commands. Thus, the Voice
Navigator system 102 provides an additional user interface. In some
cases, the "voice" events may comprise text strings to be displayed
or included with text being processed by the application
program.
[0048] At any time during the operation of the Voice Navigator
system, the Recognizer Software 120 may be trained to recognize an
utterance of a particular user and to associate a corresponding
text string with each utterance. In this mode, the Recognizer
Software 120 displays to the user a menu of the utterance names
(such as "file", "page down") which are to be recognized. These
names, and the corresponding Voice Control command strings
(indicating the appropriate actions) appear in a current word list
124. The user designates the utterance name of interest and then is
prompted to speak the utterance corresponding to that name. For
example, if the utterance name is "file", the user might utter
"FILE" or "PLEASE FILE". The digitized samples from the Voice
Navigator box 112 corresponding to that utterance are then used by
the Recognizer Software 120 to create a "macro" representing the
utterance, which is stored in the voice file 122 and subsequently
associated with the utterance name in the word list 124.
Ordinarily, the utterance is repeated more than once, in order to
create a macro for the utterance that accommodates variation in a
particular speaker's voice.
[0049] The meaning of the spoken utterance need not correspond to
the utterance name, and the text of the utterance name need not
correspond to the Voice Control command strings stored in the word
list. For example, the user may wish a command string that causes
the operating system to save a file to have the utterance name
"save file"; the associated command string may be "@MENU(file,2)";
and the utterance that the user trains for this utterance name may
be the spoken phrase "immortalize". The Recognizer Software and
Voice Control cause that utterance, name, and command string to be
properly associated in the voice file and word list 124.
[0050] Referring to FIG. 2A, the word lists 124 used by the Voice
Navigator are created by the Language Maker desk accessory 140
running under the operating system. Each word list 124 is
hierarchical, that is, some utterance names in the list link to
sub-lists of other utterance names. Only the list of utterance
names at a currently active level of the hierarchy can be
recognized. (In the current embodiment, the number of utterance
names at each level of the hierarchy can be as large as 1000.) In
the operation of Voice Control, some utterances, such as "file",
may summon the file menu on the screen, and link to a subsequent
list of utterance names at a lower hierarchical level. For example,
the file menu may list subsequent commands such as "save", "open",
or "save as", each associated with an utterance.
[0051] Language Maker enables the user to create a hierarchical
language of utterance names and associated command strings,
rearrange the hierarchy of the language, and add new utterance
names. Then, when the language is in the form that the user
desires, the language is converted to a word list 124. Because the
hierarchy of the utterance names and command strings can be
adjusted, when using the Voice Navigator system the user is not
bound by the preset menu hierarchy of an application. For example,
the user may want to create a "save" command at the top level of
the utterance hierarchy that directly saves a file without first
summoning the file menu. Also, the user may, for example, create a
new utterance name "goodbye", that saves a file and exits all at
once.
[0052] Each language created by Language Maker 140 also contains
the command strings which represent the actions (e.g. clicking the
mouse at a location, typing text on the screen) to be associated
with utterances and utterance names. In order for the training of
the Voice Navigator system to be more intuitive, the user does not
specify the command strings to-describe the actions he wishes to be
associated with an utterance and utterance name. In fact, the user
does not need to know about, and never sees, the command strings
stored in the Language Maker language or the resulting word list
124.
[0053] In a "record" mode, to associate a series of actions with an
utterance name, the user simply performs the desired actions (such
as typing the text at the keyboard, or clicking the mouse at a
menu). The actions performed are converted into the appropriate
command strings, and when the user turns off the record mode, the
command strings are associated with the selected utterance
name.
[0054] While using Language Maker, the user can cause the creation
of a language by entering utterance names by typing the names at
the keyboard 142, by using a "create default text" procedure 146
(to parse a text file on the clipboard, in which case one utterance
name is created for each word in the text file, and the names all
start at the same hierarchical level), or by using a "create
default menus" procedure (to parse the executable code 144 for an
application, and create a set of utterance names which equal the
names of the commands in the menus of the application, in which
case the initial hierarchy for the names is the same as the
hierarchy of the menus in the application).
[0055] If the names are typed at the keyboard or created by parsing
a text file, the names are initially associated with the keystrokes
which, when typed at the keyboard, produce the name. Therefore, the
name "text" would be initially be associated with the keystrokes
t-e-x-t. If the names are created by parsing the executable code
144 for an application, then the names are initially associated
with the command strings which execute the corresponding menu
commands for the application. These initial command strings can be
changed by simply selecting the utterance name to be changed and
putting Language Maker into record mode.
[0056] The output of Language Maker is a language file 148. This
file contains the utterance names and the corresponding command
strings. The language file 148 is formatted for input to a VOCAL
compiler 150 (available from Dragon Systems), which converts the
language file into a word list 124 for use with the Recognition
Software. The syntax of language files is specified in the Voice
Navigator Developer's Reference Manual, provided as Appendix D, and
incorporated by reference.
[0057] Referring to FIG. 2B, a macro 147 of each learned utterance
is stored in the voice file 122. A corresponding utterance name 149
and command string 151 are associated with one another and with the
utterance and are stored in the word list 124. The word list 124 is
created and modified by Language Maker 140, and the voice file 122
is created and modified by the Recognition Software 120 in its
learn mode, under the control of the Voice Control driver 128.
[0058] Referring to FIG. 3, in the Voice Navigator system 102, the
Voice Navigator hardware box 152 includes an analog-to-digital
(A/D) converter 154 for converting the analog signal from the
microphone into a digital signal for processing, a DSP section 156
for filtering and compacting the digitized signal, a SCSI manager
158 for communication with the Macintosh, and a microphone control
section 160 for controlling the microphone.
[0059] The Voice Navigator system also includes the Recognition
Software voice drivers 120 which include routines for utterance
detection 164 and command execution 166. For utterance detection
164, the voice drivers periodically poll 168 the Voice Navigator
hardware to determine if an utterance is being received by Voice
Navigator box 152, based on the amplitude of the signal received by
the microphone. When an utterance is detected 170, the voice
drivers create a speech buffer of encoded digital samples (tokens)
to be used by the command execution drivers 166. On command 166
from the Voice Control driver 128, the recognition drivers can
learn new utterances by token-to-terminal conversion 174. The token
is converted to a macro for the utterance, and stored as a terminal
in a voice file 122 (FIG. 1).
[0060] Recognition and pattern matching 172 is also performed on
command by the voice drivers. During recognition, a stored token of
incoming digitized samples is compared with macros for the
utterances in the current level of the recognition hierarchy. If a
match is found, terminal to output conversion 176 is also
performed, selecting the command string associated with the
recognized utterance from the word list 124 (FIG. 1). State
management 178, such as changing of sensitivity controls, is also
performed on command by the voice drivers.
[0061] The Voice Control driver 128 forms an interface 182 to the
voice drivers 120 through control commands, an interface 184 to the
Macintosh operating system 132 (FIG. 1) through event posting and
operating system hooks, and an interface 186 to the user through
display menus and prompts.
[0062] The interface 182 to the drivers allows Voice Control access
to the Voice Driver command functions 166. This interface allows
Voice Control to monitor 188 the status of the recognizer, for
example to check for an utterance token in the utterance queue
buffered 170 to the Macintosh. If there is an utterance, and if
processor time is available, Voice Control issues command
sdi_recognize 190, calling the recognition and pattern match
routine 172 in the voice drivers. In addition, the interface to the
drivers may issue command sdi_output 192 which controls the
terminal to output conversion routine 176 in the voice drivers,
converting a recognized utterance to an command string for use by
Voice Control. The command string may indicate mouse or keystroke
events to be posted to the operating system, or may indicate
commands to Voice Control itself (e.g. enabling or disabling voice
Control).
[0063] From the user's perspective, Voice Control is simply a
Macintosh driver with internal parameters, such as sensitivity, and
internal commands, such as commands to learn new utterances. The
actual processing which the user perceives as Voice Control may
actually be performed by Voice Control, or by the Voice Drivers,
depending upon the function. For example, the utterance learning
procedures are performed by the Voice Drivers under the control of
Voice Control.
[0064] The interface 184 to the Macintosh operating system allows
Voice Control, where appropriate, to manipulate the operating
system (e.g., by posting events or modifying event queues). The
macro interpreter 194 takes the command strings delivered from the
voice drivers via the text buffer and interprets them to decide
what actions to take. These commands may indicate text strings to
be displayed on the display or mouse movements or menu selections
to be executed.
[0065] In the interpretive execution of the command strings, Voice
Control must manipulate the Macintosh event queues. This task is
performed by OS event management 196. As discussed above, voice
events may simulate events which are ordinarily associated with the
keyboard or with the mouse. Keyboard events are handled by OS event
management 196 directly. Mouse events are handled by mouse handler
198. Mouse events require an additional level of handling because
mouse events can require operating system manipulation outside of
the standard event post routines which are accomplished by the OS
event management 196.
[0066] The main interface into the Macintosh operating system 132
is event based, and is used in the majority of the commands which
are voice recognized and issued to the Macintosh. However, there
are other "hooks" to the operating system state which are used to
control parameters such as mouse placement and mouse motion. For
example, as will be discussed later, pushing the mouse button down
generates an event, however, keeping the mouse button pushed down
and dragging the mouse across a menu requires the use of an
operating system hook. For reference, the operating system
hooks.
[0067] The operating system hooks are implemented by the trap
filters 200, which are filters used by Voice Control to force the
Macintosh operating system to accept the controls implemented by OS
event management 196 and mouse handler 198.
[0068] The Macintosh operating system traps are held in Macintosh
read only memories (ROMs), and implement high level commands for
controlling the system. Examples of these high level commands are:
drawing a string onto the screen, window zooming, moving windows to
the front and back of the screen, and polling the status of the
mouse button. In order for the Voice Control driver to properly
interface with the Macintosh operating system it must control these
operating system traps to generate the appropriate events.
[0069] To generate menu events, for example, Voice Control "seizes"
the menu select trap (i.e. takes control of the trap from the
operating system). Once Voice Control has seized the trap,
application requests for menu selections are forwarded to Voice
Control. In this way Voice Control is able to modify, where
necessary, the operating system output to the program, thereby
controlling the system behavior as desired.
[0070] The interface 186 to the user provides user control of the
Voice Control operations. Prompts 202 display the name of each
recognized utterance on the Macintosh screen so that the user may
determine if the proper utterance has been recognized. On-line
training 204 allows the user to access, at any time while using the
Macintosh, the utterance names in the word list 124 currently in
use. The user may see which utterance names have been trained and
may retrain the utterance names in an on-line manner (these
functions require Voice Control to use the Voice Driver interface,
as discussed above). User options 206 provide selection of various
Voice Control settings, such as the sensitivity and confidence
level of the recognizer (i.e., the level of certainty required to
decide that an utterance has been recognized). The optimal values
for these parameters depend upon the microphone in use and the
speaking voice of the user.
[0071] The interface 186 to the user does not operate via the
Macintosh event interface. Rather, it is simply a recursive loop
which controls the Recognition Software and the state of the Voice
Control driver.
[0072] Language Maker 140 includes an application analyzer 210 and
an event recorder 212. Application analyzer 210 parses the
executable code of applications as discussed above, and produces
suitable default utterance names and pre-programmed command
strings. The application analyzer 210 includes a menu extraction
procedure 214 which searches executable code to find text strings
corresponding to menus. The application analyzer 210 also includes
control identification procedures 216 for creating the command
strings corresponding to each menu item in an application.
[0073] The event recorder 212 is a driver for recording user
commands and creating command strings for utterances. This allows
the user to easily create and edit command strings as discussed
above.
[0074] Types of events which may be entered into the event recorder
include: text entry 218, mouse events 220 (such as clicking at a
specified place on the screen), special events 222 which may be
necessary to control a particular application, and voice events 224
which may be associated with operations of the Voice Control
driver.
[0075] Language Maker
[0076] Referring to FIG. 4, the Language Maker main event loop 230
is similar in structure to main event loops used by other desk
accessories in the Macintosh operating system. If a desk accessory
is selected from the "Apple" menu, an "open" event is transmitted
to the accessory. In general, if the application in which it
resides quits or if the user quits it using its menus, a "close"
event is transmitted to the accessory. Otherwise, the accessory is
transmitted control events. The message parameter of a control
event indicates the kind of event. As seen in FIG. 4, the Language
Maker main event loop 230 begins With an analysis 232 of the event
type.
[0077] If the event is an open event Language Maker tests 234
whether it is already opened. If Language Maker is already opened
236, the current language (i.e. the list of utterance names from
the current word list) is displayed and Language Maker returns 237
to the operating system. If Language Maker is not open 238, it is
initialized and then returns 239 to the operating system.
[0078] If the event is a close event, Language Maker prompts the
user 240 to save the current language as a language file. If the
user commands Language Maker to save the current language, the
current language is converted by the Write Production module 242 to
a language file, and then Language Maker exits 244. If the current
language is not saved, Language Maker exits directly.
[0079] If the event is a control event 246, then the way in which
Language Maker responds to the event depends upon the mode that
Language Maker is in, because Language Maker has a utility for
recording events (i.e. the mouse movements and clicks or text entry
that the user wishes to assign to an utterance), and must record
events which do not involve the Language Maker window. However,
when not recording, Language Maker should only respond to events in
its window. Therefore, Language Maker may respond to events in one
mode but not in another.
[0080] A control event 246 is forwarded to one of three branches
248, 250, 252. All menu events are forwarded to the accMenu branch
252. (Only menu events occurring in desk accessory menus will be
forwarded to Language Maker.) All window events for the Language
Maker window are forwarded to the accEvent branch 250. All other
events received by Language Maker, which correspond to events for
desktop accessories or applications other than Language Maker,
initiate activity in the accRun branch 248, to enable recording of
actions.
[0081] In the accRun branch 248, events are recorded and associated
with the selected utterance name. Before any events are recorded
Language Maker checks 254 if Language Maker is recording; if not,
Language Maker returns 256. If recording is on 258, then Language
Maker checks the current recording mode.
[0082] While recording, Language Maker seizes control of the
operating system by setting control flags that cause the operating
system to call Language Maker every tick of the Macintosh (i.e.
every {fraction (1/60)} second).
[0083] If the user has set Language Maker in dialog mode, Language
Maker can record dialog events (i.e. events which involve modal
dialog, where the user cannot do anything except respond to the
actions in modal dialog boxes). To accomplish this, the user must
be able to produce actions (i.e. mouse clicks, menu selections) in
the current application so that the dialog boxes are prompted to
the screen. Then the user can initialize recording and respond to
the dialog boxes. When modal dialog boxes should be produced,
events received by Language Maker are also forwarded to the
operating system. Otherwise, events are not forwarded to the
operating system. Language Maker's modal dialog recording is
performed by the Run Modal module 260.
[0084] If modal dialog events are not being recorded, the user
records with Language Maker in "action" mode, and Language Maker
proceeds to the Run Edit module 262.
[0085] In the accEvent branch, all events are forwarded to the
Event Handler module 264.
[0086] In the accMenu branch, the menu indicated by the desk
accessory menu event is checked 266. If the event occurred in the
Language Maker menu, it is forwarded to the Do My Menu module 268.
Other events are ignored 270.
[0087] Referring to FIG. 5, the Run Edit module 262 performs a loop
272, 274. Each action is recorded by the Record Actions submodule
272. If there are more actions in the event queue then the loop
returns to the Record Actions submodule. If a cancel action appears
276 in the event queue then Run Edit returns 277 without updating
the current language in memory. Otherwise, if the events are
completed successfully, run edit updates the language in memory and
turns off recording 278 and returns to the operating system
280.
[0088] Referring to FIG. 6, in the Record Actions submodule 272,
actions performed by the user in record mode are recorded. When the
current application makes a request for the next event on the event
queue, the event is checked by record actions. Each non-null event
(i.e. each action) is processed by Record Actions. First, the type
of action is checked 282. If the action selects a menu 284, then
the selected menu is recorded. If the action is a mouse click 286,
the In Button? routine (see FIG. 8) checks if the click occurred
inside of a button (a button is a menu selection area in the front
window) or not. If so, the button is recorded 288. If not, the
location of the click is recorded 290.
[0089] Other actions are recorded by special handlers. These
actions include group actions 292, mouse down actions 294, mouse up
actions 296, zoom actions 298, grow actions 300, and next window
actions 302.
[0090] Some actions in menus can create pop-up menus with
subchoices. These actions are handled by popping up the appropriate
pop-up menu so that the user may select the desired subchoice. Move
actions 304, pause actions 306, scroll actions 308, text actions
310 and voice actions 312 pop up respective menus and Record
Actions checks 314 for the menu selection made by the user (with a
mouse drag). If no menu selection is made, then no action is
recorded 316. Otherwise, the choice is recorded 318.
[0091] Other actions may launch applications. In this case 320 the
selected application is determined. If no application has been
selected then no action is recorded 322, otherwise the selected
application is recorded 324.
[0092] Referring to FIG. 7, the Run Modal procedure 260 allows
recording of the modal dialogs of the Macintosh computer. During
modal dialogs, the user cannot do anything except respond to the
actions in the modal dialog box. In order to record responses to
those actions, Run Modal has several phases, each phase
corresponding to a step in the recording process.
[0093] In the first phase, when the user selects dialog recording,
Run Modal prompts the user with a Language Maker dialog box that
gives the user the options "record" and "cancel" (see FIG. 25). The
user may then interact with the current application until arriving
at the dialog click that is to be recorded. During this phase, all
calls to Run Modal are routed through Select Dialog 326, which
produces the initial Language Maker dialog box, and then returns
327, ignoring further actions.
[0094] To enter the second, recording, phase, the user clicks on
the "record" button in the Language Maker dialog box, indicating
that the following dialog responses are to be recorded. In this
phase, calls to Run Modal are routed to Record 328, which uses the
In Button? routine 330 to check if a button in current
application's dialog box has been selected. If the click occurred
in a button, then the button is recorded 332, and Run Modal returns
333. Otherwise, the location of the click is recorded 334 and Run
Modal returns 335.
[0095] Finally, when all clicks are recorded, the user clicks on
the "cancel" button in the Language Maker dialog box, entering the
third phase of the recording session. The click in the "cancel"
button causes Run Modal to route to Cancel 336, which updates 338
the current language in memory, then returns 340.
[0096] Referring to FIG. 8, the In Button? procedure 286 determines
whether a mouse click event occurred on a button. In Button? gets
the current window control list 342 (a Macintosh global which
contains the locations of all of the button rectangles in the
current window, refer to Appendix B) from the operating system and
parses the list with a loop 344-350. Each control is fetched 350,
and then the rectangle of the control is found 346. Each rectangle
is analyzed 348 to determine if the click occurred in the
rectangle. If not, the next control is fetched 350, and the loop
recurses. If, 344, the list is emptied, then the click did not
occur on a button, and no is returned 352. However, if the click
did occur in a rectangle, then, if, 351, the rectangle is named,
the click occurred on a button, and yes is returned 354; if the
rectangle is not named 356, the click did not occur on a button,
and no is returned 356.
[0097] Referring to FIG. 9, the Event Handler module 264 deals with
standard Macintosh events in the Language Maker display window. The
Language Maker display window lists the utterance names in the
current language. As shown in FIG. 9, Event Handler determines 358
whether the event is a mouse or keyboard event and subsequently
performs the proper action on the Language Maker window.
[0098] Mouse events include: dragging the window 360, growing the
window 362, scrolling the window 364, clicking on the window 368
(which selects an utterance name), and dragging on the window 370
(which moves an utterance name from one location on the screen to
another, potentially changing the utterance's position in the
language hierarchy). Double-clicking 366 on an utterance name in
the window selects that utterance name for action recording, and
therefore starts the Run Edit module.
[0099] Keyboard events include the standard cut 372, copy 374, and
paste 376 routines, as well as cursor movements down 380, up 382,
right 384, and left 386. Pressing return at the keyboard 378, as
with a double click at the mouse, selects the current utterance
name for action recording by Run Edit. After the appropriate
command handler is called, Event Handler returns 388. The
modifications to the language hierarchy performed by the Event
Handler module are reflected in hierarchical structure of the
language file produced by the Write Production module during close
and save operations.
[0100] Referring to FIG. 10, the Do My Menu module 268 controls all
of the menu choices supported by Language Maker. After summoning
the appropriate submodule (discussed in detail in FIGS. 11A through
11I), Do My Menu returns 408.
[0101] Referring to FIG. 11A, the New submodule 390 creates a new
language. The New submodule first checks 410 if Language Maker is
open. If so, it prompts the user 412 to save the current language
as a language file. If the user saves the current language, New
calls Write Production module 414 to save the language. New then
calls Create Global Words 416 and forms a new language 418. Create
Global Words 416 will automatically enter a few global (i.e.
resident in all languages) utterance names and command strings into
the new language. These utterance names and command strings allow
the user to make Voice Control commands, and correspond to
utterances such as "show me the active words" and "bring up the
voice options" (the utterance macros for the corresponding voice
file are trained by the user, or copied from an existing voice
file, after the new language is saved).
[0102] Referring to FIG. 11B, the Open submodule 392 opens an
existing language for modification. The Open submodule 392 checks
420 if Language Maker is open. If so, it prompts the user 422 to
save the current language, calling Write Production 424 if yes.
Open then prompts the user to open the selected language 426. If
the user cancels, Open returns 428. Otherwise, the language is
loaded 430 and open returns 432.
[0103] Referring to FIG. 11c, the Save submodule 394 saves the
current language in memory as a language file. Save prompts the
user to save the current language 434. If the user cancels, Save
returns 436, otherwise, Save calls Write Production 438 to convert
the language into a state machine control file suitable for use by
VOCAL (FIG. 2). Finally, Save returns 440.
[0104] Referring to FIG. 11D, the New Action submodule 396
initializes the event recorders to begin recording a new sequence
of actions. New Action initializes the event recorder by displaying
an action window to the user 442, setting up a tool palette for the
user to use, and initializing recording of actions. Then New Action
returns 444. After New Action is started, actions are not delivered
to the operating system directly; rather they are filtered through
Language Maker.
[0105] Referring to FIG. 1E, the Record Dialog submodule 398
records responses to dialog boxes through the use of the Run Modal
module. Record Dialog 398 gives the user a way to record actions in
modal dialog; otherwise the user would be prevented from performing
the actions which bring up the dialog boxes. Record Dialog displays
446 the dialog action window (see FIG. 25) and turns recording on.
Then Record Dialog returns 448.
[0106] Referring to FIG. 11F, the Create Default Menus submodule
400 extracts default utterance names (and generates associated
command strings) from the executable code for an application.
Create Default Menus 270 is ordinarily the first choice selected by
a user when creating a language for a particular application. This
submodule looks at the executable code of an application and
creates an utterance name for each menu command in the application,
associating the utterance name with a command string that will
select that menu command. When called, Create Default Menus gets
450 the menu bar from the executable code of the application, and
initializes the current menu to be the first menu (X=1). Next, each
menu is processed recursively. When all menus are processed, Create
Default Menus returns 454. A first loop 452, 456, 458, 460 locates
the current (X.sup.th) menu handle 456, initializes menu parsing,
checks if the current menu is fully parsed 458, and reiterates by
updating the current menu to the next menu. A second loop 458, 462,
464 finds each menu name 462, and checks 464 if the name is
hierarchical (i.e. if the name points to further menus). If the
names are not hierarchical, the loop recurses. Otherwise, the
hierarchical menu is fetched 466, and a third loop 470, 472 starts.
In the third loop, each item name in the hierarchical menu is
fetched 472, and the loop checks if all hierarchical item names
have been fetched 470.
[0107] Referring to FIG. 11G, the Create Default Text submodule 402
allows the user to convert a text file on the clipboard into a list
of utterance names. Create default text 402 creates an utterance
name for each unique word in the clipboard 474, and then returns
476. The utterance names are associated with the keyboard entries
which will type out the name. For example, a business letter can be
copied from the clipboard into default text. Utterances would then
be associated with each of the common business terms in the letter.
After ten or twelve business letters have been converted the
majority of the business letter words would be stored as a set of
utterances.
[0108] Referring to FIG. 11H, the Alphabetize Group submodule 404
allows the user to alphabetize the utterance names in a language.
The selected group of names (created by dragging the mouse over
utterance names in the Language Maker window) is alphabetized 478,
and then Alphabetize Group returns 480.
[0109] Referring to FIG. 11I, the Preferences submodule 406 allows
the user to select standard graphic user interface preferences such
as font style 482 and font size 484. The Preferences submenu 486
allows the user to state the metric by which mouse locations of
recorded actions are stored. The coordinates for mouse actions can
be relative to the global window coordinates or relative to the
application window coordinates. In the case where application menu
selections are performed by mouse clicks, the mouse clicks must
always be in relative coordinates so that the window may be moved
on the screen without affecting the function of the mouse click.
The Preferences submenu 486 also determines whether, when a mouse
action is recorded, the mouse is left at the location of a click or
returned to its original location after a click. When the
preference selections are done 488, the user is prompted whether he
wants to update the current preference settings for Language Maker.
If so, the file is updated 490 and Preferences returns 492. If not,
Preferences returns directly to the operating system 494 without
saving.
[0110] Referring to FIG. 12, the Write Production module 242 is
called when a file is saved. Write Production saves the current
language and converts it from an outline processor format such as
that used in the Language Maker application to a hierarchical text
format suitable for use with the state machine based Recognition
Software. Language files are associated with applications and new
language files can be created or edited for each additional
application to incorporate the various commands of the application
into voice recognition.
[0111] The embodiment of the Write Production module depends upon
the Recognition Software in use. In general, the Write Production
module is written to convert the current language to suitable
format for the Recognition Software in use. The particular
embodiment of Write Production shown in FIG. 12 applies to the
syntax of the VOCAL compiler for the Dragon Systems Recognition
Software.
[0112] Write Production first tests the language 494 to determine
if there are any sub-levels. If not, the Write Terminal submodule
496 saves the top level language, and Write Production returns 498.
If sub-levels exist in the language, then each sub-level is
processed by a tail-recursive loop. If a root entry exists in the
language 500 (i.e. if only one utterance name exists at the current
level) then Write Production writes 502 the string "Root=(" to the
file, and checks for sub-levels 512. Otherwise, if no root exists,
Write Terminal is called 504 to save the names in the current level
of the language. Next, the string "TERMINAL =" is written 506, and
if, 508, the language level is terminal, the string "(" is written.
Next, Write Production checks 512 for sublevels in the language. If
no sub-levels exist, Write Production returns 514. Otherwise, the
sub-levels are processed by another call 516 to Write Production on
the sub-level of the language. After the sub-level is processed,
Write Production writes the string ")" and returns 518.
[0113] Referring to FIG. 13, the Write Terminal submodule 496
writes each utterance name and the associated command string to the
language file. First, Write Terminal checks 520 if it is at a
terminal. If not, it returns 530. Otherwise, Write Terminal writes
522 the string corresponding to the utterance name to the language
file. Next, if, 524, there is an associated command string, Write
Terminal writes the command string (i.e. "output") to the language
file. Finally, Write Terminal writes 528 the string ";" to the
language file and returns 530.
[0114] Voice Control
[0115] The Voice Control software serves as a gate between the
operating system and the applications running on the operating
system. This is accomplished by setting the Macintosh operating
system's get_next_event procedure equal to a filter procedure
created by Voice Control. The get_next_event procedure runs when
each next_event request is generated by the operating system or by
applications. Ordinarily the get_next_event procedure is null, and
next_event requests go directly to the operating system. The filter
procedure passes control to Voice Control on every request. This
allows Voice Control to perform voice actions by intercepting mouse
and keyboard events, and create new events corresponding to spoken
commands.
[0116] The Voice Control filter procedure is shown in FIG. 14.
[0117] After installation 538, the get_next_event filter procedure
540 is called before an event is generated by the operating system.
The event is first checked 54Z to see if it is a null event. If so,
the Process Input module 544 is called directly. The Process Input
routine 544 checks for new speech input and processes any that has
been received. After Process Input, the Voice Control driver
proceeds through normal filter processing 546 (i.e., any filter
processing caused by other applications) and returns 548. If the
next event is not a null event, then displays are hidden 550. This
allows Voice Control to hide any Voice Control displays (such as
current language lists) which could have been generated by a
previous non-null action. Therefore, if any prompt windows have
been produced by Voice Control, when a non-null event occurs, the
prompt windows are hidden. Next, key down events are checked 552.
Because the recognizer is controlled (i.e. turned on and off) by
certain special key down events, if the event is a key down event
then Voice Control must do further processing. Otherwise, the Voice
Control drive procedure moves directly to Process Input 544. If a
key down event has occurred 554, where appropriate, software
latches which control the recognizer are set. This allows
activation of the Recognizer Software, the selection of Recognizer
options, or the display of languages. Thereafter, the Voice Control
driver moves to Process Input 544.
[0118] Referring to FIG. 15, the Process Input routine is the heart
of the Voice Control driver. It manages all voice input for the
Voice Navigator. The Process Input module is called each time an
event is processed by the operating system. First 546, any latches
which need to be set are processed, and the Macintosh waits for a
number of delay ticks, if necessary. Delay ticks are included, for
example, where a menu drag is being performed by Voice Control, to
allow the menu to be drawn on the screen before starting the drag.
Also, some applications require delay between mouse or keyboard
events. Next, if recognition is activated 548 the process input
routine proceeds to do recognition 562. If recognition is
deactivated, Process Input returns 560.
[0119] The recognition routine 562 prompts the recognition drivers
to check for an utterance (i.e., sound that could be speech input).
If there is recognized speech input 564, Process Input checks the
vertical blanking interrupt VBL handler 566, and deactivates it
where appropriate.
[0120] The vertical blanking interrupt cycle is a very low level
cycle in the operating system. Every time the screen is refreshed,
as the raster is moving from the bottom right to the top left of
the screen, the vertical blanking interrupt time occurs. During
this blanking time, very short and very high priority routines can
be executed. The cycle is used by the Process Input routine to move
the mouse continuously by very slowly incrementing of the mouse
coordinates where appropriate. To accomplish this, mouse move
events are installed onto the VBL queue. Therefore, where
appropriate, the VBL handler must be deactivated to move the
mouse.
[0121] Other speech input is placed 568 on a speech queue, which
stores speech related events for the processor until they can be
handled by the ProcessQ routine. However, regardless of whether
speech is recognized, ProcessQ 570 is always called by Process
Input. Therefore, the speech events queued to ProcessQ are
eventually executed, but not necessarily in the same Process Input
cycle. After calling ProcessQ, Process Input returns 571.
[0122] Referring to FIG. 16, the Recognize submodule 562 checks for
encoded utterances queued by the Voice Navigator box, and then
calls the recognition drivers to attempt to recognize any
utterances. Recognize returns the number of commands in (i.e. the
length of) the command string returned from the recognizer. If,
572, no utterance is returned from the recognizer, then Recognize
returns a length of zero (574), indicating no recognition has
occurred. If an utterance is available, then Recognize calls
sdi_recognize 576, instructing the Recognizer Software to attempt
recognition on the utterance. If, 578, recognition is successful,
then the name of the utterance is displayed 582 to the user. At the
same time, any close call windows (i.e. windows associated with
close call choices, prompted by Voice Control in response to the
Recognizer Software) are cleared from the display. If recognition
is unsuccessful, the Macintosh beeps 580 and zero length is
returned 574.
[0123] If recognition is successful, Recognize searches 584 for an
output string associated with the utterance. If there is an output
string, recognize checks if it is asleep 586. If it is not asleep
590, the output count is set to the length of the output string
and, if the command is a control command 592 (such as "go to sleep"
or "wake up"), it is handled by the Process Voice Commands routine
594.
[0124] If there is no output string for the recognized utterance,
or if the recognizer is asleep, then the output of Recognize is
zero (588). After the output count is determined 596, the state of
the recognizer is processed 596. At this time, if the Voice Control
state flags have been modified by any of the Recognize subroutines,
the appropriate actions are initialized. Finally, Recognize returns
598.
[0125] Referring to FIG. 17, the Process Voice Commands module
deals with commands that control the recognizer. The module may
perform actions, or may flag actions to be performed by the Process
States block 596 (FIG. 16). If the recognizer is put to sleep 600
or awakened 604, the appropriate flags are set 602, 606, and zero
is returned 626, 628 for the length of the command string,
indicating to Process States to take no further actions. Otherwise,
if the command is scratch_that 608 (ignore last utterance), first
level 612 (go to top of language hierarchy, i.e. set the Voice
Control state to the root state for the language), word_list 616
(show the current language), or voice_options 620, the appropriate
flags are set and 610, 614, 618, 622, and a string length of -1 is
returned 624, 628, indicating that the recognizer state should be
changed by Process States 596 (FIG. 16).
[0126] Referring to FIG. 18 the ProcessQ module 570 pulls speech
input from the speech queue and processes it. If, 630, the event
queue is empty then ProcessQ may proceed, otherwise ProcessQ aborts
632 because the event queue may overflow if speech events are
placed on the queue along with other events. If, 634, the speech
queue has any events then process queue checks to see if, 636,
delay ticks for menu drawing or other related activities have
expired. If no events are on the speech queue the ProcessQ aborts
636. If delay ticks have expired, then ProcessQ calls Get Next 642
and returns 644. Otherwise, if delay ticks have not expired,
ProcessQ aborts 640.
[0127] Referring to FIG. 19, the Get Next submodule 642 gets
characters from the speech queue and processes them. If, 646, there
are no characters in the speech queue then the procedure simply
returns 648. If there are characters in the speech queue then Get
Next checks 650 to see if the characters are command characters. If
they are, then Get Next calls Check Command 660. If not, then the
characters are text, and Get Next sets the meta bits 652 where
appropriate.
[0128] When the Macintosh posts an event, the meta bits (see
Appendix B) are used as flags for conditioning keystrokes such as
the condition key, the option key, or the command key. These keys
condition the character pressed at the keyboard and create control
characters. To create the proper operating system events,
therefore, the meta bits must be set where necessary. Once the meta
bits are set 652, a key down event is posted 654 to the Macintosh
event queue, simulating a keypush at the keyboard. Following this,
a key up is posted 656 to the event queue, simulating a key up. If,
658, there is still room in the event queue, then further speech
characters are obtained and processed 646. If not, then the Get
Next procedure returns 676.
[0129] If the command string input corresponds to a command rather
than simple key strokes, the string is handled by the Check Command
procedure 660 as illustrated in FIG. 19. In the Check Command
procedure 660 the next four characters from the speech queue (four
characters is the length of all command strings, see Appendix A)
are fetched 662 and compared 664 to a command table. If, 666, the
characters equal a voice command, then a command is recognized, and
processing is continued by the Handle Command routine 668.
Otherwise, the characters are interpreted as text and processing
returns to the meta bits step 652.
[0130] In the Handle Command procedure 668 each command is
referenced into a table of command procedures by first computing
670 the command handler offset into the table and then referencing
the table, and calling the appropriate command handler 672. After
calling the appropriate command handler, Get Next exits the Process
Input module directly 674 (the structure of the software is such
that a return from Handle Command would return to the meta bits
step 652, which would be incorrect).
[0131] The command handlers available to the Handle Command routine
are illustrated in FIG. 20. Each command handler is detailed by a
flow diagram in FIGS. 21A through 21G. The syntax for the commands
is detailed in Appendix A.
[0132] Referring to FIG. 21A, the Menu command will pull down a
menu, for example, @MENU(apple,0) (where apple is the menu number
for the apple menu) will pull down the apple menu. Menu command
will also select an item from the menu, for example,
@MENU(apple,calculator) (where calculator is the itemnumber for the
calculator in the apple menu) will select the calculator from the
apple menu. Menu command initializes by running the Find Menu
routine 678 which queues the menu id and the item number for the
selected menu. (If the item number in the menu is 0 then Find Menu
simply clicks on the menu bar.) After Find Menu returns, if 680,
there are no menus queued for posting, the Menu command simply
returns 690. However, if menus are queued for posting, Menu command
intercepts 682 one of the Macintosh internal traps called Menu
Select. The Menu Select trap is set equal to the My Menu Select
routine 692. Next the cursor coordinates are hidden 684 so that the
mouse cannot be seen as it moves on the screen. Next, Menu command
posts 686 a mouse down (i.e. pushes the mouse button down) on the
menu bar. When the mouse down occurs on the menu bar the Macintosh
operating system generates a menu event for the application. Each
application receiving a menu event requests service from the
operating system to find out what the menu event is. To do this the
application issues a Menu Select trap. The menu select trap then
places the location of the mouse on the stack. However, when the
application issues a menu select trap in this case, it is serviced
by the My Menu Select routine 692 instead, thereby allowing Menu
command to insert the desired menu coordinates in the place of the
real coordinates. After posting a mouse down in the appropriate
menu bar, Menu Command sets 688 the wait ticks to 30, which gives
the operating system time to draw the menu, and returns 690.
[0133] In the My Menu Select trap 692 the menuselect global state
is reset 694 to clear any previously selected menus, and the
desired menu id and the item number are moved to the Macintosh
stack 696, thus selecting the desired menu item.
[0134] The Find Menu routine 700 collects 702 the command
parameters for the desired menu. Next, the menuname is compared 704
to the menu name list. If, 706, there is no menu with the name
"menuname", Find Menu exits 708. Otherwise, Find Menu compares 710
the itemname to the names of the items in the menu. If, 712, the
located item number is greater than 0, then Find Menu queues 718
the menu id and item number for use by Menu command, and returns
720. Otherwise, if the item number is 0 then Find Menu simply sets
714 the internal Voice Control flags "mousedown" and "global" flags
to true. This indicates to Voice Control that the mouse location
should be globally referenced, and that the mouse button should be
held down. Then Find Menu calls 716 the Post Mouse routine, which
references these flags to manipulate the operating system's mouse
state accordingly.
[0135] Referring to FIG. 21B, the Control command 722 performs a
button push within a menu, invoking actions such as the save
command in the file menu of an application. To do this, the Control
command gets the command parameters 724 from the control string,
finds the front window 726, gets the window command list 728, and
checks 730 if the control name exists in the control list. If the
control name does exist in the control list then the control
rectangle coordinates are calculated 732, the Post Mouse routine
734 clicks the mouse in the proper coordinates, and the Control
command returns 736. If the control name is not found, the Control
command returns directly.
[0136] The Keypad command 738 simulates numerical entries at the
Macintosh keypad. Keypad finds the command parameters for the
command string 740, gets the keycode value 742 for the desired key,
posts a key down event 744 to the Macintosh event queue, and
returns 746.
[0137] The Zoom command 748 zooms the front window. Zoom obtains
the front window pointer 750 in order to reference the mouse to the
front window, calculates the location of the zoom box 752, uses
Post Mouse to click in the zoom box 754, and returns 756.
[0138] The Local Mouse command 758 clicks the mouse at a locally
referenced location. Local Mouse obtains the command parameters for
the desired mouse location 760, uses Post Mouse to click at the
desired coordinate 762, and returns 764.
[0139] The Global Mouse command 766 clicks the mouse at a globally
referenced location. Global Mouse obtains the command parameters
for the desired mouse location 768, sets the global flag to true
770 (to signal to Post Mouse that the coordinates are global), uses
Post Mouse to click at the desired coordinate 772, and returns
774.
[0140] The Double Click command double clicks the mouse at a
locally referenced location. Double Click obtains the command
parameters for the desired mouse location 778, calls Post Mouse
twice 780, 782 (to click twice in the desired location), and
returns 784.
[0141] The Mouse Down command 786 sets the mouse button down. Mouse
Down sets the mousedown flag to true 788 (to signal to Post Mouse
that mouse button should be held down), uses Post Mouse to set the
button down 790, and returns 792.
[0142] The Mouse Up command 794 sets the mouse button up. Mouse Up
sets the mbState global (see Appendix B) to Mouse Button UP 796 (to
signal to the operating system that mouse button should be set up),
posts a mouse up event to the Macintosh event queue 798 (to signal
to applications that the mouse button has gone up), and returns
800.
[0143] Referring to FIG. 21D, the Screen Down command 802 scrolls
the contents of the current window down. Screen Down first looks
804 for the vertical scroll bat in the front window. If, 806, the
scroll bar is not found, Screen Down simply returns 814. If the
scroll bar is found, Screen Down calculates the coordinates of the
down arrow 808, sets the mousedown flag to true 810 (indicating to
Post Mouse that the mouse button should be held down), uses Post
Mouse to set the mouse button down 812, and returns 814. The Screen
Up command 816 scrolls the contents of the current window up.
Screen Up first looks 818 for the vertical scroll bar in the front
window. If, 820, the scroll bar is not found, Screen Up simply
returns 828. If the scroll bar is found, Screen Up calculates the
coordinates of the up arrow 822, sets the mousedown flag to true
824 (indicating to Post Mouse that the mouse button should be held
down), uses Post Mouse to set the mouse button down 826, and
returns 828.
[0144] The Screen Left command 830 scrolls the contents of the
current window left. Screen Left first looks 832 for the horizontal
scroll bar in the front window. If, 834, the scroll bar is not
found, Screen Left simply returns 842. If the scroll bar is found,
Screen Left calculates the coordinates of the left arrow 836, sets
the mousedown flag to true 838 (indicating to Post Mouse that the
mouse button should be held down), uses Post Mouse to set the mouse
button down 840, and returns 842.
[0145] The Screen Right command 844 scrolls the contents of the
current window right. Screen Right first looks 846 for the
horizontal scroll bar in the front window. If, 848, the scroll bar
is not found, Screen Right simply returns 856. If the scroll bar is
found, Screen Right calculates the coordinates of the right arrow
850, sets the mousedown flag to true 852 (indicating to Post Mouse
that the mouse button should be set down), uses Post Mouse to set
the mouse button down 854, and returns 856.
[0146] Referring to FIG. 21E, the Page Down command 858 moves the
contents of the current window down a page. Page Down first looks
860 for the vertical scroll bar in the front window. If, 862, the
scroll bar is not found, Page Down simply returns 868. If the
scroll bar is found, Page Down calculates the page down button
coordinates 864, uses Post Mouse to click the mouse button down
866, and returns 868.
[0147] The Page Up command 870 moves the contents of the current
window up a page. Page Up first looks 872 for the vertical scroll
bar in the front window. If, 874, the scroll bar is not found, Page
Up simply returns 880. If the scroll bar is found, Page Up
calculates the page up button coordinates 876, uses Post Mouse to
click the mouse button down 878, and returns 880.
[0148] The Page Left command 882 moves the contents of the current
window left a page. Page Left first looks 884 for the horizontal
scroll bar in the front window. If, 886, the scroll bar is not
found, Page Left simply returns 892. If the scroll bar is found,
Page Left calculates the page left button coordinates 888, uses
Post Mouse to click the mouse button down 890, and returns 892.
[0149] The Page Right command 894 moves the contents of the current
window right a page. Page Right first looks 896 for the horizontal
scroll bar in the front window. If, 898, the scroll bar is not
found, Page Right simply returns 904. If the scroll bar is found,
Page Right calculates the page right button coordinates 900, uses
Post Mouse to click the mouse button down 902, and returns 904.
[0150] Referring to FIG. 21F, the Move command 906 moves the mouse
from its current location (y,x), to a new location
(y+.delta.y,x+.delta.x). First, Move gets the command parameters
908, then Move sets the mouse speed to tablet 910 (this cancels the
mouse acceleration, which otherwise would make mouse movements
uncontrollable), adds the offset parameters to the current mouse
location 912, forces a new cursor position and resets the mouse
speed 914, and returns 916.
[0151] The Move to Global Coordinate command 918 moves the cursor
to the global coordinates given by the Voice Control command
string. First, Move to Global gets the command parameters 920, then
Move to Global checks 922 if there is a position parameter. If
there is a position parameter, the screen position coordinates are
fetched 924. In either case, the global coordinates are calculated
926, the mouse speed is set to tablet 928, the mouse position is
set to the new coordinates 930, the cursor is forced to the new
position 932, and Move to Global returns 934.
[0152] The Move to Local Coordinate command 936 moves the cursor to
the local coordinates given by the Voice Control command string.
First, Move to Local gets the command parameters 938, then Move to
Local checks 940 if there is a position parameter. If there is a
position parameter, the local position coordinates are fetched 942.
In either case, the global coordinates are calculated 944, the
mouse speed is set to tablet 946, the mouse position is set to the
new coordinates 948, the cursor is forced to the new position 950,
and Move to Global returns 952.
[0153] The Move Continuous command 954 moves the mouse continuously
from its present location, moving .delta.y,.delta.x every refresh
of the screen. This is accomplished by inserting 956 the VBL Move
routine 960 in the Vertical Blanking Interrupt queue of the
Macintosh and returning 958. Once in the queue, the VBL Move
routine 960 will be executed every screen refresh. The VBL Move
routine simply adds the .delta.y and .delta.x values to the current
cursor position 962, resets the cursor 964, and returns 966.
[0154] Referring to FIG. 21G, the Option Key Down command 968 sets
the option key down. This is done by setting the option key bit in
the keyboard bit map to TRUE 970, and returning 972.
[0155] The Option Key Up command 974 sets the option key up. This
is done by setting the option key bit in the keyboard bit map to
FALSE 976, and returning 978.
[0156] The Shift Key Down command 980 sets the shift key down. This
is done by setting the shift key bit in the keyboard bit map to
TRUE 982, and returning 984.
[0157] The Shift Key Up command 986 sets the shift key up. This is
done by setting the shift key bit in the keyboard bit map to FALSE
988, and returning 990.
[0158] The Command Key Down command 992 sets the command key down.
This is done by setting the command key bit in the keyboard bit map
to TRUE 994, and returning 996.
[0159] The Command Key Up command 998 sets the command key up. This
is done by setting the command key bit in the keyboard bit map to
FALSE 1000, and returning 1002.
[0160] The Control Key Down command 1004 sets the control key down.
This is done by setting the control key bit in the keyboard bit map
to TRUE 1006, and returning 1008.
[0161] The Control Key Up command 1010 sets the control key up.
This is done by setting the control key bit in the keyboard bit map
to FALSE 1012, and returning 1014.
[0162] The Next Window command 1016 moves the front window to the
back. This is done by getting the front window 1018 and sending it
to the back 1020, and returning 1022.
[0163] The Erase command 1024 erases numchars characters from the
screen. The number of characters typed by the most recent voice
command is stored by Voice Control. Therefore, Erase will erase the
characters from the most recent voice command. This is done by a
loop which posts delete key keydown events 1026 and checks 1028 if
the number posted equals numchars. When numchars deletes have been
posted, Erase returns 1030.
[0164] The Capitalize command 1032 capitalizes the next keystroke.
This is done by setting the caps flag to TRUE 1034, and returning
1036.
[0165] The Launch command 1038 launches an application. The
application must be on the boot drive no more than one level
deep.
[0166] This is done by getting the name of the application 1040
("appl_name"), searching for appl_name on the boot volume 1042,
and, if, 1044, the application is found, setting the volume to the
application folder 1048, launching the application 1050 (no return
is necessary because the new application will clear the Macintosh
queue). If the application is not found, Launch simply returns
1046.
[0167] Referring to FIG. 22, the Post Mouse routine 1052 posts
mouse down events to the Macintosh event queue and can set traps to
monitor mouse activity and to keep the mouse down. The actions of
Post Mouse are determined by the Voice Control flags global and
mousedown, which are set by command handlers before calling Post
Mouse. After a Post Mouse, when an application does a
get_next_event it will see a mouse down event in the event queue,
leading to events such as clicks, mouse downs or double clicks.
[0168] First, Post Mouse saves the current mouse location 1054 so
that the mouse may be returned to its initial location after the
mouse events are produced. Next the cursor is hidden 1056 to shield
the user from seeing the mouse moving around the screen. Next the
global flag is checked. If, 1058, the coordinates are local (i.e.
global=FALSE) then they are converted 1060 to global coordinates.
Next, the mouse speed is set to tablet 1062 (to avoid acceleration
problems), and the mouse down is posted to the Macintosh event
queue 1064. If, 1066, the mousedown flag is TRUE (i.e. if the mouse
button should be held down) then the Set Mouse Down routine is
called 1072 and Post Mouse returns 1070. Otherwise, if the mouse
down flag is FALSE, then a click is created by posting a mouse up
event to the Macintosh event queue 1068 and returning 1070.
[0169] Referring to FIG. 23, the Set Mouse Down routine 1072 holds
the mouse button down by replacing 1074 the Macintosh button trap
with a Voice Control trap named My Button. The My Button trap then
recognizes further voice commands and creates mouse drags or clicks
as appropriate. After initializing My Button, Set Mouse Down checks
1076 if the Macintosh is aMacintosh Plus, in which case the Post
Event trap must also be reset 1078 to the Voice Control My Post
Event trap. (The Macintosh Plus will not simply check the mbState
global flag to determine the mouse button state. Rather, the Post
Event trap in a Macintosh Plus will poll the actual mouse button to
determine its state, and will post mouse up events if the mouse
button is up. Therefore, to force the Macintosh Plus to accept the
mouse button state as dictated by Voice Control, during voice
actions, the Post Event trap is replaced with a My Post Event trap,
which will not poll the status of the mouse button.) Next, the
mbState flag is set to MouseDown 1080 (indicating that the mouse
button is down) and Set Mouse Down returns 1082.
[0170] The My Button trap 1084 replaces the Macintosh button trap,
thereby seizing control of the button state from the operating
system. Each time My Button is called, it checks 1086 the Macintosh
mouse button state bit mbstate. If mbState has been set to UP, My
Button moves to the End Button routine 1106 which sets mbState to
UP 1108, removes any VBL routine which has been installed 1110,
resets the Button and Post Event traps to the original Macintosh
traps 1112, resets the mouse speed and couples the cursor to the
mouse 1114, shows the cursor 1102, and returns 1104.
[0171] However, if the mouse button is to remain down, My Button
checks for the expiration of wait ticks (which allow the Macintosh
time to draw menus on the screen) 1088, and calls the recognize
routine 1090 to recognize further speech commands. After further
speech commands are recognized, My Button determines 1092 its next
action based on the length of the command string. If the command
string length is less than zero, then the next voice command was a
Voice Control internal command, and the mouse button is released by
calling End Button 1106. If the command string length is greater
than zero, then a command was recognized, and the command is queued
onto the voice que 1094, and the voice queue is checked for further
commands 1096. If nothing was recognized (command string length of
zero), then My Button skips directly to checking the voice queue
1096. If there is nothing in the voice queue, then My Button
returns 1104. However, if there is a command in the voice queue,
then My Button checks 1098 if the command is a mouse movement
command (which would cause a mouse drag). If it is not a mouse
movement, then the mouse button is released by calling End Button
1106. If the command is a mouse movement, then the command is
executed 1100 (which drags the mouse), the cursor is displayed
1102, and My Button returns.
[0172] Screen Displays
[0173] Referring to FIG. 24, a screen display of a record actions
session is shown. The user is recording a local mouse click 1106,
and the click is being acknowledged in the action list 1108 and in
the action window 1110.
[0174] Referring to FIG. 25, a record actions session using dialog
boxes is shown. The dialog boxes 1112 for recording a manual
printer feed are displayed to the user, as well as the Voice
Control Run Modal dialog box 1114 prompting the user to record the
dialogs. The user is preparing to record a click on the Manual Feed
button 1116.
[0175] Referring to FIG. 26, the Language Maker menu 1118 is
shown.
[0176] Referring to FIG. 27, the user has requested the current
language, which is displayed by Voice Control in a pop-up display
1120.
[0177] Referring to FIG. 28, the user has clicked on the utterance
name "apple" 1122, requesting a retraining of the utterance for
"apple". Voice control has responded with a dialog box 1124 asking
the user to say "apple" twice into the microphone.
[0178] Referring to FIG. 29, the text format of a Write Production
output file 1126 (to be compiled by VOCAL) and the corresponding
Language Maker display for the file 1128 are shown. It is clear
from FIG. 29 that the Language Maker display is far more
intuitive.
[0179] Referring to FIG. 30, a listing of the Write Production
output file as displayed in FIG. 29 is provided.
[0180] Other Embodiments
[0181] Other embodiments of the invention are within the scope of
the claims which follow the appendices. For example, the graphic
user interface controlled by a voice recognition system could be
other than that of the Apple Macintosh computer. The recognizer
could be other than that marketed by Dragon Systems.
[0182] Included in the Appendices are Appendix A, which sets forth
the Voice Control command language syntax, Appendix B, which lists
some of the Macintosh OS globals used by the Voice Navigator
system, Appendix C, which is a fiche of the Voice Navigator
executable code, Appendix D, which is the Developer's Reference
Manual for the Voice Navigator system, and Appendix E, which is the
Voice Navigator User's Manual, all incorporated by reference
herein.
[0183] A portion of the disclosure of this patent document contains
material which is subject to copyright protection (for example, the
microfiche Appendix, the User's Manual, and the Reference Manual).
The copyright owner has no objection to the facsimile reproduction
by anyone of the patent document or patent disclosure, as it
appears in the Patent and Trademark Office patent file or records,
but otherwise reserves all copyright rights whatsoever.
[0184] Appendix A: Voice Control Command Language Syntax
[0185] Menu Command--@MENU(menuname,itemnum).
[0186] Finds item named itemnum in the menu named menuname and
selects it. If itemnum is 0, hold the menu down.
[0187] Control Command--@CTRL(ctlname)
[0188] Finds the control named ctlname and clicks in its
rectangle.
[0189] Key Pad Command--@KYPD(n), where n=0-9, -, +, *, /, =, and c
for clear
[0190] Posts a Keydown for keys on the numeric keypad.
[0191] Zoom Command--@ZOOM
[0192] Clicks in the zoom box of the front window.
[0193] Local Mouse Click Command--@LMSE(y,x)
[0194] Clicks at local coordinates (y,x) of the front window.
[0195] Global Mouse Click Command--@GMSE(y,x)
[0196] Clicks at the global coordinates (y,x) of the current
screen.
[0197] Double Click Command--@DCLK(y,x)
[0198] Double clicks at the global coordinates (y,x) of the current
screen. If y=x=0, double click at the current Mouse location.
[0199] Mouse Down Command--@MSDN
[0200] Set the mouse button state to down and set up traps to keep
it down.
[0201] Mouse Up Command--@MSUP
[0202] Set the mouse button state to up.
[0203] Scroll Down Command--@SCDN
[0204] Post a mouse down in the down arrow portion of the front
window's scroll bar.
[0205] Scroll Up Command--@SCUP
[0206] Post a mouse down in the up arrow portion of the front
window's scroll bar.
[0207] Scroll Left Command--@SCUP
[0208] Post a mouse down in the left arrow portion of the front
window's scroll bar.
[0209] Scroll Right Command--@SCRT
[0210] Post a mouse down in the right arrow portion of the front
window's scroll bar.
[0211] Page Down Command--@PGDN
[0212] Click in the page down portion of the front window's scroll
bar.
[0213] Page Up Command--@PGUP
[0214] Click in the page up portion of the front window's scroll
bar.
[0215] Page Left Command--@PGLF
[0216] Click in the page left portion of the front window's scroll
bar.
[0217] Page Right Command--@PGRT
[0218] Click in the page right portion of the front window's scroll
bar.
[0219] Move Command--@MOVE(.delta.y,.delta.x)
[0220] Move the Mouse from its current location (y,x), to a new
location (y+.delta.y,x+.delta.x) where .delta.y and .delta.x are
pixels and can be either positive or negative values.
[0221] Move Continuous Command--MOVI(.delta.y,.delta.x)
[0222] Move the mouse continuously from its present location,
moving .delta.y,.delta.x every refresh of the screen.
[0223] Move to Local Coordinate
Command--MOVL(y,x<,windowname>) or
[0224] MOVL(n<,y,x<,windowname>> where
n=N,S,E,W,NE,SE,SW,NW,C- ,G
[0225] Move the cursor to the local coordinates given by (y,x) or
by (n.v+y,n.h+x). Use the grafPort of the window named
"windowname". If there is no "windowname" use the grafPort of the
front window.
[0226] Move to Global Coordinate Command--@MOVG(n,<y,x>)
[0227] where n=N,S,E,W,NE,SE,SW,NW,C,G
[0228] move the cursor to the global coordinates given by (y,x) or
by (n.v+y,n.h+x). Use the grafPort of the screen.
[0229] Option Key Down Command--@OPTD
[0230] Press (and hold) the option key.
[0231] Option Key Up Command--@OPTU
[0232] Release the option key.
[0233] Shift Key Down Command--@SHFD
[0234] Press (and hold) the shift key.
[0235] Shift Key Up Command--@SHFU
[0236] Release the shift key.
[0237] Command Key Down Command--@CMDD
[0238] Press (and hold) the command key.
[0239] Command Key Up Command--@CMDU
[0240] Release the command key.
[0241] Control Key Down Command--@CTLD
[0242] Press (and hold) the control key.
[0243] Control Key Up Command--@CTLU
[0244] Release the control key.
[0245] Next Window Command--@NEXT
[0246] Sends the front window to the back.
[0247] Erase Command--@ERAS
[0248] Erase the last numchars typed.
[0249] Capitalize Command--@CAPS
[0250] Capitalize the next letter typed.
[0251] Launch Command--@LAUN(application name)
[0252] Launch the application named application_name. The
application must be on the boot drive no more than one level
deep.
[0253] Wait Command--@WAIT(nnn)
[0254] Wait for nnn ticks to elapse before doing anything else in
recognition.
[0255] Appendix B: Macintosh OS Globals
[0256] Interfacing to the Macintosh Operating System requires that
certain low memory globals be managed by Voice Control. The
following describes the most important globals. Further information
is available in "Inside Macintosh", Vols. I-V.
[0257] Mouse Globals
[0258] MickeyBytes EQU $D6A--a pointer to the cursor value; used to
control the acceleration of the mouse. Set to point to tablet
whenever the mouse is moved more than 10 pixels. [pointer]
[0259] MTemp EQU $828--a low-level interrupt mouse location; used
to move the mouse during VBL handling while executing a @MOVI
command. [long]
[0260] Mouse EQU $830--the processed mouse coordinate; used to move
the mouse for all other @MOVx commands. [long]
[0261] MBState EQU $172--current mouse button state; used to set
the MouseDown for @MSDN and for @MENU when itemname=0. [byte]
[0262] Keyboard Globals
[0263] KeyMap EQU $174--keyboard bit map, with one bit mapped to
each key on the keyboard. Set the bit to TRUE to set the Meta keys
(option, command, shift, control) down. [2 longs]
[0264] Filter Globals
[0265] JGNEFilter EQU $29A--Get Next Event filter proc; set to
Voice Control's main loop to intercept calls to Get Next Event.
[pointer]
[0266] Event Queue Globals
[0267] evtMax EQU $1E--maximum number of events in the event queue.
When this number is reached, stop Posting events.
[0268] EventQueue EQU $14A--event queue header, the location of the
Macintosh event queue. [10 bytes]
[0269] Time Globals
[0270] Ticks EQU $16A--Tick count, time since boot. Used to measure
elapsed time between Voice Control actions. [long]
[0271] Cursor Globals
[0272] CrsrCouple EQU $8CF--cursor coupled to mouse? Used to
disconnect cursor when doing remote clicks with @LMSE and @GMSE.
[byte]
[0273] CrsrNew EQU $8CE--Cursor changed? Force a new cursor after
moving the cursor. [byte]
[0274] Menu Globals
[0275] MenuList EQU $A1 Current menuBar list structure. This handle
can be de-referenced to find all the menus associated with an
application. Use for @MENU commands [handle]
[0276] Window Globals
[0277] WindowList EQU $9D6--Z-ordered linked list of windows. This
pointer will lead to a chain of all existing windows for an
application. Use to find a window queue for all local commands.
[pointer]
[0278] Window Offsets
[0279] These values are offsets within the window records that
describe characteristics of the window. Once a window is located,
these offsets are used to calculate:
[0280] thePort EQU 0--GrafPtr; local coordinates for @LMSE and
@MOVL commands.
[0281] portRect EQU $10--port's rectangle [rect]; window relative
forms of the @MOVL command. I
[0282] controlList EQU 140--used to find the controls associated
with a window.
[0283] contrlTitle EQU 40--used to compare control Titles for @CTRL
commands.
[0284] contrlRect EQU 8--used to calculate the click locations in a
control.
[0285] nextwindow EQU 144--used to locate the next window for the
@NEXT command.
* * * * *