U.S. patent application number 12/753744 was filed with the patent office on 2011-08-18 for real-time typing assistance.
This patent application is currently assigned to Microsoft Corporation. Invention is credited to Itai Almog, Eric Norman Badger, Matthew J. Bennett, Shawna Julie Davis, Bryan W. Nealer, Timothy S. Paek, Tirthankar Sengupta.
Application Number | 20110201387 12/753744 |
Document ID | / |
Family ID | 44370018 |
Filed Date | 2011-08-18 |
United States Patent
Application |
20110201387 |
Kind Code |
A1 |
Paek; Timothy S. ; et
al. |
August 18, 2011 |
REAL-TIME TYPING ASSISTANCE
Abstract
An apparatus and method are disclosed for providing feedback and
guidance to touch screen device users to improve the text entry
user experience and performance through the use of indicators such
as feedback semaphores. Also disclosed are suggestion candidates,
which allow a user to quickly select next words to add to text
input data, or replacement words for words that have been
designated as incorrect. According to one embodiment, a method
comprises receiving text input data, providing an indicator for
possible correction of the text input data, displaying suggestion
candidates associated with alternative words for the data,
receiving a single touch screen input selecting one of the
suggestion candidates, and modifying the input data using the word
associated with the selected suggestion candidate.
Inventors: |
Paek; Timothy S.;
(Sammamish, WA) ; Almog; Itai; (Redmond, WA)
; Badger; Eric Norman; (Issaquah, WA) ; Sengupta;
Tirthankar; (Redmond, WA) ; Davis; Shawna Julie;
(Seattle, WA) ; Bennett; Matthew J.; (Seattle,
WA) ; Nealer; Bryan W.; (Seattle, WA) |
Assignee: |
Microsoft Corporation
Redmond
WA
|
Family ID: |
44370018 |
Appl. No.: |
12/753744 |
Filed: |
April 2, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61304341 |
Feb 12, 2010 |
|
|
|
Current U.S.
Class: |
455/566 ;
345/168; 704/9; 704/E11.001 |
Current CPC
Class: |
G06F 40/274 20200101;
G06F 3/04883 20130101; G06F 3/04886 20130101; G06F 3/0237 20130101;
G06F 40/247 20200101; G06F 40/166 20200101; G06N 20/00 20190101;
G06F 3/048 20130101 |
Class at
Publication: |
455/566 ;
345/168; 704/9; 704/E11.001 |
International
Class: |
G06F 3/02 20060101
G06F003/02; G06F 17/27 20060101 G06F017/27; H04M 1/00 20060101
H04M001/00 |
Claims
1. A method, comprising: receiving first input data comprising one
or more input words from a keyboard; providing one or more feedback
semaphores, wherein the feedback semaphores are operable to alert a
user using the keyboard that one or more suggestion candidates are
available; automatically displaying the one or more suggestion
candidates, wherein the suggestion candidates are each associated
with an alternative word for one or more of the input words;
automatically receiving a touch screen input selecting one of the
suggestion candidates; and modifying the first input data using the
alternative word associated with the selected suggestion
candidate.
2. The method of claim 1, further comprising: based on the first
input data, generating one or more alternative words for one or
more of the input words; auto-correcting the first input data by
automatically replacing the input word with one of the alternative
words; and wherein the providing the feedback semaphores occurs
immediately after the auto-correcting.
3. The method of claim 2, wherein: the first input data comprises a
character designated as a delimiter for the input word; and the
providing the feedback semaphores occurs based on receiving the
delimiter character in the first input data and on word probability
data associated with the input word.
4. The method of claim 1, wherein the feedback semaphores include
one or more of the following: playing a sound, generating haptic
feedback, highlighting one or more keys on the keyboard,
highlighting a background area of the keyboard, and highlighting a
suggestion candidates area.
5. The method of claim 1, wherein: the feedback semaphores include
highlighting one or more delimiter keys on the keyboard; and the
receiving a single touch screen input comprises receiving a key
press for one of the highlighted delimiter keys.
6. The method of claim 1, wherein the feedback semaphores are not
provided until the input word is designated incorrect using a
dictionary, a common speller application programming interface, or
an input history data source.
7. The method of claim 6, wherein: the first input data comprises a
character designated as a delimiter for the input word; and the
providing the feedback semaphores occurs based on receiving the
delimiter character in the first input data.
8. The method of claim 1, wherein the providing the feedback
semaphores does not occur until a delimiter key for the input word
is received in the first input data.
9. The method of claim 1, wherein: the keyboard is a touch screen
keyboard; and the receiving text input data further comprises
playing a randomly selected keypress sound for a character of the
first input data, wherein the keypress sound is selected from a
group including at least two or more keypress sounds.
10. The method of claim 1, further comprising: receiving typing
speed data for at least a portion of the first input data; and
based on the typing speed data, selecting one or more feedback
semaphores from a group including at least two or more of the
following: a tooltip balloon, audio feedback, haptic feedback,
highlighting one or more delimiter keys, highlighting the keyboard,
and wherein the one or more feedback semaphores are the selected
feedback semaphores.
11. The method of claim 1, further comprising: detecting whether
the user is using the keyboard with one hand or two hands, and;
based on the detecting, selecting one or more feedback semaphores
from a group including at least two or more of the following: a
tooltip balloon, audio feedback, haptic feedback, highlighting one
or more delimiter keys, highlighting the keyboard, and wherein the
one or more feedback semaphores are the selected feedback
semaphores.
12. A computer-readable storage media storing computer-readable
instructions that when executed by a computer cause the computer to
perform the method of claim 1.
13. A computer-readable storage media storing computer-readable
instruction that when executed by a computer cause the computer to
perform a method, the method comprising: receiving text input data
comprising at least one word using a keyboard coupled to a touch
screen; detecting that an event has occurred based on the text
input data and based on the detected event, automatically
displaying on the touch screen one or more suggestion candidates
for the at least one word based on the text input data and one or
more candidate sources, wherein each of the candidates is
associated with at least one next word designated as likely to
follow the at least one word; receiving a single touch screen input
selecting one of the suggestion candidates; and based on the single
touch screen input, automatically modifying the text input data by
adding the at least one next word associated with the selected
suggestion candidate to the text input data.
14. The computer-readable storage media of claim 13, wherein the
event is detected based on a measured keystroke latency exceeding a
threshold value.
15. The computer-readable storage media of claim 13, wherein the
event is an auto-correction of the text input data or receiving a
delimiter key in the text input data.
16. The computer-readable storage media of claim 13, further
comprising automatically displaying on the touch screen one or more
replacement suggestion candidates for the at least one word based
on the text input data and one or more candidate sources, wherein
each of the replacement suggestion candidates is associated with at
least one replacement word for the at least one word; and wherein
the displaying includes displaying the replacement suggestion
candidates in a manner that distinguishes the replacement
suggestion candidates from the suggestion candidates.
17. The computer-readable storage media of claim 13, further
comprising: immediately after the automatically adding the at least
one word, displaying one or more suggestion candidates for the text
input data, which includes the selected next word; and repeating
the automatically modifying the text input data.
18. A mobile device, comprising: one or more processing units
operable to execute computer-executable instructions for text entry
and correction; one or more memory units coupled to the processing
units; one or more touch screens coupled to the mobile device
configurable to have a text display area, a suggestion candidates
area, and a touch screen keyboard area, wherein the text display
area, the suggestion candidates area, and the touch screen keyboard
area occupy distinct, non-overlapping areas of the touch screens,
and wherein the one or more touch screens are operable to receive
touch input over at least a portion of the touch screen keyboard
area and the suggestion candidates area; storage for storing the
computer-executable instructions for text entry and correction
using: a text input module for receiving text input using the touch
screen keyboard and displaying at least a portion of the text input
in the text display area; a candidate generation module for
generating one or more suggestion candidates comprising one or more
replacement suggestions and/or one or more next word suggestions
for an input word of the text input; an indicator generation module
for producing one or more indicators that can notify a user that
the suggestion candidates are available; a suggestion presentation
module for presenting the suggestion candidates associated with the
input word in the suggestion candidates area and receiving touch
screen user input for selecting one of the presented suggestions,
wherein: if the selected suggestion is a replacement suggestion,
replacing the input word with the selected suggestion; and if the
selected suggestion is a next word suggestion, adding the selected
suggestion as a next word subsequent to the input word in the text
input.
19. The mobile device of claim 18, wherein the indicator generation
module and the suggestion presentation module are configured to not
produce indicators and to not present suggestion candidates,
respectively, until one of the input words is designated incorrect
by a dictionary, a common speller application programming
interface, or an input history data source.
20. The mobile device of claim 18, wherein the indicator generation
module and the suggestion presentation module are configured to not
produce indicators and to not present suggestion candidates,
respectively, until a delimiter character is received immediately
after the input word.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application No. 61/304,341, filed Feb. 12, 2010, and entitled
"TYPING ASSISTANCE FOR EDITING," which is hereby incorporated
herein by reference in its entirety.
FIELD
[0002] The present disclosure pertains to devices and methods for
enhancing text entry using a touch screen device.
BACKGROUND
[0003] With the increasing popularity of mobile devices, including
cellphone devices, handheld devices, handheld computers,
smartphones, PDAs, etc., there is a need for improving the user
interface experience by increasing user text input speed, reducing
text entry errors, and improving the overall user experience.
[0004] Mobile devices with capacitive or resistive touch
capabilities often utilize a touch screen keyboard, a hardware
keyboard, speech recognition, handwriting recognition, or
combination of the four, for entry of text input. Touch screen
keyboards enable larger displays for videos, web pages, email,
etc., without the requirement of a physical keyboard. Because touch
screen keyboards are software-based, they can be easily adjusted
for different languages, screen orientation, and key layouts.
Furthermore, touch screen keyboards can be augmented with widgets
for word prediction and disambiguation candidates.
[0005] Users of devices with touch screens, especially mobile
devices, have varying abilities and styles of entering text. In
particular, some users prefer to type large chunks of text input
fairly rapidly, and do not to review and correct the entered text
until complete phrases, sentences, or complete messages have been
entered. Similarly, users entering text using speech recognition or
handwriting recognition do not want to stop to review their text
input until having entered completed phrases, sentences, or
complete messages. Predictive typing assistance software such as T9
only offer word prediction candidates as users type. After users
finish typing, they are usually left without any assistance, and
must then struggle to edit text by placing cursors in-between
characters in order to proof and correct text.
[0006] Therefore, there exists ample opportunity for improvement in
technologies related to facilitating user input on electronic
devices by providing more helpful and accurate assistance in the
text correction process in order to accelerate user text entry and
reduce user input error rates.
SUMMARY
[0007] An apparatus and method are disclosed for providing feedback
and guidance to touch screen device users to improve the text entry
user experience and performance.
[0008] One exemplary embodiment disclosed herein is a method
comprising receiving first input data comprising one or more input
words from a keyboard, providing one or more feedback semaphores,
wherein the feedback semaphores are operable to alert a user using
the keyboard that one or more suggestion candidates are available,
automatically displaying the one or more suggestion candidates,
wherein the suggestion candidates are each associated with an
alternative word for one or more of the input words, automatically
receiving a touch screen input selecting one of the suggestion
candidates, and modifying the first input data using the
alternative word associated with the selected suggestion candidate.
In some examples, one or more alternative words for one of the
input words are generated, the first input data is immediately
auto-corrected are providing the feedback semaphores by
automatically replacing the input word with one of the alternative
words. In some examples, the first input data comprises a character
designated as a delimiter and the providing feedback semaphores
occurs based on a word probability for the input word and receiving
the delimiter character. In some examples, the feedback semaphores
can include one or more of the following: playing a sound,
generating haptic feedback, highlighting one or more keys on the
keyboard, highlighting a background area of the keyboard, and
highlighting a suggestion candidates area. In some examples, the
feedback semaphores include highlighting a delimiter key and
receiving a single touch screen input for one of the highlighted
delimiter keys. In some examples, the feedback semaphores are not
provided until the input word is designated incorrect using a
candidate source. In some examples, the feedback semaphores
provided can be selected based on receiving typing speed data for
the first input data. In some examples, the feedback semaphores
provided can be selected based on detecting whether the user is
typing on the keyboard with one or two hands.
[0009] Another exemplary embodiment disclosed herein is a
computer-readable storage media storing computer-readable
instruction that when executed by a computer cause the computer to
perform a method comprising: receiving text input data comprising
at least one word using a keyboard coupled to a touch screen,
detecting that an event has occurred based on the text input data
and based on the detected event, automatically displaying on the
touch screen one or more suggestion candidates for the at least one
word based on the text input data and one or more candidate
sources, where each of the candidates is associated with at least
one next word designated as likely to follow the at least one word,
receiving a single touch screen input selecting one of the
suggestion candidates, and, based on the single touch screen input,
automatically modifying the text input data by adding the at least
one next word associated with the selected suggestion candidate to
the text input data. In some examples, the event is detected based
on a measured keystroke latency. In some examples, the event is an
auto-correction of the text input data. In some examples, the event
is receiving a delimiter key in the text input data. In some
examples, a method further comprises automatically displaying on
the touch screen one or more replacement suggestion candidates for
the at least one word based on the text input data and one or more
candidate sources, where each of the replacement suggestion
candidates is associated with at least one replacement word for the
at least one word, and the displaying includes displaying the
replacement suggestion candidates in a manner that distinguishes
the replacement suggestion candidates from the suggestion
candidates. In some examples, a method comprises immediately after
the automatically adding the at least one word, displaying one or
more suggestion candidates for the text input data, which includes
the selected next word, and repeating the automatically modifying
the text input data.
[0010] A further exemplary embodiment disclosed herein is a mobile
device comprising one or more processing units operable to execute
computer-executable instructions for text entry and correction, one
or more memory units coupled to the processing units, one or more
touch screens coupled to the mobile device configurable to have a
text display area, a suggestion candidates area, and a touch screen
keyboard area, wherein the text display area, the suggestion
candidates area, and the touch screen keyboard area occupy
distinct, non-overlapping areas of the touch screens, and wherein
the one or more touch screens are operable to receive touch input
over at least a portion of the touch screen keyboard area and the
suggestion candidates area. A mobile device further comprises
storage for storing the computer-executable instructions for text
entry and correction using a text input module for receiving text
input using the touch screen keyboard and displaying at least a
portion of the text input in the text display area a candidate
generation module for generating one or more suggestion candidates
comprising one or more replacement suggestions and/or one or more
next word suggestions for an input word of the text input an
indicator generation module for producing one or more indicators
that can notify a user that the suggestion candidates are
available, a suggestion presentation module for presenting the
suggestion candidates associated with the input word in the
suggestion candidates area and receiving touch screen user input
for selecting one of the presented suggestions wherein if the
selected suggestion is a replacement suggestion, replacing the
input word with the selected suggestion, and if the selected
suggestion is a next word suggestion, adding the selected
suggestion as a next word subsequent to the input word in the text
input. In some examples, the indicator generation module and the
suggestion presentation module are configured to not produce
indicators and to not present suggestion candidates, respectively,
until one of the input words is designated incorrect by a
dictionary, a common speller application programming interface, or
an input history data source. In some examples, the indicator
generation module and the suggestion presentation module are
configured to not produce indicators and to not present suggestion
candidates, respectively, until a delimiter character is received
immediately after the input word.
[0011] The described techniques and tools for solutions for
improving text entry user experience and performance can be
implemented separately, or in various combinations with each other.
As will be described more fully below, the described techniques and
tools can be implemented on hardware that includes software touch
screen keyboards or hardware keyboards. As will be readily apparent
to one of ordinary skill in the art, the disclosed technology can
be implemented using various platforms coupled with a touch screen
including, but not limited to, mobile devices (cellphones,
smartphones, PDAs, handheld devices, handheld computers, PDAs,
touch screen tablet devices), tablet or laptop computers, desktop
computers, and home theater systems. As used herein, a touch screen
includes a display coupled with touch sense capabilities (for
example, displays using capacitive or resistive sensors).
[0012] The foregoing and other objects, features, and advantages
will become more apparent from the following detailed description,
which proceeds with reference to the accompanying figures.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 is a system diagram depicting an exemplary mobile
device including a variety of optional hardware and software
components.
[0014] FIG. 2 illustrates a generalized example of a suitable
computing environment in which described embodiments, techniques,
and technologies may be implemented.
[0015] FIG. 3 illustrates a generalized example of a suitable
implementation environment for a text entry device connected to a
computing cloud.
[0016] FIG. 4 illustrates a generalized example of a text entry
device having a touch screen including a touch screen keyboard, a
suggestion candidates area, and a text entry area.
[0017] FIGS. 5A-5C illustrate generalized examples of text entry
devices, including a touch screen device with a touch screen
keyboard, a mobile device, and a personal computer.
[0018] FIG. 6 is a flow chart of an exemplary implementation of a
method for generating possible corrections for input data,
providing indicators including feedback semaphores that indicate
the availability of the possible corrections, and displaying
suggestion candidates.
[0019] FIGS. 7A-7E depict a method of providing unexpected-key
feedback, including providing feedback semaphores, when an
unexpected key is pressed.
[0020] FIGS. 7F-7H depict three alternative designs for providing
feedback semaphores.
[0021] FIG. 8 is a flow chart that outlines an exemplary
implementation of the method shown in FIGS. 7A-7E.
[0022] FIGS. 9A-9D depict an exemplary implementation of a method
of providing auto-correction notification, including providing
feedback semaphores, when auto-correction of text input data
occurs.
[0023] FIG. 10 is a flow chart that outlines an exemplary
implementation of the method shown in FIGS. 9A-9D.
[0024] FIGS. 11A-11C depict an exemplary implementation of a method
of providing likely next phrases, including using suggestion
candidates after receiving a text input word and a delimiter.
[0025] FIGS. 11D-11F depict an exemplary implementation of a method
of providing likely next words or phrases, including using
suggestion candidates after receiving a text input word and a
delimiter, and then providing an additional next word.
[0026] FIG. 12 is a flow chart that outlines an exemplary
implementation of the methods shown in FIGS. 11A-11C and
11D-11F.
[0027] FIG. 13 is a chart representing events that occur during an
exemplary implementation of a method that includes playing random
click sounds as a user presses keys on a touch screen keyboard, as
well as the use of keystroke latency to determine when to present
suggestion candidates.
[0028] FIG. 14 is a flow chart that outlines an exemplary
implementation of the methods shown in FIG. 13.
[0029] FIG. 15 illustrates a generalized example of a suitable
implementation environment including a computing cloud and various
connected devices in which described embodiments, techniques, and
technologies can be implemented.
[0030] FIGS. 16A-16D are charts depicting some experimental results
obtained using the disclosed technologies.
DETAILED DESCRIPTION
I. General Considerations
[0031] This disclosure is set forth in the context of
representative embodiments that are not intended to be limiting in
any way.
[0032] As used in this application and in the claims, the singular
forms "a," "an," and "the" include the plural forms unless the
context clearly dictates otherwise. Additionally, the term
"includes" means "comprises." Further, the term "coupled"
encompasses mechanical, electrical, as well as other practical ways
of coupling or linking items together, and does not exclude the
presence of intermediate elements between the coupled items.
[0033] The described things and methods described herein should not
be construed as being limiting in any way. Instead, this disclosure
is directed toward all novel and non-obvious features and aspects
of the various disclosed embodiments, alone and in various
combinations and sub-combinations with one another. The disclosed
systems, methods, and apparatus are not limited to any specific
aspect or feature or combinations thereof, nor do the disclosed
things and methods require that any one or more specific advantages
be present or problems be solved.
[0034] Although the operations of some of the disclosed methods are
described in a particular, sequential order for convenient
presentation, it should be understood that this manner of
description encompasses rearrangement, unless a particular ordering
is required by specific language set forth below. For example,
operations described sequentially may in some cases be rearranged
or performed concurrently. Moreover, for the sake of simplicity,
the attached figures may not show the various ways in which the
disclosed things and methods can be used in conjunction with other
things and methods. Additionally, the description sometimes uses
terms like "produce," "generate," "select," "highlight," and
"provide" to describe the disclosed methods. These terms are
high-level abstractions of the actual operations that are
performed. The actual operations that correspond to these terms
will vary depending on the particular implementation and are
readily discernible by one of ordinary skill in the art.
[0035] Theories of operation, scientific principles or other
theoretical descriptions presented herein in reference to the
apparatus or methods of this disclosure have been provided for the
purposes of better understanding and are not intended to be
limiting in scope. The apparatus and methods in the appended claims
are not limited to those apparatus and methods that function in the
manner described by such theories of operation.
[0036] In the following description, certain terms may be used such
as "up," "down," "upper," "lower," "horizontal," "vertical,"
"left," "right," "over," "on," "near," and the like. These terms
are used, where applicable, to provide some clarity of description
when dealing with relative relationships. But, these terms are not
intended to imply absolute relationships, positions, and/or
orientations.
[0037] As used in this disclosure, the term "wait" may be used to
describe the action a device takes while waiting for particular
value or type of input before proceeding with a particular
operation. This waiting should not be construed as limiting the
device to only waiting for the particular type of input, rather,
the device may receive other input or perform other actions
concurrently with the waiting.
[0038] As used in this disclosure, the term "automatically" is used
to describe actions that can proceed immediately, without receiving
further user input. As used in this disclosure, the term
"immediately" means that an action occurs within a short time
period following a preceding action without needing to receive
intervening user input. In some cases, there may be intervening
actions performed between or concurrently with the preceding action
and the action occurring "immediately," for example, screen refresh
or redraw, sound playback, etc.
[0039] As used in this disclosure, the term "incorrect" is used to
describe a designation of a word or phrase as being incorrect. A
word designated as incorrect can be automatically highlighted or
auto-corrected, even though the word designated as incorrect by a
correction module might actually be considered to be correct by the
user. For example, a word can be designated as incorrect because it
does not exist in a dictionary, CSAPI (common speller application
programming interface), or IHDS (input history data source).
Alternatively, a word can be designated as "incorrect" even though
it exists in a dictionary, CSAPI, or IHDS, because of other
checking rules implemented in a correction module or candidate
generation module, or because of the context of the word within a
phrase.
[0040] As used in this disclosure, the term "indicator" is used to
describe output that is intended to capture the attention of a user
of a text entry device, and can include visual, audio, and haptic
feedback. Some examples techniques for providing visual indicators
include highlighting an object on a display by changing the color
of the object, shading of the object or background of an object,
changing the color of one or more elements of an object (e.g., by
coloring the letter "F" on the "F" key of a touch screen keyboard),
flashing the display, or by drawing a new object (e.g., using
underlines, balloons, or suggestion candidates) on the display.
Some examples of audio indicators include playing back a
pre-recorded or synthesized sound, including a randomly selected
sound. Some examples of haptic indicators include vibrating a
handheld device or area of a touch screen keyboard, as well as
increasing or decreasing the amount of pressure sufficient to
register a key press on a keyboard, including touch screen
keyboards. An example of varying key press resistance to prevent
typing errors may be found in Hoffmann, et al., "TypeRight: A
Keyboard with Tactile Error Prevention," Proceedings of the 27th
Int'l Conf. on Human Factors in Computing Systems, pp. 2265-2268
(2009).
[0041] As used in this disclosure, the term "semaphore" is used to
describe the use of one or more indicators to provide feedback
and/or guidance signals on a text entry device user to help
discover and avoid text entry errors. Generally speaking, "feedback
semaphore" is used to describe an indicator or signal that is
provided after an event is received (for example, an event could be
an auto-correction or other action that a text entry device
performs, or the receiving of a key press or touch screen touch
input). In contrast, "guidance signal" is used to describe an
indicator that is provided before the occurrence of an event. For
example, a text entry device can predict which key a user is likely
to press next and highlight the most likely key, thereby providing
a guidance signal, so the user can more easily find the most likely
key.
[0042] As used in this disclosure, the term "over" is used to
describe the positioning of one or more objects (for example, a
finger, thumb, or stylus) over, on, or near a location on a touch
screen. In some embodiments, this object need not come into contact
with the touch screen for the object's position to be determined.
In other embodiments, the object described as "over" the touch
screen may be in contact with the surface of the touch screen. In
some embodiments, the object determined to be "over" a location of
the touch screen may not actually be positioned directly over the
touch screen location, but determined to be "over" the location on
the touch screen, for example, by a position correction module of
the text entry device or touch screen.
[0043] The disclosed technology includes various approaches to
improving typing accuracy or typing speed when using devices having
a touch screen by using suggestion candidates to augment other
input devices. These suggestion candidates are typically
represented in a candidates area, which need not be permanently
reserved for that purpose, or can appear in varying location on the
touch screen. After entering one or more words to form a text
entry, the user can review the text entry by viewing the touch
screen and deciding whether to select word(s) for "suggestions."
Although some examples disclosed herein describe "a word" or "a
selected word," it should be understood that in some examples,
selecting a word can include but is not limited to selecting a
single word with a single touch screen input, selecting multiple
words of a phrase with a single touch screen input, or selecting
multiple words of a phrase using touch screen input comprising
plural single touch screen inputs. For example, auto-correction or
unexpected-key feedback can be generated for a single word, or for
a phrase comprising multiple words and spaces, but that are related
in some way.
[0044] In some examples, after an indicator is provided, one or
more suggestion candidates are displayed on a touch screen display.
The suggestion candidates can be presented as "buttons" which
include a word related to the word selected by the user. Suggestion
candidates can be determined to be related to the selected word
using a candidate generation module, which can use a dictionary,
thesaurus, common speller application programming interface, input
history data source, or other sources or methods to generate
suggestion candidates. The candidate generation module can also
determine the rank order in which suggestion candidates are
presented. For example, the suggestion candidates can be presented
from left to right, with the suggestion candidate determined to be
the most likely presented furthest to the left, and the least
likely suggestion candidate presented furthest to the right. The
user reviews the suggestion candidates, and selects one of the
candidates for replacement using a single touch screen input over
the desired suggestion candidate on the touch screen.
[0045] As used in this disclosure, a single touch screen input
refers to the input received when a user positions an object over
the surface of a touch screen such that the touch screen device can
determine the position of the object. In some embodiments, the
object can be the user's finger or thumb. In other embodiments, the
object can be a stylus or puck. In some embodiments, the single
touch screen input is received after the user "taps" the touch
screen over a word or suggestion candidates. In other embodiments,
the single touch screen input is received when the user presses the
screen with a finger, thumb, or stylus. Receiving a single touch
screen input is sufficient to determine which suggestion candidate
the user is indicating on the touch screen--no additional keyboard
input, mouse input, trackball input, voice input, or additional
touches are necessary. Using a single touch screen input to
determine user selections simplifies the input process and allows
for the fast correction of text entries without the need to use
submenus, popup menus, or additional input devices.
II. Example Mobile Device
[0046] FIG. 1 is a system diagram depicting an exemplary mobile
device 100 including a variety of optional hardware and software
components, shown generally at 102. Any components 102 in the
mobile device can communicate with any other component, although
not all connections are shown, for ease of illustration. The mobile
device can be any of a variety of computing devices (e.g., cell
phone, smartphone, handheld computer, Personal Digital Assistant
(PDA), etc.) and can allow wireless two-way communications with one
or more mobile communications networks 104, such as a cellular or
satellite network.
[0047] The illustrated mobile device 100 can include a controller
or processor 110 (e.g., signal processor, microprocessor, ASIC, or
other control and processing logic circuitry) for performing such
tasks as signal coding, data processing, input/output processing,
power control, and/or other functions. An operating system 112 can
control the allocation and usage of the components 102 and support
for one or more application programs 114. The application programs
can include common mobile computing applications (e.g., email
applications, calendars, contact managers, web browsers, messaging
applications), or any other computing application.
[0048] The illustrated mobile device 100 can include memory 120.
Memory 120 can include non-removable memory 122 and/or removable
memory 124. The non-removable memory 122 can include RAM, ROM,
flash memory, a hard disk, or other well-known memory storage
technologies. The removable memory 124 can include flash memory or
a Subscriber Identity Module (SIM) card, which is well known in GSM
communication systems, or other well-known memory storage
technologies, such as "smart cards." The memory 120 can be used for
storing data and/or code for running the operating system 112 and
the application programs 114. Example data can include web pages,
text, images, sound files, video data, or other data sets to be
sent to and/or received from one or more network servers or other
devices via one or more wired or wireless networks. The memory 120
can be used to store a subscriber identifier, such as an
International Mobile Subscriber Identity (IMSI), and an equipment
identifier, such as an International Mobile Equipment Identifier
(IMEI). Such identifiers can be transmitted to a network server to
identify users and equipment.
[0049] The memory 120 can also be used for the candidate sources
116, which are used for generating and suppressing auto-corrections
and generation suggestion candidates. Candidate sources 116 can
include but are not limited to: a system dictionary, a user
dictionary, a common speller application programming interface
(CSAPI), touch models, and an input history data source.
[0050] The mobile device 100 can support one or more input devices
130, such as a touch screen 132, microphone 134, camera 136,
physical keyboard 138, and/or trackball 140 and one or more output
devices 150, such as a speaker 152 and a display 154. Other
possible output devices can include haptic output devices such as a
piezoelectric transducer 156 or other suitable device. Some devices
can serve more than one input/output function. For example, touch
screen 132 and display 154 can be combined in a single input/output
device.
[0051] A wireless modem 160 can be coupled to an antenna (not
shown) and can support two-way communications between the processor
110 and external devices, as is well understood in the art. The
modem 160 is shown generically and can include a cellular modem for
communicating with the mobile communication network 104 and/or
other radio-based modems (e.g., Bluetooth 164 or Wi-Fi 162). The
wireless modem 160 is typically configured for communication with
one or more cellular networks, such as a GSM network for data and
voice communications within a single cellular network, between
cellular networks, or between the mobile device and a public
switched telephone network (PSTN).
[0052] The mobile device can further include at least one
input/output port 180, a power supply 182, a satellite navigation
system receiver 184, such as a Global Positioning System (GPS)
receiver, an accelerometer 186, and/or a physical connector 190,
which can be a USB port, IEEE 1394 (FireWire) port, and/or RS-232
port. The illustrated components 102 are not required or
all-inclusive, as any components can deleted and other components
can be added.
III. Example Computing Environment
[0053] FIG. 2 illustrates a generalized example of a suitable
computing environment 200 in which described embodiments,
techniques, and technologies may be implemented. For example, the
computing environment 200 can implement unexpected-key feedback,
auto-correction notification, invoking likely next phrases, random
playback of click sounds, and waiting for a keystroke latency
before presenting suggestion candidates, as described below.
[0054] The computing environment 200 is not intended to suggest any
limitation as to scope of use or functionality of the technology,
as the technology may be implemented in diverse general-purpose or
special-purpose computing environments. For example, the disclosed
technology may be implemented with other computer system
configurations, including hand held devices, multiprocessor
systems, microprocessor-based or programmable consumer electronics,
network PCs, minicomputers, mainframe computers, and the like. The
disclosed technology may also be practiced in distributed computing
environments where tasks are performed by remote processing devices
that are linked through a communications network. In a distributed
computing environment, program modules may be located in both local
and remote memory storage devices.
[0055] With reference to FIG. 2, the computing environment 200
includes at least one central processing unit 210 and memory 220.
In FIG. 2, this most basic configuration 230 is included within a
dashed line. The central processing unit 210 executes
computer-executable instructions and may be a real or a virtual
processor. In a multi-processing system, multiple processing units
execute computer-executable instructions to increase processing
power and as such, multiple processors can be running
simultaneously. The memory 220 may be volatile memory (e.g.,
registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM,
flash memory, etc.), or some combination of the two. The memory 220
stores software 280 and candidate sources 285 that can, for
example, implement the technologies described herein. A computing
environment may have additional features. For example, the
computing environment 200 includes storage 240, one or more input
devices 250, one or more output devices 260, one or more
communication connections 270, and one or more touch screens 290.
An interconnection mechanism (not shown) such as a bus, a
controller, or a network, interconnects the components of the
computing environment 200. Typically, operating system software
(not shown) provides an operating environment for other software
executing in the computing environment 200, and coordinates
activities of the components of the computing environment 200.
[0056] The storage 240 may be removable or non-removable, and
includes magnetic disks, magnetic tapes or cassettes, CD-ROMs,
CD-RWs, DVDs, or any other medium which can be used to store
information and that can be accessed within the computing
environment 200. The storage 240 stores instructions for the
software 280 and candidate sources 285, which can implement
technologies described herein.
[0057] The input device(s) 250 may be a touch input device, such as
a keyboard, keypad, mouse, pen, or trackball, a voice input device,
a scanning device, or another device, that provides input to the
computing environment 200. For audio, the input device(s) 250 may
be a sound card or similar device that accepts audio input in
analog or digital form, or a CD-ROM reader that provides audio
samples to the computing environment 200. The output device(s) 260
may be a display, printer, speaker, CD-writer, or another device
that provides output from the computing environment 200. The touch
screen 290 can act as an input device (receiving touch screen
input) and as an output device (displaying the text entry area,
suggestion candidates area, and/or touch keyboard).
[0058] The communication connection(s) 270 enable communication
over a communication medium (e.g., a connecting network) to another
computing entity. The communication medium conveys information such
as computer-executable instructions, compressed graphics
information, or other data in a modulated data signal.
[0059] Computer-readable media are any available media that can be
accessed within a computing environment 200. By way of example, and
not limitation, with the computing environment 200,
computer-readable media include memory 220, storage 240,
communication media (not shown), and combinations of any of the
above.
[0060] Computer-readable media are any available media that can be
accessed within a computing environment 200. By way of example, and
not limitation, with the computing environment 200,
computer-readable media include memory 220 and/or storage 240. As
should be readily understood, the term computer-readable storage
media includes the media for data storage such as memory 220 and
storage 240, and not transmission media such as modulated data
signals.
IV. Example Text Entry Device
[0061] FIG. 3 illustrates a generalized example of a suitable
implementation environment 300 of a text entry device 305 connected
to a computing cloud 325. The text entry device 305 includes
several modules stored on a computer-readable storage medium 310,
including a text input module 330 for receiving text entry input,
touch input module 332 for receiving touch screen input from a
touch screen (not shown), and an output module 334 for providing
output to a touch screen. The communication module 320 adapts the
text entry device 305 so that it can communicate with service
providers located in the cloud 325. The computer-readable storage
medium 310 also includes an indicator generation module 340 for
generating feedback semaphores and other indicators, a correction
module 342 for checking and correcting text entries, and a
candidate generation module 344 for generating suggestion
candidates. Indicator generation module 340, correction module 342,
and candidate generation module 344 can communicate with multiple
modules to determine correction and suggestion candidates,
including a grammar checking module 350, a system dictionary module
351, a user dictionary module 352, a CSAPI (Common Speller API)
module 353, and an IHDS (input history data source) module 354. In
some embodiments, one or all of these source modules 350-354 can be
provided by a service provider in an alternate location 380 in the
cloud 325.
[0062] FIG. 15 illustrates a generalized example of a suitable
implementation environment 1500 in which described embodiments,
techniques, and technologies may be implemented.
[0063] In example environment 1500, various types of services
(e.g., computing services) are provided by a computing cloud 1510.
For example, the cloud 1510 can comprise a collection of computing
devices, which may be located centrally or distributed, that
provide cloud-based services to various types of users and devices
connected via a network such as the Internet. The implementation
environment 1500 can be used in different ways to accomplish
computing tasks. For example, some tasks (e.g., processing user
input and presenting a user interface) can be performed on local
computing devices (e.g., connected devices 1530-1532) while other
tasks (e.g., storage of data to be used in subsequent processing,
including candidate sources) can be performed in the cloud
1510.
[0064] In example environment 1500, the cloud 1510 provides
services for connected devices 1530-1532 with a variety of screen
capabilities. Connected device 1530 represents a device with a
computer screen (e.g., a mid-size screen). For example, connected
device 1530 could be a personal computer such as desktop computer,
laptop, notebook, netbook, or the like. Connected device 1531
represents a device with a mobile device screen (e.g., a small size
screen). For example, connected device 1531 could be a mobile
phone, smart phone, personal digital assistant, tablet computer,
and the like. Connected device 1532 represents a device with a
large screen. For example, connected device 1532 could be a
television screen (e.g., a smart television) or another device
connected to a television (e.g., a set-top box or gaming console)
or the like. One or more of the connected devices 1530-1532 can
include touch screen capabilities. Devices without screen
capabilities also can be used in example environment 1500. For
example, the cloud 1510 can provide services for one or more
computers (e.g., server computers) without displays.
[0065] Services can be provided by the cloud 1510 through service
providers 1520, or through other providers of online services (not
depicted). For example, cloud services can be customized to the
screen size, display capability, and/or touch screen capability of
a particular connected device (e.g., connected devices
1530-1532).
[0066] In example environment 1500, the cloud 1510 provides the
technologies and solutions described herein to the various
connected devices 1530-1532 using, at least in part, the service
providers 1520. For example, the service providers 1520 can provide
a centralized solution for various cloud-based services. The
service providers 1520 can manage service subscriptions for users
and/or devices (e.g., for the connected devices 1530-1532 and/or
their respective users).
[0067] Although the operations of some of the disclosed methods are
described in a particular, sequential order for convenient
presentation, it should be understood that this manner of
description encompasses rearrangement, unless a particular ordering
is required by specific language set forth below. For example,
operations described sequentially may in some cases be rearranged
or performed concurrently. Moreover, for the sake of simplicity,
the attached figures may not show the various ways in which the
disclosed methods can be used in conjunction with other
methods.
[0068] Any of the disclosed methods can be implemented as
computer-executable instructions stored on one or more
computer-readable media (e.g., non-transitory computer-readable
media, such as one or more optical media discs, volatile memory
components (such as DRAM or SRAM), or nonvolatile memory components
(such as hard drives)) and executed on a computer (e.g., any
commercially available computer, including smart phones or other
mobile devices that include computing hardware). Any of the
computer-executable instructions for implementing the disclosed
techniques as well as any data created and used during
implementation of the disclosed embodiments can be stored on one or
more computer-readable media (e.g., non-transitory
computer-readable media). The computer-executable instructions can
be part of, for example, a dedicated software application or a
software application that is accessed or downloaded via a web
browser or other software application (such as a remote computing
application). Such software can be executed, for example, on a
single local computer (e.g., any suitable commercially available
computer) or in a network environment (e.g., via the Internet, a
wide-area network, a local-area network, a client-server network
(such as a cloud computing network), or other such network) using
one or more network computers.
[0069] For clarity, only certain selected aspects of the
software-based implementations are described. Other details that
are well known in the art are omitted. For example, it should be
understood that the disclosed technology is not limited to any
specific computer language or program. For instance, the disclosed
technology can be implemented by software written in C++, Java,
Perl, JavaScript, Adobe Flash, or any other suitable programming
language. Likewise, the disclosed technology is not limited to any
particular computer or type of hardware. Certain details of
suitable computers and hardware are well known and need not be set
forth in detail in this disclosure.
[0070] Furthermore, any of the software-based embodiments
(comprising, for example, computer-executable instructions for
causing a computer to perform any of the disclosed methods) can be
uploaded, downloaded, or remotely accessed through a suitable
communication means. Such suitable communication means include, for
example, the Internet, the World Wide Web, an intranet, software
applications, cable (including fiber optic cable), magnetic
communications, electromagnetic communications (including RF,
microwave, and infrared communications), electronic communications,
or other such communication means.
[0071] The disclosed methods, apparatus, and systems should not be
construed as limiting in any way. Instead, the present disclosure
is directed toward all novel and nonobvious features and aspects of
the various disclosed embodiments, alone and in various
combinations and subcombinations with one another. The disclosed
methods, apparatus, and systems are not limited to any specific
aspect or feature or combination thereof, nor do the disclosed
embodiments require that any one or more specific advantages be
present or problems be solved.
V. Example Touch Screen Text Entry Device
[0072] FIG. 4 depicts an exemplary embodiment 400 of a text entry
device 401 having a touch screen 402. The touch screen 402 includes
a display area for a touch screen keyboard 420, a suggestion
candidates area 450, and a text display area 403. The text display
area 403 is shown displaying a phrase "smsing while dr" 404. In
some embodiments, only a portion of the text input is displayed
because of, for example, screen size limitations. A carat (cursor)
406 is also shown in the text entry area 403. In some embodiments,
the carat 406 can be placed at various positions in the text entry
area using a single touch screen input. In some embodiments, the
touch screen 402 has a substantially planar surface, and the
display capability can be implemented using LED, LCD, electronic
ink, DLP, Plasma, CRT, or other suitable display technology.
[0073] The text entry device 401 has a touch screen 402 that
displays a touch screen keyboard 420 having several keys 424, 426,
428, 430, 432, 434, etc. Some of the keys, including the backspace
key 430, return key 432, and space key 434 are also designated as
delimiter keys. As shown, the touch screen keyboard displays the
keys 424, 426, 428, 430, 432, 434, etc. as images on the touch
screen 402, as the user's finger 442 is touching the key "s" 428.
The touch screen can include capacitive, resistive, inductive, or
other suitable technologies for determining the position of one or
more touch inputs detected over the surface of the keyboard and
converting this touch input into text input. In some embodiments,
the touch input is created using a stylus or puck, while in other
embodiments the touch input can be created using a finger or thumb.
In other embodiments, the touch screen keyboard 420 can be
implemented as a hardware keyboard including mechanical keys.
[0074] The suggestion candidates area 450 is depicted in FIG. 4
showing three suggestion candidates, including "drive" 455, "drug,"
"draft," "drop," and "driving." As shown, the suggestion candidates
area 450 includes widgets directly above the keyboard area
containing word prediction and disambiguation candidates. Placing
the suggestion candidates area 450 close to the touch screen
keyboard 420 can improve typing speed and reduce errors by allowing
users to maintain their focus of attention near the keyboard area
while correcting a phrase 410 in the text entry area 403. In other
embodiments, the suggestion candidates area 450 can be placed
nearby the phrase 404 in the text entry area 403.
[0075] The location of the text entry area 403, the keyboard 420,
etc., can be varied based on the particular implementation and
design.
[0076] FIGS. 5A-5C illustrate additional exemplary implementations
500, 530, 560 of the disclosed technology. FIG. 5A depicts an
exemplary implementation 500 of a text entry device, which includes
a touch screen 506 displaying a window 510, including a designated
text entry area displaying the phrase "smsing while driving" 512.
The touch screen 506 is also displaying a candidates area 520, and
a touch screen keyboard 524. A finger 522 is shown selecting the
candidate "driving" in the candidates area 520.
[0077] FIG. 5B depicts an exemplary implementation 530, which
includes a mobile device 531 having a touch screen 532 with a text
entry area 534 (displaying the phrase "smsing while driving" 536),
a suggestion candidates area 542, and a hardware keyboard 540.
[0078] FIG. 5C depicts an exemplary implementation 570 of a text
entry device, which includes a personal computer (PC) device having
a display screen 571 and a keyboard 580. The display screen 571 is
shown displaying the phrase "smsing while dr" 575 in a window 574.
Also shown is a suggestion candidates area 586 displaying
candidates "drive," "draft," "drop," and "driving." In some
examples, the text entry device is not limited to receiving text
input using a touch screen keyboard, but can also use hardware
keyboards, handwriting recognition, or speech recognition to
receive text input.
[0079] Methods and apparatus for performing handwriting recognition
can include but are not limited to those based on: Bayesian
networks, neural nets, hidden Markov models, or k-nearest-neighbor
approaches. Methods and apparatus for performing speech recognition
can include but are not limited to those based on a dynamic time
warping approach or hidden Markov models.
VI. Example Indicators, Signals, and Feedback Semaphores
[0080] Feedback and guidance signals are examples of indicators
that can be implemented in many ways. For example, mobile tactile
feedback, such as piezoelectric tactile displays that responds to a
stylus, or a stylus that produces tactile feedback, can be
employed. For touch screen keyboards in particular, artificial
tactile feedback via vibration actuators can improve text entry
performance in both static environments and mobile environments
(for example, typing in a moving vehicle such as an automobile or
train). In some examples, keyboard events can be defined including
a fingertip-over event to signal when a fingertip has touched a
key, a fingertip-click event to signal that a key has been
registered, a fingertip-slip event to signal when the fingertip
moved over the edge of a key, and a home-key event for indicating
where the home keys are located on the touch screen keyboard. For
example, in an English-language keyboard, the home keys of "F" and
"J," one for each finger, can be associated with haptic feedback
that is provided to indicate a user's fingers are over the home
keys. Using these events can improve tactile touch screen keyboard
accuracies obtained to close to those obtained using a physical
keyboard. Examples of using haptic feedback with a touch screen
keyboard may be found in: Brewster, et al., "Tactile feedback for
mobile interactions," Proc. of SIGCHI Conf. on Human Factors in
Computing Systems, pp. 159-162. (2007); Brown & Brewster,
"Multidimensional Tactons for Non-visual Information Display in
Mobile Devices," Proc. of MobileHCI Conf. 2004, pp. 231-238 (2004);
and Hoggan et al., "Investigating the Effectiveness of Tactile
Feedback for Mobile Touchscreens," Proc. of the 26th Int'l Conf. on
Human Factors in Computing Systems, pp. 1573-1582 (2008).
[0081] Although employing different keyboard layouts can assist
text entry on touch screen keyboards, these layouts have not yet
become widespread. It is desirable to create semaphores for a touch
screen keyboard that can be marketed widely.
[0082] In addition or instead of semaphores created using tactile
feedback, similar results can be achieved by providing visual and
auditory cues with a standard touch screen keyboard. While some
users complain that their fingertips obfuscate visual feedback
provided using a soft keyboard, visual and auditory feedback for
the three keyboard events discussed above can be employed. Some
examples of visual/audio feedback includes: a tooltip balloon that
appears and disappears at the fingertip-over and fingertip-slip
events, respectively, and an audible click is played at
fingertip-click events.
[0083] Obstacles to providing effective visual and auditory cues in
mobile settings include small screen size, outside noise, social
restrictions and other circumstantial demands. Moreover, it has not
been empirically established that artificial tactile feedback alone
can impart enough information to realize full parity with physical
keyboards. Furthermore, some users find tactile feedback to be
annoying. Because individual differences may account for users'
preference of one modality over another, touch screen keyboards
employing a variety of tactile, visual and auditory semaphores, and
combinations thereof, can be used.
[0084] Certain examples of the disclosed technology include touch
screen keyboards using "typing semaphores." As described further
below, typing semaphores can provide multimodal feedback and/or
guidance signals that help users discover and avoid typing errors.
Multimodal feedback refers to the use of visual, auditory, and/or
haptic (touch output) communication that are received be a text
entry device user to indicate information such as potential errors,
the occurrence of events such as auto-correction, or the
availability of suggestion candidates.
[0085] A conventional semaphore is a system for conveying
information by means of hand-held flags or lights. Widely adopted
by the maritime world in the 1800s, semaphores are still employed
today to safely guide airplanes onto naval vessels. Typing
semaphores can be viewed as multimodal signals that guide users to
enter text correctly--by analogy, a kind of flight control for
typing. State of the art touch screen keyboards make it too easy
for users to "crash," expecting them to edit their text thereafter,
which can be mentally disruptive, time-consuming, and taxing.
[0086] The purpose of typing semaphores is to improve text entry
user experience and performance as users type (e.g., in real-time)
or otherwise provide input data using both feedback signals to
alert user to possible typing errors when they occur, and guidance
signals to prevent future errors.
VII. Example Application Using Indicators and Suggestion
Candidates
[0087] FIG. 6 is a flow chart 600 that outlines a method of
generating possible corrections for input data, providing
indicators, including feedback semaphores that indicate the
availability of the possible corrections, and displaying suggestion
candidates. After displaying the suggestion candidates, an input is
received selecting one of the suggestion candidates, and the input
data is modified with one or more words associated with the
selected suggestion candidate.
[0088] At process block 610, input data comprising one or more
input word(s) is received from a source such as a touch screen
keyboard. In some embodiments, the first input data includes text
characters, text words, position data for key presses on a touch
screen keyboard, typing speed data, correction data, and/or touch
screen orientation data.
[0089] At process block 620, one or more feedback semaphores are
provided to indicate that one or more suggestion candidates are
available for one or more of the input words. The feedback
semaphores can include various indicators or signals to alert a
user while maintaining focus in the keyboard area by, for example,
highlighting one or more keys of a keyboard, playing back a sound
using a speaker, generating haptic feedback (e.g., vibrating a
handheld device), or displaying a tooltip balloon.
[0090] At process block 630, one or more suggestion candidates are
provided to a user (e.g., using a touch screen display). Each of
the suggestion candidates is associated with an alternative word
for one or more of the input words. In some examples, the
suggestion candidates are associated with alternative words for
only one of the input words, while in other examples, suggestion
candidates are associated with alternative words for more than one
of the input words. An exemplary display of a suggestion candidate
includes displaying a button using a touch screen display, where
the button includes the text of an associated alternative word
within the boundary of the button.
[0091] At process block 640, an input selecting one of the
suggestion candidates is received. For example, the selection can
received using a single touch screen input created by a user
pressing a finger over a button associated with the desired touch
screen candidate on a touch screen display.
[0092] At process block 650, the input data is modified using the
alternative word associated with the selected suggestion candidate.
In some examples, the alternative word is used to replace the input
word in the input data. In other examples, the alternative word is
added to the input data preceding or subsequent to the input word.
In some examples, the "alternative" word is a word that was
previously auto-corrected, and the alternative word is therefore
used to effectively undo a word inserted in the input data using an
auto-correction routine. Thus, as described above, a quick and
effective way of providing suggestion candidates and indicators of
their availability is provided that allows users to quickly modify
input data using the suggestion candidates without moving their
focus of attention away from a touch screen keyboard.
VIII. Example Unexpected Key Feedback
[0093] Because touch screen keyboards lack tactile feedback, it is
difficult for users to tell when they hit their desired keys.
Consequently, they have to pay more attention to the keyboard area
than when using hardware keyboards. FIGS. 7A-7E depict a method 700
for providing visual, audio, and tactile feedback to a user that
has pressed an expected key on a mobile device 702. FIG. 8 is a
flow chart 800 corresponding to the method 700 depicted in FIGS.
7A-7E.
[0094] As shown in FIG. 7A, a mobile device 702 comprises a text
display area 704, suggestion candidates area 708, and touch screen
keyboard 710. The text display area 704 is shown displaying a
phrase "smsing whil" 706, with a carat (cursor) 712 immediately
after the phrase as the user enters it on the keyboard 710.
[0095] FIG. 7B depicts the mobile device 702 providing the user
with unexpected key feedback semaphores as an unexpected key, "r"
726, is pressed with a user's finger 722. The types of feedback
semaphores shown include: a tooltip balloon 728 displaying the
unexpected key, an audio "clunk clunk" sound 740 (played back over
a speaker coupled to the mobile device), haptic feedback (e.g., a
vibration created by an oscillating transducer coupled to the
mobile device) indicated by squiggle lines 755 and 756, suggestion
candidates "whip" 730, "whilst" 731, and "while" 732, and a
highlighted key indicator shown as shading of the unexpected "r"
key 726. The unexpected "r" key 726 is designated as unexpected
based on the current context of the text input data shown on the
text display area 704 (e.g., the word "whilr" does not exist in the
English language). Here, the suggestion candidates 730-732 are
actually based on the data input phrase "smsing whil" 706 before
the "r" was received, but as shown in FIG. 7C, will be updated
based on the unexpected key input.
[0096] FIG. 7C depicts the mobile device 702 shortly after
providing one or more feedback semaphores as discussed above. As
shown, the phrase 742 has been updated with the unexpected key "r,"
and the carat 712 is placed immediately after the phrase. Further,
suggestion candidates "while" 750, "white" 751, and "wholesale" 752
have been provided based on the phrase 742. In addition, designated
delimiter keys including a space bar 758 and an enter key 759 have
been highlighted to indicate that the highlighted suggestion
candidate will replace the word "whilr" if one of the delimiter
keys is pressed next. Alternatively, the user can press one of the
suggestion candidates 750-752 in order to select the corresponding
word shown on the candidate button to replace the word "whilr."
[0097] FIG. 7D depicts the mobile device 702 as a user uses a
finger 723 to provide a single touch screen input over a delimiter
key (the space bar 758).
[0098] FIG. 7E depicts the mobile device 702 after receiving the
delimiter key. As shown, the phrase 782 has been updated to replace
the word "whilr" with the word "while," which was the word
associated with the highlighted suggestion candidate 750. The carat
712 has also been advanced one space past the phrase 782.
[0099] The manner of providing feedback semaphores used can be
selected based on their effectiveness in alerting users without
annoying them. For example, when tooltip balloons (e.g., tooltip
balloon 728) appear above the keys of the keyboard 710, users do
not always see these visual cues, especially if they are typing
quickly. It is desirable to use feedback semaphores that can alert
users to unexpected key presses so that they can keep their focus
on the keyboard 710, or immediately switch their focus of attention
to the candidates area 708. A key is designated "unexpected" when
the letters entered so far (e.g., up to the previous word boundary)
do not match the prefix of any word in a candidate source (e.g., a
dictionary, CSAPI, or input history data source).
[0100] FIGS. 7F-7H depict different designs 760, 770, and 780 of
highlighting a touch screen that can be used to provide
unexpected-key feedback semaphores to a user. In FIGS. 7F-7H, a
mobile device 702 is shown as the user types the "r" key after
entering the phrase "smsing whil" 762. As shown in the design 760
of FIG. 7F, a shaded tooltip balloon 766 and both the background
and keys of the touch screen keyboard 764 are highlighted to
provide an unexpected key feedback semaphore. Similarly, the design
770 of FIG. 7G depicts the mobile device 702, but with a shaded
tooltip balloon 776 and only the background of the touch screen
keyboard 774 highlighted. Finally, the design 780 of FIG. 7H
depicts the mobile device 702, but with a shaded tooltip balloon
786 and only the border of the background of the touch screen
keyboard 784 highlighted. The highlighting and shading shown in
FIGS. 7A-7H can be based on using a pattern and/or color(s), and
can slowly fade back to the original key colors over time to
present a more pleasing visual appearance.
[0101] Usability studies have indicated that some users found the
design 760 shown in FIG. 7F, including a shaded keyboard 764,
shaded "r" key, and shaded tooltip balloon 766, "too disruptive."
Although the design 770 of FIG. 7G, including shading for only the
background of the keyboard 774, shaded "r" key, and shaded tooltip
balloon 776, was found to be less disruptive, users still found
this design more distracting than the design shown in FIG. 7H,
which includes only a shaded border around the keyboard 784, shaded
"r" key, and shaded tooltip balloon 786. Thus, by performing
usability studies, desirable combinations of visual and auditory
feedback that are agreeable to users can be determined.
[0102] FIG. 8 is a flow chart 800 further detailing the method 700
shown in FIGS. 7A-7E. FIG. 8 depicts process blocks for receiving
input data, evaluating the data using one or more candidate sources
to determine unexpected input, and based on the evaluating,
displaying feedback semaphores and suggestion candidates allowing a
word to be replaced with a selected suggestion candidate.
[0103] At process block 810, input data is received from a source
such as a touch screen keyboard. In some embodiments, the first
input data includes text characters, text words, position data for
key presses on a touch screen keyboard, typing speed data,
correction data, and/or touch screen orientation data.
[0104] At process block 820, the input data is analyzed and
compared against one or more of the following candidate sources: a
system dictionary; a user dictionary; a common speller application
programming interface (CSAPI); or an input history data source that
can include previously generated input history data for one or more
users including word probability data, key probability data, edit
distance data, and touch model probability data. In some examples,
the analysis and comparison is repeated continuously for each
character of text input, while in other examples, the analysis and
comparison may only occur after an event. Examples of events
include receiving a delimiter key (e.g., a space, backspace, enter,
ESC, or other designated key(s)) or determining a latency in user
typing speed that exceeds a threshold value.
[0105] At process block 830, the analysis and comparison data from
process block 820 is evaluated to determine if any unexpected input
data was received. If no unexpected input data is received (e.g.,
all the words of the text input data can be found in one or more of
the candidate sources used at process block 820), the method
proceeds to process block 810 in order to continue receiving input
data. If unexpected input data is detected, the method proceeds to
process block 840. In some embodiments, a word can be designated as
unexpected input data even though the word may exist in one of the
candidate sources, for example, using a grammar checker module. An
unusual word such as "whilst" may still be designated as unexpected
input data because other similar words, for example, "while" have a
higher probability of correctness, based on word probability data,
key probability data, edit distance data, and touch model
probability data. In some examples, the feedback semaphores for
unusual words designated as unexpected may be modified in
comparison to feedback semaphores used for words that do not exist
in any of the candidate sources at all.
[0106] At process block 840, an indicator such as a feedback
semaphore is provided in order to alert a user that suggestion
candidates are available. As discussed above, the feedback
semaphores provided can include visual displays, including the
highlighting of touch screen keyboard keys or keyboard background;
audio playback, such as a distinctive "clunk clunk" sound (which
indicates to the user that something may be incorrect in the input
data), or haptic feedback.
[0107] At process block 850, suggestion candidates are provided,
using for example, a candidates area on a touch screen display.
Each suggestion candidate is associated with an alternative word
that will be used to modify the input data if that particular
suggestion candidate is selected. In some examples, one of the
candidates can by displayed differently (or highlighted
differently) than other candidates to indicate that it is the most
likely suggestion candidate. This distinguishing display not only
provides a visual cue to the user, but can also be used to indicate
that the input data will be modified with the suggestion candidate
if other designated keys are pressed, for example, a delimiter key
such as the space key, punctuation keys, or the return or enter
keys.
[0108] At process block 860, the selection (or absence of a
selection) of one of the suggestion candidates is detected.
Suggestion candidates can be selected several ways, including
receiving a single touch screen input over the suggestion candidate
on the touch screen display, or by receiving a keystroke for a
designated delimiter key. If no suggestion candidate is selected,
the method proceeds to process block 810 in order to receive more
input data. If a suggestion candidate selection is detected, the
method proceeds to process block 870.
[0109] At process block 870, the input data is modified using an
alternative word associated with the selected suggestion candidate.
In some examples, the alternative word is used to replace the input
word in the input data. In other examples, the alternative word is
added to the input data preceding or subsequent to the input word.
In some examples, the "alternative" word is a word that was
previously auto-corrected, and the alternative word is therefore
used to effectively undo a word inserted in the input data using an
auto-correction routine. Thus, as described above, a quick and
effective way of providing suggestion candidates and indicators of
their availability is provided that allows users to quickly modify
input data using the suggestion candidates without moving their
focus of attention away from a touch screen keyboard.
IX. Example Auto-Correction Feedback
[0110] Users often keep their focus of attention on the keyboard
area when typing on a touch screen keyboard, which often causes
users to fail to see auto-corrections that may be replacing
legitimate words, such as names and technical terms that do not
exist in the dictionary. This can lead to tremendous frustration,
especially if users do not notice the text replacements until much
later.
[0111] FIGS. 9A-9D depict a method 900 of providing the user with
an auto-correction feedback semaphore. FIG. 10 is a flowchart 1000
corresponding to the method 900 depicted in FIGS. 9A-9D.
[0112] As shown in FIG. 9A, a mobile device 902 comprises a text
display area 904, suggestion candidates area 910, and touch screen
keyboard 908. The display area 904 is shown displaying a phrase
"smsing" 920 after a user has typed the unknown word "smsing," but
before typing a delimiter, so a carat 916 is displayed immediately
after the phrase. Shown in the suggestion candidates area 910 are
the suggestion candidates "ending" 912, "facing" 913, and
"smashing" 914.
[0113] FIG. 9B depicts the mobile device 902 as the user presses
the space bar 932 with a finger 934, which causes the space bar to
immediately be highlighted red, the word "smsing" in the phrase 920
to be replaced with a replacement word "ending," 922 a "swish"
sound 950 to be played over a speaker, and the mobile device 902 to
vibrate, as shown by squiggle lines 955 and 956. The carat 916 is
advanced by a space, and the mobile device also displays the
suggestion candidates "smsing" 940, "facing" 941, and "smashing"
942 in the candidates area. The candidate "smsing" 940 is the word
that was just replaced by auto-correction, and the candidate 940 is
colored green to indicate that it is the replaced word.
[0114] FIG. 9C depicts the mobile device 902 after the user has
decided to undo the auto-correction. To do so, the user uses a
finger 962 to make a single touch screen input over the suggestion
candidate "smsing" 966, which will undo the auto-correction.
[0115] FIG. 9D depicts the mobile device 902 after the
auto-correction has been undone. As shown, the word "ending" has
been replaced with the original word "smsing" 982, the suggestion
candidates area 910 is cleared, and the carat 916 remains a single
space past the replaced word.
[0116] FIG. 10 is a flow chart 1000 further detailing the method
900 shown in FIGS. 9A-9D. At process block 1010, a mobile device
receives input data from a source such as a touch screen keyboard.
In some embodiments, the first input data includes text characters,
text words, position data for key presses on a touch screen
keyboard, typing speed data, correction data, and/or touch screen
orientation data.
[0117] At process block 1020, the input data is analyzed to
determine if a designated delimiter key (e.g., a space key or an
enter key) has been received in the input data. If not, then the
method proceeds process block 1010 to receive more input data. If a
delimiter key was received in the input data, the method proceeds
to process block 1030.
[0118] At process block 1030, the last word of the input data is
compared against one or more candidate sources, and an
auto-correction is applied to replace a word designated as
incorrect with a replacement word from a candidate source.
[0119] At process block 1040, one or more auto-correction feedback
semaphores are displayed or played back, in a manner that is likely
to keep the user's focus of attention on the keyboard. For example,
a "swish" sound can be played, haptic feedback provided, and red
highlighting displayed on the delimiter key that was pressed by the
user. The feedback semaphores are displayed in a manner that does
not require the user to shift their focus of attention away from
the keyboard, and therefore the user is able to more easily
recognize that an auto-correction has occurred, as well as to
easily view suggestion candidates that will be presented at process
block 1050.
[0120] At process block 1050, one or more suggestion candidates are
presented to the user, including the original word that was
auto-corrected. One of the suggestion candidates is also
highlighted using, for example, shading or a different color. The
highlighted suggestion candidate is designated as the most likely
substitution that the user will select. In some examples, the
highlighted suggestion candidate is original word that was
auto-corrected. In other examples, the highlighted suggestion
candidate is a word from a candidate source designated to be the
most likely substitution. The method proceeds to process block 1060
after receiving touch screen input.
[0121] At process block 1060, the method determines whether the
user selected a delimiter key. If a delimiter key was pressed, the
method proceeds to process block 1070, where the word inserted by
auto-correction at process block 1030 is itself replaced with the
word shown as the highlighted suggestion candidate, and the method
then proceeds to process block 1010, where more input data is
received. If a delimiter key was not pressed, the method proceeds
to process block 1080.
[0122] At process block 1080, the method determines whether the
user selected a suggestion candidate (e.g., by providing a single
touch screen input over the suggestion candidate). If a suggestion
candidate is selected, the method proceeds to process block 1090,
where the word inserted by auto-correction at process block 1030 is
itself replaced with the word associated with the selected
suggestion candidate. If a suggestion candidate is not pressed, the
method proceeds to process block 1010 to process the selected key
(e.g., by adding the selected key to the input data) and continue
receiving input data.
X. Example Invoking Next Phrases
[0123] FIGS. 11A-11F depict exemplary methods 1100 and 1150 of
providing a user with predicted next phrases after receiving a
completed word of text input using suggestion candidates. In some
examples, users can select two or more suggestion candidates
sequentially, thereby improving typing speed performance. FIG. 12
is a flow chart 1200 illustrating an exemplary implementation of a
method corresponding to the methods 1100 and 1150 shown in FIGS.
11A-11F.
[0124] As shown in FIG. 11A, a mobile device 1102 has a text
display area 1104, a suggestion candidates area 1110, and a
keyboard 1108. The mobile device 1102 is shown after a user has
typed one word of the phrase "happy" 1106, and the carat 1116 is
positioned immediately after the word "happy."
[0125] FIG. 11B depicts the mobile device 1102 after the user has
typed a delimiter key (e.g., the space bar). As shown, three
suggestion candidates "birthday" 1120, "new" 1121, and "go lucky"
1122 are displayed in the candidates area 1110. The user is shown
using a finger 1130 to select the first suggestion candidate
1120.
[0126] FIG. 11C depicts the mobile device 1102 after adding an
additional next word "birthday" 1138 after the word "happy" in the
phrase 1106 based on the selected suggestion candidate and
positioned the carat 1116 one space away from the end of the word
1138.
[0127] FIG. 11D depicts an exemplary method 1150 including an
alternative selection of a suggestion candidate by a user compared
to that shown in FIGS. 11A-11C. FIG. 11D shows a mobile device 1152
entering a delimiter character immediately after typing the word
"happy" 1154. The carat 1156 is positioned one space past the word.
A user is shown selecting the suggestion candidate "new" 1162 with
a finger 1165 in the suggestion candidates area 1160.
[0128] FIG. 11E depicts the mobile device 1152 after the user has
selected the suggestion candidates "new" 1162. The word "new" 1170
is added to the text input after the word "happy," another search
of one or more candidate sources is performed for the phrase "happy
new," and several suggested next words for the phrase "happy new"
are presented as suggestion candidates: "year" 1182, "baby" 1183,
and "day" 1184. The user is shown selecting the suggestion
candidate "year" 1182 with a finger 1175.
[0129] FIG. 11F depicts the mobile device 1152 after adding the
word "year" 1190 associated with the selected suggestion candidate
1182 to the text input, to form the phrase "happy new year."
Another search of the one or more candidates sources is performed,
but no matches are found, so the suggestion candidates area 1160 is
cleared, and the carat 1156 is placed one space from the word
"year" 1190.
[0130] FIG. 12 is a flow chart 1200 further detailing the exemplary
methods 1100 and 1150 shown in FIGS. 11A-11F. At process block
1210, input data is received, and the method proceeds to process
block 1220, where the presence or absence of a delimiter character
in the input data detected. If no new delimiter key is received in
the input data, the method proceeds to process block 1210 to
receive additional input data. If a delimiter key is received, then
the method proceeds to process block 1230 to generate suggestion
candidates for the input data. In some examples, only the last word
typed is used to search for suggestion candidates, while in other
examples, additional words in the input data can be included in the
search.
[0131] At process block 1240, the method determines if any
suggestion candidates were generated. If no suggestion candidates
are generated, the method proceeds to process block 1210 to receive
additional input data. If one or more candidates were generated at
process block 1230, the method proceeds to process block 1250. At
process block 1250, the suggestion candidates are presented to the
user using a touch screen display coupled to the mobile device.
[0132] At process block 1260, a suggestion candidate selection (or
absence of a selection) is detected. Suggestion candidates can be
selected several ways, including receiving a single touch screen
input over the suggestion candidate on the touch screen display, or
by receiving a keystroke for a designated delimiter key. If no
suggestion candidate is selected, the method proceeds to process
block 1210 in order to receive more input data. If a suggestion
candidate selection is detected, the method proceeds to process
block 1270.
[0133] At process block 1270, the input data is modified using an
alternative word associated with the selected suggestion candidate.
In some examples, the alternative word is used to replace the input
word in the input data. In other examples, the alternative word is
added to the input data preceding or subsequent to the input word.
In some examples, the "alternative" word is a word that was
previously auto-corrected, and the alternative word is therefore
used to effectively undo a word inserted in the input data using an
auto-correction routine. After modifying the input data, the method
immediately proceeds to process block 1230, where more suggestion
candidates are generated based on the modified input data. In this
manner, a user can select multiple suggestion candidates from a
changing set of suggestion candidates, thereby improving typing
speed and accuracy, especially for highly frequent combinations of
words and phrases.
XI. Example Use of Typing Speed and Simulating Click Sounds
[0134] FIG. 13 is a chart 1300 representing some events that occur
during a method that includes the use of random click sounds played
back as the user presses keys on a touch screen keyboard, as well
as the use of keystroke latency to determine when to present
suggestion candidate to a touch screen keyboard user. FIG. 14 is a
flow chart 1400 that corresponds to the methods shown in FIG.
13.
[0135] Shown in FIG. 13 is a time axis 1330 that indicates when
events occur or are received by a mobile device. The user enters
several successive key presses on the touch screen: "b" 1340, "a"
1341, "n" 1342, and "a" 1343 in relatively rapid succession. At the
same time, the mobile device randomly selects and plays a "click"
sound that simulates the sound that a key press makes when typing
on a hardware keyboard. For example, the mobile device can randomly
select one of the five different click sounds labeled #1-#5. As
shown, the click sounds 1310-1314, etc., are not always the same
for a given key that is pressed. For example, the first keypress
"a" 1341 is followed by playback of click sound #2 1311, while the
second keypress "a" 1344 is followed by playback of click sound #3
1314. Similarly, the two keypresses shown for "n" playback sound #3
1312 the first keypress 1342 and sound #5 1313 for the second
keypress 1344. By providing playback of random click sounds that
sound like a hardware keyboard, the user receives more naturalist
audio feedback that more closely simulates typing on a hardware
keyboard.
[0136] Also shown in FIG. 13 is the presentation of suggestion
candidates based on the user's keystroke latency. The mobile device
monitors a user's keystroke latency as words are typed, and if the
user pauses for a length of time that exceeds a pre-determined
threshold, suggestion candidates are presented for the input data
of the most recent word input. The techniques for presenting the
suggestion candidates are similar to those discussed above. In some
examples, suggestion candidates are only presented when the partial
data input is determined to be incomplete, i.e., when the partial
input data does not appear in a candidate source such as a
dictionary or input history data. In other examples, suggestion
candidates are presented after a pre-determined latency if no
delimiter has been received for the word. In some examples, the
user can define the latency threshold value, while others examples
use a pre-defined threshold value.
[0137] As shown, the user types the keys "b" 1340, "a" 1341, "n"
1342, and "a" 1343 in relatively rapid succession, followed by a
relatively longer pause. After the latency threshold has been
reached based on the delay after the keypress 1343, the mobile
device presents several suggestion candidates for the partial input
data "bana," includes candidates "banana" 1350, "bananas" 1351, and
"banal" 1352. As shown, the user decides to ignore the suggestion
candidates and continues typing "na<space>s" followed by
another pause after keypress "s" 1345. Another time period exceeds
the latency threshold value after the keypress "s" 1345, and
different suggestion candidates 1360-1363 are generated. Thus, the
suggestion candidate generation module can consider not only the
partial input word "s," but also the previous word input "banana."
(e.g., "banana s" is the input being considered by the candidate
generation module). Thus, very accurate suggestion candidates can
be provided even though the user has only entered a single letter
"s" of the latest input word. As shown, the user selects the
candidate "slug" 1362 by pressing the "slug" suggestion candidate
1370 on the touch screen display, and the input data is modified to
"banana slug."
[0138] FIG. 14. is a flow chart 1400 further detailing the method
shown in FIG. 13. At process block 1410, input data is received
from a source such as a touch screen keyboard. In some embodiments,
the first input data includes text characters, text words, position
data for key presses on a touch screen keyboard, typing speed data,
correction data, and/or touch screen orientation data.
[0139] At process block 1420, a random click sound is selected from
a collection of two or more sounds and played back to the user
using, e.g., a speaker. As discussed above regarding FIG. 13, the
click sounds are not associated with any specific key, and any
particular key press on the same key can have multiple random
sounds played back.
[0140] At process block 1430, the latency between the most recently
received keystroke, and last keystroke received immediately
preceding that keystroke, is measured. At process block 1440, the
measured latency is compared to a pre-determined threshold value.
In some examples, this latency threshold can be selected by the
user. In other examples, the threshold can come from a default
value. In other examples, the threshold can be selected based on a
user's previous or average typing speed. If the measured latency
does not exceed the threshold, the method proceeds to process block
1410, and more input data is received. If the measured latency does
exceed the threshold, the method proceeds to process block
1450.
[0141] In some examples, the keystroke latency or typing speed data
is also used to select which types of feedback semaphore(s) are
presented. For example, if a user is typing relatively rapidly,
only one or two more subtle semaphores are provided (e.g., only
delimiter key highlighting can be used for fast typists). If a user
is typing relatively slowly, more semaphores, or semaphores that
use stronger visual, audio, and/or haptic cues are provided (e.g.,
louder audio, use of haptic force, and/or brighter or
different-colored highlighting can be used for slower typists).
Alternatively, or in addition, typing speed data, touch point data,
device orientation, etc., can also be used to detect whether a user
is typing using one or two hands. Thus, different types of feedback
semaphores can be provided based on whether the user is typing
using one or two hands.
[0142] At process block 1450, the input data is analyzed and
suggestion candidates are generated and presented for the input
data in a similar fashion to those techniques for generating
suggestion candidates discussed above. In some examples, candidates
are only generated if the input data is partial and cannot be found
in a candidate generation source. In other examples, candidates are
generated unless the last key pressed is a delimiter key. In some
examples, only partial input data is considered, while other
examples can analyze other words in the input data (e.g., the word
immediately preceding or the word immediately subsequent to the
current input word). After the candidates are generated, they are
presented to the user using, e.g., a candidates area on a touch
screen device.
[0143] At process block 1460, the selection (or absence of
selection) of one of the suggestion candidates is detected. If a
suggestion candidate selection is detected, the method proceeds to
process block 1470, where the partial input data is modified by
replacing the input word with the word associated with the selected
suggestion candidate, and the method proceeds to process block 1410
to receive more input data. If a suggestion candidate is not
detected as selected, then the method proceeds to process block
1410 to receive more data.
XII. First Experimental Results
[0144] A. Introduction
[0145] Experiments were conducted in order to measure the text
entry performance improvement in a controlled experiment using
several semaphore designs on naive participants in a usability
study.
[0146] One of the semaphores studied is called mobile key-trail
feedback. To help users keep track of the keys they just typed in
the keyboard area, the on-screen keyboard utilizes a feedback
technique where pressed keys briefly light up and gradually fade
away.
[0147] Another semaphore studied is a guidance semaphore called
key-prediction guidance. An example of key-prediction guidance is
highlighting or bolding the next likely letter on a keyboard.
[0148] The other two semaphores studied were the feedback
semaphores unexpected-key feedback and auto-correction
feedback.
[0149] B. Experiment Methodology
[0150] Four female and seven male participants from a metropolitan
area were recruited by a professional contracting service. Three
owned touch screen phones, five had owned a QWERTY phone at some
point in their lives, and three owned only 12-key numeric phones.
The participants came from various occupational backgrounds from
housewife to IT professional, and were within an age range of
19-39. All participants were compensated for their time.
[0151] Participants were shown a short phrase on a desktop computer
screen from the well-known phrase set described in MacKenzie &
Soukoreff, "Phrase Sets for Evaluating Text Entry Techniques,"
Extended Abstracts of CHI 2003, pp. 754-755 (2003) (hereinafter
"MacKenzie I"). The set contains 500 short English phrases with no
punctuation, varying from 16-43 characters with a high letter
frequency correlation with an English corpus. The phrase set was
supplemented with news headline phrases culled from the Internet
containing words not found in the test dictionary (e.g., "smsing
while driving is risky," "obama is inaugurated"). Four phrases and
one supplemental phrase were randomly selected for each of sixteen
conditions: one for each semaphore (4-choose-1), plus combinations
thereof (4-choose-2,4-choose-3, and 4-choose-4). These combinations
were tested in order to examine if any of the semaphores conflicted
with each other, and to investigate how people felt about having a
semaphore be absent in another condition. The order of the
conditions was not counter-balanced. Using the "think-aloud"
protocol described in Lewis & Rieman, Task-Centered User
Interface Design: A Practical Introduction (1993) (available via
anonymous ftp at: ftp.cs.colorado.edu), participants were asked in
each of the sixteen conditions to type in the phrases "as quickly
as possible," but instructed that they should pause and verbalize
any new thoughts they had about the semaphores at any moment.
During the study, specific questions were asked about each
semaphore, and at the end of the study, participants were asked to
rank-order any semaphores they would leave on by default.
[0152] C. Results
[0153] Because this was a usability study aimed at refining the
user experience and design of the semaphores, the visual and
auditory parameters of some semaphores were continually adjusted
based on user feedback. As such, it is difficult to accurately
interpret raw statistics. However, the numbers do convey general
trends reported here.
[0154] 1. Unexpected-Key Feedback
[0155] Seven of the eleven participants stated that they would
leave the unexpected-key feedback semaphore turned on by default,
even when using the more "distracting" versions shown in FIGS.
7F-7H. Five of the eleven listed this semaphore as their top
choice, stating that they "depended" on it. In particular, they
noted how it would alert them to the candidates area where they
would almost certainly find their desired word as a choice. In
fact, many participants claimed that this was their "strategy" for
typing as quickly and as accurately as possible. Interestingly,
after the settling on a visual design, half the participants said
they did not perceive the visual cues and only relied on the
auditory cue, and half said that they would turn off the auditory
cue, as they considered the visual cue informative enough.
[0156] 2. Auto-Correction Feedback
[0157] All the eleven participants loved the auto-correction
semaphore and said that they "depended" on knowing when words were
being auto-corrected. Furthermore, several participants noted that
they could even predict when certain words would be auto-corrected
(e.g., "obama"). This helped them to be prepared to select their
replaced word in the candidate area, as shown in, for example, FIG.
9C. Many participants asked to have the replaced word be
automatically added to the dictionary when selected from the
candidate area.
[0158] 3. Key-Trail Feedback
[0159] Only five of the eleven participants stated that they would
leave the semaphore on by default, with two of the five claiming
that they "depended" on it, and three of the five claiming that it
was "cool eye candy." Informally, it was observed that those two
subjects had very slow baseline typing speeds. Of the six who did
not choose to leave the semaphore on, two really disliked it
(although this may have been due to the fading rate, which were
later adjusted).
[0160] 4. Key-Prediction Guidance
[0161] Only five of the eleven participants stated that they would
leave the semaphore on by default, with three of the five listing
it as their top choice. Of the five, only one claimed he "depended"
on it, particularly for one-handed use, with the rest stating that
they would leave it on "just in case [they] were unsure of how to
spell a long word."
[0162] 5. Combinations
[0163] Initially, two participants disliked the combination of the
key-prediction feedback with the key-trail feedback, but that was
when the keys were entirely colored. Once the color blue was
applied only to the letters, no complaints were received. No other
combinations seemed to bother the participants.
[0164] 6. Discussion
[0165] Given the uniformly positive response for the
auto-correction feedback, it was decided not to test this semaphore
further. Indeed, for the subsequent experiment, this semaphore was
left on all the time. The diversity of preferences for the other
semaphores suggests that users should be given the option of
turning specific semaphores on and off in a control panel. With the
unexpected-key feedback, some people seemed to be more tuned to the
audio cue and some to the visual, suggesting that giving users
fine-grained control over which cue is on would be valuable.
[0166] In watching and listening to the usability participants, it
was observed that the semaphores seemed to have more utility for
people who typed slower than those who were accustomed to typing
quickly on touch screen keyboards. Interestingly, all of the touch
screen phone owners stated that they would leave the unexpected-key
feedback on by default, with two of the three listing it as their
top choice.
XI. Second Experimental Results
[0167] A. Introduction
[0168] Given the feedback received from the usability study
discussed above, it was decided to compare the text entry
performance of only the unexpected-key feedback and the
key-prediction guidance semaphores. The auto-correction feedback
semaphore was incorporated into the baseline for two reasons: (1)
the usability results for auto-correction feedback were compelling,
and (2) to see how well it would work with the other two semaphores
for timed text entry. Key-trail feedback was not tested in the
second experiment. Although the key-trail feedback received
usability responses that were as ambivalent as the key-prediction
feedback, key-prediction feedback was tested instead because of
related previous research as discussed in MacKenzie & Zhang,
"Eye Typing Using Word and Letter Prediction and a Fixation
Algorithm," Proc. of the ACM Symposium on Eye Tracking Research and
Applications--ETRA 2008, pp. 55-58 (2008) (hereinafter "MacKenzie
II") and Magnien, et al., "Mobile Text Input with Soft Keyboards:
Optimization by Means of Visual Clues," Proc. of MobileHCI, pp.
337-341 (2004) (hereinafter "Magnien").
[0169] B. Methodology
[0170] Eighteen participants (nine males and nine females) between
the ages of 21 and 39 were recruited using the same professional
contracting service as in Experiment One. Participants again hailed
from a wide variety of occupational backgrounds. All participants
were compensated for their time. Five owned touch screen phones at
some time in their life, nine owned QWERTY phones at some time in
their life, and seven owned 12-key numeric phones only (note that
these are not exclusive categories). During recruiting, all
participants answered that they were familiar with the QWERTY
layout and could type on a normal-size keyboard without frequently
looking at the keys.
[0171] The MacKenzie and Soukoreff phrase set was again utilized,
except in order to make sure that participants had a chance to hit
every letter on the keyboard, a script to select the shortest
sequences of phrases was used that covered the entire alphabet from
A to Z. Supplemental phrases with words not found in the test
dictionary were not included, to attempt to reproduce previous
results for the key-prediction guidance semaphore on a mobile
QWERTY touch screen keyboard. For each condition, subjects received
eight practice and twenty stimuli items.
[0172] All participants were first taught the basics of using the
touch screen keyboard on a mobile device, in particular, a
prototype before-market phone with a 3.5 inch resistive screen
having 800.times.480 WVGA resolution. A target phrase was displayed
on a desktop computer screen and participants were asked to
memorize it. They had as much time as they needed to memorize the
phrase. Participants were asked to memorize the phrases in order to
mimic the real experience of entering intended text. When
participants felt they were "ready," their task was to type the
phrase into the mobile device "as quickly and as accurately as
possible." The phrase was left displayed on the computer screen,
because in previous experiments with the same task, some naive
participants (who were not all university students) experienced
difficulties with memorization under timed conditions. Timing began
as soon as they entered the first letter of the phrase and ended
when they hit the "Enter" button twice. The entire experiment took
slightly under two hours.
[0173] The primary independent variable used was Semaphore,
consisting of Baseline, Unexpected-Key Feedback and Key-Prediction
Guidance. As dependent measures, time to enter text, accuracy, and
the efficiency measure: keystrokes-per-character or KSPC as
described in MacKenzie & Tanaka-Ishii, Text Entry Systems:
Mobility, Accessibility, Universality (Morgan Kaufmann Publishers
2007) (hereinafter "MacKenzie III") were examined.
[0174] The number of times users pressed the backspace button was
used as an additional measure. Note that users were not allowed to
place the cursor onto their typed text for editing since not all
participants were proficient at this task. In short, a simple,
within-subject experiment was conducted where all participants
encountered the three Semaphore conditions in counter-balanced
order.
[0175] C. Results
[0176] 1. Time to Enter Text
[0177] Given the previously reported success of key-prediction, it
was hypothesized that the key-prediction guidance semaphore would
significantly reduce the average time participants spent entering
in each phrase. It was also predicted that the unexpected-key
feedback semaphore would reduce time to enter text as well since
several usability participants had already mentioned using this
feedback as a strategy for selecting word prediction choices in the
candidates area. Indeed, using repeated measures analysis of
variance (ANOVA), a significant main effect for Semaphore was
observed (F.sub.2,1057=4.97, p<0.01). However, Tukey post-hoc
pair-wise comparisons only revealed significant differences between
the Baseline and Unexpected-Key Feedback (p<0.01) and between
the Unexpected-Key Feedback and the Key-Prediction Guidance
(p<0.05), but not between the Baseline and Key-Prediction
Guidance. FIG. 16A is a chart 1600 that shows the average time
spent per phrase for the Baseline, Key-Prediction Guidance and the
Unexpected-Key Feedback conditions, which were 26.58, 25.81, and
23.95 respectively.
[0178] One possible reason for not observing a significant
difference in time to enter text for the Key-Prediction Guidance
over the Baseline is that participants were all familiar with the
QWERTY layout, unlike the dynamically changing keyboard task
described in Magnien. Hence, participants did not experience a need
to limit the visual search space.
[0179] 2. Accuracy
[0180] Accuracy was measured in terms of whether or not the
participant ultimately typed in the correct phrase. Finer-grained
measures of correctness were also looked at, such as the Minimal
String Distance Error Rate described in MacKenzie III, which
computes the distance between two strings in terms of the lowest
number of edit operations to turn one string into the other.
However, these measures could not be used since some of the errors
made by users were those related to pressing a word prediction or
disambiguation candidate, which swaps the typed text completely. In
this case, one keystroke engenders a lot of errors. Hence,
finer-grained measures could be unreliable.
[0181] It was hypothesized that both semaphores would result in
higher accuracy since the Unexpected-Key Feedback is immediately
informing users of errors and the Key-Prediction Guidance is
leading users to the correct spelling. However, in analyzing the
data, a main effect for Semaphore was not observed. All three
conditions obtained accuracies between 0.12 and 0.14 with fairly
wide standard deviations. This deviates from previous results
described in MacKenzie II in which Key-Prediction Guidance reduced
error rates for eye-typing, although it is hard to compare across
modalities.
[0182] 3. KSPC
[0183] It was also hypothesized that the two semaphores would
reduce keystrokes-per-character. Indeed, a significant main effect
was observed for Semaphore (F.sub.2,1057=6.74, p<0.001), with
significant Tukey post-hoc differences between the Baseline and the
Unexpected-Key Feedback (p<0.05) and between the Baseline and
the Key-Prediction Guidance (p<0.05). FIG. 16B shows a chart
1610 displaying the average keystrokes-per-character for the
Baseline, Key-Prediction Guidance and the Unexpected-Key Feedback
conditions, which were 1.22, 1.10, and 1.09 respectively.
[0184] Assuming that participants accurately hit only the letter
they are supposed to and not any of the word prediction candidates,
the baseline KSPC for a QWERTY layout keyboard is theoretically 1
according to MacKenzie III. Note that all three conditions had a
KSPC above 1, indicating that typing on a small 3.5 inch resistive
screen with fat fingers is not easy, even with the word prediction
and disambiguation candidates.
[0185] 4. Number of Backspaces
[0186] The number of backspaces was measured as a way to tease
apart intentionally corrected mistakes from KSPC, which encompasses
that as well as keystrokes for selecting word prediction
candidates. It was hypothesized that Unexpected-Key Feedback would
reduce the number of backspaces by alerting users to a mistake
before they enter more characters to the end of that mistake, and
that Key-Prediction Guidance would also reduce the number of
backspaces by leading users to correct spelling. Indeed, a main
effect was observed for Semaphore (F.sub.2,1057=7.26, p<0.01)
but with just a significant Tukey post-hoc difference between the
Baseline and Unexpected-Key Feedback (p<0.001). FIG. 16C is a
chart 1620 that shows the average number of backspaces for the
Baseline, Key-Prediction Guidance and the Unexpected-Key Feedback
conditions, which were 3.54, 2.79, and 2.29 respectively.
[0187] 5. One Versus Two Hands
[0188] During the process of running the experiment, an interesting
trend was noticed; those participants who chose to perform the
entire study with two thumbs tended to make less use of the word
prediction and disambiguation candidates. Because participants had
to maintain the same way of holding the device throughout the
experiment, incoming subjects were asked thereafter to use either
two hands (i.e., two thumbs) or one hand for input. No participant
held the device with one hand and typed with the thumb for the
entire experiment, though during practice some participants
certainly tried out that position. All one-handed participants used
one hand to hold the device and another to enter text with their
index finger.
[0189] While it would have been ideal to run a within-subjects,
3.times.2 factorial design experiment with the OneVsTwoHands added
as another independent variable, because fair amount of data had
already been collected, between-subjects analysis of the results
was conducted, making sure that both one-handed and two-handed
users were represented in all possible orderings of the three
conditions studied. Hence, additional statistical analyses treating
OneVsTwoHands as an additional factor in a univariate ANOVA was
performed.
[0190] For time to enter text, a significant main effect for
OneVsTwoHands was observed (F.sub.1,1057=5.66, p<0.05), as well
as a significant interaction effect with Semaphore
(F.sub.2,1057=3.68, p<0.05). FIG. 16D is a chart 1630 that shows
a breakdown of the previous time taken to enter text result in
terms of the number of hands. When using two hands, participants do
not seem to be reducing time to enter text in either the
Key-Prediction Guidance or the Unexpected-Key Feedback conditions,
as shown by the bars 1632, 1633, and 1634 for Baseline,
Key-Prediction Guidance, and Unexpected Key Feedback, respectively.
However, when participants used one hand, the differences with the
Baseline become more accentuated, as shown by the bars 1636, 1637,
and 1638 for Baseline, Key-Prediction Guidance, and Unexpected Key
Feedback, respectively. Although significant main effect for
OneVsTwoHands on KSPC (F.sub.1,1057=2.77, p<0.01) and number of
backspaces (F.sub.1,1057=5.46, p<0.05) was observed as well, the
same kind of interaction effect with Semaphore was not observed for
any dependent variable other than time to enter text. Hence, for
KSPC and number of backspaces, the significant post-hoc differences
observed previously recur, regardless of the number of hands
used.
[0191] 6. Final Questionnaire
[0192] After the experiment, users were asked to pick their
favorite condition and to rank-order which of the two semaphores
they would leave on by default. This was done to confirm
qualitative findings from the usability study. Thirteen of the
eighteen participants found the Unexpected-Key Feedback condition
to be the most favorable. No one picked the Baseline. With respect
to rank-ordering, eight of the eighteen participants said they
would leave on the key-prediction feedback semaphore and nine of
the eighteen said they would leave on the unexpected-key feedback
semaphore. The fact that only nine, and not thirteen, participants
said they would leave it on implies that although some people found
the semaphore useful, it is disruptive enough to them that they
would turn it off by default and turn it on as needed.
[0193] D. Discussion and Design Implications
[0194] From conducting the usability study, it was observed that
visual cues for semaphores cannot grab too much attention. If they
do, users generally dislike them. Toning down the semaphores allows
users who want to use the visual cues, to mentally grab hold of
them, and those who do not, to ignore them. As such, it is
desirable that semaphores be designed to be very visually
subtle.
[0195] Thus, these experimental results indicate that both the
unexpected-key semaphore and the key-prediction guidance semaphore
have the ability to improve text entry performance.
XIII. Example Alternatives and Combinations
[0196] Any of the methods described herein can be performed via one
or more computer-readable media (e.g., storage or other tangible
media) comprising (e.g., having or storing) computer-executable
instructions for performing (e.g., causing a computing device to
perform) such methods. Operation can be fully automatic,
semi-automatic, or involve manual intervention.
[0197] Having described and illustrated the principles of our
innovations in the detailed description and accompanying drawings,
it will be recognized that the various embodiments can be modified
in arrangement and detail without departing from such principles.
It should be understood that the programs, processes, or methods
described herein are not related or limited to any particular type
of computing environment, unless indicated otherwise. Various types
of general purpose or specialized computing environments may be
used with or perform operations in accordance with the teachings
described herein. Elements of embodiments shown in software may be
implemented in hardware and vice versa.
[0198] In view of the many possible embodiments to which the
principles of our invention may be applied, we claim as our
invention all such embodiments as may come within the scope of the
following claims and equivalents thereto.
* * * * *