U.S. patent application number 12/458460 was filed with the patent office on 2009-12-03 for system and method for interaction of graphical objects on a computer controlled system.
This patent application is currently assigned to Xerox Corporation. Invention is credited to Patrick Baudisch, Lance E. Good, Jock D. Mackinlay, Mark J. Stefik, Polle T. Zellweger.
Application Number | 20090295826 12/458460 |
Document ID | / |
Family ID | 40942678 |
Filed Date | 2009-12-03 |
United States Patent
Application |
20090295826 |
Kind Code |
A1 |
Good; Lance E. ; et
al. |
December 3, 2009 |
System and method for interaction of graphical objects on a
computer controlled system
Abstract
A user interface method and system for performing sensemaking
tasks using visual information objects. The present invention
provides techniques for the system to perform operations for
creating, rearranging visual information objects, reallocating
display space and scaling user input automatically so that users
need not interrupt their creative activities. Such operations are
intuitive and non-intrusive. Certain of the behavior results of the
operations are specified in contracts associated with each visual
information object.
Inventors: |
Good; Lance E.; (Cupertino,
CA) ; Stefik; Mark J.; (Portola Valley, CA) ;
Baudisch; Patrick; (Seattle, WA) ; Mackinlay; Jock
D.; (Palo Alto, CA) ; Zellweger; Polle T.;
(Palo Alto, CA) |
Correspondence
Address: |
OLIFF & BERRIDGE, PLC
P.O. BOX 320850
ALEXANDRIA
VA
22320-4850
US
|
Assignee: |
Xerox Corporation
Norwalk
CT
|
Family ID: |
40942678 |
Appl. No.: |
12/458460 |
Filed: |
July 13, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10371017 |
Feb 21, 2003 |
7576756 |
|
|
12458460 |
|
|
|
|
60357850 |
Feb 21, 2002 |
|
|
|
Current U.S.
Class: |
345/619 ;
345/620; 345/660; 715/792 |
Current CPC
Class: |
G06F 3/0481 20130101;
G09G 2340/12 20130101; G09G 5/14 20130101 |
Class at
Publication: |
345/619 ;
345/660; 345/620; 715/792 |
International
Class: |
G09G 5/00 20060101
G09G005/00; G06F 3/048 20060101 G06F003/048 |
Claims
1. A computer controlled free form sensemaking system for user
creation and user control of visual information objects, the visual
information objects being automatically responsive to user
interactions involving objects or the information in them, the
system behavior for automatic responses being governed by
parameterized contracts, and the system comprising a repair element
for enabling a finished interaction to be replayed differently
resulting in a different automatic system behavior according to
selection of a different value in a parameter in a contract wherein
the repair element further comprises: a) a recording element for
recording interactions amongst said visual information objects; b)
an undo element for undoing an automatic interaction behavior; c) a
contract parameter selection element to enable selection of
different values for one or more contract parameters, and d) a
replay element that carries out the user interactions under the
different selected contract parameters resulting in a different
automatic system behavior without requiring the interactions to be
recorded again.
2. The computer controlled free form sensemaking system of claim 1,
wherein the parameter selected is for an object with which the user
has directly interacted.
3. The computer controlled free form sensemaking system of claim 1,
wherein the parameter selected is for an object other than an
object with which the user has directly interacted.
4. The computer controlled free form sensemaking system of claim 1,
wherein the selectable parameters govern behavior involving one or
more of automatic alignment, occlusion, clipping, growth, scaling,
shading, object arrangement, aspect ratio, panning, peripheral
awareness indicators, or representation of detail.
5. The computer controlled free form sensemaking system of claim 4,
wherein the scaling parameter controls objects that are moved into
a nested space of objects.
6. The computer controlled free form sensemaking system of claim 4,
wherein the shading parameter controls shading of objects that are
visually nested inside other objects.
7. The computer controlled free form sensemaking system of claim 1,
wherein when user interaction with an object increases the amount
of information in the object then the parameter selection governs
the automatic behavior of the object for growing the object in its
containing object by at least one of altering the scale used for
information inside the object, clipping information inside the
object, moving information in the populated space to a space
outside of the viewed space, or automatic panning to keep the
interacting objects in view.
8. The computer controlled free form sensemaking system of claim 1,
wherein the parameters that govern the behavior of text objects
select between line wrapping or horizontal growth.
9. The computer controlled free form sensemaking system of claim 1,
wherein the parameters that govern the behavior of text object
select between bumping versus zoom-anti-zoom.
10. The computer controlled free form sensemaking system of claim
1, wherein the parameters that govern the behavior of text objects
select between preservation of aspect ratio or font scaling.
11. The computer controlled free form sensemaking system of claim
1, wherein the parameters that govern scaling allow selection of
scaling parameters for each type of information object contained
within an object.
12. The computer controlled free form, sensemaking system of claim
1, wherein each of said visual information objects is an instance
of one of a plurality of visual information object types, said
plurality of information object types including one or more of: a
space object type for indicating a content based relationship
amongst visual information objects; a text object type for
representing textual information having textual behaviors; or a
structure object type for representing sets of visual information
objects having a predetermined structure.
13. The computer controlled free form, sensemaking system as
recited in claim 1, wherein said predetermined structure of said
structure object type is a list structure.
14. The computer controlled free form, sensemaking system as
recited in claim 1, wherein said space object type is displayed as
a view area of a larger populated area wherein information objects
are positioned.
15. A method of operating a computer controlled free form
sensemaking system comprising the steps of: a) generating and
displaying user perceptible representations of a plurality of
visual information objects in a viewed space that is contained
within a populated space during user free form sensemaking wherein
each user perceptible representation of the plurality of visual
information objects has one or more associated contracts for
defining automatic behavior responsive to an interaction with other
visual information objects; b) recording interactions amongst said
visual information objects; c) undoing an automatic interaction
behavior; d) detecting a user input indicating a selection of
different values for one or more contract parameters; and e)
replaying the interactions under the different selected contract
parameters resulting in a different automatic system behavior
without requiring the interactions to be recorded again.
Description
[0001] This is a Division of application Ser. No. 10/371,017 filed
Feb. 21, 2003, which claims the benefit of U.S. Provisional
Application No. 60/357,850 filed Feb. 21, 2002. The disclosure of
the prior application is hereby incorporated by reference herein in
its entirety.
CROSS REFERENCE TO RELATED APPLICATIONS
[0002] Reference is made to the following commonly-assigned
copending U.S. patent applications: (1) Ser. No. 10/371,263,
entitled "System and Method for Moving Graphical Objects on a
Computer Controlled System," by Lance E. Good et al., now U.S. Pat.
No. 7,068,288, (2) Ser. No. 10/369,613, entitled "Method and System
for Incrementally Changing Text Representation," by Benjamin B.
Bederson et al., (3) Ser. No. 10/369,614, entitled "Method and
System for Incrementally Changing Text Representation," by Benjamin
B. Bederson et al., now U.S. Pat. No. 7,549,114, (4) Ser. No.
10/369,612, entitled "Methods and Systems for Navigating a
Workspace," by Lance E. Good et al., now U.S. Pat. No. 7,228,507,
(5) Ser. No. 10/369,624, entitled "Methods and Systems for
Interactive Classification of Objects," by Lance E. Good et al.,
and (6) Ser. No. 10/396,617, entitled "Methods and Systems for
Indicating Invisible Contents of Workspace," by Lance E. Good et
al., now U.S. Pat. No. 7,487,462 all filed herewith and the
disclosures of which are incorporated herein.
FIELD OF THE INVENTION
[0003] The present invention is related to the field of computer
controlled display systems, in particular techniques which optimize
use of available display space and minimize user disruption when
performing tasks causing the interaction of graphical objects on
the computer controlled display system.
BACKGROUND OF THE INVENTION
[0004] Blackboards, electronic whiteboards and other
two-dimensional free form workspaces have long been used for
supporting brainstorming, sensemaking, and other generative tasks
where information is created and organized incrementally.
Sensemaking is a term used to describe activities that interweave
retrieving, creating and using information. As the amount of
information placed in these workspaces grows, the competition for
display space increases. If users need to interrupt their ideation
to rearrange the information in the workspace, make space for new
information, or perform other "housekeeping" functions to enable
use of the workspace, they risk disruption of their train of
thought or "losing the flow" resulting in the possible loss of
ideas. Often, the information contained in the workspace is
represented as graphical objects. Thus, there is a general need in
systems providing a two-dimensional (2D) workspace for techniques
that provide intuitive and timely manipulation and interaction of
graphical objects.
[0005] The use of computer controlled two-dimensional workspaces
for organizing information is well known in the art. Some of these
systems include the Alternate Reality Kit (SMITH, R. Experiences
with the Alternate Reality Kit: an example of the tension between
literalism and magic. Proceedings of CHI/GI '87, April 1987, pp.
61-67), Aquanet (MARSHALL, C. AND ROGERS, R. Two Years Before the
Mist: Experiences with Aquanet, Proceedings of the ACM Conference
on Hypertext, November 1992, pp. 53-62), Cognoter (STERK, M.,
BOBROW, D., FOSTER, G., LANNING, S. AND TATAR, D. WYSIWIS Revised:
Early Experiences with Multi-User Interfaces. Proceedings of the
Conference on Computer-Supported Cooperative Work, December 1986,
pp. 276-290), Flatland (MYNATT, E., IGARASHI, T., EDWARDS, W.,
LAMARCA, A. Flatland: New Dimensions in Office Whiteboards.
Proceedings of the CHI 99 Conference on Human Factors and Computing
Systems, May 1999, pp. 346-353), MUSE (FURNAS, G. AND ZHANG, X.
MuSE: A Multiscale Editor, Proceedings of UIST '98, November 1998,
pp. 107-116), PowerPoint (MICROSOFT POWERPOINT.
http://www.microsoft.com/office/powerpoint/), Storyspace (EASTGATE
SYSTEMS STORYSPACE. http://www.eastgate.com/Storyspace.html),
Tivoli (PEDERSON, E., McCALL, K., MORAN, T. AND HALASZ, F. Tivoli:
An Electronic Whiteboard for Informal Workgroup Meetings,
Proceedings of the Conference on Human Factors in Computing
Systems, April 1993, pp. 391-398), VIKI (MARSHALL, C., SHIPMAN, F.,
AND COOMBS, J. VIKI: Spatial Hypertext Supporting Emergent
Structure, Proceedings of the 1994 ACM European Conference on
Hypermedia Technology, September 1994, pp. 13-23), and Web Squirrel
(EASTGATE SYSTEMS WEB SQUIRREL
http://www.eastgate.com/sguirrel/Welcome.html).
[0006] In order to minimize "losing the flow" when using such
systems, the manner in which a user interacts and the system
responds is critical. A number of techniques have been disclosed in
the art which purportedly facilitate the use of such systems. Some
techniques provide for certain operations to be automatically
performed. One such operation is the handling of related
information in some coordinated fashion, e.g. as a list or group.
For example, PowerPoint, Tivoli, and VIKI all have notions of a
list structure to automate the alignment and ordering of list
items. Although PowerPoint and Tivoli also use this list structure
to assist in inserting items into the list, neither of these
systems provide an interactive preview of the resulting insertion.
Such a preview is useful in understanding the effect of updating a
list structure. Storyspace and VIKI also have the notion of a
subspace (i.e. a separate and distinct interaction space within
another space) to automate the maintenance of a spatial,
hierarchical collection of objects.
[0007] Several systems have also employed features for automating
object interactions. The Flatland system uses a bumping concept to
avoid the occlusion of objects on a whiteboard surface. Moving or
growing an object could cause it to transitively bump other objects
and if an object hits the edge of the whiteboard (i.e. the visible
display area) it would automatically shrink. Data Mountain
(ROBERTSON, G., CZERWINSKI, M., LARSON, K., ROBBINS, D., THIEL, D.,
VAN DANTZICH, M. Data Mountain, Using Spatial Memory for Document
Management, Proceedings of UIST '98, November 1998, pp. 153-162)
also employs bumping to reduce the occlusion of web pages on an
inclined plane; facile movement and placement of the pages is
permitted while keeping at least some part of the page visible. The
Alternate Reality Kit provided a more diverse collection of object
interaction contracts, such as gravity or spring forces, to
simulate object interactions in a virtual world with alternate
physics principles. Lastly, PowerPoint provides automatic panning,
a type of object interaction to keep the current text entry point
in view. However, PowerPoint does not return your position to the
original viewpoint when the text entry is completed.
[0008] Another technique known in the art is to provide specialized
tools to enable movement or the organization of objects. The
alignment stick (RAISAMO, R. AND RAIHA, K. A New Direct
Manipulation Technique for Aligning Objects in Drawing Programs.
Proceedings of UIST '96, November 1996, pp. 157-164 and RAISAMO, R.
An alternative way of drawing. CHI'99 Proceedings, 1999, 175-182)
provides a tool for providing direct manipulation of layouts. The
broom alignment tool (ROBBINS, J., KANTOR, M. AND REDMILES, D.
Sweeping away disorder with the broom alignment tool. Conference
Companion of CHI'99) is similar to the alignment stick, but it can
also distribute objects along its length. The user can cycle
through different distribution policies by pressing the space bar.
Furthermore, moving the broom backward automatically undoes the
alignment operation.
SUMMARY OF THE INVENTION
[0009] A computer controlled system embodying a method for
graphical object interaction is disclosed. The system provides
intuitive and non-intrusive techniques to perform operations for
rearranging graphical objects, reallocating display space and
scaling user input automatically so that users need not interrupt
their creative activities. The user interface techniques may be
applied to any application which includes interaction between
graphical visual information objects. An important principle is
that graphical objects should not unnecessarily occlude or overlap
other graphical objects when they are created or manipulated so
that they interact with other graphical objects.
[0010] The system of the present invention is comprised of: one or
more input elements for enabling creation and interaction of said
visual information objects; a display for showing a viewed space
that is contained within a populated space, each of said visual
information objects positioned within said populated space; a
processing element for generating display information for said
display so that visual information objects within said viewed space
maintain user perceptible information throughout the creation and
interaction of visual information objects.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1 is a transition graph that is used to described the
sensemaking activity.
[0012] FIG. 2 illustrates a sensemaking application and associated
workspaces for performing sensemaking tasks on a computer
controlled display system of the currently preferred
embodiment.
[0013] FIG. 3 illustrates text box visual information objects as
may be implemented in the currently preferred embodiment.
[0014] FIG. 4 illustrates how text is created in a text box in the
currently preferred embodiment.
[0015] FIG. 5 illustrates stretching of a text box to change the
aspect ratio as may be performed in the currently preferred
embodiment.
[0016] FIG. 6 illustrates stretching of a text box to create a
pre-sized region for the text as may be performed in the currently
preferred embodiment.
[0017] FIG. 7 illustrates an example of a list visual information
object as may be implemented in the currently preferred
embodiment.
[0018] FIG. 8 illustrates a step in making a list from a text box
as may be performed in the currently preferred embodiment.
[0019] FIG. 9 illustrates a list visual information object
containing a sub-list as may occur in the currently preferred
embodiment.
[0020] FIG. 10 illustrates scaling the size of text contained in a
list as may be performed in the currently preferred embodiment.
[0021] FIG. 11 illustrates the nesting of spaces and resulting
reduction in scale of subspaces as may occur in the currently
preferred embodiment.
[0022] FIG. 12 illustrates the relationship between a "viewed
space" and a "populated space" for an instance of a space visual
information object as may be utilized in the currently preferred
embodiment.
[0023] FIG. 13 is an illustration of peripheral awareness
indicators for clipped visual information objects contained in a
space object as may be utilized in the currently preferred
embodiment.
[0024] FIG. 14 is an illustration of a "viewed space" that is
panned to various positions in a "populated space" resulting in
clipped objects as may be performed in the currently preferred
embodiment.
[0025] FIGS. 15-16 are an illustration of before and after
"zooming" a space so that it covers the entire writing workspace as
may be performed in the currently preferred embodiment.
[0026] FIGS. 17-19 illustrate the visual cues indicating movement
and the visual scaling that occurs as a visual information object
is positioned into another visual information object as may be
performed in the currently preferred embodiment.
[0027] FIG. 20 is an illustration building on FIGS. 17-19 wherein
animation occurs when a text box visual information object is moved
and inserted into a list visual information object as may be
performed in the currently preferred embodiment.
[0028] FIGS. 21-23 illustrates the steps for moving a group of
objects into a list object as may be performed in the currently
preferred embodiment.
[0029] FIGS. 24-25 illustrates the resultant automatic object
movement when placing an object between two or more other objects
as may be performed in the currently preferred embodiment.
[0030] FIGS. 26-29 shows a sequence of steps that illustrate the
use of a visual information object to cause movement of other
visual information objects through "bumping" as may be performed in
the currently preferred embodiment.
[0031] FIGS. 30-32 illustrate the automatic panning of a viewed
space in order to maintain text entry in view as may be performed
in the currently preferred embodiment.
[0032] FIGS. 33-36 illustrate a technique for entering text into a
text box that avoids "bumping" other objects yet maintains a
constant user perception of text entry size as may be performed in
the currently preferred embodiment.
[0033] FIG. 37-38 illustrate a user interface for entering
parameters for a contract for a "Space" visual information object
as may be utilized in the currently preferred embodiment.
[0034] FIGS. 39-40 illustrate a user interface for entering
parameters for a contract for a "List" visual information object as
may be utilized in the currently preferred embodiment.
[0035] FIG. 41 illustrates a user interface for entering parameters
for a contract for a "Text Box" visual information object as may be
utilized in the currently preferred embodiment.
[0036] FIG. 42 illustrates the steps of a "repair cycle" for
enabling a sequence of actions to be replayed according to
different contract parameters as may be utilized in the currently
preferred embodiment.
[0037] FIG. 43 is a block diagram illustrating a software
implementation of the currently preferred embodiment.
[0038] FIG. 44 is flowchart illustrating the basic user flow
involved in the movement of visual information objects in the
currently preferred embodiment.
[0039] FIG. 45 is a block diagram of a computer controlled system
upon which the currently preferred embodiment may be
implemented.
DETAILED DESCRIPTION OF THE INVENTION
[0040] A system and method for graphical object interaction that
enables a user to effectively and efficiently organize and process
large amounts of data is described herein. When utilizing the
system a user is able to reduce disruptions to their activity
caused by the limited display space constraints of a computer
controlled display system. So for example, when a user is entering
or organizing information on a workspace area of the system, they
need not be distracted by the need to rearrange items in order to
enter new information. This allows the user to stay "in the flow"
of providing such new information.
[0041] The system and method of the currently preferred embodiment
are implemented in a sensemaking system as described below. In such
a sensemaking environment, instances of information content are
referred to herein as "visual information objects". Visual
information objects are in turn handled by the system as graphical
objects. It should also be noted and emphasized that certain of the
techniques described herein may be applied generally to the field
of user interfaces for computer controlled display systems and may
be utilized for other types of applications that utilize graphical
objects other than the sensemaking activities described herein.
Description Of A Sensemaking Activity
[0042] FIG. 1 portrays a transition graph that is used to describe
sensemaking as an activity. The box 101 represents a writing
workspace in which a user writes information down (generates
content), arranges the writings/content, and navigates through the
generated content. The box 102 represents a retrieving workspace in
which the user issues queries, browses, and collects information.
During the course of the sensemaking activity, the user can shift
attention from one workspace to the other. For example, the user
could search and browse for information in the retrieving workspace
until finding something to add and arrange in the writing
workspace. Alternatively, a user could pause from writing in order
to search for relevant information in the retrieving workspace. In
an implementation, the writing and retrieval workspaces need not be
concurrently displayed nor do they need to be separate and distinct
workspaces. The retrieval workspace has at least an element for
defining and submitting queries and an area for display of some
representation of the result of a query.
[0043] It has been determined that users transition around such a
graph and between the two workspaces in the course of developing
their ideas. When the transition steps are too cumbersome, or when
resource conflicts in external cognition interfere with progress,
the steps take longer or must be repeated. This may cause the user
to "lose the flow" or become disrupted from their sensemaking
activity.
[0044] It has been determined that user interface techniques for
working with such workspaces can help a user stay in the flow. For
example, when generating content, a key requirement for staying in
the flow is to maintain a visible region of work--that is, to keep
the item(s) that the user is working on, together with as much
context as possible, visible to the user. The system may adjust the
view or move objects to accomplish this. A policy favoring limited
occlusion or non-occlusion helps prevent items from getting lost
during system manipulations.
[0045] The following are assumptions about key cognitive resources
for a sensemaking activity on a workspace. A cognitive resource is
one that aids in the performance of some mental task.
[0046] Display space. The system's display space is a primary
resource for external cognition. Information requires display space
and display space is limited. In general, a display space is too
small to show all of the information at once. Many of the design
elements in the currently preferred embodiment involving zooming,
panning, and peripheral awareness indicators are intended to make
trade-offs in the allocation of display space. In making these
trade-offs the user and system determine what information elements
to display and how much display space to allocate to each element
in order to support the active sensemaking task. The management of
a display space influences the cost structure of all three activity
steps in the writing workspace: generating, arranging, and
navigating.
[0047] Persistence of spatial arrangement. A user uses a system's
interactive blackboard-like writing workspace to create and arrange
items. The meaning of the information is held both in the content
(that is, words or graphical objects such as diagrams, graphs,
etc.) as well as in the spatial arrangement. A user tends to locate
closely related items near each other so that they can be
conveniently used together. Parallel arrangements of items suggest
parallel meanings. Persistence of arrangement speeds access in that
the user roughly remembers the location of items and does not need
to search for them. It also speeds understanding when a user uses
parallel visual structures to imply parallel meanings. The user
invests time to arrange items, and presumably benefits both from
convenient use and speedier access because he remembers where
things are. If the system automatically changes the arrangement
radically or in some non-intuitive or illogical fashion, the user
will invest more time and energy to learn the new arrangement and
possibly to rearrange it.
[0048] Limited focus. A user cannot think about everything at once.
The system provides several kinds of "focus+context" mechanisms to
support a user in focusing on what is important. These mechanisms
include grouping operators for objects, peripheral awareness
indicators on spaces, and automatic reduction of the scale of the
contents of hierarchical spaces so that users can focus on the "big
picture" or overall categories while maintaining limited awareness
about internal details. The management of this resource especially
influences the navigation step in the workspace.
[0049] FIG. 2 illustrates a sensemaking application and
accompanying workspaces. Referring to FIG. 2, a writing area 201
provides an area where a user performing the writing tasks as
described with reference to box 101 of FIG. 1. A retrieval area 202
provides an area where a user is performing the retrieval tasks as
described with reference to box 102 of FIG. 1. The writing area 201
is used to display the various visual information objects as will
be described in more detail below. Also, it should be noted that
the writing area 201 may itself be an instance of a space. The
retrieval area 202 provides an input area 203 for entering query
terms as well as results area 204 for receiving the results from
the query. Note that in the currently preferred embodiment the
query could be to a search engine that is designed to access
information available on a local computer, the Internet, an
internal network, or any combination thereof.
[0050] Further illustrated is a menu bar 210 which provides a
plurality of function menus that may be selected in a point and
click or pull down fashion, as well as a toolbar 211 for directly
invoking commonly used functions. The manner of user operation of
the menu bar 210 and corresponding function menus and toolbar 211
is according to well known user interface techniques, thus further
description of their operation is not deemed necessary. The various
functions provided in menu bar 210 include those for creating or
manipulating visual information objects, viewing or modifying
contracts and entering a repair cycle. Toolbar 211 includes
functions for entering text, creating space objects and moving
objects.
System Operation Overview
[0051] During the course of a sensemaking activity, the user will
alternate between steps of generating content, arranging the visual
information objects, and navigating between different parts of the
workspace. The system manages key resources for external cognition
with contracts on its visual information objects to minimize
"housekeeping functions" and to help keep a user in the flow.
[0052] The first step is the generation of content. In the system,
for example, content corresponds to the words contained in text
boxes or various organizational structures. These could also be
graphical information objects, such as images, charts, etc. The
contracts in the system's visual information objects are intended
to support a user in staying in the flow while generating content.
The system does not require that a user "make space" before
entering information or otherwise generating content. A user can
start at any point in the workspace and start typing or otherwise
entering objects. As a text box or a list grows, the system bumps
other objects or pans the containing space as needed so that the
user can maintain focus and stay in the flow of generating content.
Similarly, as a user is adding elements to a list, the contracts
make space for the new items in the list and enforce vertical
alignment. The user does not need to stop entering to arrange the
items.
[0053] The system also simplifies the controls for arranging
objects. Contracts are active during content generation and
automatically arrange objects without additional user action using
bumping, transitive bumping, and panning. The system also
simplifies the "move" operation by providing visual feedback of the
process. The simple pick-up-and-move operation has visual feedback
with drop shadows that provide cues showing which objects are being
moved. As the user moves an object or set of objects over a space,
the moving objects are scaled while in motion to the appropriate
scale for the space so that the user can see how objects will fit
in. When a user moves objects to put them into a list, the list
opens up and makes room for them. In one smooth action, the user
can move an object and decide just where to drop it in the list.
The list dynamically moves its elements around to prepare room for
it. When the user releases the mouse button, all of the objects are
dropped into the selection point in the list and scaled to have the
appropriate font size for the list. If a user drops an object in a
space in a way that would occlude some other objects, an action
suggestive of stepping on a bar of soap moves objects out of the
way. In addition, if a user wants to create some space in the
middle of arranging objects, the user can dynamically turn the
movement of an object into a "snowplow" mode. The drop shadow
indicating selection disappears and the user can push other objects
out of the way. This action does not require the user to fetch a
tool or to drop the object first and then select the objects to be
moved. The user uses the object to push other objects to make room.
These automatic actions keep the user in the flow during
arrangement, without requiring the distraction of selecting,
moving, or resizing other objects.
[0054] The system also provides simplified navigation. Navigation
or panning, is the work by a user to get an appropriate view of
some part of the workspace. One way that the system supports
navigation is by the automatic reduced scaling of the contents of
nested workspaces. This gives the usual benefits of a hierarchical
view where it is easier to see either the higher level structure or
to drill down to see details. The system provides a simple
one-click interface to dive down to a subspace so that it grows to
fill the display space or to return back to the containing space.
The system also provides "City Lights" peripheral awareness
indicators to help keep a user oriented to the presence of objects
that are clipped and not currently in view.
[0055] The system has been implemented for operation on computer
based systems having a graphical user interface such as those
utilizing the Windows.RTM. Operating System from Microsoft, Inc. or
the Mac OS operating System from Apple Computer, Inc. Such
graphical user interfaces have the characteristic that a user may
interact with the computer system using a cursor control device
and/or via a touch-screen display, rather than solely via keyboard
input device. Such systems also have the characteristic that they
may have multiple "windows" wherein discrete operating functions or
applications may occur. Further, such computer based system will be
preferably of a "free form" type wherein a user may create and
manipulate objects independent of positional constraints. Examples
of such free form systems include electronic whiteboards or paint
and/or draw application programs. A more complete description of
such a computer-based system upon which the present invention may
be implemented is described below with reference to FIG. 45.
Visual Information Objects
[0056] To support sensemaking activities, three basic types of
visual information objects are commonly used--text objects, lists,
and spaces. A fourth type, graphical objects such as images,
diagrams, drawings and the like, may also be used. However, the
currently preferred embodiment will be described with respect to
the first three types, but it would be apparent to one of skill in
the art to use all or any subset of the four types. The behavior of
these objects under different kinds of interactions such as
creation, movement or growing, is governed by policies, represented
as contracts. Each type of visual information object has its own
kind of contract. Further, each instance of a visual information
object may have its own instance of a contract that is different
from contracts of other visual information objects of the same
type.
[0057] Text objects are visual information objects that contain
textual information. Instantiations of text objects are referred to
as a "text box(es)". List objects are visual information objects
that arrange and present items in some ordered sequence, e.g. in a
vertical tabular format. Instantiations of list objects are
referred to as a "list(s)". A space object is a visual information
object that is used to arrange other visual information objects.
Instantiations of space objects are referred to as a
"space(s)".
[0058] Contracts in the currently preferred embodiment of the
present invention govern both objects themselves and the "children"
or elements of composite objects. For example, contracts for a list
govern both the behavior of the visual list object and also the
behavior of objects that are members of the list. Note that if an
element is "removed" as an element of a composite object it is no
longer governed by the parent object. Such relationships can be
established explicitly (e.g. in a list), or implicitly (e.g.
subspace within a space).
[0059] Listed below are examples of properties of visual
information objects governed by contracts.
[0060] Alignment. It is desirable to have constraints that govern
the vertical or horizontal alignment of objects with other objects.
For example, the left boundaries of the elements of a list should
be vertically aligned.
[0061] Occlusion. It has been determined that for sensemaking
activities, contracts often preclude occlusion. This prevents
temporary "loss" of information when one objects obscures another.
However, it is recognized that there may be instances where some
occlusion may be desirable, e.g. "piles" or "stacks" of related
visual information objects.
[0062] Clipping. Because display space is limited, clipping is
inevitable. Clipping refers to objects that cannot be completely
displayed, typically as a result of performing some operation.
Contracts about clipping determine what information is portrayed
about clipped objects.
[0063] Growth. The size of an object can change if the amount of
information that it holds changes or if the user stretches the
object in the display space. Objects differ in their growth
contracts.
[0064] Scale. Objects can be shown at different scales. There is a
default scale for objects. The scale of an object can be changed by
explicit user actions. Scale contracts can also be transformed when
an object is moved into a region where additional scaling contracts
apply. Scale contracts can also apply differently to different
parts of an object.
[0065] Aspect Ratio. The ratio of height to width of an object is
preserved by most operations.
[0066] Type. Most operations preserve the type of an object.
[0067] Representation of Detail. Contracts governing level of
detail can control the use of overviews and summaries.
[0068] The visual information objects of the present invention are
implemented using well known software development techniques and
data structures and are described in more detail below.
The No-Occlusion Principle
[0069] When the populated space contains more information than can
be displayed in the view space using a reasonable scale, something
has to give. Other systems have addressed this by using 2.5D or 3D
techniques such as having windows that partially overlap or cover
up other windows. Using such systems, users can cause occlusion
unintentionally, such as when a new object is created on top of an
existing one. In such cases, the user is focused on manipulating
some objects and may not notice that another object has been
occluded. As a result, objects that the user needs for the next
steps of sensemaking may become difficult to access or may have
disappeared for no obvious reason. Detecting occlusions requires
the user's attention and resolving them may require breaking their
flow of concentration.
[0070] Many of the contracts described below enforce a no-occlusion
principle. In practice, this often means that instead of objects
covering up other objects, they "bump" them aside. Bumping behavior
arises when objects are moved, when objects grow to contain more
information, and when they are re-sized by the user.
Text Box Objects
[0071] Text box objects are visual information objects for
containing text. An easy way to start developing and capturing
ideas is to create text boxes anywhere in an open space for the
ideas that come to mind, as shown in FIG. 3. Referring lo FIG. 3,
text boxes 301-305 are displayed.
[0072] FIG. 4 illustrates how text is created. FIG. 4 shows two
text boxes 401 and 402. The text box 401 is illustrated prior to
entry of text and is displayed as a vertical line. This vertical
line appears as the left hand edge of a text box when a user clicks
in the space and before entering any text. In the currently
preferred embodiment, creation of a text box is enabled when a user
positions the cursor in an open area of a space and clicks the
mouse button.
[0073] What happens to a text box when a user types depends on
which contract is in force. In a usual case for the currently
preferred embodiment, the user just begins typing. In this case,
the vertical line disappears and a cursor appears at the point
where text is entered as the user enters text. As suggested by the
text box 402, a user can type and the text box will grow to the
right as the user adds characters, and shrink to the left if the
user backspaces. In the currently preferred embodiment, the box can
grow to the right until reaching a maximum number of default
characters, e.g. 33. If the user continues typing, line wrap will
occur and a second line of text will appear left aligned and below
the first line. Referring briefly to FIG. 5, the text box 402 can
later be stretched to change the aspect ratio. This operation does
not change the text size (font size), but does cause the line wrap
to be recomputed. The stretching of a text box is accomplished by
"selection" of the text box, typically by a point and click
operation using the cursor control device, and then selecting and
dragging an anchor point, e.g. point 501, to a desired width.
Alternative methods of stretching, e.g. after selection clicking
the mouse button to cause stretching to the right by predetermined
character increments, would not depart from the spirit and scope of
the present invention.
[0074] In another stretching variant interaction shown in FIG. 6, a
user can immediately stretch vertical bar 601 to the right,
creating a pre-sized region for the text. In this example, the
contract lets the text grow to the right to fill the region. When
the characters reach the right hand side, the text box grows
downwards and system will wrap lines (i.e. begin entering text on a
new line).
[0075] None of the operations relating to the "length" of the text
box affect the visual size (roughly "font size") of the text
itself. In the currently preferred embodiment a text box scaling
operation that uses predetermined keys on the keyboard, e.g. the
Page Up and Page Down keys, can be used to grow or shrink the scale
of the text box and the text within it. The scaling contract
preserves the aspect ratio and line wrapping within a text box.
Lists
[0076] A list is a visual information object in the system that
arranges items in an ordered sequence. In the currently preferred
embodiment, this arrangement is vertically oriented. A list has a
title (written in a title bar) and zero (0) or more members. FIG. 7
shows an example of a list titled "Things to Pack."
[0077] Lists can be created by selecting a set of items, e.g. text
boxes, and invoking an operation to cause them to be treated as a
list. In the currently preferred embodiment of the present
invention, this is accomplished by using a "listify" menu operation
on a set of selected items.
[0078] In the currently preferred embodiment, an alternative way to
make a list is to start typing an item in a text box, and then
ending with a carriage return (typically the enter key of a
keyboard coupled to a computer controlled display system). FIG. 8
shows a step in the making of a list from a text box. The user. has
typed the word "Sodas" 801 followed by depressing the Enter key
followed by entering the term "Coke" 802. The system created a list
whose title is "List" 803 with the two members "Sodas" 801 and
"Coke" 802.
[0079] The title "List" is the default for a list structure that is
provided upon initial creation of the list structure. The title of
the list can then be set by a separate operation--either dragging
one of the list elements into the title position to make it the
title or positioning the cursor and entering text directly in the
title position.
[0080] The basic contract for lists causes all members to be left
aligned. Items in a list have a default size (font size) and new
items that are added to the list are given the same size. As new
items are added to a list, the list grows downwards.
[0081] Lists are recursive in that they can have other lists as
members. This is illustrated in FIG. 9. FIG. 9 shows a list 901
where one of the members of the list, i.e. list 902, is itself a
list. The outer list 901 is titled "Things to Pack" and the inner
list 902 is titled "Gear." It should be noted that any visual
information object can be an item in a list, so Spaces can be
members of lists as well.
[0082] Like text boxes, lists have contracts that govern scaling.
In the currently preferred embodiment, lists can be scaled by first
selecting them and then using the Page Up and Page Down keys. When
the scale of a list is reduced, the system preferentially reduces
the size of the list members more than the size of the title, as
shown in FIG. 10. FIG. 10 illustrates lists 1001-1006 wherein the
lists are successively reduced in scale. In this way, when a list
is made very small, the user can still read what the list
represents and get a rough sense of its size although the elements
in the list may be too small to read.
Spaces
[0083] The top-level writing workspace of the system of the
currently preferred embodiment is itself an instance of a space. A
space is conceptually similar to a blackboard for creating and
arranging visual information objects, such as text boxes, lists,
and other spaces.
[0084] One feature of spaces in the system is that nested spaces
reduce the scale of their members. This is illustrated in FIG. 11.
This reduction in scale reflects the sense that each level of
nesting corresponds to a deeper level of detail. Details are made
smaller so as to provide more space for the "higher" or bigger
ideas. Nested spaces also darken the background of their open space
as another visual cue to the depth of the containment relationship.
FIG. 11 shows how a list and some text boxes are scaled and the
backgrounds of spaces are shaded differently in nested spaces.
[0085] It should be noted that spaces and lists are both composite
objects. Their behaviors are different in several important ways
and it is instructive to compare some of the differences in their
contracts. Like lists, spaces have members. Unlike lists, spaces do
not enforce a vertical alignment property for arranging the
members. Further, spaces do not grow in size when more members
(i.e. information) are added to them.
[0086] One way to understand a space is to think of it as a finite
window over an infinite canvas. Because the window has a finite
size, any visual information objects on the canvas that are not
directly under the window are clipped. That part of the canvas on
which there are visual information objects is referred to as
"populated space" and that part which can be seen at any given time
is referred to as "viewed space." Many of the contract elements for
spaces are intended to govern trade-offs in managing what
information about populated space appears in the viewed space.
[0087] FIG. 12 illustrates the relationship of the viewed space to
populated space for a space object. Visual information objects that
are in populated space 1201 but not the viewed space 1202 are
clipped (i.e. they are not displayed). A panning operation makes it
possible to move the viewed space around in the populated space
1201 analogous to moving the "window" over the "canvas" as
described earlier.
[0088] To provide peripheral awareness of clipped objects in the
populated space, a peripheral awareness indicator that is referred
to as "City Lights" is provided. City Lights are described more
generally in co-pending application entitled "Methods and Systems
for Indicating Invisible Contents of Workspace", Ser. No.
XX/XXX,XXX. As the name suggests, a peripheral awareness indicator
operates such that a person can see the "glow" of the lights of a
distant city in the night sky before they can see the city itself.
A peripheral awareness indicator provides information about clipped
objects. Such information about a clipped object could include its
direction from the boundary of the viewed space, its size, its
distance, its type, its recency or time since last edit, or a
summary of its information content.
[0089] The peripheral awareness indicator of the currently
preferred embodiment provides limited information about the size,
direction, and distance of clipped objects as illustrated by FIG.
13. In FIG. 13, the boxes 1301-1302 represent items (visual
information objects) that are inside the viewed space, and the
boxes 1303-1306 indicate clipped items in populated space. The city
lights are shown as visually distinct markings on the outer
bounding box of the viewed space, see markings 1307-1310. An
orthogonal projection from the clipped items onto the bounding box
of populated space delineates the edges of the city lights. City
lights for objects that are in the "corner" and have no orthogonal
projection onto the sides of the viewed space are shown as small
corner indicators (see marking 1309). City lights for clipped
objects that are nearer than a near boundary (i.e. "near objects")
have a first visually distinct characteristic, e.g. a first color.
City lights for clipped objects that are farther away (i.e. "far
objects") have a second different visually distinct characteristic,
e.g. a second color different from the first color. Indicators for
near objects are permitted to occlude indicators for far objects
because some indicator of an object in the direction of the far
object remains. In the currently preferred embodiment, the position
of the near/far boundary defaults to half of the width of the
viewed space.
[0090] Preferably, the computation of the peripheral awareness
indicators in the currently preferred embodiment is fast enough to
allow them to be computed and re-displayed in real time when a
space is being panned. FIG. 14 illustrates several versions of
spaces with the same contents but panned to different points.
Referring to FIG. 14, space 1401 has no clipping of information
objects 1402 or 1403. At 1404 a panning of space 1401 to the right
causes clipping of information object 1403 and a resulting
indicator 1405. At 1406 a panning of space 1401 up and to the left
causes information object 1402 to be clipped and a resulting
indicator 1407. At 1408, a panning of space 1401 far to the left
causes both the information objects 1402 and 1403 to be clipped and
display of resulting indicators 1409 and 1410.
[0091] As a user works in the writing space, it is sometimes
important to be able to zoom down to work on the details of a
contained subspace. A zooming operation on a space will cause it to
expand to fill the display area of the entire writing workspace.
FIG. 15 shows an example of a session where much of the information
in the "restaurants" space 1501 is clipped.
[0092] FIG. 16 shows the effects of zooming in on the restaurant
space 1501 to enable working on the details of its contents. The
user can zoom out upon completion of working on the details to
restore the view to the overall space. The manner in which zooming
is performed and enabled is inherent in the graphical toolkit used
to implement the sensemaking application.
Operations Governing Interaction Behaviors
[0093] The contracts described above focus mainly on the allocation
of static display space. In the following, what is described are
operations that govern the moving, growing, and bumping of
information display objects.
[0094] Since the spatial organization of objects is used to convey
meaning, the system provides operations for moving objects. The
user operations for moving an object follows a sequence typical for
those used in graphical user interfaces. The user first moves the
cursor over the target object and clicks on it to select it. Still
pressing the mouse button, the user then moves the object so that
the cursor is over the place where the object is to be placed.
Releasing the mouse button drops the object. It should be noted
that in the currently preferred embodiment, that prior to releasing
the mouse button and dropping the object, the described actions
will occur. This could enable a user to decide if this is the
desired course of action to be taken. If the user does not drop the
object, the system, and the visually displayed objects, will revert
back to their original state. This is illustrated in FIGS. 17 and
18.
[0095] FIG. 17 shows the first stage of moving an object. The text
box 1701 "Plaza Gardens" has been selected. While the mouse button
is depressed, the text box 1701 is floated "above" the space as
indicated by the drop shadow 1702. This procedure for selecting an
object works the same for all kinds of visual information objects.
It also works when a collection of objects is selected and moved at
once.
Automatic Scaling: Moving Objects Into Subspaces
[0096] In this example, the user proceeds to move the text box 1701
into the "Places to Eat" space 1702. In the system of the currently
preferred embodiment, spaces reduce the scale for objects contained
within them, so the floating image of the text box 1701 immediately
reduces in size as shown in FIG. 18. The text box 1701 reduces in
size further as the user moves deeper into the hierarchy of spaces
to the "Restaurants in Disneyland" space 1703 as shown in FIG.
19.
Automatic List Growth: Moving Objects Into Lists
[0097] The user next moves the "Plaza Gardens" text box 1701 over
the list "Main Street Restaurants" 1704. The list 1704 fluidly
grows to make space for the new item. FIG. 20 shows how list items
move out of the way to make space as the floating text box is moved
over the list.
[0098] These examples show how the contracts for display of an
object interact in a fluid way when objects are moved in the
system--reducing the scale of objects as they move over spaces or
making room for an object in a list. Collectively, they support the
user in being able to gracefully move an object using a single,
simple gesture, without needing to attend to details of scaling,
making room, or re-arranging other objects.
[0099] FIG. 21-23 shows three steps of moving a group of items into
a list. Referring to FIG. 21, first the user selects a set of items
2101. In this case, the user first selects the "Plaza Gardens" text
box and then extends the group with the "Plaza Pavilion" text box.
This is carried out in the familiar fashion by holding the shift
key while making the second selection. While the user moves the
group the system displays a drop shadow under each of the items.
Referring to FIG. 22, as the cursor moves over the list "Main
Street Restaurants" 2102 the system opens enough space for the two
items in the list. The user can move the cursor up and down over
the list to select the location of the placement. Finally,
referring to FIG. 23, when the user releases the mouse button, the
items are dropped into the list. The list contracts align the items
and assure that their size is made uniform with the other items in
the list.
Stepping on the Soap: Placing an Object Between Two Objects
[0100] Interactions can also occur over white space. When an object
is dropped between two objects and there is not enough room to
display the object in that location, the objects move out of the
way to make room for it. This action is referred to as "stepping on
the soap", to suggest the way a slippery bar of soap can shoot out
from under your foot if you step on it in the shower. The system
selects an object to move and a direction to move it that minimizes
the overall perturbation as illustrated in FIGS. 24 and 25.
[0101] Referring to FIG. 24, a user positions a floating object
2401 between two stationary objects 2402 and 2403 before dropping
it. Referring to FIG. 25, when the user drops the floating object
2401, the action causes the objects 2402 and 2403 to move out of
the way.
The Snowplow: Using an Object to Move Other Objects
[0102] "Snowplow" is an alternate contract for moving an object
"on" the surface, and may be instigated by the user selecting an
object and initiating an action indicating that the snowplow mode
be invoked. Invoking the snowplow mode causes the drop shadow to
disappear and the user can use the moving object to push other
objects around. When any edge of the moving object touches another
object, the other object is pushed in whatever direction the user
pushes. The user can push a number of objects into any
configuration. Terminating the snowplow mode, e.g. by lifting the
shift key, ends snowplow mode and causes the moving object to again
float above the surface and display a drop shadow. In either mode,
the moving object is released when the cursor control button is
released indicating the object is no longer selected.
[0103] The "snowplow" operation is illustrated with reference to
FIGS. 26-29. Referring to FIG. 26, an object 2601 has been selected
and the shift key depressed. The desired end effect is to have
object 2601 positioned between objects 2602 and 2603. Note that in
FIG. 26, there is not enough room between the objects 2602 or 2603
for it to be placed. For example, for object 2601 representing a
space, it may be desirable to have it positioned between is another
space, object 2602 and a text box 2603. In any event, referring to
FIG. 27, object 2601 is used to move object 2602 to the left. In
FIG. 28, object 2601 is used to move object 2603 to the right. As
illustrated in FIG. 29, there is now enough room between objects
2602 and 2603 for object 2601 to be positioned. It should be noted
that the "soap" operation may also have been used to obtain the
desired effect for the example illustrated in FIGS. 26-29.
[0104] Many of the actions of movement--making space in a list,
stepping on the soap, or snowplow mode--are all effects of the "no
occlusion" rule in contracts for visual information objects.
Another consequence of this rule is that whenever the movement of
one object causes it to bump into another, the second object is
also moved. Bumping is transitive: all of the objects that bump
cause the other objects that they bump to move.
[0105] The no occlusion rule in contracts can have effects for any
operation that changes the area that a visual information object
might occupy. The previous section illustrated actions triggered
when a sensemaker moves an object, including scaling and bumping.
Described below are further effects of the no occlusion rule.
Automatic Bumping: Growing an Object Into Another
[0106] Bumping can also be triggered when an object grows. A text
box can grow when a user adds more text to it and a list grows when
a user adds more items to it. Any visual information object can
grow if the user explicitly changes its scale. An object can also
occupy more space if a user selects it and then reallocates the
space assigned to it. Grabbing and stretching the corners of a text
box can cause it to change its aspect ratio. Grabbing and
stretching the corners of a space changes the area available for
the "viewed space." When any of these operations would cause an
object to occlude another object, the system bumps the second
object so that no occlusion occurs.
Automatic Panning: Keeping Text Entry in View
[0107] Growth of a visual information object can also cause panning
of the viewed space. FIGS. 30-32 shows a sequence in which panning
occurs. Referring to FIG. 30, a user starts typing the text "Su
Hong's Restaurant" into a text box 3001 at the edge of the "Places
to Eat" space 3002. As illustrated in FIG. 31, in order to keep the
typing region in view as the text box grows to the right, the
system pans the containing space to the left. When the user
finishes typing, as illustrated in FIG. 32, the space is restored
to its former position so that the text box 3001 is clipped.
[0108] The same effect occurs if a user adds information to a list
that is near the edge of a containing space. The space pans
automatically while the user is working on the list and restores
the viewed space to the original pan position when the user selects
another object.
Zoom-Anti-Zoom: An Alternative to Bumping and Panning for Adding
Material
[0109] An alternative technique for governing the growth of text
boxes is one that is termed "Zoom-Anti-Zoom" or "ZAZ." The intended
effect of ZAZ is to enable a user to stay focused on entering new
text without causing the text box to bump into other objects.
Instead, the ZAZ contract automatically changes the scale (roughly
the font size) and camera position for rendering of the text boxes.
The manner of representation and rendering of visual information
objects is described in greater detail below. In effect, the text
gets smaller as the box gets more full. Since this approach would
eventually have the effect of making the text too small to see, the
system also "moves the user closer" to the text so that its
apparent size stays about the same. When the user selects another
object, the containing space is restored to its original scale and
the text appears in its greatly reduced size. The user can then
move it or scale it as desired.
[0110] This combination of scaling the text and the background
simultaneously is illustrated by the sequence in FIGS. 33-36.
Referring to FIG. 33, a user is operating in space 3301 and
entering text into text box 3302. It is noticed that as text is
entered it will "bump" into subspace 3303. Referring now to FIG.
34, it is observed that the "size" of the text in the text box 3302
appears to be the same (the effect of the increased scaling), but
also, subspace 3303 appears to grow in size (the effect of zooming
in). Referring to FIG. 35, the scaling of text in text box 3302
continues as does the zooming. Zooming is now to the extent that
vertical limits of subspace 3303 exceed the bounds of the view area
of space 3301. Thus, city lights indicators 3304 are displayed.
Finally, referring to FIG. 36 when the user leaves text entry mode,
the original view is restored. Note that the text is now displayed
in a much reduced scale corresponding to the amount of text
entered.
Representing Interaction Behaviors and Display Characteristics as
Parameterized Contracts
[0111] In the currently preferred embodiment, contracts are made
explicit to represent interaction and display policies as
parameterized contracts. The term contracts and policies in this
context is meant to indicate how the visual information object will
interact with other objects and be displayed when it is itself
being interacted with directly or indirectly by the user. The
parameters within the contracts may apply to all the visual
information objects. Alternatively, a contract may be created to
control behavior of an individual visual information object. A
description of exemplary user interfaces to these contracts is
presented below. It would have been apparent to one of ordinary
skill in the art to utilize other well known user interface
techniques to provide parameters for such contracts,
Contracts for Spaces
[0112] FIGS. 37-39 illustrate an exemplary user interface for
contracts for space visual information objects. FIG. 37 shows a
first part of an interface to a contract for spaces. Referring to
FIG. 37, there are multiple tabs indicating the different general
policies associated with space object interaction behavior and
display characteristics, namely growth tab 3701, interior tab 3702
and city lights tab 3703. What is illustrated in FIG. 37 are the
items under the growth tab 3701, namely growth policies. Note that
traversal to the other policies is accomplished by a familiar point
and click operation to the tab representing the other policies. In
FIG. 37 a first policy governs the expansion of the space and is
indicated in area 3704. By selecting the radio button 3705 for
"expand," a user can indicate that a space should grow its display
allocation to accommodate crowding on its interior. The "don't
expand" radio button 3706 means that it should clip its member
contents rather than grow. The "max size" parameters 3707 enable a
user to set a maximum size for the space. So a user can either
specify no maximum 3708 or some particular maximum 3709.
[0113] FIG. 38 shows parameters governing the treatment of interior
objects in a space. This is displayed when selecting the interior
tab 3702. Referring to FIG. 38, the "scale factor" parameter 3801
governs the reduction in magnification of objects when they are
placed into a space. The "nested shading" number 3802 governs the
percentage by which increasingly nested spaces become darker.
Finally, the radio buttons 3803 and 3804 govern whether the space
attempts to display all contained objects, or whether it employs
clipping. If the user selects "don't expand" in the growth policy,
and "show all items" on the interior policy, then the space will
decrease the view's magnification or employ "semantic zooming" in
order to show all of the objects in the limited space. Semantic
zooming refers to a technique wherein the users viewpoint will be
moved.
[0114] FIG. 39 shows the parameters governing the City Lights
peripheral Awareness Indicators when City Lights Tab 3703 is
selected. The "City Lights" tab interface controls parameters for
City Lights, such as the indicator type 3901, the information to be
represented by the indicator 3902, and the projection scheme to be
used to map objects onto the space's perimeter 3903. The indicator
type 3901 allows the user to select the physical appearance of the
peripheral awareness indicator to be used. As described herein, it
is a change of color to the portion of the boundary where the
clipped object is mapped. Other indicator types, such as arrows
pointing to the clipped object may also be used. The Information
indicator 3902 provides for allowing different types of information
to be displayed with the indicator. This could include some textual
description of the clipped object (e.g. "s" for a space, "I" for a
list or "t" for a text box). Finally, the projection scheme
indicator 3903 allows a user to alter the projection scheme used.
Typically, the scheme is for objects that are orthogonal to a
corresponding boundary. In any event, various alternatives for
presenting peripheral awareness indicators can be found in
co-pending application entitled "Methods and Systems for Indicating
Invisible Contents of Workspace", Ser. No. XX/XXX,XXX.
Contracts for Lists
[0115] Lists and spaces in the system of the currently preferred
embodiment are similar in that both kinds of objects are
composite--holding other visual information objects in their
interiors. Lists differ from spaces in that they impose an
alignment on their interior objects. The contract for growth for
lists and for spaces is similar. FIG. 40 shows an example of a user
interface to the contract governing interior objects in a list.
[0116] The user interface to the interior of a list offers controls
for formatting,. such as whether the elements of a list have a
leading "bullet" 4001 and whether items are indented 4002. It also
offers controls governing the magnification of items moved into a
list 4003. Behavior controls 4004 are offered that can govern
scrolling and panning behavior when viewing the contents of a large
list based on known techniques such as a fish-eye, focus-context
approach. Finally, the radio buttons 4005 and 4006 govern whether
the list attempts to display all objects in the area provided for
the list, or whether it employs clipping. The resulting behaviors
are similar as those for the Space object.
Contracts for Text Boxes
[0117] FIG. 41 illustrates a user interface for specifying a Text
Box contract. Referring to FIG. 41, a first parameter 4101 that may
be specified is for the number of characters per line. As
illustrated in FIG. 41, the line length is 33 characters. Next, a
parameter 4102 may be used to specify if the default text box size
is created. In this case, the default text box size is the same as
a single line of text. Finally, parameters 4103, 4104 and 4105 are
used to specify the textual attributes of font, size and
highlights.
Selectable Contracts and the Repair Cycle
[0118] A useful feature of the system is a "repair cycle" for
easily managing the system's automated behaviors. The user has the
option of entering the repair cycle whenever the system makes
automated contract adjustments in response to a user action. Once
in the repair cycle, the user can undo this previous transaction
under the current contract and replay the action under other
contracts to find the most satisfying result. A repair cycle is
illustrated in FIG. 42.
[0119] Referring to FIG. 42, the cycle begins when the user
completes a transaction 4201 with the system resulting in automatic
contract-mediated adjustments 4202. The user can undo the
transaction and modify the current contract to preview the results
of other contract options 4202. The cycle ends when the user is
satisfied with the current contract.
[0120] In the example sequence shown in FIGS. 33-36, the current
contract on the text box specifies that the text box grow
horizontally to accept new text and perform zoom-anti-zoom when
competing for space with other objects. In the repair cycle, the
user can preview and select alternate contracts such as line
wrapping instead of horizontal growth, or bumping instead of
zoom-anti-zoom.
[0121] The repair cycle is invoked using conventional well known
means such as a function option via the menu structure of the
sensemaking application and/or a predetermined key combination or
sequence on the keyboard. When invoked the user will be presented
with an interface to undo the last transaction performed. Once the
transaction is "undone", the contract user interface (as described
with respect to FIGS. 37-41) for the visual information object
initiating the transaction will be presented to the user, to enable
the user to select the desired changes. Alternatively, a user may
select to alter the contract(s) of a different visual information
object.
Software Implementation of the Currently Preferred Embodiment
[0122] The currently preferred embodiment is implemented on a
computer based system having a "windows" supported Graphical User
Interface (GUI), such as Windows from Microsoft Corporation of
Redmond Washington, or MAC OS from Apple Computer of Cupertino,
California. Such a GUI will typically support operations such as
"tap" for selecting/deselecting a graphic object or "double tap"
for invoking an operation that may be associated with the graphic
object being tapped on. Once an object is selected it may be
dragged and dropped to a desired location in the workspace.
Mechanisms for providing such user interface operations are well
known in the art. Thus no further description for creating such UI
operations is deemed necessary.
[0123] The currently preferred embodiment has been implemented
using object oriented programming techniques of the Java
programming environment in combination with a 2D graphics toolkit
available from the University of Maryland known as JAZZ. The JAZZ
toolkit can be used to create graphics based applications. JAZZ
utilizes a "scene graph" model that contains the graphic object
information used to render a scene. In this case, the scene being
rendered is the sensemaking workspace. A scene graph is comprised
of a tree structure that contains information about the objects to
be rendered for display as well as "cameras" for providing
different views. In the currently preferred embodiment, each Visual
Information object and camera is represented by a node on the tree
structure. As objects are moved, their position within the scene
graph will correspondingly change. JAZZ provides a set of tools
that may be used to build applications with a zoomable interface.
This feature is exploited when implementing the various panning and
zooming operations. Further information regarding JAZZ can be found
on the World Wide Web at the website for the University of
Maryland.
[0124] Each Visual Information Object type is defined by a class
and each Visual Information Object is an instance of a class. So in
the currently preferred embodiment, there is an object class
defined for each of the space, list and text objects. As described
above, each Visual Information Object has a distinct user
interface. In the system of the currently preferred embodiment,
such user interface is implemented using tools provided in the
Graphics User Interface (GUI) Library known as SWING (which is
available on the Java 2 platform from Sun Microsystems). Such user
interface tools are part of the class definition for each Visual
Information Object type.
[0125] Input from a user is responded to by an Event Handler that
is also provided by JAZZ. The input could correspond to movement of
the cursor control device, selection, deletion, movement or
creation of a Visual Information Object.
[0126] FIG. 43 is a block diagram illustrating the basic components
of the software implementation of the currently preferred
embodiment and does not include basic "operating system"
functionality. Referring to FIG. 43, a user 4301 causes an input
4302 which is provided to an Event Handler 4303. The input 4302 may
be a movement of the cursor control device, a click and/or hold of
a switch associated with a cursor control device (e.g. a mouse
button), the depression of a key on a keyboard, etc. In any event,
the Event Handler 4303 interprets the input, and passes the
information on to the Sensemaking application 4306. The Sensemaking
Application 4306 includes all the application specific software
implemented logic for the system. Based on the current state of the
system, the Sensemaking application 4306 will cause changes to the
scene graph 4304 to occur. Such a change may reflect the movement
of a Visual information object to a different location within the
scene graph or the indication of the use of a different "camera" to
be used to render the image to be displayed. In any event, the
rendering module 4305 generates the display information for the
visual information objects to be shown on display 4307 based on the
current state of the scene graph 4305 and information identifying
the "camera" from which the scene is to be viewed.
Movement of Visual Information Objects
[0127] FIG. 44 is a flowchart which describes the basic user flow
of movement of visual information objects. Note that this
description is general so as to encompass all types of visual
information objects. Referring to FIG. 44, the first system detects
selection of a visual information object, step 4401. This is by
conventional means in graphical user interfaces, such as pointing
at the visual information object with a cursor control device and
depressing a switch associated with the device. Responsive to a
selection, the system will display the visual information object in
a "select" mode, step 4402. This is also done using conventional
means such as displaying the visual information object with a drop
shadow. As the visual information object is moved, the system will
seek to detect if the user has entered the "snow plow" mode, step
4403. The snow plow mode will enable a user to "push" other visual
information objects in an effort to create more room. A user may
enter snow plow mode by e.g. depressing a predetermined key on the
keyboard, e.g. the shift key, or by other similar means. In any
event, if the snow plow mode is detected, the visual information
object is then displayed in snow plow mode, step 4404. This is
another visually distinct state such as an outlining of the visual
information object. As the visual information object is moved by
the user via the cursor control device, visual information objects
in the path of movement are bumped, step 4405. As described earlier
the bumping is done in the direction of movement and is intended to
emulate the physical action of bumping another visual information
object. Recall that bumping is transitive, so that any other visual
information objects in the path of movement are also bumped, step
4406. The system seeks to detect if snow plow mode is exited, step
4407. If it isn't in snow plow mode then bumping continues per step
4405. If it is, then the visual information object is returned to
be displayed in "select mode", step 4408.
[0128] The system then determines where the visual information
object is positioned on the workspace and respond accordingly. In
one instance it looks to see if it is positioned over a List
structure, step 4409. If it is positioned over a list structure,
then a preview of the insertion is presented, step 4410. The system
then detects whether it is "dropped" into the list, step 4411. If
it is dropped into the List structure, then it is "incorporated"
into the list structure, step 4412 and any transitive bumping of
other visual information objects as a result of the growth of the
list occurs, step 4413. Note that incorporation means that the
visual information object dropped takes on various characteristics
or attributes of the list into which it is inserted. For example,
if a text box is inserted into a text list, then the scale and font
of the inserted text box takes on the scale and font of the
list.
[0129] As movement continues, it is detected if the visual
information object is dropped into a space visual information
object, step 4414. If it is, then it is incorporated and displayed
into the space visual information object, step 4415. Note that in
this instance incorporation not only includes incorporating
attributes of the space, e.g. scaling, but it will also determine
whether or not there is room on the area within the space visual
information object and bumping of other visual information objects
as described above may occur.
[0130] It is determined that the visual information object is to be
dropped on an open area on the workspace, step 4416. It is first
checked to see if there is enough room in the open area to
accommodate the visual information object, step 4417. If yes, the
visual information object is dropped and displayed in normal
unselected mode, step 4419. If there is not enough room, visual
information objects are bumped to make space, step 4418. The visual
information object is then displayed in original mode per step
4419. If the visual information object is not dropped, then the
system continues detecting for user action per the steps starting
at step 4403.
Computer Controlled Display System
[0131] The computer controlled display system as may be utilized in
the present invention is embodied in the computer based system
illustrated with reference to FIG. 45. Referring to FIG. 45, the
computer based system is comprised of a plurality of components
coupled via a bus 4501. While there may in some embodiments be
multiple instances of the same type of component, only a single
instance of each type is described herein. The bus 4501 illustrated
here is simplified in order not to obscure the present invention.
The bus 4501 may consist of a plurality of parallel buses (e.g.
address, data and status buses) as well as a hierarchy of buses
(e.g. a processor bus, a local bus and an I/O bus). In any event,
the computer system is further comprised of a processor 4502 for
executing instructions provided via bus 4501 from Internal memory
4503 (note that the Internal memory 4503 is typically a combination
of Random Access or Read Only Memories). When in operation, program
instructions for carrying out the various functional components of
the present invention are stored in internal memory 4503. The
processor 4502 and Internal memory 4503 may be discrete components
or a single integrated device. The processor 4502 and internal
memory 4503 comprise circuitry for performing the various
processing functions described herein. Also coupled to the bus 4501
is external storage 4507. The external storage 4507 is typically a
high capacity storage media such as magnetic or optical disk
storage.
[0132] Also coupled to the bus 4501 is a display 4504 and a
pointing device 4505. The pointing device 4505 may be a stylus,
mouse, track-ball or other cursor control device. The network
connection 4508 provides means for attaching to a network, e.g. a
Local Area Network (LAN) card or modem card with appropriate
software. Such access may be to the Internet or to local Intranets
or both. Besides communication access, the network connection 4508
may be used to access various resources (i.e. servers) such as
shared computing, storage or printing resources.
[0133] Moreover, the computer controlled display upon which the
computer controlled system may be implemented includes the various
forms of interactive systems such as electronic whiteboards, laptop
or workstation computers, Personal Digital Assistants, Notebook
computers, mainframe systems, client-server systems and the like.
Further, the system may be "dedicated" to the sensemaking activity
or could be part of a general purpose computer based system.
[0134] Finally, it should be noted that the invention has been
described in relation to software implementations, but the
invention may be implemented with specialized hardware built to
operate in a systems architecture as described above.
[0135] Although the invention has been described in relation to
various implementations, together with modifications, variations,
and extensions thereof, other implementations, modifications,
variations, and extensions are within the scope of the invention.
The invention is therefore not limited by the description contained
herein or by the drawings, but only by the claims.
* * * * *
References