U.S. patent application number 12/875115 was filed with the patent office on 2011-03-17 for value-driven visualization primitives for spreadsheets, tabular data, and advanced spreadsheet visualization.
Invention is credited to Lester F. Ludwig.
Application Number | 20110066933 12/875115 |
Document ID | / |
Family ID | 43731675 |
Filed Date | 2011-03-17 |
United States Patent
Application |
20110066933 |
Kind Code |
A1 |
Ludwig; Lester F. |
March 17, 2011 |
VALUE-DRIVEN VISUALIZATION PRIMITIVES FOR SPREADSHEETS, TABULAR
DATA, AND ADVANCED SPREADSHEET VISUALIZATION
Abstract
A method for visually enhancing the display of tabular data of
an electronic spreadsheet is disclosed. The method includes
obtaining data presented in at least one cell of the spreadsheet,
computing at least one visual parameter based on the obtained data,
using the at least one visual parameter to control a visual effect
of the displayed data, and applying the at least one visual effect
to the displayed data to visually enhance the displayed data. Each
visual parameter is a number within a range of numbers and is
determined according to a numerically calculated mathematical
function. The visual enhancements include varying the background
color or texture or border color, thickness, grouping scope of
cells enveloping characters conveying data, as well as font color,
type, embellishment, size, format, location, decimal places,
rounding or supplemental symbols symbolic or image element rendered
within the cell of characters conveying data.
Inventors: |
Ludwig; Lester F.;
(Belomont, CA) |
Family ID: |
43731675 |
Appl. No.: |
12/875115 |
Filed: |
September 2, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61239349 |
Sep 2, 2009 |
|
|
|
Current U.S.
Class: |
715/212 |
Current CPC
Class: |
G06F 40/103 20200101;
G06F 40/18 20200101 |
Class at
Publication: |
715/212 |
International
Class: |
G06F 3/14 20060101
G06F003/14 |
Claims
1. A method for visually enhancing the display of tabular data of
an electronic spreadsheet comprising cells arranged in rows and
columns, the method comprising: obtaining at least one data value
of the tabular data; computing at least one visual parameter
responsive to the at least one data value, wherein each visual
parameter is a number within a range of numbers and is determined
according to a numerically calculated mathematical function;
controlling a visual effect of the at least one data value
according to the at least one visual parameter; and applying the
visual effect to the display of the at least one data value to
visually enhance the display of tabular data of the electronic
spreadsheet.
2. The method of claim 1 wherein the visual effect is a color of at
least one of a font, a background, and a border of a spreadsheet
cell in which the at least one data value is displayed.
3. The method of claim 1 wherein the visual effect is at least one
of a font style of, a font size of, and a font embellishment of the
at least one displayed data value.
4. The method of claim 1 wherein the visual effect is at least one
of a format of, a degree of rounding of, a number of displayed
decimal places of, and a rounding method of the at least one
displayed data value.
5. The method of claim 1 wherein the visual effect is a thickness a
border of a spreadsheet cell in which the at least one data value
is displayed.
6. The method of claim 6 wherein the border encompasses a plurality
of spreadsheet cells in which more than one data value is
displayed.
7. The method of claim 1 wherein the visual effect is displaying a
background texture of a spreadsheet cell in which the at least one
data value is displayed.
8. The method of claim 1 wherein the visual effect is aligning the
at least one data value within one or more spreadsheet cells in
which the at least one data value is displayed.
9. The method of claim 1 wherein the visual effect is at least one
of appending a symbol to the at least one data value displayed in a
spreadsheet cell, displaying a symbol in a spreadsheet cell, and
displaying an image in a spreadsheet cell.
10. The method of claim 1 wherein the visual effect is applied to
the at least one of the row and the column in which the at least
one data value is displayed.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims benefit of priority of U.S.
provisional application Ser. No. 61/239,349 filed on Sep. 2, 2009,
incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The invention relates to data visualization and in
particular its implementation by and application to conventional
spreadsheets and displays of tabular data.
[0004] 2. Background of the Invention
[0005] Interactive electronic spreadsheets such as the 1979
VisiCalc numerical spreadsheet
(http://www.danbricklin.com/visicalc.htm), Supercalc numerical
spreadsheet (1980), Microsoft Multiplan.TM. 91982), Lotus 1-2-3.TM.
(1983), the AppleWorks.TM. spreadsheet module (1984), Microsoft
Excel.TM. (1985), OpenOffice.org.TM. Calc, Google's Google
Spreadsheets, etc., present a tabular presentation of data and
calculation formula flow in arrangements that have proven to be
very useful for data analysis. Interactive electronic spreadsheets
typically include plotting routines that create graphical
representation of selected data points. These plots can be used to
provide visual representations of data and mathematical functions.
In a sense, these interactive electronic spreadsheet functions
provide a form of data visualization based on data in the
interactive electronic spreadsheet.
[0006] Further, interactive electronic spreadsheets present
numerical, textual, and symbolic data to be presented in a tabular
array of cells, with the numerical, textual, and symbolic data
within the cells provided in various font styles and colors, and
with various background colors and border styles of the associated
cell. Using interactive electronic spreadsheet functions known as
"conditional formatting" these font styles and colors, cell
background colors, and cell border styles can be controlled by
values of data according to hand-specified conditional tests
performed on data within one or more specified cells. Such
hand-specified conditional tests are entered through interactive
electronic spreadsheet GUIs (Graphical User Interfaces), VBA
(Visual Basic for Applications), APIs (Application programmer
interfaces), etc. These interactive electronic spreadsheet
functions provide another form of data visualization based on data
in the interactive electronic spreadsheet.
[0007] These visual data representation successes of the
interactive electronic spreadsheet paradigm suggest additional
opportunities for providing and combining additional data
visualization capabilities with the well-established functionality
and embedded deployment of interactive electronic spreadsheet
software. The present invention provides several such additional
data visualization capabilities to traditional interactive
electronic spreadsheets. The visual effects provided by the
invention can be automatically varied over a range responsive to
values of data or formulas according to mathematical functions
rather than conditional tests found in interactive electronic
spreadsheet products.
[0008] Unlike the logical-test structure inherent in "conditional
formatting", the invention provides for use of numerical values
directly from tabular data or as calculated by hidden mathematical
function, compositions of mathematical functions, or traditional
spreadsheet formulas to easily provide rich detailed control of
useful parameterized visual effects rendered in the context of
conventional interactive electronic spreadsheet and tabular data
display.
[0009] The visual effects can include variation of background color
or texture or border color, thickness, grouping scope of cells
enveloping characters conveying data, as well as font color, type,
embellishment, size, format, location, decimal places, or
supplemental symbols symbolic or image element rendered within the
cell of characters conveying data. Additionally, the invention
provides for data-driven 3D plots rendered in the context of or
projection from interactive electronic spreadsheet or tabular data.
The invention provides for these visualization operations to be
generated by an algorithm directly or indirectly in communication
with data or parts of a program rendering an interactive electronic
spreadsheet or visually displayed data table. The invention also
provides for various optional additional functions, for example:
[0010] Data-driven automatic "live" sorting of rows and columns;
[0011] Incorporation of externally-provided stored data and live
data feeds; [0012] Real-time response to live input data,
interactive data entry, or visualization manipulation via
interactive user interfaces; [0013] Collaboration capabilities;
[0014] Web-access capabilities; [0015] Meaningful data sonification
capabilities, including those involving multichannel timbre
modulation.
[0016] Because many aspects of the invention are described in terms
of, or pertain to interactive electronic spreadsheet programs, the
shorthand "spreadsheet" will be understood to mean interactive
electronic spreadsheet programs.
SUMMARY OF THE INVENTION
[0017] The invention provides for a method to visually enhance the
display of tabular data of an electronic spreadsheet comprising
cells arranged in rows and columns. At least one data value is used
to compute at least one visual parameter, and each visual parameter
is a number within a range of numbers and is determined according
to a numerically calculated mathematical function. A visual effect
of the at least one data value is controlled according to the at
least one visual parameter, and the visual effect is applied to the
display of the at least one data value to visually enhance the
display of tabular data of the electronic spreadsheet.
[0018] Various aspects and features of the invention are described
below. These aspects and features may be found either singly or in
combination in the invention, and the descriptions of the various
aspects and features are intended to be illustrative and not to
limit the scope of the invention.
[0019] The visual effects can be automatically varied over a range
responsive to data values or formulas according to mathematical
functions rather than conditional tests found in interactive
electronic spreadsheet products. The visual parameters of a data or
cell presentation attribute may be varied over a pre-defined range
(for example, between 0 and 1, between 0 and 100, between -1 and
+1, etc).
[0020] Various attributes of the font in which the tabular data is
displayed can be controlled. For example, one or more of the font
color, the font type or style, font embellishments, and the font
size may be controlled by the visual parameters. If the tabular
data is numeric data, one or more of the number format, the number
of decimal places, and the method and degree of rounding may also
be controlled by the visual parameters.
[0021] The background of the cell in which tabular data is
displayed may also be visually enhanced by the visual parameters.
For example, one or more of the background color and background
texture may be controlled by the visual parameters.
[0022] Further, one or more of border color and border line
thickness may be controlled by the visual parameters. The border
may enclose a single spreadsheet cell displaying tabular data or
may enclose a plurality of contiguous cells displaying tabular
data.
[0023] According to the visual parameters, a symbol may be appended
to the displayed tabular data, or an image or a data symbol may be
independently displayed instead of the tabular data in a
spreadsheet cell.
[0024] While each of the above aspects and features of the
invention may be applied to individual spreadsheet cells, these
aspects and features may be applied to rows and columns containing
tabular data.
[0025] Other aspects and features of the invention include: [0026]
visual parameters are generated by an algorithm in communication
with the spreadsheet program, or from stored data; [0027] storage
and recall of sessions. [0028] the support of data-driven automatic
"live" rows and columns sorting; [0029] live data feeds; [0030]
externally provided data; [0031] data sonification; [0032] web
access; [0033] collaboration with other users; [0034] 3D
capabilities; and [0035] at least one user interface that providing
the selection of function blocks and the general connectivity among
them to be specified using a drawing tool and a palette of function
blocks, wherein clicking on each function blocks would cause dialog
windows to be displayed for setting parameters by means of typed-in
values, sliders, mouse manipulation, or advanced high-dimension
user interface devices.
BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The above and other aspects, features and advantages of the
present invention will become more apparent upon consideration of
the following description of preferred embodiments taken in
conjunction with the accompanying drawing figures. The accompanying
figures are examples of the various aspects and features of the
present invention and are not limiting either individually or in
combination.
[0037] FIG. 1 provides a demography of ways in which data
visualization and spreadsheets can be integrated together.
[0038] FIG. 2A depicts a region of a spreadsheet or table of
tabular data.
[0039] FIG. 2B depicts a cell and its immediate neighborhood within
a spreadsheet or table of tabular data.
[0040] FIG. 3 depicts of data and cell presentation attributes
serving as candidates for visualization parameters within a
spreadsheet display.
[0041] FIGS. 4A-4E illustrate several application settings where
the invention is well-suited to add value.
[0042] FIG. 5 illustrates an arrangement of selected more general
aspects of the invention that are not restricted to spreadsheet
visualization.
[0043] FIG. 6 illustrates an adaptation of the arrangement of FIG.
5 to an environment supporting collaboration features.
[0044] FIG. 7 depicts a topological interconnection of data flow
paths linking various elements depicted in FIG. 5.
[0045] FIGS. 8A-8C depict approaches for mapping a data value lying
within a pre-defined range to a value within a pre-defined range
for a parameterized data or cell presentation attribute.
[0046] FIG. 9 depicts a more general view and organization of
pre-visualization operations provided for by the invention.
[0047] FIG. 10 depicts an embodiment wherein a selected metaphor is
used to automatically generate parameter assignments and graphics
rendering operations.
[0048] FIG. 11A illustrates a user interface that permits users to
select the geometric element boundary of a captured or rendered
tabular matrix of tabular matrix data elements or cells within a
spreadsheet. FIG. 11B illustrates a user interface that permits
users to select array operations to be performed.
[0049] FIGS. 12A-12G depict transformations of graphics objects via
array operations as provided for by the invention.
[0050] FIG. 13 depicts a spreadsheet architecture and information
flow as it might be used in a data visualization setting.
[0051] FIGS. 14A-14D depict various types of architectural
approaches for combining and integrating spreadsheet programs,
visualization operations provided by the invention, and a data set
used by each.
[0052] FIG. 15 depicts an embodiment of an augmentation of a
spreadsheet program with additional spreadsheet features as
provided for by the invention.
[0053] FIGS. 16A-16B depict ways in which one or more user
interface input devices (such as a mouse, trackball, touchpad,
etc.) can be managed so as to control graphical user interfaces of
a plural software application in the context of the invention.
[0054] FIGS. 17A-17F illustrate the six independently adjustable
degrees of freedom of touch from a single finger that can be
simultaneously measured by the HDTP technology.
[0055] FIG. 18 suggests general ways in which two or more of these
independently adjustable degrees of freedom can be adjusted at
once.
[0056] FIG. 19 demonstrates a few two-finger multi-touch postures
and/or gestures from the many that can be readily recognized by
HTDP technology.
[0057] FIG. 20 shows how raw measurements of the six quantities of
FIGS. 17A-17F, together with shape recognition for distinguishing
contact with various parts of the hand and the touchpad, can be
used to create a rich information flux of parameters, rates, and
symbols.
[0058] FIG. 21 shows an approach for incorporating posture
recognition, gesture recognition, state machines, and parsers to
create an even richer human/machine tactile interface system
capable of incorporating syntax and grammars.
[0059] FIG. 22 depicts a user interface input arrangement
incorporating one or more HDTPs that provides user interface input
event and quantity routing of the type described earlier in
conjunction with FIGS. 16A-16B.
[0060] FIGS. 23A-23L depict a number of arrangements employing the
HDTP technology suitable for use with visualization
environments.
[0061] FIGS. 24A and 24B illustrate examples of scroll-wheel mice
provided with an additional scroll-wheel.
[0062] FIGS. 25A-25C illustrate a single trackball incorporated
into the back of a conventional computer mouse.
[0063] FIGS. 26A-26C illustrate two trackballs incorporated into
the back of a conventional computer mouse.
[0064] FIG. 27 depicts an arrangement wherein the HDTP or
alternatives such as advanced mice can interface with browser-based
applications via a browser plug-in.
[0065] FIG. 28 depicts an implementation for rendering
visualization in a browser.
[0066] FIG. 29 depicts a web-based implementation of a
visualization environment leveraging browser arrangements such as
that depicted in FIGS. 27-28, their variations, alternatives,
etc.
[0067] FIG. 30A depicts a client-side collaboration implementation
wherein a general purpose collaboration tool can be used to share
the visualization environment session running on one computer with
one or more other computers.
[0068] FIG. 30B depicts a server-side collaboration implementation
wherein a general purpose or specialized server-side application
sharing environment is used to share a server-based interactive
data visualization application.
[0069] FIG. 30C depicts a server-side collaboration implementation
comprising a server-based multi-user interactive data visualization
application.
[0070] FIG. 31 illustrates a general framework for data
sonification wherein a parameterized metaphor is used to manage
parameter assignment and sound rendering and a parameterized data
indexing operation.
[0071] FIG. 32 depicts an arrangement wherein the hue of a color
can be varied through natural colors of red through violet for the
bulk of the [0,1] range and a range of artificial colors, such as
the purple through magenta continuum, are appended for the
remaining portion of the [0,1] range.
[0072] FIG. 33 illustrates an arrangement wherein line width, line
dashing period, line dashing duty-cycle, and line dashing
sub-duty-cycle can be determined by a uniform parameter in the
range of [0,1].
[0073] FIG. 34 illustrates an arrangement wherein stipple line
width, stipple gap, and stipple angle can be determined by a
uniform parameter in the range of [0,1].
[0074] FIG. 35 depicts an arrangement wherein at least one border
rendering option is to be determined by a uniform parameter in the
range of [0,1]. The border can be the border of a spreadsheet cell
as depicted, or can be the border of another visually rendered
graphical object.
[0075] FIG. 36A depicts an arrangement wherein a boldness parameter
is under the control of uniform parameters in the range of [0,1].
FIG. 36B depicts an arrangement wherein an italics angle parameter
is under the control of uniform parameters in the range of [0,1].
FIG. 36C depicts an arrangement wherein an font family
interpolation parameter is under the control of uniform parameters
in the range of [0,1].
[0076] FIG. 37A depicts an arrangement wherein exemplary vertical
and horizontal dilation/contract parameters are each respectively
under the control of uniform parameters in the range of [0,1]. FIG.
37B depicts an arrangement wherein font underline line width and
font underline separation gap parameters are each respectively
under the control of uniform parameters in the range of [0,1].
[0077] FIG. 38 depicts an attribute control panel wherein there are
pre-selected values and a value which can be customized.
[0078] FIG. 39 depicts an arrangement wherein a change in color of
a font can be specified for a cell, a group of cells, or globally
across an entire worksheet.
[0079] FIG. 40 depicts an interface that determines a font type of
the specified cell(s).
[0080] FIG. 41 depicts an interface that determines the
embellishment of a font in rendering some data element within the
spreadsheet or data table.
[0081] FIG. 42 depicts an interface to control the size of a font
to be displayed.
[0082] FIG. 43 depicts an wherein data in a cells in a selected
column can have symbol appended to data within the cells of the
column.
[0083] FIG. 44 depicts an example use of controlling the background
texture or stipples of cells as a function of data values in a
spreadsheet or other form of tabular data.
[0084] FIG. 45 depicts an interface that can be used to specify the
background texture or stipples to be applied to a cell or a group
of cells.
[0085] FIG. 46 depicts an interface that can be used to specify the
position of text being displayed in a cell.
[0086] FIG. 47 depicts an interface that can be used to specify the
rotation of text being displayed combined with the vertical or
horizontal placement of the text.
[0087] FIG. 48 depicts a visualization wherein an attribute of a
column area of data elements is varied as a group.
[0088] FIG. 49 depicts a visualization wherein the shading of a of
a row area of data elements is varied as a group as a function of a
data value and also wherein the font boldness of a column area of
data elements is varied as a group as a function of another data
value.
[0089] FIG. 50 illustrates a priority job queue model tabular-data
visualization wherein the row can automatically reorder among
themselves as underlying priority vales change.
[0090] FIG. 51A illustrates a basic map-based visualization example
generated from previous data. FIG. 51B illustrates the
visualization of FIG. 51A augmented with additional spatially
located symbols or parameterized glyphs.
[0091] FIG. 52 depicts a 2D-surface representing a 3D data plot of
at least 3-dimensional data rendered above a planar array of
associated tabular data or spreadsheet, these separated by a
vertical gap within a 3D visual field as provided for by the
invention.
[0092] FIG. 53 depicts with dashed lines a few ways in which the
observation viewpoint can be moved with respect to these abstract
objects.
[0093] FIG. 54 depicts a translation and rotation of an exemplary
3D data plot within the 3D visual field.
[0094] FIG. 55A depicts an arrangement wherein data plotted in a
2D-surface representing a 3D data plot within the 3D visual field
can directly echo data displayed in a planar array of associated
tabular data or spreadsheet and/or can originate from another set
of tabular data or spreadsheet region and/or can originate from
other data.
[0095] FIG. 55B depicts an arrangement wherein processing of data
via mathematical transformations, statistical processing, signal
processing, etc. is made prior to creation of the 2D-surface
representing at least 3-dimensional data.
[0096] FIG. 56 illustrates a height measuring visual in its use
with 2D-surface representing at least 3-dimensional data and useful
for representing at least 4-dimensional data.
[0097] FIG. 57 depicts a slicing function here used to provide a
(planar-slice) level set curve.
[0098] FIG. 58A illustrates two 2D-surfaces, each representing at
least 3-dimensional data. FIG. 58B illustrates the interactively
created intersection of the two exemplary 2D-surfaces and the
resultant intersection curve.
[0099] FIG. 59 depicts an arrangement wherein a curve can be
generated by tabular data and be suspended over tabular data used
to create it or otherwise associated with the curve, and can be
visually linked to a geometric rendering of tabular data
values.
[0100] FIG. 60 depicts an arrangement wherein a curve can be given
a thickness and or color under the control of data values.
[0101] FIG. 61 depicts an arrangement wherein the thickness of a
curve can be varied as a function of one or more data values.
[0102] FIG. 62 depicts a curve intersection tool that can be used
as a numerical solution operator for plotted, interpolated, and
processed data.
DETAILED DESCRIPTION
[0103] In the following description, reference is made to the
accompanying drawing figures which form a part hereof, and which
show by way of illustration specific embodiments of the invention.
It is to be understood by those of ordinary skill in this
technological field that other embodiments can be utilized, and
structural, electrical, as well as procedural changes can be made
without departing from the scope of the present invention. The
aspects and features described herein may be used singly or in
combination unless specifically stated otherwise.
[0104] The visual data representation successes of spreadsheets
suggest additional opportunities for providing and combining
additional data visualization capabilities with the
well-established functionality and embedded deployment of
spreadsheet software. FIG. 1 provides an example of ways in which
data visualization and spreadsheets can be integrated. The present
invention provides several additional data visualization
capabilities to traditional spreadsheets.
[0105] First, traditional spreadsheets present data in a tabular
form leveraging row, column, and sheet organization. Traditional
spreadsheets typically include plotting routines that create
graphical representation of selected data points. These plots can
be used to provide visual representations of data and mathematical
functions. These spreadsheet functions provide a base-level form of
data visualization based on data in the spreadsheet, as represented
in rows A and B of FIG. 1.
[0106] Additional subsequent work has been done to employ data
visualization characterizing internals of spreadsheet structure
such as information flow, cell formula dependences, and semantic
classification. These do not provide visualization of data per se,
but can be very useful in understanding the data handling of a
given spreadsheet. This is represented in rows C and D of FIG.
1.
[0107] Further subsequent work has been done to supplement basic
spreadsheet plotting utilities providing richer data plotting
capabilities (for example, utilizing 3D graphics). These
capabilities enhance the possible data visualization based on data
in the spreadsheet. This is represented in row E of FIG. 1.
[0108] Yet other subsequent work has been done adopting the
interactive format and metaphor of a spreadsheet for use in a
tabular presentation of complex data visualization renderings. This
is represented in row F of FIG. 1. One aspect of the invention
focuses on adapting and significantly expanding this capability.
This is represented in row G of FIG. 1.
[0109] Still other subsequent work has been done superimposing a
third (height) dimension atop the 2-dimensional tabular data layout
of an interactive electronic spreadsheet. The third dimension can
be used to render color-coded line and surface plots, and the
resulting 3D graphics can be viewed from various virtual
observation points for inspection. This is represented in row H of
FIG. 1. Another aspect of the invention focuses on adapting and
expanding this capability.
[0110] Traditional spreadsheets also present numerical, textual,
and symbolic data in a tabular array of cells, with the numerical,
textual, and/or symbolic data within the cells provided in various
font styles and colors, and with various background colors and
border styles of the associated cell. Using spreadsheet functions
known as "conditional formatting" these font styles and colors,
cell background colors, and cell border styles can be controlled by
values of data according to hand-specified (through spreadsheet
GUI, VBA, APIs, etc.) conditional tests performed on data within
one or more specified cells. These spreadsheet functions provide
another form of data visualization based on data in the
spreadsheet. One aspect of the invention focuses on adapting and
expanding this capability. This is represented in row I of FIG.
1.
[0111] FIG. 2A illustrates a region 200 of a traditional
spreadsheet or table of tabular data. Such spreadsheet or tabular
data can be created by desktop applications such as Microsoft Excel
or online applications such as Google Spreadsheet. Aspects of the
present invention can be implemented via an add-on package of
existing software or additional plug-in script on any online
applications. The minimum block that makes up this whole region is
referred to as a cell, and cells can be merged to compose a cell
with bigger area expanding multiple rows, columns, or both. Region
200 of an example spreadsheet contains four columns 281-284 and
five rows 291-295. Cell 230 is confined within one row 293 and on
column 283 and accordingly is defined by a left border 213, a right
border 214, an upper border 222, and a lower border 223. Each of
the four columns 281-284 represents a different categorical subset
of data (for example, an item, region, revenue area, inventory
level, etc.). In contrast to exemplary cell 230, another exemplary
cell 240 in the left-most column comprises the height of three rows
293, 294, 295.
[0112] The border of cell 230 is bolded with respect to the other
cells depicted so as to call this cell out from the others. This
can be rendered for any of a number of reasons, such as signifying
the selection of the cell via a pointing device of a user
interface, selection resulting from a query, the signifying of
special status or conditions, etc. The data displayed in cell 230
is a decimal number formatted as a national financial currency and
with a thousands-separating comma. In contrast, the data displayed
in cell 260 is an unformatted non-decimal number without a
thousands-separating comma. Also in contrast, the data displayed in
cell 250 comprises a text string, while the data displayed in cell
240 comprises an alphabetic text string and an unformatted
non-decimal number. In the latter, the unformatted non-decimal
number can in fact be part of the preceding text string, or can be
a calculated, retrieved, or can be a fixed number composed with a
calculated, retrieved, or fixed text string.
[0113] In general the data displayed in the foreground of a cell
can contain numbers, text, symbols, or a combination of these, and
this data or portions of it can be further "emphasized" with
attributes. Such attributes include choice of the font family, font
size, bolding, italicizing, stretching, color, transparency with
respect to the cell background, position within the cell, various
types of animations (modulations of aforementioned attributes,
blinking, shimmering, vibrating, rotating, etc.) Each individual
font attribute can be fixed or determined by data values via
conditional formatting. The data displayed in the foreground of a
cell can be fixed, replicated, or result from processing of other
data by (no displayed) formulas and/or functions.
[0114] FIG. 2B depicts additional detail of cell 230 and its
immediate neighborhood. In order to render the bolded border of the
cell, the dividing border lines 213, 214, 222, 223 of FIG. 2A can
be segmented into a plurality of sections--for example dividing
border line 213 of FIG. 2A can be segmented into at least adjacent
sections 213a, 213b, 213c; similarly dividing border line 214 of
FIG. 2A can be segmented into at least adjacent sections 214a,
214b, 214c, dividing border line 222 of FIG. 2A can be segmented
into at least adjacent sections 222a, 222b, 222c, and dividing
border line 223 of FIG. 2A can be segmented into at least adjacent
sections 223a, 223b, 223c. The exemplary cell 230 is visually
defined as the geometric area enclosed by upper, lower, left, and
right dividing border lines 213b, 214b, 222b, and 223b--these
border line sections can be accented via thickening, darkening,
coloring, etc. so as to visually contrast with the other sections
213a, 213c, 214b, 214c, 222a, 222c, 223a, 223c. Such border line
section accenting can be fixed or determined by data values via
conditional formatting.
[0115] The background 245 of each cell can be used to display a
background color, shading pattern, image, etc. which can be fixed
or determined by data values via conditional formatting. The
background can vary from cell to cell.
[0116] The displayed data in this particular case is, again, a
decimal 241 number formatted as a national financial currency via a
signifying currency symbol 242 and further employing a
thousands-separating comma 243.
[0117] FIG. 3 depicts of data and cell presentation attributes
serving as candidates for visualization parameters within a
spreadsheet display. Some of these attributes can be controllable
under conditional formatting in traditional spreadsheets while
others may not be controllable under conditional formatting in
traditional spreadsheets. In one aspect of the invention, a data or
cell presentation attribute provided a pre-defined range of
variability and is controlled by a parameter that can vary over a
pre-defined range (for example, between 0 and 1, between 0 and 100,
between -1 and +1, etc). This parameter-controlled variability of a
data or cell presentation attribute allows that presentation
attribute to be used as a visualization parameter.
[0118] For those data and cell presentation attributes that have
been controllable under conditional formatting in traditional
spreadsheets, the parameter-controlled variability provided for by
the invention is fundamentally different in a number of ways. A few
of these include: [0119] Conditional formatting provides a
particular custom-specified result for satisfaction of a
custom-specified rule; [0120] For each individual result desired, a
corresponding hand-specified rule must be entered; for example in
order to span 64 color steps, 64 hand-specified rules must be
hand-entered; [0121] In the case of colors, shades, fonts,
stipples, etc., conditional formatting provided (or extended by
VBA, APIs, etc.) in traditional spreadsheets offers only a small
discrete set of choices. In the case of colors, Microsoft Excel for
example provides a fixed pre-selected collection of 56 colors, some
of which are duplicates. Even if Microsoft Excel provided 64
colors, this would only permit 4 steps per primary color in an RGB
color model or a very limited color wheel in a HSB color model.
Additionally, the fixed collection of pre-selected colors will
always have a significant portion of colors that will be unusable
as foreground or background colors as they will not be able to
visibly stand out, respectively, against a corresponding range of
background or foreground colors. Thus there are typically
considerably less than the full collection of pre-selected colors
to work with in a given color-control data presentation
visualization if numeric or text information is co-displayed.
Clearly the present invention provides a far superior approach with
greater capability, practicality, and ease of use.
[0122] Among other capabilities, the invention provides new
visualization capabilities to spreadsheet presentation, spreadsheet
formatting, spreadsheet data handling, and spreadsheet interactive
use in ways well-suited for use as a numeric-intensive data
visualization tool. Many of these spreadsheet features can be
implemented with simple vector graphics rendering operations or
easily managed mixtures of vector and raster graphics, permitting:
[0123] Implementation of a number of the invention's spreadsheet
presentation features through use of APIs of existing spreadsheet
products; [0124] Implementation of many of the invention's
spreadsheet presentation features with relatively small
augmentation and modification of existing product software by
product manufacturers; [0125] Implementation of most of the
invention's spreadsheet presentation features with well-established
graphics utilities such as SVG (http://www.w3.org/TR/SVG11/) and
the graphics utilities of various computer operating systems
provided by companies such as Microsoft and Apple; [0126] Via these
and/or other graphics utilities, implementation of most of the
invention's spreadsheet presentation features in browser-rendered
web applications, thus making the invention's spreadsheet
presentation features available in a web page; [0127] Via web
and/or other implementation approaches, implementation of the
invention's presentation features in a collaborative viewing
environment.
[0128] The invention provides for a value of a data element, or
result of a formula calculated from one or more data elements in a
spreadsheet or data table, to be used to determine spreadsheet
presentation attributes, such as one or more of the following
examples:
[0129] Attributes of a Font and/or Symbol: [0130] the color of a
font employed in rendering some data element within the spreadsheet
or data table; [0131] the opacity of a font employed in rendering
some data element within the spreadsheet or data table; [0132] the
selection of a font employed in rendering some data element within
the spreadsheet or data table; [0133] the embellishment of a font
(bolding, italicizing, underlining, etc.) employed in rendering
some data element within the spreadsheet or data table; [0134] the
size of a font employed in rendering some data element within the
spreadsheet or data table;
[0135] Attributes of Appended Symbols: [0136] the condition for the
addition of an appending symbol or string employed in rendering
some data element within the spreadsheet or data table; [0137]
where to append symbol or string [0138] color(s) of symbol or
string [0139] opacity of symbol or string [0140] other parameters
of the symbol or contents of string;
[0141] Attributes of Numerical Quantization of Cell Data: [0142]
the number of decimal places employed in rendering some data
element within the spreadsheet or data table; [0143] the method of
numerical rounding employed in rendering some data element within
the spreadsheet or data table; [0144] the degree of numerical
rounding employed in rendering some data element within the
spreadsheet or data table; [0145] the selection of a symbolic data
element rendered within the spreadsheet or data table; [0146] the
selection of an image element rendered within the spreadsheet or
data table;
[0147] Attributes of a Cell Border: [0148] the color of at least
one border of a cell enveloping a font employed in rendering some
data element within the spreadsheet or data table; [0149] the
thickness/width of at least one border wall of a cell enveloping a
font employed in rendering some data element within the spreadsheet
or data table; [0150] the thickness/width of at least one border
outlining of a plurality of contiguous cells, where each cell
envelopes a font employed in rendering of data elements within the
spreadsheet or data table;
[0151] Attributes of Cell Interior: [0152] the background color of
a cell enveloping some data element within the spreadsheet or data
table; [0153] the background texture of a cell enveloping some data
element within the spreadsheet or data table; [0154] the location
of data element within a cell enveloping that data element within
the spreadsheet or data table;
[0155] Attributes of Rows & Columns: [0156] an attribute of a
column employed in rendering a plurality of data elements within
the spreadsheet or data table; [0157] an attribute of a row
employed in rendering a plurality of data elements within the
spreadsheet or data table. [0158] an attribute of a row-by-column
area employed in rendering a plurality of data elements within the
spreadsheet or data table.
[0159] The invention also provides for one or more of the following
capabilities: [0160] reordering of tabular rows and/or columns as a
function of data values; [0161] superimposing a third (height)
dimension atop the 2-dimensional tabular data layout of an
interactive electronic spreadsheet. [0162] adopting the interactive
format and metaphor of an interactive electronic spreadsheet for
use in a tabular presentation of complex data visualization
renderings;
[0163] Additionally, the invention provides for the control of data
and cell presentation attributes through use of a uniform
parameterization framework. This allows pre-visualization
operations, such as scaling, translation, filtering, array (matrix,
tensor) operations, nonlinear warping, etc. to be employed in a
modular, cascadable fashion independent of the particular choice of
data and cell presentation attributes. The invention further
provides for pre-visualization operations to themselves have
parameters that can be adjusted in real time and/or be stored in
files for recall. The invention further provides for a network of
pre-visualization operations to be stored in files for recall.
[0164] The invention additionally provides for advanced user
interface devices, particularly those providing large numbers of
simultaneously-adjustable interactive control parameters, to be
used to control the viewing, presentation, and creation of the
visualization as well as controlling the underlying data source
such as databases, statistical packages, simulations, etc.
[0165] Further, via web and other implementation approaches, the
implementation of the invention's presentation features in a
collaborative interactive use environment.
[0166] The invention thus can serve a variety of callings. FIGS.
4a-4e illustrate several example application settings where the
invention is well-suited to add value. In one scenario, depicted in
FIG. 4A, a spreadsheet is initially used as a tool of choice or
convenience 401, and at some point some simple data visualizations
are incorporated 402. Such capabilities can be within the range
provided by available spreadsheet software products 400. As study
of the data or development of the model intensifies, additional
degrees of visualization are incorporated 403, potentially
involving a need for a rich suite of visualization capabilities to
create data visualization of substantial richness 404. In FIG. 4B,
a spreadsheet is chosen as a vehicle or venue for visualization as
it matches one dominating user needs (for example, original data
already in or readily imported into a spreadsheet, quick assembly,
tabular nature of data, tabular nature of formula replication,
etc.). In FIG. 4C, a spreadsheet is chosen as a vehicle or venue
for visualization as it is comfortable or intuitive for a
particular audience. In FIG. 4D, a spreadsheet is chosen as a
vehicle or venue for visualization as other aspects of the
invention allow for it to be rendered in a web-based application.
In FIG. 4E, a spreadsheet is chosen as a vehicle or venue for
visualization as other aspects of the invention allow for it to be
rendered in a collaborative context.
[0167] The spreadsheet visualization aspects of the present
invention are a special case of other more general aspects of the
invention including: [0168] the use of arbitrary or integrated data
sources (such as static databases, dynamic databases, streaming
databases, live sensing data streams, numerical simulations, signal
processing, statistical processing, linear and nonlinear
transformations, etc.); [0169] uniform parameterizations of
selected or all visualization presentation parameters; [0170] the
support for real-time updates to integrated data sources (such as
static databases, dynamic databases, streaming databases, live
sensing data streams, numerical simulations, signal processing,
statistical processing, linear and nonlinear transformations,
etc.); [0171] the use of data flow paths to link arbitrary data
sources with arbitrary data destinations via arbitrary topologies
(graphically, via an interconnection, specification, and/or
data-flow language, etc.); [0172] the providing of shared GUI
environments for controlling two or more of visualization
rendering, pre-visualization operations, and data sources.
[0173] FIG. 5 illustrates an arrangement of these more general
aspects of the invention. Implicit in FIG. 5 are more general
aspects of the invention, not restricted to spreadsheet
visualization, that support visual rendering in a browser window
and as a web application. As shown in the figure, graphical user
interfaces (GUIs) are provided for including one or more relating
to data sources and one or more relating to visualization
processes. Additionally, the invention also provides for
multifunction GUIs that provide roles involving both data source
matters and visualization processes. The overall collection of user
interfaces provides the following capabilities: [0174] Setup of a
fixed data analysis configuration; [0175] Setup of fixed data
visualization configuration; [0176] Setup of fixed data
visualization presentation; [0177] Setup of an interactive data
analysis configuration; [0178] Setup of an interactive data
analysis session; [0179] Setup of an interactive data visualization
configuration; [0180] Setup of an interactive visualization
session; [0181] Interactive control of data source selection and
usage; [0182] Interactive control of data analysis sessions; [0183]
Interactive control of visualization sessions; [0184] Storage and
recall of configurations; [0185] Storage and recall of data; [0186]
Storage and recall of visualizations; [0187] Storage and recall of
sessions.
[0188] Yet additional aspects of the invention, not restricted to
spreadsheet visualization, also include (real-time and
non-real-time) collaboration capabilities. FIG. 6 illustrates an
adaptation of the arrangement of FIG. 5 to a collaboration
environment supporting collaboration features such as one or more
of file exchange, real-time or stored display sharing, real-time
control sharing, real-time or stored collaborative annotation, and
archives of collaboration sessions and transactions.
Uniform Parameterizations of Cell and Data Presentation
Attributes
[0189] The invention provides for uniform parameterizations of
selected or all visualization presentation parameters. This allows
pre-visualization operations, such as scaling, translation,
filtering, array (matrix, tensor) operations, nonlinear warping,
etc. to be employed in a modular, cascadable fashion independent of
the particular choice of data and cell presentation attributes.
Data Flow Paths to Implement Arbitrary Interconnection
Topologies
[0190] The invention provides for the use of data flow paths to
link arbitrary data sources with arbitrary data destinations via
arbitrary topologies. This allows the selection and/or fusion of
data sources, their interconnection with selected signal
processing, statistical processing, pre-visualization operations,
and visualization parameters (such as, among other examples, the
cell and data presentation parameters of the spreadsheet
visualization described earlier and in more detail to follow).
[0191] FIG. 7 depicts an exemplary topological interconnection of
data flow paths linking various elements depicted in FIG. 5. In
various embodiments, functions such as data re-indexing,
statistical processing, and signal processing can be provided as
the data sources depicted in FIG. 5 or as the pre-visualization
functions depicted in FIG. 5. Similarly, numerical simulations, for
example (but not limited to) those rendered by the computer running
a spreadsheet program, a high-performance or other computer, can
serve as the data sources depicted in FIG. 5. Certain
pre-visualization functions, for example linear predictors, can in
an embodiment be regarded as a numerical simulation. Additionally,
in a spreadsheet visualization setting, calculations in the
spreadsheet can provide a form of numerical simulation prior to
visualization.
[0192] The invention provides for some or all of the data flow
paths (such as depicted in the example of FIG. 7) to be specified
in any convenient way, for example graphically via an interactive
GUI or via a character-based language (interconnection,
specification, and/or data-flow, etc.). In an exemplary embodiment,
a GUI can permit the rendering of a graphic similar to that of FIG.
7. A GUI can permit creation and customization of instances of
functional blocks such as the ones depicted in FIG. 7 from a
library, menu, and/or graphical pallet. In an exemplary embodiment,
a GUI can be used to create and link these customized instances of
functional blocks, via link-by-link "drawing," with a data path
topology such as the ones depicted in FIG. 7.
[0193] Various types of user interfaces may be used to create
configurations such as that of FIG. 7 and set the various
parameters within it. As one example approach, the selection of
function blocks and the general connectivity among them can be
specified using a drawing tool and a palette of function blocks,
and clicking on each function block would cause dialog windows to
appear that can be used for setting parameters. The parameters can
be set by means of typed-in values, sliders, mouse manipulation, or
the advanced high-dimension user interface devices to be described
later in the specification. More generally, a wide range of
approaches, styles, and forms of configuration-specification,
feature-selection, and parameter-setting GUIs are known in the art.
A few example user interfaces or aspects of them will be mentioned
to showcase different aspects, but a comprehensive treatment of
user interface approaches is not necessary for enablement as
general forms of configuration specification GUIs and parameter
setting GUIs are known in the art. A number of example
configuration-specification, feature-selection, and
parameter-setting GUIs will be the subject of a later companion
patent application.
Pre-Visualization Operations
[0194] Attention is now directed to consideration of
pre-visualization operations. FIGS. 8A-8C depict an approach for
mapping a data value lying within a pre-defined range to a value
within a pre-defined range for a parameterized data or cell
presentation attribute. In most cases the input data range must be
at least scaled and shifted to match the pre-defined range for a
parameterized presentation attribute. This (linear or affine
transformation) arrangement is depicted in FIG. 8A. In some
circumstances it can also be desirable to warp the data range with
nonlinearity. In one embodiment a library of fixed or adjustable
nonlinearities are provided which are such that the input and
output of the nonlinearity both match the pre-defined range for a
parameterized presentation attribute. This arrangement is depicted
in FIG. 8B. In another embodiment the warping effect is provided
with additional flexibility by allowing pre-scaling and
pre-shifting prior to applying a selected nonlinearity and
subjecting the outcome of the nonlinear warping to post-scaling
and/or post-shifting operations in order to match the resulting
range to the pre-defined range for a parameterized presentation
attribute. This arrangement is depicted in FIG. 8C. Features and
parameters of the above operations can be selected via user
interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0195] FIG. 9 depicts a more general view and organization of
pre-visualization operations provided for by the invention. In this
example, available pre-visualization operations include: [0196]
Data indexing/re-indexing, data sorting, data suppression, and
similar types of data operations; [0197] Normalization, shifting
(translation), and other types of linear and affine
transformations; [0198] Linear filtering, convolution, linear
prediction, and other types of signal processing operations; [0199]
Warping, clipping, nonlinear transformations, nonlinear prediction,
and other nonlinear transformations. [0200] The invention provides
for other types of pre-visualization operations as well. Features
and parameters of the above operations can be selected via user
interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0201] The invention also provides for the inclusion of statistical
operations and statistical processing functions and for the linking
to external programs to perform other types of pre-visualization
operations. Features and parameters of these can be selected via
user interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0202] The invention also provides for external programs to be
added to the collection of available pre-visualization
operations.
[0203] The invention additionally provides for the inclusion and
use of visual metaphors to simplify visualization setup and user
interaction for data exploration. As an example, FIG. 10 depicts an
exemplary embodiment wherein a selected metaphor is used to
automatically generate parameter assignments and graphics rendering
operations. The invention provides for metaphors to control other
aspects of the visualization and pre-visualization operations. The
invention provides for a metaphor to base its operations on
characteristics of a data set being visualized, previously
visualized, and anticipated to be visualized. The invention
additionally provides for metaphors to be selected and controlled
by user interaction, data values, or other means. Features and
parameters of the above operations can be selected via user
interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art. The invention also
provides for array (vector, matrix, tensor) operations such as
(vector, matrix, tensor) linear combinations, (vector, matrix,
tensor) multiplication, scalar multiplication, finding (matrix,
tensor) determinants, finding (matrix, tensor) inverses and
psuedoinverses, row reduction, factorization, change of basis, or
calculation of an eigensystem (eigenvalues, eigenvectors,
eigentensors).
[0204] Users can select the region of the spreadsheet to be stored
in an array and give a name to the array. Different programs can
require different formatting of arrays, thus, users can also choose
the separating characters, such as a comma and a space or tab. The
key of the array can be numbers from 0 to (the number of
entries--1) or the values of any column user selects. Such array
created can be exported to an external program. Similarly, data of
one or multiple arrays can be implemented to be imported from an
external program and the values of elements are displayed in
appropriate numbers of rows and columns. FIG. 11A illustrates an
example of a user interface that permits users to select the
geometric element boundary of a captured or rendered tabular matrix
data elements or cells within a spreadsheet. Other features and
parameters of the above operations can be selected via user
interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0205] In the case of spreadsheet visualization, spreadsheets
natively provide certain types of array calculations with relative
ease while other standard matrix and array operations, such as
matrix/tensor multiplication, eigensystem calculations,
convolutions, etc., have no native support within the spreadsheet
paradigm. The invention provides for a matrix boundary tool for use
in capturing and labeling matrix data and rendering matrix
operation results. For example, users can specify the group of
cells as the elements of one matrix or multiple matrices. FIG. 11B
illustrates an exemplary user interface that permits users to
select array operations to be performed on selected array data, for
example as may be selected by the tool of FIG. 11A. Other features
and parameters of the above operations can be selected via user
interface dialog windows. General forms
configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0206] The invention can additionally transform a graphics object
via matrix operations. As examples, FIG. 12A illustrates a figure
being resized, rotated, and shifted through array-based scaling,
rotating, and translating linear transformations. FIG. 12B
illustrates processes of a figure being rotated around x-axis,
y-axis, line of y=x, and line of y=-x. and FIG. 12c illustrates
processes of figure being rescaled vertically or horizontally. FIG.
12D illustrates a transformation process acting on 2D graphics
objects such as a rectangle-bounded horizontal line array and a
graphic. The invention also provides for embodiments to extend
these and related systems and methods to transform 3D-graphics via
scalings, rotations, stretch, etc.
[0207] The invention provides for dimensional transformations among
points, 1D, 2D, and 3D graphics objects. FIG. 12E illustrates a
matrix and/or tensor transformation process and how it changes a 3D
graphics object into a 2D object via dimension dropping. FIG. 12F
illustrates a 3D graphics object rendered of a 2D surface created
by a vertical line rotated around the y-axis. FIG. 12G illustrates
a 3D graphics object rendered by perspective projection defined by
a point.
[0208] The invention also provides for 3D graphics generation from
3D vector draw-lists provided by other programs, spreadsheet data,
and/or the user. In an embodiment, the invention also provides for
3D graphics generation from equations provided by the user.
[0209] The invention further provides for pre-visualization
operations to themselves have parameters that can be adjusted in
real time and/or be stored in files for recall.
[0210] The invention further provides for a network of
pre-visualization operations to be stored in files for recall.
Spreadsheet Visualization Integration Architectures
[0211] The invention provides implementation of at least some of
the invention's spreadsheet presentation features through use of
APIs of existing spreadsheet products. The invention provides
implementation of at least some of the invention's spreadsheet
presentation features with relatively small augmentation and
modification of existing product software by product
manufacturers.
[0212] FIG. 13 depicts an example of spreadsheet architecture
(right side) and an information flow as it might be used in a data
visualization setting. One of or some combination of entered data,
measured data, and/or computed data can be stored as stored data or
stored in a queriable data base. In some situations live data feeds
can also provide data. One or some combination of these data
sources can be used to provide a data set used by the spreadsheet
program. The spreadsheet program internally comprises at least a
GUI component, a calculation compound, and a visual rendering
component.
[0213] FIGS. 14A-14D depicts various types of exemplary
relationships and information flows among a system or method for
rendering a conventional spreadsheet or table of tabular data, a
system or method for rendering the visualization operations of the
invention, and a data set used by each. Visualization and
pre-visualization operations can be introduced between the data and
the spreadsheet program as depicted in FIG. 14A. In another
embodiment, visualization and pre-visualization operations can
obtain data from the spreadsheet program, process data, and return
the generated visual operation back to the spreadsheet program as
depicted in FIG. 14B. An example of this is using an API or the use
of VBA in conjunction with Microsoft Excel.
[0214] Visualization and pre-visualization operations can also be
implemented as parallel operations to the spreadsheet program.
Visualization and pre-visualization operations can be configured
separately as an independent operation(s) with direct access to
data (i.e., able to obtain data and send the result to the data
without having to go through the spreadsheet program), as shown in
FIG. 11C. One advantage of this is that relevant categories of data
can be extracted to generate complicated visualizations without the
data being arranged into table form. Visualization and
pre-visualization operations can be part of a spreadsheet program
without direct access to the data, as depicted in FIG. 14D.
Spreadsheet Visualization User Interface Architectures: Traditional
and High-Dimension User Interface Devices (HDTP and Advanced
Mice)
[0215] Next, user interface architectures for general visualization
and spreadsheet visualization environments are considered. FIG. 15
depicts an exemplary embodiment of an augmentation of a spreadsheet
program with additional spreadsheet features as provided for by the
invention. A data set is subjected to data operations so as to
produce modified data. The data operations can be controlled by the
spreadsheet GUI and spreadsheet calculation as well as by the
visualization GUI and Visualization calculations.
[0216] FIGS. 16A-16B depict ways in which one or more user
interface input devices (such as a mouse, trackball, touchpad,
etc.) can be managed to control graphical user interfaces of a
plural software application in the context of the invention. FIG.
16A, the windowing system directs the input events from a single
user interface input device to various applications, for example, a
Database GUI, a Simulation GUI, a Spreadsheet GUI, and a
Visualization GUI. FIG. 16B depicts an alternative implementation
wherein a Visualization Environment Input Router/Parser accepts
input events from one or more user interface input devices and
directs these events to each of the various applications. The
Visualization Environment Input Router/Parser can direct events
from multiple user interface input devices to different inputs to
the same application. In another embodiment the Visualization
Environment Input Router/Parser can split events originating from
an individual user interface device and separately route these to
inputs of separate applications. In another embodiment, a
Visualization Environment Input Router/Parser works together with
the Window System to route input events.
[0217] The above arrangements are general and apply to use of
conventional mice, trackballs, touchpads, etc. However,
visualization and CAD workstations have often been provided with
more sophisticated user input devices that provide a higher number
of interactive simultaneously-adjustable parameters. Classic
examples of this are knob-boxes (as used in HP and SGI
workstations), the DataGlove (VPL, General Reality), the SpaceBall
(Logitech3Dconnexion, Labtec, HP/Compaq), etc., although few of
these have survived product cycles to remain in active use or with
wide availability. More recently enhanced touch-based interfaces
have attracted a great deal of attention, mostly for their
multi-touch and gesture recognition capabilities. However, some
enhanced touch-based interfaces such as the HDTP ("High Dimensional
Touch Pad," U.S. Pat. No. 6,570,078; U.S. patent application Ser.
Nos. 11/761,978 U.S. Ser. No. 12/418,605, and U.S. Ser. No.
12/502,203 among others) employ a tactile sensor array (pressure,
proximity, etc.) and real-time image and mathematical processing to
provide a powerful user input device with both a higher number of
interactive simultaneously-adjustable parameters and a rich range
of syntactic and metaphorical capabilities well-suited to use with
interactive visualization. Additionally, the HDTP technology can be
readily implemented as a touchscreen through use of, for example,
inexpensive transparent capacitive proximity-sensor arrays. The
present invention provides for the incorporation and use of the
HDTP and other metaphor-rich user interface input devices into
visualization environments. Attention is now directed to
incorporation of the HDTP as part of a visualization environment
user interface and then brief attention is directed to alternative
use of a few alternative metaphor-rich enhanced user interface
technology approaches.
[0218] FIGS. 17A-17F illustrate the six independently adjustable
degrees of freedom of touch from a single finger that can be
simultaneously measured by the HDTP technology. The depiction in
these figures is from the side of the touchpad. FIGS. 17A-17C show
actions of positional change (amounting to applied pressure in the
case of FIG. 17C) while FIGS. 17D-17F show actions of angular
change. Each of these can be used to control a user interface
parameter, allowing the touch of a single fingertip to control up
to six simultaneously-adjustable quantities in an interactive user
interface. FIG. 18 suggests general ways in which two or more of
these six independently adjustable degrees of freedom 1811-1816 can
be adjusted at once for a finger 1800 in contact with an HDTP
sensor surface 1801. More advanced implementations of the HTDP
provide for multi-touch capabilities far more sophisticated than
those popularized by the Apple iPhone, NYU, and others.
[0219] FIG. 19 demonstrates a few exemplary two-finger multi-touch
postures and/or gestures from the hundreds that can be readily
recognized by HTDP technology. The two fingers may be spread apart
somewhat, as in the cases 1901-1903 and 1931-1933, or may be
brought together, as in the cases 1911-1913 and 1941-1943. The
separation of the two fingers adds an additional parameter that can
be controlled by the user, and the yaw (pivot) parameter may be
varied independently from the newly introduced finger separation
parameter. For example the hand can pivot clockwise 1902, 1912, or
counter-clockwise 1903, 1913 from the corresponding neutral
positions 1901, 1911. Similarly, the pitch of the pair of fingers
can be raised 1932, 1942 or lowered 1933, 1943 with respect to the
corresponding neutral positions 1931, 1941, while independently
from these motions the two fingers are spread apart 1931-1933 or
brought together 1941-1943. Note the finger spread can be varied
considerably in this two-finger posture.
[0220] The pair of fingers can be moved as a group through all six
degrees of freedom (left-right, forward-back, downward pressure,
roll, pitch, yaw), and comfortably allow for two differences
between the fingers (differences in downward pressure, and one
angle of finger separation spread) and even three differences
between the fingers (differences in downward pressure and two
coordinates of separation when fingers curl to form independently
controlled "x" and "y" components). Thus two-finger postures
considered above can readily provide a nine-parameter set relating
to the pair of fingers as a separate composite object adjustable
within an ergonomically comfortable range. One example
nine-parameter set the two-finger postures comprises: [0221]
composite (group) average x position; [0222] inter-finger
differential x position; [0223] composite (group) average y
position; [0224] inter-finger differential y position; [0225]
composite (group) average pressure; [0226] inter-finger
differential pressure; [0227] composite (group) roll; [0228]
composite (group) pitch; [0229] composite (group) yaw.
[0230] HTDP technology can also be configured to recognize and
measure postures and/or gestures involving three or more fingers,
various parts of the hand, the entire hand, multiple hands, etc. In
general, multifinger contact can be used to provide control of up
to three additional independently adjustable parameters for each
additional finger.
[0231] FIG. 20 shows an example of how raw measurements of the six
quantities of FIGS. 17A-17F, together with shape recognition for
distinguishing contact with various parts of the hand and the
touchpad, can be used to create a rich information flux of
parameters, rates, and symbols. FIG. 21 shows an exemplary approach
for incorporating posture recognition, gesture recognition, state
machines, and parsers to create an even richer human/machine
tactile interface system capable of incorporating syntax and
grammars.
[0232] FIG. 22 depicts an exemplary user interface input
arrangement incorporating one or more HDTPs that provides a user
interface input event and quantity routing of the type described
earlier in conjunction with FIGS. 16a-16b.
[0233] FIGS. 23A-23L depict a number of arrangements employing the
HDTP technology suitable for use with visualization environments.
FIG. 23A illustrates an HDTP as a peripheral that can be used with
a desktop computer (shown) or laptop (not shown). FIG. 23B depicts
an HDTP integrated into a laptop in place of the traditional
touchpad pointing device. In FIGS. 23A-23B the HDTP tactile sensor
can be a stand-alone component or can be integrated over a display
to form a touchscreen. FIG. 23C depicts an HDTP integrated into a
desktop computer display to form a touchscreen. FIG. 23D shows the
HDTP integrated into a laptop computer display to form a
touchscreen. FIG. 23E depicts an HDTP integrated into a cellphone,
smartphone, PDA, or other hand-held consumer device. FIG. 23F shows
an HDTP integrated into a test instrument, portable
service-tracking device, portable service-entry device, field
instrument, or other hand-held industrial device. In FIGS. 23E-23F
the HDTP tactile sensor can be a stand-alone component or can be
integrated over a display to form a touchscreen. FIG. 23G depicts
an HDTP touchscreen configuration that can be used in a tablet
computer, wall-mount computer monitor, digital television, video
conferencing screen, kiosk, etc. In at least the arrangements of
FIGS. 23A, 23C, 23D, and 23G, or other sufficiently large tactile
sensor implementation of the HDTP, more than one hand can be used
and individually recognized as such.
[0234] FIGS. 23H-23K depict various integrations of an HDTP into
the back of a conventional computer mouse. In FIGS. 23H-23K the
HDTP tactile sensor can be a stand-alone component or can be
integrated over a display to form a touchscreen. More than two
touchpads can be included, such as the example of FIG. 23L as
taught in U.S. Pat. No. 7,557,797.
[0235] The types of human-machine geometric interaction between the
hand and the HDTP facilitate many useful applications within a
visualization environment. A few of these include control of
visualization observation viewpoint location, orientation of the
visualization, and controlling fixed or selectable ensembles of one
or more of viewing parameters, visualization rendering parameters,
pre-visualization operations parameters, data selection parameters,
simulation control parameters, etc. As one example, the 6D
orientation of a finger can be naturally associated with
visualization observation viewpoint location and orientation,
location and orientation of the visualization graphics, etc. As
another example, the 6D orientation of a finger can be naturally
associated with a vector field orientation for introducing
synthetic measurements in a numerical simulation. As another
example, at least some aspects of the 6D orientation of a finger
can be naturally associated with the orientation of a robotically
positioned sensor providing actual measurement data. As another
example, the 6D orientation of a finger can be naturally associated
with an object location and orientation in a numerical simulation.
As another example, the large number of interactive parameters can
be abstractly associated with viewing parameters, visualization
rendering parameters, pre-visualization operations parameters, data
selection parameters, numeric simulation control parameters,
etc.
[0236] In another example, the x and y parameters provided by the
HDTP can be used for focus selection and the remaining parameters
can be used to control parameters within a selected GUI.
[0237] In another example, the x and y parameters provided by the
HDTP can be regarded as specifying a position within an underlying
base plane and the roll and pitch angles can be regarded as
specifying a position within a superimposed parallel plane. In a
first exemplary extension of the previous two-plane example, the
yaw angle can be regarded as the rotational angle between the base
and superimposed planes. In a second exemplary extension of the
previous two-plane example, the finger pressure can be employed to
determine the distance between the base and superimposed planes. In
a variation of the previous two-plane example, the base and
superimposed planes are not fixed as being parallel but rather
intersect at an angle associated with the yaw angle of the finger.
In each of these examples, either or both of the two planes can be
used to represent an index or indexed data, a position, pair of
parameters, etc. of a viewing aspect, visualization rendering
aspect, pre-visualization operations, data selection, numeric
simulation control, etc.
[0238] A large number of other examples are possible as is
appreciated by one skilled in the art.
[0239] One use of the HDTP in the above examples is simply to
supply more than the usual two user interface parameters provided
by a conventional user interface input device such as a
conventional computer mouse, trackball, touchpad, etc. The present
invention provides for the use of other user interface input
arrangements and devices as alternatives to or in conjunction with
one or more HDTPs.
[0240] In a simple example, the scroll-wheel of a scroll-wheel
mouse is used to provide a third simultaneously adjustable user
interface parameter. In another example, a second or yet more
additional scroll-wheels can be added to a conventional
scroll-wheel mouse. The resultant collection of scroll-wheels can
be relatively positioned in parallel, oriented at orthogonal angles
to support a coordinate-metaphor, positioned on the sides of the
mouse body, etc. FIGS. 24A and 24B illustrate examples 2410, 2420
of a conventional scroll-wheel mouse 2401, 2402 with a traditional
up-down scrollwheel 2421 and at least standard mouse buttons 2411,
2412 provided with an added left-right scroll-wheel 2422 as taught
in U.S. Pat. No. 7,557,797. Such an arrangement can employ a
connecting cable, or the device can be wireless. Each of the
arrangements of FIGS. 24A and 24B as well as the ones described in
the next paragraph, plus variations and adaptations, will be
referred to as an "advanced mouse" or in the plural "advanced
mice."
[0241] In another example of an advanced mouse, one or more
trackballs can be added to a conventional computer mouse. FIGS.
25A-25C illustrate examples 2510, 2520, 2530 wherein a single
trackball 2511, 2521, 2531 is incorporated into the back of a
conventional computer mouse 2510, 2520, 2530 with at least standard
mouse buttons 2512-2513, 2522-2523, 2532-2533 as taught in U.S.
Pat. No. 7,557,797. FIGS. 26A-26C illustrate examples where two
trackballs 2605a, 2605b are incorporated into the back of a
conventional computer mouse as taught in U.S. Pat. No. 7,557,797.
The trackballs in the arrangements of FIGS. 25A-25C and FIGS.
26A-26C can be the conventional two degree of freedom type (roll
left-right, roll away-towards) or can provide three to six degrees
of freedom as taught in U.S. Pat. No. 7,557,797; U.S. patent
application Ser. No. 10/806,694. Useful exemplary advanced mouse
arrangements include the trackball 2605a-2605b, touchpad 2605c,
scrollwheel 2668 mouse of FIG. 26C, the trackball/touchpad mouse of
FIG. 26d, and the multiple slider configuration of FIG. 26E, each
as taught in U.S. Pat. No. 7,557,797. Each of these arrangements
can employ a connecting cable, or the advanced mouse device can be
wireless.
[0242] The additional parameters provided by the HDTP and the above
alternatives are more than the usual number supported by
conventional window systems (for example as described in
conjunction with FIG. 16A). To implement additional parameter
handling, for example such as in the HDTP arrangement of FIG. 22 or
its equivalents for the alternative advanced mice as described
above, additional arrangements must be made beyond the conventional
operating system pointer device handling.
[0243] In an additional approach, the invention provides for the
HDTP or alternatives such as the advanced mice described above to
interface with a browser via a browser plug-in. This arrangement
can be used to capture the additional user interface input
parameters and pass these on to an application interfacing to the
browser. An example of such an arrangement is depicted in FIG. 27.
The browser can interface with local or web-based applications that
drive the visualization and/or control the data source(s), process
the data, etc. The browser can be provided with client-side
software such as JAVA Script, AJAX, etc. Such an arrangement is
particularly useful when combined with browser-based rendering of
visualizations, as described in the next section.
[0244] The invention provides for HDTP parameters to be separated
into groups which are individually directed to pointer device
interfaces on multiple computers.
Browser-Rendered Implementations
[0245] The invention provides for visualizations to be rendered in
a browser. This allows for implementations wherein the browser is
used as a viewer. The browser can interface with local or web-based
applications that drive the visualization. An exemplary arrangement
is depicted in FIG. 28. In an embodiment, the browser can be
provided with Simple Vector Graphics ("SVG") utilities (natively or
via an SVG plug-in) to render basic 2D vector and raster graphics.
In another embodiment, the browser can be provided with a 3D
graphics capability, for example via the Cortona 3D browser
plug-in. These embodiments or alternatives can be provided with
client-side software such as JAVA Script, AJAX, etc. The example of
FIG. 28 also provides for other plug-ins (represented by the
dashed-line box) such as the user interface input device plug-in
described earlier.
Web-Based Implementations
[0246] The invention provides for web-based implementations of the
visualization environment. FIG. 29 depicts an exemplary web-based
implementation of a visualization environment leveraging browser
arrangements such as that depicted in FIGS. 27-28, their
variations, alternatives, etc. As depicted in the arrangement of
FIG. 29, the client environment can employ such a browser to access
a visualization server environment or visualization-related
(databases, live data feeds, simulations, statistical or signal
processing, etc.) server environment. In an embodiment, the client
environment browser can connect over the internet with two or more
visualization server environments and/or visualization-related
server environments and/or local applications. In an embodiment, a
visualization or visualization-related server environment can
connect over the internet with two or more browsers in the client
environment. Other variations and alternative implementations are
also possible as is clear to one skilled in the art.
[0247] A server environment can include web server foundation
software such as Apache, IIS, etc. A server environment can also
include server-side scripting and dynamic software such as CGI,
JAVA/JAVA Script, Python, PHP, Perl, JSP, ASP, etc. Other
variations and alternative implementations are also possible as is
clear to one skilled in the art.
[0248] The additional interactively-controlled parameters provided
by the HDTP provide more than the usual number supported by
conventional browser systems and browser networking environments.
This may be addressed in a number of ways.
[0249] In a first approach, an HDTP interfaces with a browser both
in a traditional way and additionally via a browser plug-in. Such
an arrangement may be used to capture the additional user interface
input parameters and pass these on to an application interfacing to
the browser. An example of such an arrangement is depicted in FIG.
30D.
[0250] In a second approach, an HDTP interfaces with a browser in a
traditional way and directs additional GUI parameters though other
network channels. Such an arrangement may be used to capture the
additional user interface input parameters and pass these on to an
application interfacing to the browser. An example of such an
arrangement is depicted in FIG. 30E.
[0251] In a third approach, an HDTP interfaces all parameters to
the browser directly. Such an arrangement may be used to capture
the additional user interface input parameters and pass these on to
an application interfacing to the browser. An example of such an
arrangement is depicted in FIG. 30F.
[0252] The browser may interface with local or web-based
applications that drive the visualization and/or control the data
source(s), process the data, etc. The browser may be provided with
client-side software such as JAVA Script, AJAX, etc. The browser
may provide also be configured advanced graphics to be rendered
within the browser display environment, allowing the browser to be
used as a viewer for data visualizations, advanced animations,
etc., leveraging the additional multiple parameter capabilities of
the HDTP. The browser may interface with local or web-based
applications that drive the advanced graphics. In an embodiment,
the browser may be provided with Simple Vector Graphics ("SVG")
utilities (natively or via an SVG plug-in) so as to render basic 2D
vector and raster graphics. In another embodiment, the browser may
be provided with a 3D graphics capability, for example via the
Cortona 3D browser plug-in.
Collaboration Implementations
[0253] The invention provides for collaboration implementations and
the use of collaboration tools for collaborative use of
visualization features of the invention and for the creation of a
collaborative visualization environment.
[0254] FIG. 30A shows a client-side collaboration implementation
wherein a general purpose collaboration tool can be used to share
the visualization environment session running on one computer with
one or more other computers. The visualization environment can be a
stand-alone visualization application or can comprise a browser
implementation employing, for example, any of the various
arrangements described in conjunction with FIGS. 27-29.
[0255] In the case of real-time collaboration, the general purpose
collaboration tool employed in the arrangement of FIG. 30 can
comprise application sharing software (such as Microsoft's
NetMeeting.TM., IBM/Lotus' SameTime.TM.,.sup.SM, HP's Shared-X,
etc.) or display sharing software. In the case of real-time
collaboration, audio and/or video conferencing can also be used to
facilitate interpersonal communication among collaborating session
participants.
[0256] FIG. 30B shows server-side collaboration implementation
wherein a general purpose or specialized server-side application
sharing environment is used to share a server-based interactive
data visualization application.
[0257] FIG. 30C shows a server-side collaboration implementation
comprising a server-based multi-user interactive data visualization
application.
Interfacing with Sonification
[0258] The invention further provides for interfacing with
established and advanced data sonification utilities. Data
sonification has received considerable attention and analysis but
for the most part is fairly simplistic and often contributes little
practical value. More advanced data sonification techniques, such
as the multi-channel data sonification system described in U.S.
Patent Application 61/268,856 employed in the environmental GIS
(Geographic Information System) process monitoring and modeling
system of U.S. Patent Application 61/268,873, provides new
opportunities for practical use of data sonification.
[0259] FIG. 31 illustrates a general framework for data
sonification where a parameterized metaphor is used to manage
parameter assignment and sound rendering and a parameterized data
indexing operation.
[0260] Data sonification may be applied to the same data used to
generate visualization. The data directed to data sonification may
be selected by interacting with a rendered visualization via a user
interface input device. Data sonification may be provided by a
multichannel data sonification system such as the one described in
U.S. Patent Application 61/268,856. Data sonification output may be
shared using an audio channel of a real-time collaboration system,
may be transmitted from a web-based application using an audio
channel as provided by a high-fidelity VoIP system, may be shared
using an audio channel as provided by a high-fidelity VoIP system,
may be produced local to the user computer under the control of a
web-based application, or may be produced local to the user
computer under the control of a web-based application employing
MIDI protocol.
[0261] Some user interface considerations relating to
configurations, features and parameters of data sonification
operations are described in the aforementioned U.S. Patent
Application 61/268,856. Beyond these, and more generally,
configurations, features and parameters of data sonification
operations can be selected via user interface dialog windows.
General forms configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
Uniform Parameterizations of Cell and Data Presentation
Attributes
[0262] The invention provides for visualization rendering
parameters to be uniform over a common variational range. This
permits uniform handling of visualization rendering parameters and
numerical operation compositions. Any of these can be set, varied,
or modulated as selected or as advantageous to represent data or
information derived from data. In an embodiment, the invention
provides for visualization rendering uniform parameters in the
range of [0,1]. The uniform parameter range of [0,1] is used in the
examples below, but it is understood that other choices for the
uniform range are also possible and are provided for by the
invention.
[0263] Color Morphing
[0264] The invention provides for at least one color selection
option to be determined by a uniform parameter in the range of
[0,1]. FIG. 32 depicts an exemplary embodiment wherein the hue of a
color can be varied through natural colors of red through violet
for the bulk of the [0,1] range and a range of artificial colors,
such as the purple through magenta continuum, are appended for the
remaining portion of the [0,1] range. The invention also provides
for other color models, such as RGB, HSB, LUV, LAB, grayscale, etc.
to be used with one to three uniform parameters, each in the range
of [0,1]. Such uniform parameter colors can be assigned one or more
of text, symbols, borders, backgrounds, gridlines, glyphs, lines,
components of lines, components of stipples, components of
gradients, surfaces, and other geometric primitives employed in the
visualization of data. Other variations and alternative
arrangements are possible as is clear to one skilled in the
art.
[0265] A uniform parameter range of [0,1] is used in the examples
above, but it is understood that other choices for the uniform
range are also possible and are provided for by the invention.
[0266] Line Morphing
[0267] The invention provides for at least one line rendering
option to be determined by a uniform parameter in the range of
[0,1]. FIG. 33 illustrates an example wherein line width is
determined by a uniform parameter in the range of [0,1]. FIG. 33
also illustrates an example wherein line dashing period is
determined by a uniform parameter in the range of [0,1]. FIG. 33
further illustrates an example wherein line dashing duty-cycle is
determined by a uniform parameter in the range of [0,1]. FIG. 33
also illustrates an example wherein line dashing sub-duty-cycle is
determined by a uniform parameter in the range of [0,1]. Other
variations and alternative arrangements are possible as is clear to
one skilled in the art.
[0268] The invention provides for line color to be determined by a
uniform parameter color model as described above and for line
dashing to include a plurality of colors.
[0269] Stipple Morphing
[0270] The invention provides for at least stipple pattern
rendering option to be determined by a uniform parameter in the
range of [0,1]. FIG. 34 illustrates an example wherein stipple line
width is determined by a uniform parameter in the range of [0,1].
FIG. 34 also illustrates an example wherein stipple gap is
determined by a uniform parameter in the range of [0,1]. A period
and duty cycle model can be used instead and controlled via
respectively associated uniform parameters varying over the range
of [0,1].
[0271] FIG. 34 further illustrates an example wherein stipple angle
is determined by a uniform parameter in the range of [0,1]. Other
variations and alternative arrangements are possible as is clear to
one skilled in the art.
[0272] The invention provides for at least one of stipple
foreground and background colors to be determined by a uniform
parameter color model as described above. The invention provides
for a stipple pattern to comprise more than two colors, for the
inclusion of line dashing in a stipple pattern, and for line
dashing to include a plurality of colors.
[0273] Other variations and alternative arrangements are possible
as is clear to one skilled in the art. A uniform parameter range of
[0,1] is used in the examples above, but it is understood that
other choices for the uniform range are also possible and are
provided for by the invention.
[0274] Border Morphing
[0275] The invention provides for at least one border rendering
option to be determined by a uniform parameter in the range of
[0,1]. The border rendering option can comprise the border of a
spreadsheet cell as depicted in FIG. 35, or can comprise the border
of another visually rendered graphical object.
[0276] In the complete cell depicted in the upper right corner of
FIG. 35, all four sides of the cell border are uniform. Such a
border to have a common width W.sub.0 controlled by a uniform
parameter in the range of [0,1] and to have a common color
controlled by one or more uniform parameters in the range of [0,1].
The cell border can be rendered as a gradient among two or more
selected colors, wherein the colors and/or gradient characteristic
can be respectfully determined by an associated uniform parameter
in the range of [0,1].
[0277] In the complete cell depicted in the lower right corner of
FIG. 35, all four sides of the cell border are distinct. The
invention provides for each such border segment to have an
associated width (W.sub.1-W.sub.4), each respectively controlled by
an associated uniform parameter in the range of [0,1]. The
invention further provides for each such border segment to have a
common color controlled by one or more uniform parameters in the
range of [0,1]. A border segment can be rendered as a gradient
among two or more selected colors, wherein the colors and/or
gradient characteristic can be respectfully determined by an
associated uniform parameter in the range of [0,1].
[0278] The invention further provides for such a cell (interior)
background to be filled with a color controlled by one or more
uniform parameters in the range of [0,1]. In an embodiment, the
invention provides for stipple patterns to be used as a cell
background. The cell background can be rendered as a gradient among
two or more selected colors, wherein the colors and/or gradient
characteristic can be respectfully determined by an associated
uniform parameter in the range of [0,1].
[0279] The invention also provides for the four sides of a cell
border to be segmented in other ways, for example as in the two
complete cells in the left side of FIG. 35.
[0280] Other variations and alternative arrangements are possible
as is clear to one skilled in the art. A uniform parameter range of
[0,1] is used in the examples above, but it is understood that
other choices for the uniform range are also possible and are
provided for by the invention.
[0281] Font Morphing
[0282] The invention provides for font morphing and font
interpolation under the control of uniform parameters in the range
of [0,1]. Other choices for the uniform range are also possible and
are provided for by the invention.
[0283] FIG. 36A depicts a boldness parameter under the control of
uniform parameters in the range of [0,1]. In an embodiment, the
variation in boldness can be implemented so that it is governed by
the optical properties of de Groot Interpolation Theory. In an
embodiment, a minimum and maximum degree of boldness are assigned
to the extremes of the parameter range.
[0284] FIG. 36B depicts an italics angle parameter under the
control of uniform parameters in the range of [0,1]. In an
embodiment, a minimum and maximum degree of italics angle is
assigned to extremes of the parameter range.
[0285] FIG. 36C depicts an font family interpolation parameter
under the control of uniform parameters in the range of [0,1]. One
font family is assigned to the minimum extreme of the parameter
range and a second font family is assigned to the maximum extreme
of the parameter range.
[0286] FIG. 37A depicts vertical and horizontal
dilation/contraction parameters, each respectively under the
control of uniform parameters in the range of [0,1]. In an
embodiment, a minimum and maximum degree of italics angle are
assigned to the extremes of the parameter range.
[0287] FIG. 37B shows font underline line width and font underline
separation gap parameters, each respectively under the control of
uniform parameters in the range of [0,1]. In an embodiment, a
minimum and maximum degree of italics angle are assigned to the
extremes of the parameter range.
[0288] Other variations and alternative arrangements are possible
as is clear to one skilled in the art. A uniform parameter range of
[0,1] is used in the examples above, but it is understood that
other choices for the uniform range are also possible and are
provided for by the invention.
User Interfaces for Adjusting Cell and Data Presentation
Attributes
[0289] Configurations, features and parameters for related
operations, such as data set selection, data ranges, offsets,
scalings, warping functions, etc., can be, for example, selected
via traditional types of user interface dialog windows. General
forms configuration-specification, feature-selection, and
parameter-setting GUIs are known in the art.
[0290] The invention provides for user interfaces to simplify
operation with a friendly user interface for new or occasional
users. A typical attribute control panel, as shown in FIG. 38,
contains a few pre-selected values and a value which can be
customized usually by selecting a value from a drop-down list.
[0291] Also, users can specify if the changes in attributes are to
be applied globally or are to be applied to a single cell or a
group of cells in a spreadsheet. The change to be made can also be
determined by the value of a data element or by the result of a
formula calculated from one or more data elements.
[0292] FIG. 39 illustrates an example of how change in color of a
font can be specified for a cell, a group of cells, or globally
across an entire worksheet. Users can select the color of a font
simply by clicking on a region in the gray level or Hue color chart
or by moving the slider that determines the gray level or
parameters in Hue. A formula that determines such value can also be
included. For example, the ratio of the data in a cell to the range
of the data in a category can determine the value of gray level
accordingly.
[0293] FIG. 40 depicts an interface for determining a font type of
the specified cell(s). Such interface can have a pre-selected list
or a list to be customized or a combination of both, and can be
implemented in a wide variety of ways.
[0294] FIG. 41 depicts an example of an interface for determining
the embellishment of a font in rendering some data element within
the spreadsheet or data table. The data element can be displayed in
bold face, italics or underlined. The percentage of bolding or the
thickness of the underline can be varied. For example a data
element can be emphasized by being displayed in bold face, or a
title of a book or quotation can be displayed in italics, or a link
to a website or a reference to another cell or region can be
underlined.
[0295] FIG. 42 depicts an example interface to control the size of
a font to be displayed. Several preset sizes are listed on the
panel as well as a slider that enables users to control the font
size gradually. Font size can be incremental with step sizes or
continuous, which can be adjusted by controlling a slider, as long
as the value is positive and less than reasonable size compared to
the area of the text displayed.
[0296] The invention provides for a user interface to determine
addition of an appending symbol. The symbol(s) can be different
types, numbers, or sizes of graphical objects. The factor that
determines can depend on the significance or the ratio of the
element to the category or the value of another category in the
dataset. In this example, the symbols appended to the right end of
the chart represent different categories of expenses. If the symbol
appended is configured to represent quantity, the number of symbols
to be appended will depend on the ratio of the value of the cell to
the unit or base value. If the symbol appended is configured to
represent significance, the size of symbols to be appended will
depend on the significance of the value of the cell in that
category. FIG. 43 depicts an example wherein data in a cells in a
selected column can have symbol appended to numbers or other data
within the cells of the column.
[0297] Different formatting options can be applied to different
types of data. The factor that determines the type of data can be
obtained from the result of a hidden formula, from associated data
such as the data from another column in the same row, or simply
selected by the user. The number of decimals can be determined
according to such factors. Also symbols can be attached according
to the type of data. If the data in a cell represents currency, a
currency symbol can be attached, and if the data represents
percentage, a percentage sign will be attached. Further if the
country is specified in another column, the corresponding country's
currency sign can be attached.
[0298] The invention provides for the varying the color and
opacity/transparency of the background color of a cell or group of
cells as a function of data values. Specifying background color and
opacity/transparency can be done similarly with the case for color
of the font is determined, as illustrated in FIG. 39. For example,
a user can affect the control the background color and
opacity/transparency with the sliders or by simply clicking on a
region of color or grayscale chart. The background color control
can also be incorporate auto-scaling. Optionally, and the color can
be determined depending on the ratio of the value of the cell to
the values in the column.
[0299] Similarly, the invention provides for the background texture
or stipples of a cell or group of cells can also be varied as a
function of data values. FIG. 44 depicts an example use of
controlling the background texture or stipples of cells as a
function of data values in a spreadsheet or other form of tabular
data. FIG. 45 depicts an exemplary interface that can be used to
specify the background texture or stipples to be applied to a cell
or a group of cells. The stipples of a cell can be made up of dots,
shapes, or grid lines. The size, thickness, density, or opacity of
the background can be adjusted with sliders as well. Stipples can
be used alone or in combination with the background color.
[0300] The invention provides for the specifying the location of a
displayed data element within a cell. FIG. 46 depicts an exemplary
interface that can be used to specify the position of text being
displayed in a cell. Users can move sliders by clicking and
dragging on the icon and moving it vertically or horizontally to
specify vertical or horizontal position of text displayed in a
cell. Users can specify the point of vertical and horizontal
centering by clicking on a region in the provided box and selecting
how texts are aligned horizontally or vertically relative to the
point specified. In an embodiment, values representing the
left-right or up-down ratio can be simply entered in text boxes by
the users.
[0301] The invention provides for the specifying the location of a
displayed rotation of text, symbols, or other information within a
cell. FIG. 47 depicts an exemplary interface that can be used to
specify the rotation of text, symbols, or other information
displayed combined with the vertical or horizontal placement of the
text. By clicking and dragging the right tip of the slider handle,
the user can control the text to be rotated counter-clockwise, and
vice versa. Vertical or horizontal shift of the position of the
text can be controlled by clicking and dragging the middle of the
slider handle to the desired direction.
[0302] The invention provides for a user interface to select of the
color of at least one border of a cell, the thickness of at least
one border of a cell enveloping a font, the thickness of at least
one border outlining of a plurality of contiguous cells, and for a
rendered image element.
[0303] The invention provides for data of a specified category to
be extracted from cells and matched with hidden formula to obtain
data used in the visualization. Data from another associated group
of cells can be used for another attribute in the visualization.
For example, data from one category can be used to determine the
type of object displayed, another category can be used to determine
the location of the object displayed, another category can be used
to determine the number of objects displayed, and data from another
category can be used to determine the size of the object(s)
displayed. Existing spreadsheet applications offer various types of
charts, but the placing of the objects are limited to aligning them
in certain orders or directions.
[0304] The invention provides for visualizations to be co-rendered
with the spreadsheet program. The invention provides for
visualizations displayed on a specified region of a spreadsheet. In
an embodiment, the invention provides for the result of
visualizations to be exported to outside of the spreadsheet
program.
Spreadsheet Visualizations
[0305] The invention provides for visual aspects of groups of cells
to be varied together as a function of a data value. For example,
FIG. 48 depicts an exemplary visualization wherein an attribute of
a column area of data elements is varied as a group. In this
example a set of collected data containing the state, main
industry, population, the increase in population, number in current
work force, unemployment rate, and the increase in unemployment
rate can be used for different purposes. Depending on what category
the emphasis is on, an attribute of that column can be varied. Such
emphasis can be done by varying the font boldness, font size, or
font color. FIG. 49 depicts an exemplary visualization wherein the
shading of a row area of data elements is varied as a group as a
function of a data value and also wherein the font boldness of a
column area of data elements is varied as a group as a function of
another data value.
[0306] The invention provides for a reordering of tabular rows
and/or columns as a function of data values. Unlike a traditional
spreadsheet data sort, such a reordering operates as a "live"
function, responsive to interactive changes of the underlying data
values. FIG. 50 illustrates a model of a priority queue wherein the
rows can automatically reorder among themselves as underlying
priority sales change. Jobs to be done can have different
attributes, such as location or types of jobs along with numbers
that indicate the rating of priority. A user would expect jobs with
a higher priority rating to be displayed in a higher position of
the priority queue tree listing and to reorder as calculated
priority values change.
[0307] In another aspect of the invention function or mapping
utilities can be included in the program. FIG. 51A illustrates a
basic map-based visualization example generated from previous data.
FIG. 51B illustrates the visualization of FIG. 51A augmented with
additional spatially located symbols or parameterized glyphs.
Interactive Tabular Spreadsheet Metaphor for Presentation of
Visualization Renderings
[0308] The invention provides for interactive tabular spreadsheet
metaphor for presentation of visualization renderings. A cell may
be selected and the data used to create the enclosed visualization
can be captured (as structured for the enclosed visualization) to
be used in a subsequent visualization or calculation. A cell may be
selected and the pre-visualization dataflow and processing steps
used to create the enclosed visualization can be captured (as
structured for the enclosed visualization) to be used in a
subsequent visualization or calculation. A cell may be selected and
the visualization format employed by the enclosed visualization can
be captured (as structured for the enclosed visualization) to be
used in a subsequent visualization or calculation.
[0309] Two cells can be selected and the data used to create the
enclosed visualizations can be captured (as structured for the
enclosed visualization) and provided to a mathematical operation
(for example, adding, subtracting, multiplying, convolving, etc.)
to form a new dataset to be used in a subsequent visualization or
calculation. Similarly, multiple cells can be selected and the data
used to create the enclosed visualizations can be captured (as
structured for the enclosed visualization) and merged to form a new
dataset to be used in a subsequent visualization or calculation.
Embodiments of the invention can also include one or more of the
following optional features: [0310] At least part of the captured
data can be re-sampled. In an embodiment, redundant data occurring
in the merged dataset is removed; [0311] Gaps in data occurring in
the merged dataset are noted. In an embodiment, gaps in data
occurring in the merged dataset can be filled in via interpolating
functions; [0312] At least one partial row and at least one partial
column can be selected and an operation applied to the underlying
data; [0313] At least one partial row and at least one partial
column can be selected and an operation applied to the individually
enclosed visualizations; [0314] At least one partial row and at
least one partial column can be selected and at least one attribute
and/or parameter of a common visualization operation can be
adjusted and applied to each of the individually enclosed
visualizations; [0315] At least one partial row and at least one
partial column can be selected and at least one attribute and/or
parameter of a common pre-visualization operation can be adjusted
and applied to each of the individually enclosed visualizations;
[0316] At least one partial row and at least one partial column can
be selected and at least one attribute and/or parameter of a common
data source selection can be adjusted and applied to each of the
individually enclosed visualizations.
Exemplary 3D Graphics Augmentations to Spreadsheet
Visualizations
[0317] The invention provides for the inclusion of useful novel 3D
graphics visualization functions for multidimensional data
representation, data query, and numerical solution tools for
simultaneous interactive numerical equations. These can be
incorporated in various ways, as described below, to expand
spreadsheet--based visualization capabilities yet further.
[0318] A first set of the useful novel 3D graphics visualization
functions provided for by the invention pertain to multidimensional
data visualizations based on 2D-surfaces embedded in a 3D visual
field. FIG. 52 data or spreadsheet, these separated by a vertical
gap within a 3D visual field as provided for by the invention. The
invention also provides for interactively shifting the observation
point with respect to these objects for more detailed feature,
theme, or trend inspection. The viewpoint can be changed according
to the 6 degrees of freedom (three translations, three angles) of
rigid motion. FIG. 53 depicts with dashed lines a few ways in which
the observation viewpoint can be moved with respect to these
abstract objects. The viewpoint is changed under the control of an
HDTP or can be changed under the control of an advanced mouse.
[0319] The invention also provides for moving the 2D-surface
representing a 3D data plot within the 3D visual field. The
2D-surface can be moved according to the 6 degrees of freedom
(three translations, three angles) of rigid motion. FIG. 54 depicts
a translation and rotation of a 3D data plot within the 3D visual
field. The 3D data plot can be moved under the control of an HDTP
or the 3D data plot can be moved under the control of an advanced
mouse.
[0320] The 2D-surface representing a 3D data plot within the 3D
visual field can be virtually illuminated by one or more lighting
sources. The one or more lighting sources can be moved according to
the 6 degrees of freedom (three translations, three angles) of
rigid motion. The one or more lighting sources can also permit
control of the color and intensity of virtual light emitted. A
lighting source can be moved and controlled under the control of an
HDTP or under the control of an advanced mouse.
[0321] The data plotted in the 2D-surface representing a 3D data
plot within the 3D visual field can directly echo the data
displayed in the exemplary planar array of associated tabular data
or spreadsheet and/or can originate from another set of tabular
data or spreadsheet region and/or can originate from other data.
This is suggested by FIG. 55A. The invention also provides for
processing of the data via mathematical transformations,
statistical processing, signal processing, etc. prior to creation
of the 2D-surface representing at least 3-dimensional data. This is
suggested by FIG. 55B. In an embodiment, the 2D-surface
representing at least 3-dimensional data can plot filtered or
averaged versions of tabular data, spreadsheet data, or other data.
In an embodiment, and the filtering or averaging can be controlled
by an interactive parameter.
[0322] The invention provides for 2D-surface representing of at
least 3-dimensional data to be rendered along with a height
measuring visual so that the local color of the surface need not be
a function of the vertical value being plotted. This frees up the
local 2D-surface color to be used to represent an additional
dimension of data. FIG. 56 illustrates an exemplary height
measuring visual in its use with 2D-surface representing at least
3-dimensional data and useful for representing at least
4-dimensional data.
[0323] The invention provides for a 2D-surface representing at
least 3-dimensional data to be rendered with a parameterized
locally varying surface texture that can be used to represent an
additional dimension of data, with numerical values rendered on the
surface that can be used to represent an additional dimension of
data, with symbols rendered on the surface that can be used to
represent an additional dimension of data, or with parameterized
glyphs rendered on the surface that can be used to represent (an)
additional dimension(s) of data.
[0324] The invention provides for a slicing function to provide
level set data. FIG. 57 depicts an exemplary slicing function, here
used to provide a (planar-slice) level set curve. Other uses,
implementations, and applications for slicing functions are also
possible.
[0325] The invention provides for a more general surface
intersection tool that can be used as a numerical solution operator
for plotted, interpolated, and processed data. FIG. 58A illustrates
two exemplary 2D-surfaces, each representing at least 3-dimensional
data. Each of these can be interactively moved and rotated as
described earlier. FIG. 58B illustrates the interactively created
intersection of the two exemplary 2D-surfaces and the resultant
intersection curve. A 2D-surface intersection tool allows the
numerical values of selected points on a curve formed by the
intersecting surfaces to be numerically calculated and visually
displayed. The numerical values can also be used for other purposes
and other subsequent calculations.
[0326] The sample points can be selected according to the details
of the intersection. An adaptive sampling method can be employed.
The resultant intersection data can be interpolated and re-sampled
according to another sampling strategy. More than two surfaces can
be intersected simultaneously, and various pair wise and group
intersection data can be captured and displayed.
[0327] A second set of the useful novel 3D graphics visualization
functions provided for by the invention pertain to multidimensional
data visualizations based on 1D-curves embedded in a 3D visual
field.
[0328] The invention provides for a curve to be generated by
tabular data and to be suspended over tabular data used to create
it or otherwise associated with the curve. The invention provides
for the curve to be visually linked to a geometric rendering of
tabular data. These features are illustrated by FIG. 59. The
invention provides for the curve to be given a thickness and or
color under the control of one or more data values. The invention
provides for the color of the curve to be varied as a function of
tabular (or other) data values. These features are illustrated by
FIG. 60. The invention provides for the thickness of the curve to
be varied as a function of tabular (or other) data values. This
feature is illustrated in FIG. 61.
[0329] Such curve plotting utilities can be provided with
functions, features, and operations like that of the 2D-surface
utilities described above. The invention also provides for
interactively shifting the observation point with respect to these
objects for more detailed feature, theme, or trend inspection. The
viewpoint can be changed according to the 6 degrees of freedom
(three translations, three angles) of rigid motion. The viewpoint
is changed under the control of an HDTP or an advanced mouse. The
invention also provides for moving a curve within the 3D visual
field. The curve can be moved according to the 6 degrees of freedom
(three translations, three angles) of rigid motion. The 2D-surface
can be moved under the control of an HDTP or an advanced mouse.
[0330] A curve of any thickness can be treated as if it comprises a
surface that can be virtually illuminated by one or more lighting
sources. The one or more lighting sources can be moved according to
the 6 degrees of freedom (three translations, three angles) of
rigid motion. The one or more lighting sources can also permit
control of the color and intensity of virtual light emitted. In an
embodiment, a lighting source can be moved and/or controlled under
the control of an HDTP or an advanced mouse.
[0331] The data plotted in a curve can directly echo the data
displayed in the exemplary planar array of associated tabular data
or spreadsheet and/or can originate from another set of tabular
data or spreadsheet region and/or can originate from other data as
suggested earlier in FIG. 55A. The invention also provides for
processing of the data via mathematical transformations,
statistical processing, signal processing, etc. prior to creation
of the curve as was suggested earlier in FIG. 55B. A curve can plot
filtered or averaged versions of tabular data, spreadsheet data, or
other data. In an embodiment, the filtering or averaging can be
controlled by an interactive parameter.
[0332] The invention provides for a curve to be rendered along with
a height measuring visual, and also provides for a locally variable
parameterized texture to be imposed on the curve and for the
texture to be varied as a function of tabular or other data.
[0333] The invention provides for the curve to be rendered with
numerical values displayed adjacently at specific points on the
curve and that these can be used to represent an additional
dimension of data. The invention also provides for the curve to be
rendered with symbols displayed adjacently at specific points on
the curve and that these can be used to represent an additional
dimension of data. The invention further provides for the curve to
be rendered with parameterized glyphs displayed adjacently at
specific points on the curve and that these can be used to
represent (an) additional dimension(s) of data.
[0334] The invention provides for a curve intersection tool that
can be used as a numerical solution operator for plotted,
interpolated, and processed data. FIG. 62 illustrates two exemplary
curves, each representing at least 2-dimensional data. Each of
these can be interactively moved and rotated as described earlier,
interactively creating an intersection of the two exemplary curves
and at least one resultant intersection point. In an embodiment, a
2D-surface intersection tool allows the numerical values of
intersection points to be numerically calculated and visually
signified. The numerical values can also be used for other purposes
and other subsequent calculations.
[0335] In an embodiment, more than two curves can be intersected
simultaneously, and various pair-wise and group intersection data
can be captured and displayed.
[0336] For example, a curve and a surface can be intersected
simultaneously, and intersection data can be captured and
displayed. As another example, more than two curves and at least
one surface can be intersected simultaneously, and various
pair-wise and group intersection data can be captured and
displayed.
[0337] While the invention has been described in detail with
reference to disclosed embodiments, various modifications within
the scope of the invention will be apparent to those of ordinary
skill in this technological field. It is to be appreciated that
features described with respect to one embodiment typically can be
applied to other embodiments.
[0338] The invention can be embodied in other specific forms
without departing from the spirit or essential characteristics
thereof. The present embodiments are therefore to be considered in
all respects as illustrative and not restrictive, the scope of the
invention being indicated by the appended claims rather than by the
foregoing description, and all changes which come within the
meaning and range of equivalency of the claims are therefore
intended to be embraced therein. Therefore, the invention properly
is to be construed with reference to the claims.
* * * * *
References