U.S. patent application number 10/854117 was filed with the patent office on 2005-01-06 for system and method for multiple node display.
Invention is credited to Arcas, Blaise Aguera y.
Application Number | 20050001849 10/854117 |
Document ID | / |
Family ID | 33514016 |
Filed Date | 2005-01-06 |
United States Patent
Application |
20050001849 |
Kind Code |
A1 |
Arcas, Blaise Aguera y |
January 6, 2005 |
System and method for multiple node display
Abstract
A method of display of visual content is disclosed which
utilizes nodes that launch only when necessary. The nodes may be
rendered and/or displayed in the coordinate system of another node,
perhaps a parent node, when appropriate. Increase precision of
display objects is achieved.
Inventors: |
Arcas, Blaise Aguera y;
(Seattle, WA) |
Correspondence
Address: |
KAPLAN & GILMAN , L.L.P.
900 ROUTE 9 NORTH
WOODBRIDGE
NJ
07095
US
|
Family ID: |
33514016 |
Appl. No.: |
10/854117 |
Filed: |
May 26, 2004 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60474313 |
May 30, 2003 |
|
|
|
Current U.S.
Class: |
345/619 ;
345/428; 345/698 |
Current CPC
Class: |
G06T 11/20 20130101 |
Class at
Publication: |
345/619 ;
345/698; 345/428 |
International
Class: |
G09G 005/00; G06T
017/00 |
Claims
1. A method of displaying visual content, said visual content
comprising at least two nodes, the method comprising displaying the
visual content associated with at least a first node and at least a
second node using the rendering method and coordinate system of the
first node until a predetermined navigation condition is detected,
and then displaying the second node using the rendering method and
coordinate system of the second node.
2. The method of claim 1 wherein at least one of said rendering
method or said coordinate system differs between said first and
second nodes.
3. The method of claim 2 wherein said predetermined navigation
condition is size of display.
4. A method of displaying visual content organized into nodes, the
method comprising displaying visual content associated with a first
node using its own coordinate system during a prescribed range of
zooms and displaying visual content associated with said first node
using a coordinate system of a second node outside said prescribed
range of zooms.
5. The method of claim 4 wherein said first node is within said
second node.
6. The method of claim 5 wherein said first node is displayed using
its own rendering method during said range of zooms and using a
rendering method of said second node outside said range of
zooms.
7. The method of claim 4 or 6 wherein at a border of said range of
zooms, said visual content is displayed by blending from a first
rendition to a second rendition.
8. The method of claim 7 wherein said second node contains a
pointer to said first node.
9. The method of claim 8 wherein said second node also contains
information indicative a size and location of said first node.
10. The method of claim 7 wherein said first rendition and second
rendition use different coordinate systems.
11. The method of claim 7 wherein said first rendition and said
second rendition use different rendering methods.
12. The method of claim 7 wherein said first and second renditions
use different rendering methods and different coordinate
systems.
13. A method of displaying visual content, said method comprising
displaying visual content associated with a first node using a
coordinate system and rendering method of a second node during a
first range of zooms, displaying said visual content associated
with said first node using its own coordinate system and rendering
method during a second range of zooms, and ceasing to display said
visual content associated with said first node during a third range
of zooms.
14. The method of claim 13 wherein said visual content gradually
blends at borders of said first and second range of zooms, and said
second and third range of zooms.
15. A method comprising generating a viewStack of views of all
nodes pierced by a point relative to said display and which are
greater than a predetermined size, selecting a view associated with
one of said nodes, and propagating said view to other nodes, the
visual content associated with which is to be displayed.
16. The method of claim 15 wherein visual content associated with
at least two nodes is displayed substantially simultaneously.
17. The method of claim 16 wherein said view associated is
translated by each node to which it is propagated into that node's
own coordinate system.
18. The method of claim 17 wherein said selected view is propagated
up and then down to nodes associated with visual content then
viewable.
19. The method of claim 18 wherein said upward propagation and said
downward propagation cease when nodes to which propagation would
otherwise occur meet specified conditions.
20. The method of claim 19 wherein at least one of said specified
conditions is size.
21. The method of claim 20 wherein entries in the viewstack are
deleted or added based upon navigation of said visual content.
22. The method of claim 21 wherein a node represented in said
viewstack and displayed is no longer displayed after a zooming out
operation, but is nonetheless maintained on said viewStack, thereby
permitting rezooming in accurately.
23. A method of substantially simultaneously displaying visual
content associated with each of a plurality of nodes, each node
having its own coordinate system, the method comprising each node
deriving information indicative of its own position in a display
area, and displaying visual content associated each of said nodes
substantially simultaneously in said display area.
24. The method of claim 23 wherein said deriving for at least one
node comprises receiving position information from another node,
and converting said position information from a first coordinate
system to a second coordinate system.
25. The method of claim 24 wherein said position information is a
position associated with a predetermined location within said
display area.
26. The method of claim 25 wherein visual content associated with
each of said nodes is displayed using a rendering method associated
with the particular node.
27. A method of displaying visual content comprising displaying a
first node having a first coordinate system, displaying a second
node within said coordinate system, and then displaying said second
node in its own coordinate system in response to navigation by a
user.
28. The method of claim 27 wherein said second coordinate system is
more precise than said first coordinate system.
29. The method of claim 28 wherein a precision associated with a
coordinate system is measured in number of digits, and wherein said
number of digits is variable.
30. The method of claim 29 wherein said precision associated varies
up to a maximum limited by storage capacity of a computer on which
said method is operating.
31. A method of displaying visual content on a display, the method
comprising displaying plural nodes at plural sizes, each in a
specified coordinate system, each coordinate system having a
precision, the precision in which a coordinate system is displayed
being dependent upon the size at which the node is displayed.
32. The method of claim 31 wherein said precision is variable and
increases when the coordinate system in which a node is displayed
changes during navigation.
33. The method of claim 32 wherein said precision is variable and
decreases when the coordinate system in which a node is displayed
changes during navigation.
34. A method of displaying nodes comprising displaying the node in
a first coordinate system, and displaying the node in a second
coordinate system by gradually blending from said first coordinate
system to said second coordinate system during a predefined range
of navigation.
35. The method of claim 34 wherein a rendering method used for said
node also gradually changes from a first such rendering method to a
second rendering method.
36. The method of claim 35 wherein said navigation comprises either
zooming or panning.
37. The method of claim 36 wherein information about a view is
propagated among plural nodes as said navigation occurs.
38. The method of claim 37 wherein said propagation is accomplished
using a mathematical tree.
Description
RELATED APPLICATION
[0001] This application claims the benefit of provisional
application serial number 60/474,313, the disclosure of which is
hereby incorporated by reference in its entirely.
BACKGROUND OF THE INVENTION
[0002] The present invention relates to zooming user interfaces
(ZUI) for computers.
[0003] Most present day graphical computer user interfaces are
designed using visual components of a fixed spacial scale. The
visual content can be manipulated by zooming in or out or otherwise
navigating through it. However, the precision with which
coordinates of various objects can be represented is extremely
limited by the number of bits, usually between 16 and 64,
designated to represent such coordinates. Because of their limited
representational size, there is limited precision.
[0004] In the context of the zooming user interface, the user is
easily able to zoom in, causing the area which previously covered
only a single pixel to fill the entire display. Conversely, the
user may zoom out, causing the contents of the entire display to
shrink to the size of a single pixel. Since each zoom in or out may
multiply or divide the xy coordinates by numerous orders of
magnitude, just a few such zooms completely exhaust the precision
available with a 64 bit floating point number, for example.
Thereafter, round-off causes noticeable degradation of image
quality.
[0005] It is an object of the present invention to provide a ZUI in
which a larger range of zooms is possible.
[0006] It is a further object of the invention to provide a ZUI in
which the precision in which cooridnates are represented is related
to the required precision needed at a particular zoom level of
detail. It is a further object of the present invention to allow a
pannable and zoomable two-dimensional space of a finite physical
size, but of an arbitrarily high complexity or resolution, to be
embedded into a well-defined area of a larger pannable and zoomable
two-dimensional space.
[0007] A further objective of the present invention is to allow
zooming out after a deep zoom-in to behave like the "back" button
of a web browser, letting the user retrace his or her steps through
a visual navigation.
[0008] A further objective of the present invention is to allow
zooming in immediately after zooming out to behave analogously to
the "forward" button of a web browser, letting the user precisely
undo the effects of an arbitrarily long zoom-out.
[0009] A further objective of the present invention is to allow a
node, a visual object as defined more precisely below, to have a
very large number of child nodes (for example, up to 10{circumflex
over ( )}28).
[0010] A further objective of the present invention is to allow a
node to generate its own children programmatically on the fly,
enabling content to be defined, created or modified dynamically
during navigation.
[0011] A further objective of the present invention is to enable
near-immediate viewing of arbitrarily complex visual content, even
if this content is ultimately represented using a very large amount
of data, and even if the data are stored at a remote location and
shared over a low-bandwidth network.
[0012] A further objective of the present invention is to allow the
user to zoom arbitrarily far in on visual content while maintaining
interactive frame rates.
[0013] A further objective of the present invention is to allow the
user to zoom arbitrarily far out to get an overview of complex
visual content, in the process both preserving the overall
appearance of the content and maintaining interactive frame
rates.
[0014] These and other broader objectives of the present invention
will become apparent to those skilled in the art from a review of
the specification that follows.
SUMMARY OF THE INVENTION
[0015] The above and other objects of the present invention are
accomplished by displaying visual content as plural "nodes." Each
node preferably has its own coordinate system and rendering method,
but may be contained within a parent node, and may be represented
in the coordinate system and rendering method of the parent node.
As a user navigates the visual content, by for example, zooming in
or out, a node is only "launched" when the zooming results in an
appropriate level of detail. The launching of the node causes the
node to be represented in its own coordinate system and/or
rendering method, rather than in the coordinate system and/or
rendering method of a different node.
[0016] Prior to the node being launched, the node is either
represented in the coordinate system of the parent node, or not
represented at all. By launching nodes only when they are required,
the precision of a coordinate system is a function of the zoom
level of detail of what is being displayed. This allows a variable
level of precision, up to and including the maximum permissible by
the memory of the computer in which the system operates.
DESCRIPTION OF THE DRAWINGS
[0017] For the purposes of illustration, there are forms shown in
the drawings that are presently preferred, it being understood,
however, that the invention is not limited to the precise
arrangements and instrumentalities shown.
[0018] FIG. 1 is a depiction of visual content on a display;
[0019] FIG. 2 is an image of the visual content of FIG. 1 at a
different level of detail;
[0020] FIG. 3 is a representation of an embodiment of the
invention;
[0021] FIG. 4 is an exemplary embodiment of the invention showing
plural nodes on a display;
[0022] FIG. 5 is a tree diagram corresponding to the exemplary
embodiment shown in FIG. 4.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0023] We assume a user interface metaphor in which the display is
a camera, through which the user can view part of a two-dimensional
surface, or 2D universe. For convenience, although it is not
necessary to do so, we ascribe physical dimensions to this
universe, so that it may be, for example, one meter square. The
invention is equally applicable to N-dimensional
representations.
[0024] The exemplary universe in turn contains 2D objects, or
nodes, which have a visual representation, and may also be dynamic
or interactive (i.e. video clips, applications, editable text
documents, CAD drawings, or still images). For a node to be visible
it must be associated with a rendering method, which is able to
draw it in whole or in part on some area of the display. Each node
is also endowed with a local coordinate system of finite precision.
For illustrative purposes, we assume a node is rectangular and
represented by a local coordinate system.
[0025] These two parameters, the rendering method and coordinate
system, specify how to display the node, and the positions of items
in the node. Each node may have 0 or more child nodes, which are
addressed by reference. The node need not, and generally does not,
contain all the information of each child node, but instead only an
address providing information necessary to obtain the child node.
As a user navigates, for example, zooms in and out, the nodes are
displayed on the screen, as shown, for example in FIG. 1.
[0026] Generally, a "node" is the basic unit of functionality in
the present invention. Most nodes manifest visually on the user's
display during navigation, and some nodes may also be animated
and/or respond to user input. Nodes are hierarchical, in that a
node may contain child nodes. The containing node is then called a
parent node. When a parent node contains a child node, the child's
visual manifestation is also contained within the parent's visual
manifestation. Each node has a logical coordinate system, such that
the entire extent of the node is contained within an exemplary
rectangle defined in this logical coordinate system; e.g. a node
may define a logical coordinate system such that it is contained in
the rectangle (0,0)-(100,100).
[0027] Each node may have the following data defining its
properties:
[0028] the node's logical coordinate system, including its logical
size (100.times.100 in the above example);
[0029] the identities, positions and sizes of any child nodes,
specified in the (parent) node's logical coordinates;
[0030] optionally, any necessary user data;
[0031] executable code defining these operations or "methods":
[0032] initialization of the node's data based on "construction
arguments"
[0033] rendering all or a portion of the node's visual appearance
(the output of this method is a rendered tile);
[0034] optionally, responding to user input, such as keyboard or
mouse events.
[0035] The executable code defines a "node class", and may be
shared among many "node instances". Node instances differ in their
data content. Hence a node class might define the logic needed to
render a JPEG image. The "construction arguments" given to the
initialization code would then include the URL of the JPEG image to
display. A node displaying a particular image would be an instance
of the JPEG node class. Plural instances of a node may be viewable
in the same visual content, similar to the way a software
application may be instantiated numerous times simultaneously.
[0036] Note that in a complex visual document or application, it is
usually possible to divide the necessary functionality into nodes
in many different ways. For example, a scripted web-page-like
document containing multiple images, pull-down menus and buttons
could be implemented as a single node with complex rendering and
user input methods. Alternatively, it could be implemented as a
parent node which only defines the overall layout of the page, with
every constituent image and button a child node. This has the
obvious advantage of reusing or "factoring" the functionality more
effectively: the buttons may all have the same behavior, and hence
all be instances of the same node class; the images may all be in
the same format and so also be instances of a common node class,
etc. This also simplifies rearranging the layout--the parent node
can easily move or resize the child nodes.
[0037] In accordance with the present invention, visual content may
be displayed in a manner that depends upon the state of navigation
input by a user. For example, FIG. 1 shows a node 105 which may be
the image of a portion of the city. Node 105 may contain child
nodes 101-103. Node 101 may be an image of a building in the city,
node 102 could be an image of a playground, and node 103 might be a
sports arena. At the level of zoom shown, nodes 101-103 are
relatively small, so they can be represented as a small darkened
area with no detail in node 105, located at the correct location in
the coordinate system of node 105. Only the coordinate system and
the rendering method of node 105 is needed.
[0038] Consider the case where the user now zooms in so that a
different level of detail (LOD) such as that shown in FIG. 2
results. In the LOD of FIG. 2, nodes 101 and 102 are no longer
visible on the screen, due to the fact that the visual content is
displayed as much larger. Additionally, it is noted that the
because the size at which sports arena node 103 is displayed is now
much larger, the details of the sports arena, such as the
individual seats, the field, etc, now must be displayed.
[0039] In furtherance of the foregoing, sports arena node 103 would
now be displayed not as a darkened area with no detail in the
coordinate system of node 105, but rather, it would be "launched"
to be displayed using its own coordinate system and rendering
method. When displayed using its own coordinate system and
rendering method, the details such as seating, the filed of play,
etc. would be individually shown. Other functions discussed above,
and associated with the node 103, would also begin executing at the
point when node 103 is launched. The particular navigation
condition that causes the launching of node 103, or any node for
that matter, is a function of design choice and is not critical to
the present invention.
[0040] The precision with which the node 103 will be displayed is
the combined precision of the coordinate system utilized by node
105, as well as that of node 103. Thus, for example, if the
coordinate system of each of said nodes utilizes 8 bits, then the
combined precision will be 16 bits because the coordinate system of
node 103 is only utilized to specify the position of items in node
103, but the overall location of node 103 within node 105 is
specified within the coordinate system of node 105. Note that this
nesting may continue repeatedly if sports arena 103 itself contains
additional nodes within it. For example, one such node 201 may in
fact be a particular concession stand within the sports arena. It
is represented without much detail in the coordinate system and
rendering method of node 103. As a user continues zooming in on
sports arena 103, at some point node 201 will launch. If it is
displayed using 8 bits of precision, those 8 bits will specify
where within the node 201 coordinate system particular items are to
be displayed. Yet, the location of node 201 within node 103 will be
maintained to 8 bits of precision within the coordinate system of
node 103, the location of which will in turn be maintained within
the coordinate system of node 105 using 8 bits. Hence, items within
node 201 will ultimately be displayed using 24 bits of
precision.
[0041] By nesting nodes within nodes, the precision at which visual
content may ultimately be displayed is limited only by the memory
capacity of the computer. The ultimate precision with which visual
content in a node is displayed after that node is launched is
effectively the combined precision of all parent nodes and the
precision of the node that has launched. Hence, depending upon the
level of nesting, the precision may increase as needed limited only
by the storage capacity of the computer, which is almost always
much more than sufficient. Additionally, the increased precision is
only utilized when necessary, because if the image is at an LOD
that does not require launching, then in accordance with the above
description, it will only be displayed with the precision of the
node within which it is contained if that node has been launched.
Thus, for nodes nested within other nodes, as one moves from the
outermost node inward, one may traverse nodes that have launched
until finally reaching a node that has not launched yet. Any such
unlaunched node, and nodes further within it, will be displayed
only with the precision of the last traversed node that has
launched.
[0042] This results in an "accordion" type precision, wherein the
precision at which visual content is displayed expands and
contracts as necessary and as dictated by the navigational input of
the user, maximizing the efficiency of system resources by using
them only when necessary for higher precision.
[0043] It is also noted, that when a node launches the display of
that node changes from being based upon the coordinates and
rendering method of the parent node to the coordinates and
rendering method of the child node. That change is optimally made
gradual through the use of blending, as described, for example, in
copending U.S. patent application Ser. No. 10/790,253. However,
other methodologies of gradually changing from the display of the
information in the coordinate system and rendering method the
parent node to the child node are possible. The system could be
programmed, for example, that over a particular range, the blending
from parent to child occurs. Then, as the user traverses through
that range during a zoom, the changeover occurs, unless the
navigation is ceased during that range, in which case the blending
may continue until fully displayed in the appropriate coordinate
system.
[0044] An additional issue solved by the present invention relates
to a system for maintaining the spatial interrelationship among all
nodes during display. More particularly, during dynamic navigation
such as zooming and panning, many different coordinate systems are
being used to display potentially different nodes. Some nodes, as
explained above, are being displayed merely as an image in the
coordinate system of other nodes, and some are being displayed in
their own coordinate systems. Indeed, the entire visual display may
be populated with nodes displayed at different positions in
different coordinate systems, and the coordinate systems and
precisions used for the various nodes may vary during navigation as
nodes are launched. Hence, it is important to ensure that the nodes
are properly located with respect to each other, because each node
is only knowledgeable of its own coordinate system. The present
invention provides a technique for propagating relative location
information among all of the nodes and for updating that
information when needed so that each node will "know" the proper
position in the overall view at which it should render itself.
[0045] The foregoing may be accomplished with the addition of a
field to the node structure and an additional address stack data
structure. The expanded node definition includes a field which we
term the "view" field, and which is used by the node to locate
itself relative to the entire display. The view field represents,
in the coordinates of that node, the visible area of the node--that
is, the image of the display rectangle in the node's coordinates.
This rectangle may only partially overlap the node's area, as when
the node is partially off-screen. Clearly the view field cannot
always be kept updated for every node, as we cannot necessarily
traverse the entire directed graph of nodes in real time as
navigation occurs.
[0046] The stack structure is defined thus:
[0047] Stack<Address>viewStack;
[0048] where this stack is a global variable of the client (the
computer connected to the display). For exemplary purposes we
assume that navigation begins with an overview of a universe of
content, defined by a root node; then this root node is pushed onto
the viewStack, and the root node's view field might be initialized
to be the entire area of the root node, i.e.
[0049] rootNode.view=rootNode.coordSystem;
[0050] Push(viewStack, rootNode);
[0051] Schematically, the viewStack will specify the addresses of a
sequence of nodes "pierced" by a point relative to the display,
which we will take in our exemplary implementation to be the center
of the display. This sequence must begin with the root node, but
may be infinite, and therefore must be truncated. In an exemplary
embodiment, the sequence is truncated when the nodes "pierced"
become smaller than some minimum size, defined as minimumArea. The
current view is then represented by the view fields of all of the
nodes in the viewStack, each of which specify the current view in
terms of the node's local coordinate system. If a user has zoomed
very deeply into a universe, then the detailed location of the
display will be given most precisely by the view field of the last
node in the stack. The last element's view field does not, however,
specify the user's viewpoint relative to the entire universe, but
only relative to its local coordinates. The view field of the root
node, on the other hand, does specify where in the universe the
user is looking. Nodes closer to the "fine end" of the viewStack
thus specify the view position with increasing precision, but
relative to progressively smaller areas in the universe. This is
shown conceptually in FIG. 3, where it can be seen that of the
three nodes that have been launched, node 303 provides the most
accurate indication of where the user is looking, since its
coordinate system is the "finest", but node 301 provides
information, albeit not as fine, on a much larger area of the
visual content.
[0052] The problem then reduces to the following: the views (i.e.
view fields) of all visible nodes must be kept synchronized as the
user navigates through the universe, panning and zooming. Failure
to keep them synchronized would result in the appearance of nodes
moving on the display independently of each other, rather than
behaving as a cohesive and physically consistent 2D surface.
[0053] Changing the view during any navigation operation proceeds
as follows. Because the last node in the viewStack has the most
precise representation of the view, the first step is to alter the
view field of this last node; this altered view is taken to be the
correct new view, and any other visible nodes must follow along.
The second step is to propagate the new view "upward" toward the
root node, which entails making progressively smaller and smaller
changes to the view fields of nodes earlier in the stack. If the
user is deeply zoomed, then at some point in the upward propagation
the alteration to the view may be so small that it ceases to be
accurately representable; upward propagation stops at this node. At
each stage of the upward propagation, the change is also propagated
downward to other visible nodes. Hence, first, the last node's
parent's view is modified; then, in the downward propagation, the
last node's "siblings". The next upward propagation modified the
grandparent's view, and the second downward propagation modifies
first uncles, then first cousins. The downward propagation is
halted, as before, when the areas of "cousin nodes" become smaller
than minimumArea, or when a node falls entirely offscreen.
[0054] The foregoing technique involves translating the layout of
the various nodes into a tree, which conceptually is illustrated in
FIGS. 4 and 5. As can be seen from FIGS. 4 and 5, there is a
corresponding tree for a particular displayed set of nodes, and the
tree structure may be used to propagate the view information as
previously described.
[0055] A panning operation may move the last node far enough away
that it no longer belongs in the viewStack. Alternatively, zooming
in might enlarge a child to the extent that a lengthening of the
viewStack is required, or zooming out might bring the last node's
area below a minimum area requiring a truncation of the viewStack.
In all of these cases the identity of the last node changes. These
situations are detected during the downward propagation, which may
alter the viewStack accordingly, potentially leaving it longer or
shorter.
[0056] One simple case of the foregoing is that during zooming, a
node gets launched so that now it needs to be placed in the view
stack. Another example is that by zooming out, a previously visible
node becomes so small that it must be removed from the
viewStack.
[0057] An extension of the idea is to avoid truncating the
viewStack immediately in response to a long outward zoom.
Truncating the viewStack is only necessary if the user then pans.
Although a long outward zoom will cause the view fields of deeply
zoomed nodes to grow very large (and therefore numerically
inaccurate), a field
[0058] Point2D viewCenter;
[0059] can be added to the Node structure, representing the central
point of the view rectangle; zooming without panning therefore does
not alter the viewCenter field of any node. This construction
allows zooming far outward to be followed immediately by zooming
back in. Because the viewStack has been left intact, the user can
then return to precisely the starting view. This behavior is
analogous to the "back" and "forward" buttons of a web browser:
"back" is analogous to zooming out, and "forward" is analogous to
zooming back in. In a web browser, if a user uses "back" to return
to a previous web page, but then follows an alternate link, it is
at this point that "forward" ceases to work. Following an alternate
link is thus analogous to panning after zooming out.
[0060] The foregoing provides that visual content may be displayed
and navigated in a variety of fashions with substantially infinite
precision, limited only by the capacity of the computer system on
which the application is running. The visual content displayed at
any given time is then displayed as an assembly of nodes, wherein
only the nodes needed for the particular view have been launched,
and all other nodes are displayed without launching as part of
another node or not at all. It is understood that various other
embodiments will be apparent to those of ordinary skill in the art,
and that the invention is not limited to the embodiments described
herein.
* * * * *