U.S. patent application number 10/727344 was filed with the patent office on 2004-12-09 for system and method for managing a plurality of locations of interest in 3d data displays ("zoom context").
Invention is credited to Hernandez, Jackie Lou, Lee, Jerome Chan, Serra, Luis.
Application Number | 20040246269 10/727344 |
Document ID | / |
Family ID | 32719228 |
Filed Date | 2004-12-09 |
United States Patent
Application |
20040246269 |
Kind Code |
A1 |
Serra, Luis ; et
al. |
December 9, 2004 |
System and method for managing a plurality of locations of interest
in 3D data displays ("Zoom Context")
Abstract
A system and method of presentation of 3D data sets in a 3D
display is presented. In exemplary embodiments according to the
present invention the method involves displaying data in an
overview mode wherein localization markers can be set and
manipulated, and in a local mode wherein data surrounding a
localization marker can be rendered using different display
parameters. In exemplary embodiments according to the present
invention the mode in which data is displayed can be selected by a
user. In preferred exemplary embodiments according to the present
invention the method can be implemented recursively, such that
within local mode sub-localization markers can be set and
manipulated, and data can be displayed in a sub-local mode wherein
data surrounding a sub-localization marker can be rendered using
display parameters which can be different from both those of an
overview display mode and those of a local display mode.
Inventors: |
Serra, Luis; (Singapore,
SG) ; Lee, Jerome Chan; (Tokyo, JP) ;
Hernandez, Jackie Lou; (Singapore, SG) |
Correspondence
Address: |
KRAMER LEVIN NAFTALIS & FRANKEL LLP
INTELLECTUAL PROPERTY DEPARTMENT
919 THIRD AVENUE
NEW YORK
NY
10022
US
|
Family ID: |
32719228 |
Appl. No.: |
10/727344 |
Filed: |
December 1, 2003 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60505345 |
Nov 29, 2002 |
|
|
|
60505346 |
Nov 29, 2002 |
|
|
|
60505344 |
Nov 29, 2002 |
|
|
|
Current U.S.
Class: |
345/619 |
Current CPC
Class: |
G06T 2219/2016 20130101;
G06F 3/0481 20130101; G06T 19/20 20130101; G06F 3/04812 20130101;
G06F 3/04815 20130101; G06F 2203/04806 20130101; G06F 3/04845
20130101 |
Class at
Publication: |
345/619 |
International
Class: |
G09G 005/00 |
Claims
What is claimed:
1. A method of presentation of 3D models in a 3D data display,
comprising: displaying data in an overview mode where localization
markers can be set, manipulated and viewed; and displaying data in
a local mode where data in an interest region surrounding a
localization marker are rendered using different display
parameters.
2. The method of claim 1, wherein said different display parameters
provide greater detail.
3. The method of claim 2, wherein said greater detail includes one
of enlargement or display of additional or alternate properties of
the data according to a defined representational scheme.
4. The method of claim 1, wherein said different display parameters
include a scale change.
5. The method of claim 4, wherein the local mode display uses one
of the localization markers or a user-designated point as a center
of scaling.
6. The method of claim 5, wherein the local mode display moves the
center of scaling to an optimum viewing point in the display.
7. The method of claim 1, wherein in the overview mode the
localization markers are displayed without regard to their being
partially or totally occluded by opaque regions of the
model(s).
8. The method of claim 1, wherein in the overview mode the
localization markers are displayed with regard to being partially
or totally occluded by opaque regions of the model(s);
9. The method of claim 1, further comprising displaying data in a
cycling mode, wherein a user may step through local mode displays
of all current detail regions.
10. The method of claim 1, further comprising simultaneously
displaying one or more selected regions of interest using their
respective local mode display parameters, while displaying all or
part of the non-selected portions of the model(s) using overview
mode display parameters.
11. The method of claim 1, wherein the boundaries of a region of
interest are controllable by the user.
12. The method of claim 10, wherein a user may set and adjust
parameters governing region of interest boundaries globally or
specifically to each individual region of interest.
13. The method of claim 12, wherein a user may modify region of
interest boundaries in overview mode, in local mode, or in both
overview and global mode.
14. The method of claim 1, wherein in overview mode the
localization markers are displayed using an indication icon.
15. The method of claim 1, wherein in overview mode boundaries of
the region of interest surrounding each potential localization
marker point are displayed, such that a user can see what a given
region of interest would contain.
16. The method of claim 1, wherein in overview mode a localization
marker is displayed at point.
17. The method of claim 16, wherein in overview mode a potential
region of interest is displayed surrounding each localization
marker point, rendered using local mode display parameters.
18. The method of claim 17, wherein when viewing the potential
region of interest, a user can change its shape.
19. The method of claim 18, wherein in overview mode as a user
moves a cursor or other indicator through the model the displayed
potential region of interest associated therewith moves
accordingly.
20. The method of claim 1, wherein a region of interest can have
boundaries parallel to those of the overview mode display or
nonparallel to the boundaries of the overview mode display.
21. The method of claim 1, wherein each region of interest
associated with each localization marker can have unique boundaries
of arbitrary shape.
22. A computer program product comprising: a computer usable medium
having computer readable program code means embodied therein for
controlling the scaling of a 3D computer model in a 3D data display
system, the computer readable program code means in said computer
program product comprising: computer readable program code means
for causing a computer to display data in an overview mode, wherein
localization markers can be set, manipulated and viewed; and
computer readable program code means for causing a computer to
display data in a local mode wherein data surrounding a
localization marker are rendered using different display
parameters.
23. The computer program product of claim 22, further comprising
computer readable program code means for causing a computer to
display a potential region of interest in overview mode surrounding
each point a user considers.
24. The computer program product of claim 23, further comprising
computer readable program code means for causing a computer to
display each said potential region of interest using its associated
local mode display parameters.
25. The computer program product of claim 22, further comprising
computer readable program code means for causing a computer to
facilitate interactive modification by a user of at least one of
region of interest boundaries, region of interest display
parameters, and localization marker icons, said interactive
modification operable while the system is in at least one of
overview mode, local mode and both overview mode and local
mode.
26. A program storage device readable by a machine, tangibly
embodying a program of instructions executable by the machine to
implement a method to control scaling of a 3D computer model in a
3D display system, said method comprising: displaying data in an
overview mode wherein localization markers can be set, manipulated
and viewed; and displaying data in a local mode wherein data
surrounding a localization marker is rendered using different
display parameters.
27. The method of claim 1, further comprising: displaying data in a
sub-local mode, wherein data in one or more sub-regions of interest
respectively surrounding one or more sub-localization markers are
rendered using different display parameters than those of local
mode; wherein said sub-localization markers are set, manipulated
and viewed in local mode, and located within a particular region of
interest surrounding a localization marker.
28. The method of claim 27, wherein: sub-localization markers have
all properties in relation to sub-regions of interest that
localization markers have in relation to regions of interest;
sub-local mode has all properties in relation to local mode that
local mode has in relation to overview mode; and sub-localization
markers are displayed within their particular region of interest
whenever the particular region of interest that contains them is
displayed.
Description
CROSS REFERENCE TO OTHER APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Patent Applications 60/505,345, 60/505,346 and 60/505,344, each
filed on Nov. 29, 2002, and all under common assignment
herewith.
[0002] This application is related to "METHOD AND SYSTEM FOR
SCALING IN 3D DISPLAYS ("Zoom Slider"), filed on Dec. 1, 2003, Luis
Serra, Inventor, the specification of which is hereby incorporated
herein by reference. Said application will be referred to herein as
the "Zoom Slider" application.
FIELD OF THE INVENTION
[0003] The present invention relates to the field of interaction
with computer models in three dimensions, wherein objects are
displayed to a user for interaction with them. More precisely the
present invention relates to managing of multiple detailed viewing
sites in such a display.
BACKGROUND OF THE INVENTION
[0004] A three-dimensional (3D) model is a data set associating
spatial coordinates (u, v, w) in model space with values to be
displayed to a user by a shader in such a way that the resulting
colors or other sensory qualities shown appear to a user localized
at corresponding locations. Typically, relative to the display
coordinates (x, y, z), which may be chosen relative to a user's
apparent point of view, the model space has a position giving the
correspondence. Most often the correspondence is specified as a
matrix relationship 1 [ x y z ] = [ a 1 1 a 1 2 a 1 3 a 2 1 a 2 2 a
2 3 a 3 1 a 3 2 a 3 3 ] [ u v w ] + [ X Y Z ] , equivalently [ x y
z 1 ] = [ a 1 1 a 1 2 a 1 3 X a 2 1 a 2 2 a 2 3 Y a 3 1 a 3 2 a 3 3
Z 0 0 0 1 ] [ u v w 1 ] ,
[0005] or x=Au for short, sometimes using a non-linear
transformation rather than one that can be represented by a matrix.
In general, an application's user interface provides many ways to
modify the relation A, which are experienced by a user as rotating,
moving, warping or deforming the displayed model or models. A model
may be a grid of scanned density or other values from, for example,
a computerized tomography (CT) or magnetic resonance (MR) scan,
from seismography, or from numerous other sources of such data
which attach numbers to each of a regular grid of points in space.
As well, such a model may equally contain a list of lines,
polygons, algebraic or analytic surfaces, etc., which represent a
geometric form (such geometric form often being described by yet a
further system of `object` coordinates (q, r, s), whose position in
model space is described by another transformation, such as, for
example, u=Bq), together with quantitative properties distinct from
position that may vary from point to point. (Often in a given
application such properties may consist of, or be represented as,
for example, color and transparency data.) In all such cases a
shader uses the model data together with auxiliary information
about lighting, rules for what is to be transparent, modification
of colors to distinguish significant value ranges, etc., to create
an apparent 3D image, often in stereo (using one or more means to
arrange that a user's two eyes each see slightly different views,
from which a user's visual system can construct a sensation of
depth).
[0006] In interacting with such models it often arises that a user
must pay attention to a particular smaller detail region within the
model space, wherein one or more of the models displayed therein
have significant features. Often such detail region is displayed at
a greater magnification (utilizing the techniques, for example, as
described in the Zoom Slider application), a higher resolution, or
in some other manner which would exhaust a system resource if
applied to the display of the entire model. (Such as, for example,
the fact that magnification of the entire model would move parts of
it beyond the display window, thus requiring unavailable display
area for full display; or a requirement of more closely spaced
sampling of the data, or anti-aliasing, or shading with scattered
light, etc., may require more computation time between successive
views than is compatible with a smooth apparent rotation of
viewpoint, etc.).
[0007] Particularly where models include scanned volume data, it
can be laborious to select a detail region. The particular display
software, or a user, may standardize the size of detail regions in
general, but the useful location of each particular region varies
with the data set, as, for example, tumors differently located in
different brains, and with a user's concerns, as, for example,
vessel anatomy as opposed to bone fracture details. A user must
choose the position of its center, or some other convenient
anchoring point. Moreover, even in a stereo display system a user's
sense of what data feature occurs at what depth may be limited in
precision, particularly, as is common in rendering volume data,
inasmuch as some regions are rendered so as to appear neither
transparent nor opaque, but translucent. Choosing a specific
significant point may involve cropping the display to remove
obscuring data, rotating its apparent position for a more revealing
view, and other laborious maneuvers. While in a fully interactive
3D display environment it is somewhat easier to move a selected
point in the display (merely drag and drop it in any direction with
a displayed stylus experienced as being locked to a physical device
held in the hand), in a mouse-and-mousepad interface even the
simple modification of a model point's location is a multi-step
process. A common solution for such 2D interfaces is to present a
user with a `main window`, which shows (a) a perspective view from
the direction chosen to provide the most revealing possible view,
usually parallel to no axis, and (b) subsidiary windows showing
parallel views from the (x, y), (x, z) and/or (y, z) directions
separately. Moving the cursor to one of these subsidiary views
directs user input (such as that provided, for example, via a
mouse, trackball, or other relative spatial movement based input
peripheral) to control of the corresponding coordinates, and a user
can watch an object or 3D cursor move both in the subwindow and the
main window.
[0008] To adjust any (x, y, z) position in display space thus
requires two or more successive operations in such windows, while
aiming for a target more clearly visible in the main window. It
takes longer than a single movement, and occupies a great fraction
of the scarce display space available (thus reducing detail in the
main window). Rotation of the view is similarly more complex.
[0009] Moreover, even in a fully interactive 3D display environment
the specification of a region of interest is an effort that it
benefits a user to minimize, much more so in an environment where
only 2D user display control is available. In particular, many
applications require a user to select a detail region for some
specific work such as, for example, by assigning or modifying
points in several models that--if the models were exactly
aligned--would be located at the same point (u, v, w) in model
space, then to select one or more others, and then return once
again to the earlier selected detail regions for refinement of the
initial work. It can become very time-consuming to repeat, on each
return to a given detail region, the navigation interactions that
were needed to arrive and orient such region of interest the first
time.
[0010] Objects of the present invention can include reducing this
load on a user managing multiple viewing sites and facilitating a
user's ability to interact with computer models.
SUMMARY OF THE INVENTION
[0011] A system and method of presentation of 3D data sets in a 3D
display is presented. In exemplary embodiments according to the
present invention the method involves displaying data in an
overview mode wherein localization markers can be set and
manipulated, and in a local mode wherein data surrounding a
localization marker can be rendered using different display
parameters. In exemplary embodiments according to the present
invention the mode in which data is displayed can be selected by a
user. In preferred exemplary embodiments according to the present
invention the method can be implemented recursively, such that
within local mode sub-localization markers can be set and
manipulated, and data can be displayed in a sub-local mode wherein
data surrounding a sub-localization marker can be rendered using
display parameters which can be different from both those of an
overview display mode and those of a local display mode.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 shows an exemplary 3D model displayed within an
exemplary crop box according to an exemplary embodiment of the
present invention;
[0013] FIG. 2 shows the model of FIG. 1 with a movable 3D cursor
and localization markers set at various points according to an
exemplary embodiment of the present invention;
[0014] FIG. 3 shows an exemplary set of icons which may be used to
denote the location of a localization marker point according to an
exemplary embodiment of the present invention;
[0015] FIG. 4 shows an exemplary 3D model displayed with box type
icons representing detail regions, each superimposed on the model
without regard to opacity according to an exemplary embodiment of
the present invention;
[0016] FIG. 5 shows the model and boxes of FIG. 4, displayed with
regard to opacity according to an exemplary embodiment of the
present invention;
[0017] FIG. 6 is an exemplary modular software diagram according to
an exemplary embodiment of the present invention;
[0018] FIG. 7 is a process flow diagram according to an exemplary
embodiment of the present invention;
[0019] FIGS. 8-15 depict an example application of precise
measurement of distances according to an exemplary embodiment of
the present invention; and
[0020] FIG. 16-23 depict an example application of precise
insertion of markers according to an exemplary embodiment of the
present invention.
DETAILED DESCRIPTION OF THE INVENTION
[0021] A 3D data display operates in two modes, local and overview.
In overview mode, the display of a model space is accompanied by
the display of a number of localization markers. In this mode a
user may add new markers, delete existing ones, and select or move
a currently active one. Such markers have positions which are fixed
relative to the model space. In an exemplary embodiment the markers
remain visible to a user even where their location would normally
be obscured by opaque shading.
[0022] In local mode, the detail region surrounding an active
localization marker is displayed using different display parameters
than those used in overview mode, such as, for example, to provide
some type of greater or alternative detail. Such detail may
include, for example, the use of higher resolution, more
computation-expensive rendering techniques, or a larger scale
(zoom). In the case of, for example, zoom, no part of the display
outside of the detail region is rendered so as to obscure a user's
view of the enlarged detail region. To achieve this, the display
outside of region may be subject to additional cropping, or not be
rendered at all. As well, the zoomed view may be centered on a
standard convenient point in the display region, be centered on the
display point indicated by the chosen localization marker, or even
move from one to the other.
[0023] Local and overview modes are defined relative to the
currently active display parameters. Thus, if the general view is,
for example, enlarged, the display of a detail region is enlarged
by the same factor, where possible within system resources.
[0024] In exemplary preferred embodiments the method can be used
recursively, so that within a given detail region additional
sub-localization markers may also be specified. In such exemplary
embodiments the same user actions that cause transition from
overview to local mode also cause transition from the current local
mode to a sub-local mode, with new zoom, addition of detail, or
display parameters used in the sub-detail region displayed around a
given sub-localization marker.
[0025] The methods of the present invention are implementable in
any 3D data display system, such as, e.g., a volume rendering
system. In general, a volume rendering system allows for the
visualization of volumetric data. Volumetric data are digitized
data obtained from some process or application, such as MR and CT
scanners, ultrasound machines, seismic acquisition devices, high
energy industrial CT scanners, radar and sonar systems, and other
types of data input sources. One of the advantages of volume
rendering, as opposed to surface rendering, is that it allows for
the visualization of the insides of objects.
[0026] One type of such 3D data display system is what is referred
to herein as a fully functional 3D data display environment (such
as, e.g., that of the Dextroscope.TM. system of Volume Interactions
Pte Ltd of Singapore, the assignee of the present application).
Such systems allow for three-dimensional interactivity with the
display. In such systems a user generally holds in one hand, or in
each hand, a device whose position is sensed by a computer or other
data processing device. As well, the computer monitors the status
of at least one control input, such as, e.g., a button, which a
user may click, hold down, or release, etc. Such devices may not be
directly visible to a user, being hidden by a mirror; rather, in
such exemplary systems, a user sees a virtual tool (a computer
generated image drawn according to the needs of the application)
co-located with the sensed device. In such exemplary systems the
locational identity of a user's neuromuscular sense of the position
of the held device, with a user's visual sense of the position of
the virtual tool is an interactive advantage.
[0027] In exemplary embodiments, at any time the system contains a
current specification of the enhancements to be used in viewing
each detail region, such as, for example, preferred scale changes,
changes of resolution or sampling step, or the use of
anti-aliasing. These may be preset with default values, or
alternatively be modifiable by a user via, for example, a system of
menus, voice commands, sliders or other interaction means as may be
known in the art.
[0028] Via an input signal, such as, for example, a menu selection,
voice command, or other interaction, a user directs the system to
enter the overview mode. In this mode, as illustrated in FIG. 1,
all or part of the currently active model or models 101 are
displayed within the current crop box 102, as determined by
previous interactions between a user and the system, at the current
scale and viewing position. At any point prior to or during entry
into overview mode a user may invoke other methods to change the
scale, move or rotate the crop box 102 and with it the viewed part
101 of the model(s), or modify the size or position of the crop box
relative to the model space, thus changing the visible portion of
the model or models displayed.
[0029] As well, in overview mode, as depicted, for example, in FIG.
2, a user moves a cursor 201 through the 3D display, by any means
as may be available in a given application. This may be by 3D drag
and drop in a Dextroscope.TM.-like system, by the use of auxiliary
windows in a mouse or other 2D spatially-controlled system, or by
other methods as may be known in the art. When the cursor is at a
point approximately central to a desired detail region, a user
signals that she wishes to set or designate a localization marker.
She then, for example, clicks a button on a hand-held controller
such as a mouse or a 3D position sensor, issues a voice command,
presses a foot pedal, or otherwise signals the system, as may be
known in the art, that the active cursor position is to be set as a
localization marker point 211. A marker point icon 211 is then
displayed by the system fixed at that position relative to the
model space.
[0030] FIG. 2 thus depicts a number of such designated localization
markers. With reference to FIG. 3, such a localization marker may
be displayed as, for example, a simple cross pattern 301, a pattern
of triangles 302 or 303 pointing to the central point, a 3D
structure with apparent solidity 304, or alternatively a box frame
305 or other structure which marks the boundaries of the selected
detail region, or such other marker or combination of markers as
may be useful as known in the art.
[0031] A further parameter governing detail display is the size (as
a cube, or as the side lengths of a rectangular box) of the
exemplary box frame 305. A user may modify this size, either in
overview or in detail mode, by, for example, dragging an edge or
corner of the box, as is conventional with frames and windows in a
2D environment, or by such other means as may be known in the art.
Other detail region geometries such as spheres, ellipsoids,
polyhedra, etc., with interactive objects by which form and size
may be manipulated, may be equivalently substituted within the
scope of the present invention.
[0032] The cursor may be, for example, a stylus tip, an icon
identifying a point at its center, or any other means of indicating
to the user the point to be selected or moved as the center or
other reference point of the detail region. In a preferred
exemplary embodiment a box frame 305 is used. The user thus sees,
before clicking a given point (or otherwise selecting, as may be
the case in other exemplary embodiments), the detail region that
will result, and can thus judge what it will enclose. The
orientation as well as the location of the displayed box frame
cursor 305 matches that of the detail region that will be selected.
Thus, if in an exemplary embodiment the system always shows a
detail region with edges parallel to the model space axes, such an
indicator will align itself with these axes, not rotating when the
stylus turns with the user's hand. Alternatively, in other
exemplary embodiments where detail regions are not so restricted, a
detail region indicator (including both the center of the detail
region indicator as well as any detail region boundary marking
indicator) may move as if rigidly attached to the stylus, rotating
as the latter does. In an exemplary embodiment of such systems, the
selected detail region reproduces the cursor orientation current at
the moment of selection, surrounded by a box frame of exactly the
same shape, size and position as the cursor at that moment. If a
user wishes, for example, to create a detailed view of a straight
artery section that is far from parallel to any model axis, this
orientation control is of considerable value. Later dragging of the
detail region can then include rotating it.
[0033] In exemplary embodiments an interface by which the user may
modify the default shape and size of the detail region(s) selected
is provided. Such an interface may comprise, for example, sliders
or other interactive objects as may be known in the art. In a
preferred exemplary embodiment of the invention in a fully
functional 3D environment, where both rotation and translation are
easily controlled by a user's hand, these parameters (i.e., those
that control shape and size of the detail region) may be
manipulated by use of a box frame cursor, for example by a `sticky
point` interactive object at a standard location in the display.
When a user places an edge or corner of a box frame cursor 305 near
this point and signals (for example, by holding down a button) an
intention to change these parameters, the respective edge or corner
is constrained (until the user signals a change of intention, for
example, by releasing a button) to continue to pass through or lie
on the sticky point. The box frame cursor center is moved as usual
by the user. As a result, pulling the cursor away from the sticky
point enlarges the box, while pushing toward it shrinks it. Lateral
movement may increase one side length while decreasing another,
unless the box frame is constrained by the exemplary system to
remain cubical. Alternate embodiments may have a variety of
analogous means with which to control the parameters governing
detail region size as may be desirable and as may be known in the
art.
[0034] In exemplary embodiments existing localization markers may
be manipulated as follows. When, in overview mode, a cursor is, for
example, within a pre-set proximity zone of a particular
localization marker, that marker becomes the current marker if no
other is closer to the cursor. In this context a proximity zone can
be defined, for example, as where a distance vector (x, y, z)
between the cursor and the particular localization marker have less
than a currently set threshold value, where a distance vector can
be defined, for example, as x.sup.2+y.sup.2+z.sup.2- , as
.vertline.x.vertline.+.vertline.y.vertline.+.vertline.z.vertline.,
or as max (.vertline.x.vertline., .vertline.y.vertline.,
.vertline.z.vertline.). Once in a given marker's proximity zone, a
user may enter a dragging mode by, for example, holding down a
specified button, or by otherwise indicating such an intention as
may be defined by the display system, in which mode the
localization marker will move with the cursor does until a user
signals an exit from such dragging mode by, for example, releasing
a button or making some other appropriate sign. The existence of a
current localization marker does not interfere with the placement
of a new marker near it, which is simply achieved by invoking the
standard user action for such placement, as described above.
[0035] Alternatively, by invoking another signal such as, for
example, a double click, a right mouse click, a voice command or
other appropriate interactive sign, a user may delete the current
marker from the localization marker list maintained by the system.
In a preferred exemplary embodiment, if the software supports a
general "undo" command the issuing of this command immediately
after such deletion can cause reinsertion of the deleted marker in
the list.
[0036] Similarly to the proximity region used to activate drag and
drop functionality near a particular marker, defined by some
proximity criterion, each localization marker determines a detail
region, such as, for example, the set of points whose distance
vector (x, y, z) from the marker have x.sup.2+y.sup.2+z.sup.2, or
.vertline.x.vertline.+.vertline.y-
.vertline.+.vertline.z.vertline., or max (.vertline.x.vertline.,
.vertline.y.vertline., .vertline.z.vertline.) less than a currently
set value. When in local mode, the detail region surrounding an
active localization marker is displayed using different display
parameters than that used in overview mode, such as, for example,
to provide some type of greater or alternative detail, as described
above. One example of such greater detail that can be provided is a
scale change, as described in the Zoom Slider application.
[0037] Local and overview modes are defined relative to the
currently active display parameters. Thus, if the general view is,
for example, enlarged, the display of a detail region is enlarged
by the same factor, where possible within system resources.
[0038] In an alternative exemplary embodiment, the system may
additionally support a cycling mode in which the detailed region
surrounding each localization marker in the list is displayed as a
user issues a step command (by, for example, click, voice or other
means as may be known in the art). In a 2D interactive system such
as a mouse interface, where cursor position is controlled by
relative motions, it is appropriate to cause the cursor to jump to
an appropriate position for dragging. A 2D mouse user is accustomed
to sudden discontinuous movements of the cursor, such as, for
instance, jumping to a newly appearing text entry box. Thus, a
mouse-driven cursor that must appear on a detail icon to drag it
should jump from each currently selected detail region to the next,
saving motion on the mousepad. In other exemplary systems a user
may expect an absolute cursor position to always correspond to the
same absolute control position, such as in a Dextroscope.TM.-like
or touch screen interface, following the tool in the user's hand,
with no cursor movement that does not follow the hand. In such a
system, cycling behavior may desirably be omitted. The cycling mode
may be active concurrently with the other functionalities of
overview mode.
[0039] The display of localization markers involves certain
choices, which may be made differently in various alternative
exemplary embodiments of the present invention. The markers 211 in
FIG. 2, for example, are shown in an unobscured display, as are the
boxes 411 in FIG. 4. Using such a display alternative, when viewing
from a single direction, it is not easy for a user to perceive
their depth (i.e., the distance from a user's viewpoint). If the
displayed model 101 or 401, respectively, is rendered as an opaque
object, a possible perceptual default is that the marker must be
nearer a user than the object it occludes. In a display system with
stereo views, parallax or other depth cues, a user's visual system
has information that may place the perception at a greater depth,
in conflict with the occlusion cue.
[0040] To address this situation, in an alternative exemplary
embodiment, such as is depicted in FIG. 5, markers are hidden where
they would be occluded by the model 501. This presents a consistent
set of depth cues to a user, but may partially or completely
conceal one or more markers as a tradeoff. (Since a marker may have
been originally placed when the crop box was smaller, thus exposing
different parts of the model to view, or it may have been placed
when a shader was in use that made more of the model transparent, a
given marker may subsequently become invisible within an opaquely
rendered model from all directions.) In a preferred exemplary
monoscopic display embodiment, a larger scale display structure is
added to a marker, such as, for example, long lines parallel to
each axis, which will thus protrude beyond an opaquely rendered
model. In a preferred exemplary stereoscopic display embodiment,
the technique of `apparent transferred translucency` can be used,
as described more fully in the Zoom Slider application. In such a
technique a model appears to a user to be translucent (only) with
respect to light emanating from the displayed marker, thus allowing
such marker--although actually occluded--to be easily seen.
[0041] Where an enhanced display to be used in a given detail
region surrounding a localization marker involves only changes that
do not change scale (such as, for example, finer resolution or anti
aliasing), it may be included in the overview display. If system
resources are sufficient, all detail regions may simultaneously be
shown in such an exemplary enhanced manner. If they are
insufficient, such enhanced rendering may be restricted to the
detail region centered on the current marker, and optionally
include, for example, the other N detail regions last visited,
where N may be a system default number or a number dynamically
computed as a function of the current size of detail regions and of
the enhancement options currently in force.
[0042] Where the enhancement to the display in a given detail
region includes a change of scale, a user may signal a change to
local mode by click, command or other means, in which a cursor
entering the proximity zone of a given localization marker
immediately causes the corresponding detail region of that marker
to be displayed on a larger scale. Such magnified display may
remain centered on the same point, or may be centered on a
preferred viewing point near the center of the display region.
Alternatively, the system may indicate (by, for example,
highlighting or otherwise) which is the current localization marker
and corresponding detail region, and provide an interface by which
desired zoom factors can be input by a user, such as, for example,
slider or voice commands. As well, in this setting the zoomed
display may remain centered on the marker point, or on the
preferred viewing point. Alternatively, it may slide toward the
preferred viewing point as the scale is enlarged as described in
the Zoom Slider application.
[0043] In exemplary embodiments where a change to local mode is
automatically triggered simply by a cursor moving into a detail
region, without any separate command signal from a user, upon
cursor exit from the detail region the display returns to overview
mode. Alternatively, an exemplary embodiment may make provision for
a discrete action to cause this effect, such as, for example, a
click, voice command or other user generated signal. As well, in
exemplary embodiments where a discrete user signal is used to
trigger entry into local mode, a discrete signal is generally used
for exit therefrom.
[0044] Invoking the cycling mode when a detail region is zoomed
causes each detail region in turn to be displayed in a zoom mode.
System settings, optionally adjustable via a user preferences
interface, determine whether (a) all detail regions are displayed
using as a common zoom factor that which was current when cycling
was invoked; (b) whether each is to be displayed using the zoom
factor which has been most recently set for it, or (c) some other
region specific zoom factor as may be determined. Similar controls
determine whether all detail regions are to be displayed as
centered on (a) their respective localization marker points, (b)
the preferred viewing point, (c) moving from the localization
marker point to the optimum viewing point as per the techniques
described in the Zoom Slider application, or (d) as determined by a
user intervention interactive object, such as, for example, a zoom
slider.
[0045] In an exemplary embodiment, in local mode, portions of the
model(s) outside the detail region may remain visible to provide
context. In such embodiments they are so rendered so as not to
obscure the enlarged view of the detail region. This may be
achieved by, for example, clipping the overview display to render
invisible those parts of the model that are at a lesser depth than
the current marker point, or some equivalent functionality.
[0046] As well, an exemplary embodiment may include the ability to
detach a scaled view or other modified display of a detail region,
making it an additional component of the displayed scene that does
not vanish when another detail region is selected. The motion
controls of the system can then be applied separately to this view,
to its respective crop box, such that a user may move it to a
convenient place for viewing in a displayed scene including other
elements such as other scaled regions. In such an exemplary
embodiment, moving a cursor into the detached view's crop box makes
it the active region for communication with the system, determining
such aspects as the mapping between cursor position (x, y, z) and
model space position (u, v, w) as described above. This is useful
if, for example, a user needs to define a straight line in model
space by moving one end, with the aim of arranging it such that a
distant part of the line passes accurately through a particular
point. This therefore requires simultaneously observing both the
one end and the distant part in scaled views, often where the scale
required would be incompatible (too large) with including both in a
common view that includes intervening points. (The alternative, a
need to switch repeatedly between the two scaled views would make
this a very laborious, inefficient and thus undesirable process.)
The need for such geometric construction with precision at multiple
points arises in many 3D applications, from the planning of
surgical procedures or mines to the computer aided design (CAD) of
mechanical systems.
[0047] When a detail region is shown in a change of scale (zoomed)
view, its localization marker point remains visible (as do any
other marker points within the detail region), with the apparent
size of the marker icon unzoomed. It can still be manipulated as in
overview mode, as can any other marker points within the region,
with the result of changing the detail region and thus the
displayed portion of the model or models.
[0048] In preferred exemplary embodiments, the same commands and
interface(s) actions used to manipulate the localization marker in
an overview mode are available in a local mode.
[0049] As well, in exemplary embodiments, when in a zoomed view of
a detail region a user may define new marker points, which may have
model space positions (u, v, w) which are defined either (a)
absolutely or (b) relative to the localization marker point. These
new localization markers may either act as new marker points
sharing the common list with the current one, or act as a sub-list
of marker points visible only when this detail region is active,
and with smaller detail regions associated with them which may
themselves be further zoomed or presented in greater detail. This
latter feature is less often useful when working with scanned data
models (which become blurred, like 2D digital images, when greatly
zoomed) than with high-precision models such as those used in CAD,
where points on a machine that is meters wide may be specified with
micrometric precision, and substructures may require their own zoom
marker families. The new localization marker points can be caused
to so act either according to defined system behavior or as
adjustable by user settings.
[0050] FIG. 6 depicts an exemplary modular software program of
instructions which may be executed by an appropriate data
processor, as is or may be known in the art, to implement a
preferred exemplary embodiment of the present invention. The
exemplary software program may be stored, for example, on a hard
drive, flash memory, memory stick, optical storage medium, or other
data storage devices as are known or may be known in the art. When
the program is accessed by the CPU of an appropriate data processor
and run, it performs, according to a preferred exemplary embodiment
of the present invention, a method of displaying a 3D computer
model or models in a 3D data display system. The exemplary software
program has four modules, corresponding to four functionalities
associated with a preferred exemplary embodiment of the present
invention. Numerous alternate modular software program
implementations are also possible for various exemplary embodiments
and preferred exemplary embodiments of the invention.
[0051] The first module is, for example, a Localization Marker List
Manager Module 601, which can accept user inputs via a user
interface as may be known in the art, such as, for example,
localization marker positions, sub-localization marker positions,
detail region boundaries, detail region orientations, cropping
data, etc., display parameters of detail regions, (including, for
example, a zoom scaling factor or certain quantitative properties
to be displayed for the 3D data), all as described above, as well
as user or system generated mode activation/selection signals.
[0052] A second module is, for example, a Local Mode and Recursive
Display Module 602, which, once signaled by the Localization Marker
List Manager Module 601 that local mode has been signaled, displays
the data within the detail region and/or sub-detail region
according to the display parameters associated with the detail
region and/or the sub-detail region surrounding the current
localization marker and/or sub-localization marker.
[0053] A third module, the Overview Mode Display Module 603
receives inputs from the Localization Marker List Manager Module
601 regarding the location of localization markers and applies the
defined rules, including, for example, rules as to symbology,
opacity, apparent transferred translucency, etc., as described
above, to display the various localization markers then maintained
in the system.
[0054] A fourth module is, for example, a Cyclic and Multiple
Detail Region Display Module 604, which takes data inputs from, for
example, the Localization Marker List Manager Module 601, and
displays the various detail regions in succession if cyclic mode is
signaled, or displays multiple detail regions in the display scene
in response to a detach command.
[0055] FIG. 7 depicts a process flow diagram according to an
exemplary embodiment of the present invention. In the depicted
exemplary embodiment, the cursor is a box frame indicator such as
305 with reference to FIG. 3. Thus, as described above, before
selecting a given point as a localization marker point, the user
sees the detail region that will result from selecting that point,
and can judge what it will enclose. The process flow diagram
illustrates the process of selecting a local mode view, and
interacting in local mode with the model and/or the detail region,
and finally returning to overview mode. Beginning at start 701,
flow passes to 702, where a cursor or equivalent icon is displayed
at the tip of a virtual tool with which a user can move through a
3D display. At 703 the system ascertains whether the user has
selected the local mode, as described above. If "NO" process flow
returns to 702. If "YES", at 704 the system enters the local mode,
and accordingly displays an enlarged view of a model or models, as
well as the boundaries of the detail region (or region of
interest). At 705 a user may manipulate the model(s), or may
manipulate the boundaries of the region of interest, using, for
example, a tool tip. At 706, if in 705 the region of interest's
boundaries were modified, the model or models are displayed within
the new boundaries of the region of interest. At 707 the system
queries whether the operation should stop. If "NO" then flow
returns to 705. If "YES" then flow moves to 708 and terminates,
returning the user to overview mode.
[0056] The following is exemplary pseudocode which can be used to
implement an exemplary embodiment of the present invention.
[0057] Pseudocode:
1 class RegionControl { // Control Object for Local Mode public:
void Render_Region_Of_Interest ( ); bool StartActivate ( ); bool
EndActivate ( ); void Store_Model_Display_Settings ( ); void
Restore_Model_Display_Sett- ings ( ); void
Update_Model_Display_Settings ( ); }; void
RegionControl::Render_Region_Of_Interest ( ) { // display the
cursor icon } bool RegionControl::StartActivate ( ) { // returns
true if control is activated by means of pressing a button. //
otherwise, returns false. } bool RegionControl::EndActivate ( ) {
// returns true if control is deactivated by means of releasing a
button. // otherwise, returns false. } void
RegionControl::Store_Model- _Display_Settings ( ) { // store the
Overview Mode display settings of the computer model to be
modified. } void RegionControl::Restore_Model_Display_Settings ( )
{ // restore the modified display settings of the computer model. }
void RegionControl::Update_Model_Display_Settings ( ) { // modify
the display settings of the computer model. }
[0058] Program Entry Point:
2 void main ( ) { // Set up variables and states, create objects
Initialization ( ); RegionControl regionControl; // Create one
Local Mode control object while (true) { Render_Model ( ); //
display model in Overview Mode (includes crop box, etc) // Display
the 3D tool with cursor icon
regionControl.Render_Region_Of_Interest ( ); if
(regionControl.StartActivate ( ).sup..quadrature.) // widget
`listens` for user signal that it's wanted. { // Enter Local Mode
// Store the display settings of the Overview Mode for later use.
regionControl.Store_Model_Display_Settings ( ); // Change the
display settings for the model as desired // (i.e., list of local
regions, with shared or individual // scale, level of detail, crop
box, etc). regionControl.Update Model_Display_Settings ( ); } else
if (regionControl.EndActivate ( ).sup..quadrature.) { // Exit Local
Mode, go to Overview Mode // Restore the original Overview Mode
display settings of the model.
regionControl.Restore_Model_Display_Settings ( ); } Update_System (
); // Execute the display and system commands. // User sees desired
view, with/without modified region(s). } }
[0059] Exemplary Implementation and Applications
[0060] To illustrate the functionalities available in exemplary
embodiments of the present invention, two exemplary applications of
exemplary embodiments of the present invention will next be
described with reference to FIGS. 8-23. The screen shots were
acquired using an exemplary implementation of the present invention
on a Dextroscope.TM. 3D data set display system, from Volume
Interactions Pte Ltd of Singapore. Exemplary embodiments of the
present invention can be implemented on this device. Visible in the
figures are a 3D object and a virtual pen controller and sometimes
a virtual control palette which appears below it as well as other
icons.
[0061] A. Illustration of the Precise Measurement of Distances
[0062] FIGS. 8-15 depict an exemplary embodiment of the present
invention where the ability to easily shift between overview mode
and local mode is used in the precise measurement of distances
between points in 3D data sets.
[0063] FIG. 8 depicts an original object, i.e., a human skull, from
a CT data set, positioned somewhere in 3D space. A user intends to
measure with great precision the distance between two points within
the object, but would also like to keep an overview of the data
during the operation. FIG. 9 depicts a user selecting an exemplary
tool. The tool has a cube box at its tip that indicates the
relative size of the magnification that will take place during the
measurement. The size of the cube can be adjustable. FIG. 10
depicts a user moving the tool with the cube at its tip to the area
of interest, an aneurysm.
[0064] FIG. 11 depicts an exemplary situation where as soon as the
user actuates a control function, for example, here, when a user
presses a button on the tool, the display changes to a
magnification view with the size indicated by the cube on the tool.
At this magnification level, a user can see the desired detail, and
can thus precisely position a first measurement point on one side
of the object (i.e., the right side of the aneurysm from a viewer's
perspective).
[0065] FIG. 12 depicts how when a user implements a control
function, for example, by releasing a button, and thereby goes back
to overview mode, he can see the first measurement point which was
placed in the context of the entire 3D object or data set.
[0066] Similarly, FIG. 13 depicts a user moving the tool away from
the aneurysm for an overview.
[0067] FIG. 14 depicts a user moving the tool to the other desired
measurement point, and FIG. 15 depicts the effect of the user
pressing again the button on the tool, to return to a high
magnification mode, or, generally speaking, to a "local mode." In
the depicted exemplary embodiment, the system reads out to a user
the distance between the first laid measurement point and the
current location of the tool tip.
[0068] B. Example Illustration of Precise Insertion of Markers
[0069] FIGS. 16-23 depict a second exemplary use of the methods of
exemplary embodiments according to the present invention, precise
insertion of markers. The markers are, in the depicted exemplary
embodiment, placed for purposes of implementing a two-point
registration so that data from two different scan modalities can be
co-registered.
[0070] With reference to FIG. 16, a user desires to place with
accuracy two 3D markers at the center of two fiducial markers on
the CT data set. As seen in FIG. 16, these two fiducials are
respectively located substantially on the left eyebrow and the
right temple of the depicted head. To implement the marker
placement, the user selects a zoom tool (as described more fully in
the Zoom Slider application) by virtually pushing a zoom button on
the depicted virtual control palette.
[0071] FIG. 17 depicts a user moving the tool to the first fiducial
marker (substantially located on the left eyebrow of the depicted
head) on the skin of the patient. A cube at the tip of the tool
indicates the magnification level.
[0072] FIG. 18 depicts the effect of a user pressing a button on
the virtual tool, wherein he can see a magnified view. With
reference to FIG. 19, the user, operating in a magnified view,
moves the tool tip to the center of the fiducial on the surface of
the patient data (i.e., the depicted head).
[0073] With reference to FIG. 20, in the depicted exemplary
embodiment, when a marker is at the center of the desired spot
(here the fiducial) as in FIG. 19, a user can release a button to
place the marker, resulting in the view seen in FIG. 20, where the
tool is being moved towards the other fiducial. Thus, once marker
"1" has been placed, a user can proceed to place an additional
marker.
[0074] FIG. 21 depicts a user repeating the same procedure for a
second marker, where, similarly to the situation depicted in FIG.
17, the user moves the tool and associated magnification box over
the second fiducial. FIG. 22 depicts a second marker being
positioned, the user again operating in magnification mode. FIG. 23
depicts the situation where having returned to overview mode, the
placement of a second marker at the second fiducial can be seen in
the larger context of the entire head.
[0075] The present invention has been described in connection with
exemplary embodiments and exemplary preferred embodiments and
implementations, as examples only. It will be understood by those
having ordinary skill in the pertinent art that modifications to
any of the embodiments or preferred embodiments may be easily made
without materially departing from the scope and spirit of the
present invention as defined by the appended claims.
* * * * *