U.S. patent application number 11/176482 was filed with the patent office on 2006-02-09 for architecture for rendering graphics on output devices over diverse connections.
Invention is credited to Won-Suk Chun, Joshua Napoli, Thomas J. II Purtell.
Application Number | 20060028479 11/176482 |
Document ID | / |
Family ID | 35787617 |
Filed Date | 2006-02-09 |
United States Patent
Application |
20060028479 |
Kind Code |
A1 |
Chun; Won-Suk ; et
al. |
February 9, 2006 |
Architecture for rendering graphics on output devices over diverse
connections
Abstract
A system for displaying graphical information. The system
includes an asset server for storing information and a rendering
server in communication with the asset server. The rendering server
receives a graphics command and renders graphic display data in
response to the graphics command and to the information. The
rendering server is independently addressable from the asset
server.
Inventors: |
Chun; Won-Suk; (Cambridge,
MA) ; Napoli; Joshua; (Arlington, MA) ;
Purtell; Thomas J. II; (Cambridge, MA) |
Correspondence
Address: |
CANTOR COLBURN, LLP
55 GRIFFIN ROAD SOUTH
BLOOMFIELD
CT
06002
US
|
Family ID: |
35787617 |
Appl. No.: |
11/176482 |
Filed: |
July 7, 2005 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60586327 |
Jul 8, 2004 |
|
|
|
Current U.S.
Class: |
345/531 |
Current CPC
Class: |
G06F 3/14 20130101; G09G
3/003 20130101; B33Y 50/00 20141201; G06T 2210/32 20130101; G06T
15/00 20130101 |
Class at
Publication: |
345/531 |
International
Class: |
G09G 5/39 20060101
G09G005/39 |
Goverment Interests
GOVERNMENT RIGHTS
[0002] The U.S. Government may have certain rights in this
invention pursuant to Grant No. 70NANB3H3028 awarded by the
National Institute of Standards and Technology (NIST).
Claims
1. A system for displaying graphical information, the system
comprising: an asset server for storing information; and a
rendering server in communication with the asset server for
receiving a graphics command and for rendering graphic display data
in response to the graphics command and to the information, wherein
the rendering server is independently addressable from the asset
server.
2. The system of claim 1 wherein the asset server and the rendering
server are in communication via a network.
3. The system of claim 1 wherein the asset server is located in a
different geographic location than the rendering server.
4. The system of claim 1 wherein the information includes one or
more of rendering resources, geometry data, images and shader
programs.
5. The system of claim 1 wherein the rendering server further
transmits the graphic display data to a display device that is
independently addressable from the asset server and the rendering
server.
6. The system of claim 5 wherein the graphic display data is
transmitted to the display device via a network.
7. The system of claim 5 wherein the display device is located in a
different geographic location than one or more of the rendering
server and the asset server.
8. The system of claim 5 wherein the display device includes one or
more of a two-dimensional (2-D) display and a three-dimensional
(3-D) display.
9. The system of claim 5 wherein the display device includes one or
more of a 2-D printer and a 3-D printer.
10. The system of claim 1 wherein the graphics command is generated
by a client application.
11. The system of claim 10 wherein the client application transmits
the graphics command to the rendering server via a network.
12. The system of claim 10 wherein the client application is
located in a different geographic location than the rendering
server.
13. The system of claim 1 wherein the graphics command is generated
by a plurality of client applications.
14. The system of claim 1 wherein the rendering server further
transmits the graphic display data to a plurality of independently
addressable display devices.
15. A method for displaying graphical information, the method
comprising: receiving a graphics command at a rendering server;
accessing information responsive to the graphics command, wherein
the information is located in an asset server that is separately
addressable from the rendering sever; and rendering graphic display
data in response to the graphics command and to the
information.
16. The method of claim 15 wherein the asset server is located in a
different geographic location than the rendering server and the
asset server and the rendering server communicate via a
network.
17. The method of claim 15 further comprising transmitting the
graphic display data to a display device.
18. The method of claim 17 wherein the display device is located in
a different geographic location from one or more of the asset
server and the rendering server.
19. An architecture for displaying graphical information, the
architecture comprising: an asset resource layer for storing
information; and a rendering layer for receiving a graphics command
and rendering graphic display data in response to the graphics
command and to the information, wherein the communication server is
independently addressable from the asset server.
20. The architecture of claim 19 further comprising a display layer
for receiving the graphic display data and displaying the graphic
display data on a display device, wherein the display layer in
independently addressable from the asset layer and the rendering
layer.
21. A computer program product for displaying graphical
information, the computer program product comprising: a storage
medium readable by a processing circuit and storing instructions
for execution by the processing circuit for performing a method
comprising: receiving a graphics command at a rendering server;
accessing information responsive to the graphics command, wherein
the information is located in an asset server that is separately
addressable from the rendering sever; and rendering graphic display
data in response to the graphics command and to the information.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of the filing date of
U.S. provisional patent application Ser. No. 60/586,327, filed Jul.
8, 2004, the contents of which are herein incorporated by
reference.
BACKGROUND
[0003] The present disclosure relates generally to imaging and
visualization, and, more particularly, to an architecture for
rendering graphics on output devices over diverse connections.
Example output devices are two-dimensional displays,
three-dimensional displays such as volumetric, multi-view, and
holographic displays, and two- and three-dimensional printers.
[0004] Three-dimensional (3-D) information is used in a variety of
tasks, such as radiation treatment planning, mechanical
computer-aided design, computational fluid dynamics, and
battlefield visualization. As computational power and the
capability of sensors improve, the user is forced to comprehend
more information in less time. For example, a rescue team has
limited time to discover a catastrophic event, map the structure of
the context (i.e., a skyscraper), and deliver accurate instructions
to team members. Just as an interactive computer screen is better
than a paper map, a spatial 3-D display offers rescue planners the
ability to see the entire scenario at once. The 3-D locations of
the injured are more intuitively known from a spatial display than
from a flat screen, which would require rotating the "perspective
view" in order to build a mental model of the situation.
[0005] Display technologies now exist which are designed to cope
with these large datasets. Spatial 3-D displays (e.g., Actuality
Systems Inc.'s Perspecta.RTM. Spatial 3-D Display) create imagery
that fills a volume of space--such as inside a transparent
dome--and that appears 3-D without any cumbersome headwear.
[0006] It is expected that a variety of spatial displays will come
into existence in the near future. Furthermore, software
applications will emerge that will exploit the unique properties of
spatial displays. In order to allow every type of display to be
compatible with every application, a standard is needed which
dictates how (electronically and with what protocol) spatial 3-D
information is transmitted to the display device. In addition,
software applications and display devices that are not specialized
for spatial 3-D rendering will continue to be utilized. Many
customer computer environments will contain a mix of 3-D and
non-3-D display devices and software applications. It would be
desirable for application programmers to be able to write and
execute a single application program to produce graphics on a
variety of 3-D and non-3-D displays.
[0007] Further, modern graphics environments must solve the problem
that the application software generally runs on separate hardware
from the rendering algorithms. Since off-the-shelf personal
computers (PCs) are not yet specialized for spatial 3-D rendering,
the process separation is generally more complicated than sending
the data across the peripheral component interface (PCI)-express
bus. The Chromium architecture is a prior attempt to solve this
problem. Chromium abstracts a graphical execution environment.
However, the binding between an application, rendering resource and
display is statically determined by a configuration file.
Therefore, applications cannot address specific rendering
resources. Current 3-D display architectures and applications
cannot address remote or distributed resources. Such resources are
necessary for displays where ready-made rendering hardware is not
available for PCs.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Referring now to the figures, which are exemplary
embodiments and wherein like elements are numbered alike:
[0009] FIG. 1 depicts an overview of an architecture that may be
implemented by exemplary embodiments of the present invention;
[0010] FIG. 2 depicts an more detailed view of an architecture that
may be implemented by exemplary embodiments of the present
invention;
[0011] FIG. 3 is a block diagram of an exemplary spatial graphics
language implementation;
[0012] FIG. 4 is a block diagram of an exemplary compatibility
module structure;
[0013] FIG. 5 is a block diagram of an exemplary rendering
module;
[0014] FIG. 6 is an exemplary process flow diagram for processing a
command from a ported application;
[0015] FIG. 7 depicts a system that may be implemented by exemplary
embodiments of the present invention;
[0016] FIG. 8 depicts a system that may be implemented by exemplary
embodiments of the present invention;
[0017] FIG. 9 depicts a system that may be implemented by exemplary
embodiments of the present invention; and
[0018] FIG. 10 depicts a system that may be implemented by
exemplary embodiments of the present invention.
SUMMARY OF THE INVENTION
[0019] Exemplary embodiments of the present invention are directed
to a system for displaying graphical information. The system
includes an asset server for storing information and a rendering
server in communication with the asset server. The rendering server
receives a graphics command and renders graphic display data in
response to the graphics command and to the information. The
rendering server is independently addressable from the asset
server.
[0020] Other exemplary embodiments of the present invention are
directed to a method for displaying graphical information. The
method includes receiving a graphics command at a rendering server.
Information responsive to the graphics command is accessed. The
information is located in an asset server that is separately
addressable from the rendering server. Graphic display data is
rendered in response to the graphics command and the
information.
[0021] Further exemplary embodiments of the present invention are
directed to an architecture for displaying graphical information.
The architecture includes an asset resource layer for storing
information and a rendering layer. The rendering layer receives a
graphics command and renders graphic display data in response to
the graphics command and to the information. The communication
server is independently addressable from the asset resource
layer.
[0022] Still further exemplary embodiments of the present invention
include a computer program product for displaying graphical
information. The computer program product includes a storage medium
readable by a processing circuit for performing a method. The
method includes receiving a graphics command at a rendering server.
Information responsive to the graphics command is accessed. The
information is located in an asset server that is separately
addressable from the rendering server. Graphic display data is
rendered in response to the graphics command and the
information.
DETAILED DESCRIPTION
[0023] Exemplary embodiments of the present invention include a
spatial 3-D architecture to support separate asset servers and
rendering servers in a graphics environment. The architecture also
has a spatial visualization environment (SVE), that includes a 3-D
rendering API and a display virtualization layer that enables
application developers to universally exploit the unique benefits
(such as true volumetric rendering) of 3-D displays. SVE supports
the cooperative execution of multiple software applications. As
part of the SVE, a new API is defined, referred to herein as the
spatial graphics language (SpatialGL), to provide an optional,
display-agnostic interface for 3-D rendering. SpatialGL is a
graphical language that facilitates access to remote displays and
graphical data (e.g., rendering modules and assets). The
architecture further has core rendering software which includes a
collection of high-performance rendering algorithms for a variety
of 3-D displays. The architecture also includes core rendering
electronics including a motherboard that combines a graphics
processing unit (GPU) with a 64-bit processor and double-buffered
video memory to accelerate 3-D rendering for a variety of
high-resolution, color, multiplanar and/or multiview displays. Many
of today's 3-D software applications use the well-known OpenGL API.
To provide compatibility with those applications, exemplary
embodiments of the present invention include an OpenGL driver for
the Actuality Systems, Incorporated Perspecta Spatial 3-D Display
product. Embodiments of the Perspecta Spatial 3-D Display product
are described in U.S. Pat. No. 6,554,430 to Dorval et al., of
common assignment herewith.
[0024] Currently, a volume manager is available to manage
cooperative access to display resources from one or more
simultaneous software applications (see for example, U.S. Patent
Application No. 2004/0135974 A1 to Favalora et al., of common
assignment herewith). Current implementations of the volume manager
have asset and rendering resources that are not abstracted
separately from the display. The display rendering and storage
system are considered as a single concept. Therefore, the display
and rendering system must be designed together. Effectively, the
display must be designed with the maximum image complexity in mind.
Exemplary embodiments of the SVE, as described herein, remove this
restriction by providing separately named asset, computation
(rendering), and display resources. Unlike other rendering systems,
the application has the flexibility to combine these resources by
addressing each one independently. These resources may be
independently addressed, and therefore may be located in one or
more servers and accessed via one or more networks. In addition,
these resources (e.g., two or more computation resources) may be
combined to create output for a single graphics display. The
resources may also be located in different geographic locations
(e.g., different rooms in the same building, different cities,
different countries) and in communication via a network.
[0025] FIG. 1 depicts an overview of an architecture that may be
implemented by exemplary embodiments of the present invention. One
or more means is also provided for interfacing one or more central
applications with a local, remote or distributed rendering or
display systems and for interfacing external databases with a
rendering system.
[0026] The architecture depicted in FIG. 1 includes four layers, an
application software layer 102, an SVE layer 104, a rendering
architecture layer 106 and a display-specific rendering module
layer 108. The application software layer 102 includes legacy
applications 110, ported applications 112 and native applications
116. The legacy applications 110 and the ported applications 112
are written to the OpenGL API and converted into the SpatialGL API
118 by the OpenGL compatibility module 114 in the SVE layer 104.
OpenGL and SpatialGL are examples of API types. Exemplary
embodiments are not limited to these two types of APIs and may be
extended to support any graphics APIs such as the Direct3D API. The
native applications 116 are written to the SpatialGL API 118 which
is in communication with the volume manager 120. The rendering
architecture layer 106 depicted in FIG. 1 includes core rendering
software (CRS) 122, which is a device independent management layer
for performing computations/renderings based on commands received
from the SpatialGL API 118 and data in the volume manager 120. The
display-specific rendering module layer 108 includes a Perspecta
rendering module 124 for converting data from the CRS 122 for
output to a Perspecta Spatial 3-D Display and a multiview rendering
module 126 for converting data from the CRS 122 into output to
other 3-D and 2-D display devices.
[0027] Unlike prior architectures, the architecture depicted in
FIG. 1 transforms commands (e.g., graphics commands) from several
API types into a single graphical language, SpatialGL. This permits
the architecture to provide consistent access to display and
rendering resources to both legacy and native application software.
This is contrasted with the currently utilized device-specific
rendering drivers. Each driver manages rendering hardware, visual
assets (display lists, textures, vertex buffers, etc.), and display
devices. The architecture depicted in FIG. 1 includes a rendering
architecture layer 106 that is a device-independent management
layer that with core rendering software 122. This rendering
architecture layer 106 gives the graphics language (SpatialGL 118)
access to diverse, high-level resources, such as multiple display
geometries, rendering clusters and image databases. Each class of
resources: asset (e.g., volume manager 120); computational (e.g.,
core rendering software 122); and display (e.g., Perspecta
rendering 124 and multiview rendering 126) is enabled by an
independent module.
[0028] FIG. 2 depicts a more detailed view of an architecture that
may be implemented by exemplary embodiments of the present
invention. The SVE layer 104 includes a collection of compatibility
strategies between emerging displays and application software. One
aspect of SVE provides compatibility with software applications to
diverse display types through SpatialGL APIs and OpenGL APIs. The
SVE concept extends in three additional directions: application
software development can be accelerated by producing higher-level
graphical programming toolkits; a spatial user interface (UI)
library can provide applications with a consistent and intuitive UI
that works well with 3-D displays; and a streaming content library
allows the SVE to work with stored or transmitted content. This may
be utilized to enable "appliance" applications and "dumb
terminals."
[0029] In addition, the SVE is a display-agnostic and potentially
remote-rendering architecture. The SVE can communicate with 2-D and
very different 3-D displays (multiplanar, view-sequential,
lenticular, stereoscopic, holographic). The rendering server does
not need to be local to the display(s).
[0030] The CRS 122 is a collection of rendering strategies. The
cost of implementing a rendering engine for a new display geometry
breaks down into a system integration effort and an algorithm
implementation effort. CRS 122 eliminates the system integration
effort by providing a portable communication framework to bridge
the client and server domains and by abstracting computation
assets. The CRS 122 creates output for a Perspecta rendering module
124, a multiview rendering module 126 and can be tailored to create
output for future rendering modules 206. In addition, the
architecture depicted in FIG. 2 may be utilized to support future
graphics display architectures and third-party architectures
210.
[0031] The spatial transport protocol (STP) describes the
interaction between the Spatial Visualization Environment and Core
Rendering Software. The spatial transport protocol comprises a set
of commands. The STP may optionally comprise a physical definition
of the bus used to communicate STP-formatted information. The STP
commands are divided into several groups. One group of commands is
for operating the rendering hardware, and frame buffer associated
with the display. Another group of commands is for synchronizing
the STP command stream with events on the host device, rendering
hardware and frame buffer. Another group of commands is for
operating features specific to the display hardware, such as
changing to a low power mode or reading back diagnostic
information.
[0032] Different streams of graphics commands from different
applications may proceed through the architecture to be merged into
a single STP stream. Due to multitasking, the STP is able to
coherently communicate overlapping streams of graphics commands.
STP supports synchronization objects between the applications (or
any layer below the application) and the display hardware. The
application level of the system typically generates sequential
operations for the display drivers to process. Graphics commands
may be communicated with a commutative language. For efficiency,
the display hardware completes the commands out of order.
Occasionally, order is important; one graphics operation may refer
to the output of a previous graphics operation, or an application
may read information back from the hardware, expecting to receive a
result from a sequence of graphics operations.
Application Layer
[0033] Exemplary embodiments of the SVE include a 3-D rendering API
and display virtualization layer that enables application
developers to universally exploit the unique benefits (such as true
volumetric rendering) of 3-D displays. It consists of several
subsystems: SpatialGL 118, OpenGL compatibility module 114,
streaming content library and volume manager 120. Future
development may expand SVE to include scene-graph, rendering engine
and application-specific plug-in subsystems.
SpatialGL
[0034] Just as OpenGL API implementations are video-card-specific,
implementations of the SpatialGL API 118 are display, or
output-device-specific. Examples of "targets" for SpatialGL
implementations are: 2-D displays, volumetric displays,
view-sequential displays, and lenticular multi-view displays.
Exemplary embodiments of the SVE can communicate with a broad range
of output devices whose underlying physics are quite different.
[0035] FIG. 3 is a block diagram of an exemplary SpatialGL
implementation that may be utilized by exemplary embodiments of the
present invention. The blocks include a NativeApp block 116 which
is written to take full advantage of spatial displays by using the
SpatialGL API. The NativeApp block 116 may transmit data to the
client 308, SpatialEngine 304, SceneGraph 306 and the volume
manager 310. In alternate exemplary embodiments, applications can
also take advantage of higher level APIs such as SceneGraph 306 and
SpatialEngine 304 from Actuality Systems, Incorporated. SceneGraph
306 provides an interface for encoding scene graphs in SpatialGL.
SceneGraph 306 implements features such as: assemble shapes into
objects; transform the positions of an objects; and animation node.
The SpatialEngine 304 implements high level functions such as draw
volume and overlaying a scene-graph. SpatialEngine 304 is
extensible. For example, an OilToolkit can be added, which adds
functions such as: draw porosity volume, overlay drill path and
animate path plan.
[0036] As depicted in FIG. 3, SpatialGL is input to the client 308.
In exemplary embodiments of the present invention, the native API,
or SpatialGL API, provides an object oriented front-end to the STP
byte code. The SpatialGL API exposes features such as, but not
limited to: define fragment program, define vertex program, bind
geometry source, bind texture source, swap buffer and synchronize.
The client 308 sends SpatialGL commands to the volume manager 310.
The SpatialGL commands may include commands for retrieving
persistent objects to be displayed on a graphical display device.
The persistent objects include, but are not limited to, 2-D and 3-D
textures and vertex buffers. The persistent objects may be stored
on one or more of a database, a storage medium and a memory buffer.
In addition, the SpatialGL commands may include commands for
retrieving display nodes to be displayed on a graphical display
device. Display nodes refer to an instance of any display that can
be individually referenced (e.g., a Perspecta display, a 2-D
display). STP commands from the volume manger 310 are sent to the
core rendering client 312. The core rendering client 312 is the
first computation resource available to the STP execution
environment. Early data reducing filter stages can also execute
here. Stream compression and volume overlay are processes that may
be assigned computation resources at this point. The core rendering
client 312 formats the remainder of the filter graph to take into
account the physical transport 314 layer between the core rendering
client 312 and the core rendering server. At the STP interpreter
block 316, API calls are converted into STP. Each STP node is a
computation resource. STP procedures get bound to STP nodes as the
program is processed. The node executes any procedure that has been
bound to it by a previous node.
[0037] Spatial Transport Protocol may be converted for persistent
storage and written to a disk. This can be accomplished by storing
the serialized Spatial Transport Protocol byte code to disk, along
with a global context table. The global context table allows
context-specific assets to be resolved when the STP file is later
read back from disk. The global context table establishes
correspondences between local context handles referenced by the STP
byte code and persistent forms of the referenced data. For example,
a STP byte code may reference texture image number 5. The texture
image number is associated with specific data in the original local
context of the byte code. When saved to disk, texture image number
5 is associated with a particular texture image by the local
context table. This can be accomplished by storing in table
position 5, a copy of the texture image, or by storing a GUID or
URL that identifies a persistent source for the texture image.
Compatibility Module Structure
[0038] FIG. 4 is a block diagram of an exemplary compatibility
module structure. Ported applications 112 and/or legacy
applications 110 can provide input to the compatibility module
structure depicted in FIG. 4. Ported applications 112 are
applications originally written using OpenGL, but have been
extended by programmers to interface with spatial displays. Legacy
applications 110 are applications written with no knowledge of
spatial displays or vendor APIs (e.g., Actuality Systems,
Incorporated APIs). OpenGL support is provided through two dynamic
link libraries. The main library is called the GLAS library 412. It
provides drawing methods similar to the industry-standard OpenGL
API, and also contains specialized initialization and state
management routines for spatial displays. The GLAS library 412
converts OpenGL API calls into SpatialGL 118 calls. SpatialGL 118
is a low level graphics language utilized by exemplary embodiments
of the present invention. The OGLStub library 414 exports an
interface similar to the OpenGL32.dll system library 408. The
behavior of the library can be customized on a per-application
basis. The OGLStub library 414 intercepts and redirects OpenGL API
calls in a customizable manner. Calls are optionally forwarded to
the OpenGL32.dll system library 408 and/or the GLAS library 412 for
translation.
[0039] OpenGL is an industry standard low-level 3-D graphics API
for, scientific and computer aided design (CAD) applications.
OpenGL supplies a language that expresses static information. The
application must explicitly break down dynamic scenes into discrete
frames and render each one. OpenGL expresses commands such as:
input a vertex; draw a triangle; apply a texture; engage a lighting
model; and show the new rendering.
[0040] Referring to FIG. 3, OpenGL calls are duplicated for both
the system library 408 (to render on the 2-D monitor) and for the
GLAS library 412. By default, the first scene is analyzed to
determine the depth center of the application's implied coordinate
system. Since the depth center is not known until the first
swap-buffers call, it may take until the second scene for the image
in Perspecta to render properly.
[0041] The first scene is analyzed to determine the depth center of
the application's coordinate system. Once the depth center is
calculated, a fix-up transform is calculated. This transform is
applied consistently to the projection specified by the
application, so that the application's further transformations the
projection (such as scaling and zooming) are reflected properly in
the spatial rendering. After the depth center is determined, the
Stub library 414 issues a redraw call to the application to ensure
that the first scene is drawn properly in Perspecta.
[0042] The two main configurations are "ghost mode" 406 and
"extended mode" 410. Ghost mode 406 automatically duplicates OpenGL
calls for both the system library 408 and for the GLAS library 412.
In ghost mode 406, depth centering is based on x and y scale and
centered to get the majority of the vertices within the display.
Ghost mode 406 provides an unextended OpenGL interface and attempts
to make a spatial display appear as a 2-D display to the
application. Extended mode 410 allows the application to control
the call forwarding behavior. Extended mode 410 exposes an extended
OpenGL interface. A few commands are added to help the application
control a spatial display separately from a 2-D display. Example
commands include: create a context for a spatial display and draw
to a spatial display context. Output from the GLAS library 412, in
SpatialGL, is sent to the client 308 and then to the volume manager
310. The volume manager 310 assigns display resources. It filters
the STP stream to reformat the data according to the display
resource assigned to the given context. The core rendering block
312, which contains the mechanisms for decoding and executing
procedures in the STP language, receives STP commands.
[0043] The configuration is controllable for each application,
based on a central control repository. Parameters that may
configured include, but are not limited to: context selection
strategy (allows the controller to change the context selection
while the application is running); projection fix-up strategy that
overrides the projection that that application specifies, in order
to fit the image in the actual display geometry; texture processing
strategy; context STP preamble (e.g., resolution hints); and scene
STP preamble.
[0044] Some spatial displays physically realize view-dependent
lighting effects. In this case, lighting is calculated based on the
actual view directions, rather than the master direction given by
the projection matrix.
[0045] Specific rasterization constraints and rules can only be
specified relative to the unique geometry of each display type. In
general, only fragments that intersect the projection of an element
into the display's native coordinate system may be lit. When
rendering polygons, elements must not contain holes. When rendering
connected polygons where exact vertex positions are shared, the
rendered figure must not contain holes.
[0046] When anti-aliasing is used, the partial ordering of the
color value of the fragments must agree with the partial ordering
of the intersection (area or length) between the fragment and
pixels of the display's native coordinate system, when normalized
for variation in area or volume of the pixels.
Streaming Content Library
[0047] The streaming content library permits spatial stream assets.
A spatial stream asset is a time-varying source of spatial imagery.
Optionally, the spatial stream may be synchronized with one or more
audio streams. A spatial stream may either consist of a real-time
stream, a recorded stream, or a dynamically generated stream. An
example of a ream-time spatial stream is a multi-view stream that
is fed from an array of cameras. An example of a recorded stream is
a spatial movie stored on a removable disk. An example of a
dynamically generated stream is a sequence of dynamically rendered
3-D reconstructions from a PACS database.
[0048] Each stream is associated with a spatial codec. The intended
interpretation of the stream is determined by the associated
spatial codec. The spatial codec is comprised of a stream encoding
specification and a reconstruction specification. The stream
encoding specification determines the mapping from the raw binary
stream to a time-varying series of pixel arrays. The stream
encoding specification may also identify an audio stream,
synchronized with the pixel arrays. The reconstruction
specification determines the intended mapping from pixel arrays to
physical light fields. Examples of stream encoding specifications
include MPEG coded representations. The reconstruction
specification can be defined using the persistent form of Spatial
Transport Protocol.
[0049] A client of the streaming content library receives the raw
binary stream and the spatial codec. The client proceeds to
reconstruct an approximation of the intended physical light field,
by calculating pixel arrays using the stream encoding
specification. At each time step, the client consumes one or more
pixel arrays and interprets an intended light field, using the
reconstruction specification. The intended light field is rendered
into the local display's specific geometry, using Core Rendering
Software. Finally, Core Rendering Software moves the rendered image
into the spatial frame buffer, causing the display to generate a
physical manifestation of a light field.
[0050] The streaming content library includes spatial stream asset
servers. Each spatial stream asset is published by a spatial asset
server. An asset server may publish one or more streams, each with
a unique URL. A software application using SpatialGL (such as a
spatial media player) can call up a particular spatial stream asset
using its associated URL.
[0051] Spatial stream assets may be transmitted with unidirectional
signaling: for example several TV channels may be jointly used to
transmit a multi-view recording. In this case, the spatial codec
can be continuously or periodically transmitted. Spatial content
may also be broadcast with bidirectional signaling: for example, a
spatial movie may be downloaded from an Internet-based asset server
and viewed using a spatial media player using SpatialGL. In this
case, the client could potentially negotiate an optimal spatial
codec to match the client's display geometry. Bidirectional
signaling can also be used to allow a client to remotely control a
dynamically generated stream. For example, a client may
continuously send updates to a server about the desired view
direction and region of interest, while the server continuously
returns rendered images to the client through the streaming
library. Alternately, a client may receive notifications from the
spatial stream asset server when new data is available. Based on
the notifications, the client may choose to download and render the
new data or else the client may skip the new data. When receiving a
notification, the client may decide whether to download or skip the
new data, based on factors such as the currently available buffer
space, communication bandwidth, processing power, or desired level
of image quality.
Pseudo-Code Reconstruction Specification for a Multi-View
Stream
[0052] Define n views V.sub.1, . . . V.sub.n, each comprised of a
projection P.sub.i and an aperture Q.sub.i [0053] For each time
step t [0054] For each view V.sub.i, [0055] Render a plane,
textured with pixel array t*n+i, using projection P.sub.i [0056]
Render aperture Q.sub.i [0057] Swap the rendered image into the
active frame buffer Pseudo-Code Reconstruction Specification for a
Volumetric Stream [0058] Define a local 3-D texture asset T [0059]
For each time step t [0060] For each pixel array i in 1 . . . n
[0061] Load pixel array i into slice i of texture T [0062] Render a
solid cube, textured with T [0063] Swap the rendered image into the
active frame buffer Network Layer
[0064] In an exemplary embodiment of the present invention, the CRS
122 has the character of a slave peripheral and communication to
the CRS 122 is limited to proprietary channels. Alternate exemplary
embodiments of the CRS 122 have an expanded role as a network
device. In addition, it can communicate with a host over a network,
and it supports standard protocols for network configuration. The
CRS 122 has both a client part and a server part.
[0065] In exemplary embodiments of the present invention, a host PC
runs an application and is in communication with a single
multiplanar 3-D display which contains an embedded core rendering
electronics system. The client part is embodied on the host PC,
while the server part is embodied on the core rendering
electronics. CRS 122 is distinct from the SVE because the CRS 122
is meant primarily to provide a rendering engine for specific
display types that is compatible with the display-independent
graphical commands generated in the SVE.
[0066] The client side of the CRS 122 interfaces to the SVE using
the STP language. STP is used to package and transport SpatialGL
API calls. A core rendering client connects the volume manager 120
to the physical transport by acting as an STP interpreter. The core
rendering client interpreter exposes procedures (with STP linkage)
that allow an STP program to address specific servers. Exemplary
embodiments of the present invention only function when a single
server is present. Alternate exemplary embodiments of the core
rendering client communicate with servers over a network, and are
able to list and address the set of available servers.
[0067] The client also provides a boot service. This provides the
boot-image used by the net-boot feature of the servers. The
boot-image is stored in a file that can be updated by Perspecta
Software Suite upgrade disks (or via web upgrade). The boot service
can be enabled and disabled by the SVE. After the boot-image file
is upgraded, the installer must enable the boot service to allow
the display to update.
[0068] In the current example, in which there is one host PC and
one Perspecta display, all input to the system arrives through the
gigabit Ethernet connections. The embedded system acts as a normal
Internet Protocol (IP) device. The embedded system acts as a
server, while the host PC acts as a client. The server acts as a
normal IP device. In exemplary embodiments of the present
invention, the client and server must be directly connected. In
alternate exemplary embodiments of the present invention, clients
and servers are connected through a gigabit switch. This
configuration removes the requirement that the client PC contains
two Ethernet controllers, and it allows multiple clients to connect
to a single server. The server obtains an IP address using dynamic
host configuration protocol (DHCP) (unless it has been configured
to use a static address). Once an IP address has been obtained, the
CRS 122 and the client must be made aware of the identity of the
server. This is done by a symmetric system where a node (client or
server) broadcasts a datagram when it starts. The node that starts
first obtains the identity of the later node. If the server is
started first, and encounters a client datagram broadcast, it opens
a connection to the client to communicate the server's identity. A
client may simultaneously communicate with multiple servers. Each
server may only communicate with a single client at a time.
[0069] In alternate exemplary embodiments of the present invention,
the servers have a user interface and policy for attaching to
specific clients when more than one client is available. The CRS
122 provides a simple network management protocol (SNMP) interface
to manage the network settings of the server. The SNMP interface
configures the IP address, broadcast settings and security
settings. The security settings include client allow and deny
lists.
[0070] In exemplary embodiments of the present invention the host
and client support a single gigabit Ethernet connection. In
alternate exemplary embodiments, the host and client employ an
additional protocol to support two gigabit Ethernet
connections.
[0071] Once a client knows the identity of a server, the client may
open the server. The client and server communicate through
datagrams. The server is single-threaded; the client may only open
a single connection to the server and it is guaranteed exclusive
access to the entire display resource. Once the client has opened
the server, it may begin transacting rendering commands. Rendering
commands are moved between the client and server using a command
stream and a remote memory protocol.
[0072] Since the network graphics service is meant to communicate
only over a local network segment, a very low level of packet loss
is expected. The details of the communication scheme can be
arranged to ensure that the system degrades gracefully under packet
loss. Device allocation and context creation must be guaranteed to
operate correctly under packet loss. The bulk graphics data
transfer is not protected, except that a frame that is rendered
without packet loss must not be degraded by packet loss in previous
frames. Persistent texture map data is protected against packet
loss by a checksum and a failure/retry scheme.
Core Rendering Software (CRS)
[0073] CRS 122 uses the STP language as a form for communicating
graphics commands and procedures. STP allows the interfaces between
the major components of the Core Rendering Software system to be
uniform. In the initial version of Core Rendering Software, STP
serves as the inter-process-communication mechanism. STP is used to
communicate a sequence of graphics commands from the client to the
server. The initial version of STP will include conditional
execution and parallel branching prototype features. In later
versions of Core Rendering Software, modules will be written within
the STP language, thus flattening the hardware-native part of the
architecture. Conditional execution and parallel branching features
will be optimized in later versions of Core Rendering Software.
Rendering Modules
[0074] FIG. 5 is a block diagram of an exemplary rendering module.
The pipeline system structure, or pipeline framework, subsystem
provides the generic structure that is common to rendering
pipelines for the CRS 122. A rendering pipeline is implemented
through a pipeline system class 502. A pipeline system class 502 is
composed of a rendering pipeline and a fixed set of active objects.
An active object models a device that can trade time for data
movement or transformation, such as a bus, a GPU or a CPU. The
pipeline system class 502 binds stages to scheduler threads 510
(i.e., to active objects). The scheduler thread 510 is the binding
between stages and active objects.
[0075] An instance of a pipeline 504 operates on a single input
stream of homogeneous elements. An exemplary pipeline constructor
includes: initiate first-in-first-out (FIFO) length; and initialize
stage connections. As depicted in FIG. 5, fixed length FIFOs 506
constrain the resource usage of the system.
[0076] Rendering pipelines are implemented as a series of stages
508 that communicate with tasks. A stage 508 is an algorithmic unit
that transforms one stream of tasks into another stream of tasks.
Although a stage 508 may be designed to be compatible with a
specific active object 512, the binding to the active object 512 is
external to the stage 508. For example, a stage 508 may implicitly
require a binding with a GPU by making OpenGL calls, but it must
not own or manipulate an OpenGL context.
[0077] Stage objects have an unusually complicated life cycle. They
are typically created in one thread but work in a second thread.
The lifetime of a stage 508 consists of these distinct transitions:
construction, initialization, execution, de-initialization, and
destruction. A stage 508 transforms a stream of homogeneous
elements. A stage 508 utilizes the resources of a single active
object and executes several hundreds of times a second. The biding
between a stage 508 and an active object 512 is external to the
stage class. Therefore, a pipeline 504 may be considered a stage
508, but a pipeline system 502 may not be considered a stage 508.
The remote active object 512 depicted in FIG. 5 models a thread of
execution that exists outside of the CPU. Input to the active
object 512 includes data from the GL context block 514 and the
voxel engine context block 516.
[0078] Task objects are not strongly structured, outside of their
specific implementation domain. In exemplary embodiments of the
present invention, the pipeline framework includes a Fence class,
which is utilized to provide a main stream synchronization pattern.
A pipeline system 502 operates asynchronously from its enclosing
system. The enclosing system can insert a Fence into the command
stream of a pipeline 504. A pipeline passes a fence when all
processing due to tasks issued before the fence have completed. The
enclosing system can query whether the pipeline 504 has passed the
fence, or it can block until the fence has been passed.
SpatialGL Graphics Pipeline for Other Displays
[0079] As described above, a key feature of the SVE is
display-independence (or "display-agnosticism"). Implementations of
the SpatialGL API can be made for a variety of 2-D and 3-D
displays. The SpatialGL API may be utilized with a Perspecta
multiplanar volumetric display. In addition, the SpatialGL API may
be utilized with other types of displays.
[0080] Because multi-view rendering is very similar to single-view
rendering, the SpatialGL implementation is substantially simpler
than the SpatialGL implementation for multiplanar rendering. For
example, on flat, horizontal-parallax multi-view displays, such as
the Stereographics 9-view lenticular display or Actualty System's
quasi-holographic video display, a slice volume could be created as
part of the rendering process. A slice volume contains a slice for
each rendered view direction. Rendered views use sheared versions
of standard projection matrices, corresponding to the viewing
angles. "Final views" correspond to the views that are physically
generated by the display hardware. Final views are sampled from the
slice volume (for example, using texture mapping). The number of
final views may be different than the number of rendered views.
[0081] Rendering tetrahedra requires special treatment because, at
the time of writing, GPUs lack native volumetric rendering support.
In this case, SpatialGL wraps an efficient volume rendering
implementation such as ray casting.
[0082] Depending on the multiview display, image formatting can be
different. Because Stereographics' lenticular display interfaces
via digital visual interface (DVI), it does not require special
formatting hardware (such as the Voxel Engine in Actuality Systems,
Incorporated's Core Rendering Electronics). However, the
distribution of pixels and views is somewhat irregular, and
requires a reformatting process known as "interzigging."
Additionally, view anti-aliasing can occur during this step. On the
other hand, Actuality Systems' holovideo display was designed to
use the same Core Rendering Electronics as Perspecta, and can share
the same implementation.
[0083] Because SpatialGL is display-agnostic, SpatialGL can also be
used for non-3D displays. Examples include tiled display walls,
displays with heterogeneous interfaces (e.g. the Sunnybrook HDR
LCD, foveated resolution displays), and displays with unusual
geometries (e.g. dome, sphere or cube shaped displays). Finally, an
obvious example would be a standard 2-D display such as a desktop
cathode ray tube (CRT) or liquid crystal display (LCD). This would
allow the use of SpatialGL programs on standard computer hardware
without an exotic display configuration. For the most part, the
rendering of these displays only requires changes in the image
reformatting stage, and minor changes elsewhere.
[0084] FIG. 6 is an exemplary process flow diagram of a command
from a ported application. A ported application 112 renders a scene
by issuing function calls via the compatibility module 114 to the
GLAS Extended API (similar to OpenGL). These API calls specify
features of the scene, such as texture images, the position and
shape of primitive elements (such as lines points and triangles)
and the mappings between elements textures and colors. The GLAS
extended stub library 612 receives the API calls and issues them to
the GLAS translation library 412. The GLAS translation library
manages the OpenGL state machine to interpret the GLAS Extended API
calls. Interpreted calls are translated into SpatialGL API
calls.
[0085] Legacy applications invoke a similar command flow. In this
case, a legacy application 110 renders a scene by issuing function
calls via the compatibility module 114 to the GLAS Ghost API
(similar to OpenGL). The GLAS ghost stub library 613 receives the
API calls and reformats the scene in preparation for translation to
SpatialGL. For example, the stub library may apply a heuristic that
inspects the Ghost API calls to estimate the intended depth center
of the scene. This additional information is passed to the GLAS
translation library 412, along with the API calls generated by the
legacy application. Interpreted calls are translated into SpatialGL
API calls.
[0086] The SpatialGL client library 308 directs the API calls to
the volume manager 310, along with association information to
identify the software application instance that generated the
commands (the ported application).
[0087] The volume manager 310 modifies the API call stream. For
example, it may map the application's rendering output to a
specific potion of the graphics volume generated by the display
device. It may also apply an overlay to the rendering output, such
as a sprite that marks the location of a 3-D pointer.
[0088] After the volume manager 310, the core rendering client
library 312 marshals the API calls and transmits them (for example,
using Spatial Transport Protocol 314) to the server execution
environment. The core rendering software (instantiated in the
server execution environment) receives and unmarshals API
calls.
[0089] The core rendering server 604 operates rendering algorithms,
based on the API calls. The rendering algorithms are defined by a
renderer module 606. In general, there is a specialized renderer
module for each distinct class of display geometry. The rendering
algorithms cause rendered image bitmaps to be moved into the
spatial frame-buffer 611, using the voxel engine driver 610.
Distributed Configurations
[0090] Typically, graphics libraries allow multiple client
applications to share access to a single rendering server (via, for
example, the Windows Graphics Library [WGL] for OpenGL Windows). In
exemplary embodiments of the present invention, the SVE volume
manger provides this meta-service for the SpatialGL API. The
SpatialGL API is also designed to allow a single client application
to access multiple servers. Often, this will be to provide multiple
views to an application (e.g., standard 2-D view with a Perspecta
volumetric view).
[0091] However, for the SpatialGL API, servers do not necessarily
represent access to rendering/display resources; instead they may
also represent access to graphical assets. This includes geometry
data, images, shader programs, and their combinations. Like web
pages, SpatialGL objects can be referenced by Uniform Resource
Locators (URLs). These URLs may represent local resources or shared
resources. Exemplary implementations of the present invention can
distribute different parts of the rendering pipeline to different
servers that may specialize in various tasks.
[0092] A simple configuration that may be implemented by the
network layer 202 of the architecture described previously herein
includes having a host computer attached directly to a display
(e.g., a spatial display). In this configuration, the client
application on the host computer may open multiple contexts
(virtual servers) that are shared on the display. Another
configuration that may be implemented by exemplary embodiments of
the present invention is a typical client/server configuration. In
this case, multiple host computers are attached to a display over a
network and client applications on different host computers can
each open multiple contexts (e.g., via the SpatialGL API) that are
shared on the display. Other exemplary configurations include a
buffered display configuration with a host computer that is
attached to a SpatialGL rendering server. In this configuration the
rendering server can only perform rendering, and does not actually
display images. When the client application sends scenes to the
rendering server, they are rendered and stored. The stored results
can be played back on a display at a later time.
[0093] FIGS. 7-10 depict further example configurations that may be
implemented by the network layer 202 of the architecture described
previously herein. In FIG. 7, multiple host computers 702 are
attached to various displays 704 over one or more networks 706. The
displays 704 may be a mixture of various types (e.g., 2-D and 3-D).
In exemplary embodiments of the present invention, the host
computers 702 discover displays 704 through the Domain Name System
(DNS). The host computer systems 702 include one or more graphics
applications that communicate with one or more displays 704 via the
network using an API (e.g., the SpatialGL API).
[0094] The network 706 may be implemented by any type of known
network including, but not limited to, a wide area network (WAN), a
local area network (LAN), storage area network (SAN), a global
network (e.g. Internet, cellular), a virtual private network (VPN),
and an intranet. The network 706 may be implemented using a
wireless network or any kind of physical network implementation.
SpatialGL servers (e.g., asset server, rendering server) may be
attached to the network 706 in a wireless fashion.
[0095] FIG. 8 depicts a system that may be implemented by exemplary
embodiments of the present invention. The configuration depicted in
FIG. 8 includes a dedicated asset server 802. In FIG. 8 a host
computer is attached to a SpatialGL asset server 802, a SpatialGL
rendering server 804 and a display on a thin client 806. The client
application accesses assets from the asset server 802 by URL. These
assets are forwarded to the rendering server 804 (which may have
the asset locally cached). The client application sends SpatialGL
commands to the rendering server 804, which renders the SpatialGL
scenes (also referred to herein as graphic display data) and sends
the results to the display. In exemplary embodiments of the present
invention, the assets are very large compared to the bandwidth
between the host computer and the SpatialGL server (e.g.,
host/display is a doctor's home PC connected over the Internet and
the asset server 802 is SpatialGL/DICOM bridge that creates
SpatialGL volumetric textures from a hospital's picture archiving
and communication system (PACS).
[0096] FIG. 9 depicts a system that may be implemented by exemplary
embodiments of the present invention. The configuration depicted in
FIG. 8 may be referred to as a remote rendering implementation. In
FIG. 9, a host computer is in communication with a SpatialGL
rendering server 804 and a display server through a network. In the
system depicted in FIG. 9, the host computer and display server on
located on the same machine (e.g., host/display is a thin client
806 such as a table PC or cellular telephone) and the rendering
algorithms are performed remotely by the rendering server 804. The
rendering server may access the asset server 802 via the network
706. The client application, located on the thin client 806, sends
SpatialGL commands to the rendering server 804. The rendering
server 804 then renders the SpatialGL scenes and sends the results
to the Spatial GL display on the thin client 806.
[0097] FIG. 10 depicts a system that may be implemented by
exemplary embodiments of the present invention to provide
distributed rendering. Multiple host computers 702 are attached to
various SpatialGL rendering servers 1004 and displays 1002 over a
network. 706. Client applications from the host computers 702 send
SpatialGL commands to the pool of rendering servers 1004 that
load-balances and distributes the rendering tasks amongst
themselves, possibly through an arbiter. Scenes are rendered in
parallel across available and easily accessible rendering servers
1004 and sent to the appropriate displays 1002. Interesting (though
not mutually exclusive) distributions of rendering servers 1004
include workstations that perform SpatialGL rendering during idle
cycles or centrally deployed cluster of dedicated rendering servers
1004. The latter case is particularly interesting when combined
with dedicated asset servers 802.
[0098] The system configurations described above and depicted in
FIGS. 7-10 are exemplary in nature. In general, the asset resources
(e.g., asset server 802), computation resources (e.g., rendering
server 804) and/or display resources (e.g., displays 704) may be
distributed across one or more networks 706 or co-located. In
addition, one or more assets resources and/or computation resources
may create input to one or more displays. Particular use-cases and
specific examples that may be implemented using the configurations
described above include, but are not limited to:
[0099] A customer's PC connected directly to a 3-D display (for
example, a mechanical engineer running SolidWorks application
software on a desktop IBM PC workstation, containing elements of
Spatial Visualization Environment, in physical connection to a
multiplanar three-dimensional display over gigabit Ethernet.)
[0100] One or more mobile devices with rendered 3-D graphics on its
2-D display (i.e. 3-D-on-2-D) that call upon remote rendering
computational horsepower to draw animated 3-D graphics for
games.
[0101] One or more mobile devices with rendered 3-D graphics on its
3-D display (i.e. a cellphone with a lenticular 9-view
autostereoscopic 3-D display) that call upon remote rendering
computational horsepower.
[0102] Several displays of various types distributed through an oil
exploration enterprise, such as 17'' desktop LCDs, a 9-panel 2-D
video wall, ten stereoscopic displays, and a volumetric 3-D
display, all rendering projections of a 3-D application's graphical
output, where the 3-D application resides in a computer not
necessarily local to the displays.
[0103] Several displays of various types, such as 2-D and
panoramagram 3-D, throughout one or many movie retail locations,
which in response to a potentially remote source of data provides
content in a synchronized manner. For example, when a new product
is offered, all of the displays can show up-to-date advertising
content.
[0104] Exemplary embodiments of the present invention allow
applications to have access to a wider variety of assets, rendering
algorithms and displays. The assets, rendering algorithms and
displays may be located in the same geographic location or located
in separate geographic locations and in communication via a
network.
[0105] As described above, the embodiments of the invention may be
embodied in the form of hardware, software, firmware, or any
processes and/or apparatuses for practicing the embodiments.
Embodiments of the invention may also be embodied in the form of
computer program code containing instructions embodied in tangible
media, such as floppy diskettes, CD-ROMs, hard drives, or any other
computer-readable storage medium, wherein, when the computer
program code is loaded into and executed by a computer, the
computer becomes an apparatus for practicing the invention. The
present invention can also be embodied in the form of computer
program code, for example, whether stored in a storage medium,
loaded into and/or executed by a computer, or transmitted over some
transmission medium, such as over electrical wiring or cabling,
through fiber optics, or via electromagnetic radiation, wherein,
when the computer program code is loaded into and executed by a
computer, the computer becomes an apparatus for practicing the
invention. When implemented on a general-purpose microprocessor,
the computer program code segments configure the microprocessor to
create specific logic circuits.
[0106] While the invention has been described with reference to
exemplary embodiments, it will be understood by those skilled in
the art that various changes may be made and equivalents may be
substituted for elements thereof without departing from the scope
of the invention. In addition, many modifications may be made to
adapt a particular situation or material to the teachings of the
invention without departing from the essential scope thereof.
Therefore, it is intended that the invention not be limited to the
particular embodiment disclosed as the best mode contemplated for
carrying out this invention, but that the invention will include
all embodiments falling within the scope of the appended claims.
Moreover, the use of the terms first, second, etc. do not denote
any order or importance, but rather the terms first, second, etc.
are used to distinguish one element from another.
* * * * *