U.S. patent number 8,108,577 [Application Number 13/134,211] was granted by the patent office on 2012-01-31 for method and apparatus for providing a low-latency connection between a data processor and a remote graphical user interface over a network.
This patent grant is currently assigned to Teradici Corporation. Invention is credited to David Victor Hobbs.
United States Patent |
8,108,577 |
Hobbs |
January 31, 2012 |
Method and apparatus for providing a low-latency connection between
a data processor and a remote graphical user interface over a
network
Abstract
A pair of processing modules and methods that enable low latency
communications between a data processing system and devices located
at a remote graphic user interface across a standard shared network
in accordance with the present invention is disclosed. In one
embodiment, an apparatus for communicating a raster video signal
comprises a receiver enabled to receive the raster video signal
from a graphics processor and obtain display control parameters of
the raster video signal, the display control parameters comprising
frequency and resolution information for the raster video signal; a
raster encoder, coupled to the receiver, enabled to encode a
plurality of scan lines of the raster video signal to generate a
plurality of encoded scan lines; and a network controller, coupled
to the raster encoder, enabled to transmit the plurality of encoded
scan lines and the display control parameters as packets via an
IP/Ethernet network.
Inventors: |
Hobbs; David Victor (Surrey,
CA) |
Assignee: |
Teradici Corporation (Burnaby,
British Columbia, CA)
|
Family
ID: |
40512839 |
Appl.
No.: |
13/134,211 |
Filed: |
June 1, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
Issue Date |
|
|
12380885 |
Mar 4, 2009 |
7970966 |
|
|
|
11284676 |
Apr 7, 2009 |
7516255 |
|
|
|
60667157 |
Mar 30, 2005 |
|
|
|
|
Current U.S.
Class: |
710/72; 710/61;
348/474; 709/229; 725/135; 715/700 |
Current CPC
Class: |
G06F
3/1454 (20130101); H04N 21/64322 (20130101); H04N
21/4305 (20130101); H04N 21/436 (20130101); G09G
5/006 (20130101) |
Current International
Class: |
G06F
13/12 (20060101); G06F 15/16 (20060101); G06F
3/00 (20060101); H04N 7/00 (20060101); H04N
7/16 (20060101) |
References Cited
[Referenced By]
U.S. Patent Documents
Other References
US. Office Action mailed Jun. 2, 2008 for U.S. Appl. No. 11/284,676
filed Nov. 21, 2005, 5 pages. cited by other .
U.S. Office Action mailed Jun. 4, 2008 for U.S. Appl. No.
11/549,055 filed Oct. 12, 2006, 36 pages. cited by other .
Thekkath, Chandramohan A. And Levy, Henry M., "Limits to
Low-Latency Communication on High-Speed Networks," University of
Washington, Seattle, ACM Transactions on Computer Systems, vol. 11,
No. 2, May 1993, pp. 179-203. cited by other .
Nieh, J., Yang, S. J., and Novik, N., "A Comparison of Thin-Client
Computing Architectures," Technical Report CUCS-022-00, Network
Computing Laboratory, Columbia University, Nov. 2000, 16 pages.
cited by other .
Stegmaier, S., Magallon, M. and Ertl, T., "A Generic Solution for
Hardware-Accelerated Remote Visualization," Joint
Eurographics--IEEE TCVG Symposium on Visualization, 2002, pp.
87-94. cited by other .
Yang, S. J., Nieh, J., Selsky, M. and Tiwari, N., "The Performance
of Remote Display Mechanisms for Thin-Client Computing," Department
of Computer Science, Columbia University, Proceedings of the 2002
USENIX Annual Technical Conference, Jun. 2002. cited by other .
"LongView.RTM. IP Desktop Extension over IP Solutions," Copyright
2007 Avocent Corporation, 2 pages. cited by other .
"HP Remote Graphics Software High performance remote access
software with 'just like local' feel," Copyright 2007
Hewlett-Packard Development Company, L.P., 4 pages. cited by other
.
"Advantages and Implementation of Hp Remote Graphics Software,"
Copyright 2006 Hewlett-Packard Development Company, L.P., pp. 1-9.
cited by other .
Gilbert, Jeffrey Michael, "Text / Graphics and Image Transmission
over Bandlimited Lossy Links," A thesis submitted in partial
satisfaction of the requirements for the degree of Doctor of
Philosophy in Engineering Electrical Engineering and Computer
Sciences in the Graduate Division of the University of California,
Berkeley, Spring 2000, 276 pages, Berkeley, CA. cited by other
.
Buck, Ian, Humphreys, Greg, and Hanrahan, Pat, "Tracking Graphics
State for Networked Rendering," Proceedings of
SIGGRAPH/Eurographics Workshop on Graphic Hardware, Aug. 2000, 9
pages. cited by other .
"OpenGL Vizserver.TM. 3.1 Application-Transparent Remote
Interactive Visualization and Collaboration," White Paper, Silicon
Graphics Inc., 2003, 15 pages. cited by other .
Humphreys, Greg, and Hanrahan, Pat, "A Distributed Graphics System
for Large Tiled Displays," Proceedings IEEE Visualization '99,
1999, 9 pages. cited by other .
U.S. Office Action mailed Sep. 7, 2010 for U.S. Appl. No.
12/380,885 filed Mar. 04, 2009, 11 pages. cited by other .
U.S. Office Action mailed Jun. 2, 2008 for U.S. Appl. No.
11/284,676 filed Nov. 21, 2005, 10 pages. cited by other .
"Universal Serial Bus Specification", Revision 2.0, Apr. 27, 2000,
14 pages. cited by other .
Wikipedia article on Local Area Network (LAN) from Aug. 5, 2004,
downloaded from:
http://replay.web.archive.org/20040805111742/http://en.wikipedia.org/wiki-
/local.sub.--area.sub.--network, 2 pages. cited by other .
Wikipedia article on Digital Visual Interface (DVI) from Feb. 13
2004, downloaded from:
http://relay.web.archive.org/20040213055627/http://en.
wikipedia,org/wiki/digital.sub.--visual.sub.--interface, 2 pages.
cited by other .
Wikipedia article on Ethernet from Feb. 15, 2004, downloaded from:
http://replay.web.archive.org/20040215072203/http://en.wikipedia.org/wiki-
/Ethernet, 8 pages. cited by other .
"DX-10 Digital Upgrade" from Component Engineering, downloaded
from: http://componentengineering.com/dx10.html, 2 pages. cited by
other .
U.S. Office Action mailed May 9, 2011 for U.S. Appl. No. 12/383,291
filed Mar. 23, 2009, 58 pages. cited by other .
USB/IP--"A Peripheral Bus Extension For Device Sharing Over IP
Network", Takahiro Hirofuchi et al., Nara Institute of Science and
Technology, Ikoma, Japan, Freenix Track: 2005 Usenix Annual
Technical Conference, pp. 47-60. cited by other.
|
Primary Examiner: Chen; Alan
Attorney, Agent or Firm: Moser Taboada
Parent Case Text
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser.
No. 12/380,885, filed Mar. 4, 2009 now U.S. Pat. No. 7,970,966,
which is a continuation of U.S. patent application Ser. No.
11/284,676, filed Nov. 21, 2005 now U.S. Pat. No. 7,516,255, issued
Apr. 7, 2009, which claims benefit of U.S. provisional patent
application Ser. No. 60/667,157, filed Mar. 30, 2005. Each of the
aforementioned related patent applications is herein incorporated
in its entirety by reference.
Claims
The invention claimed is:
1. Apparatus for communicating a raster video signal, comprising: a
receiver enabled to receive the raster video signal from a graphics
processor and obtain display control parameters of the raster video
signal, the display control parameters comprising frequency and
resolution information for the raster video signal; a raster
encoder, coupled to the receiver, enabled to encode a plurality of
scan lines of the raster video signal to generate a plurality of
encoded scan lines; and a network controller, coupled to the raster
encoder, enabled to transmit the plurality of encoded scan lines
and the display control parameters as packets via an IP/Ethernet
network.
2. The apparatus of claim 1 wherein the raster video signal
comprises a plurality of raster video signal partitions, and
wherein each of the plurality of raster video signal partitions
comprises a portion of the plurality of scan lines.
3. The apparatus of claim 2 wherein the raster encoder is enabled
to generate sequence information associated with the each of the
plurality of raster video signal partitions for use by a remote
computer to detect at least one of data loss or image updates.
4. The apparatus of claim 1, further comprising a controller
enabled to communicate display control information with the
graphics processor and communicate the display control information
with a remote computer for use with a display coupled to the remote
computer.
5. The apparatus of claim 1, wherein the raster encoder and the
network controller are enabled to minimize transmission latencies
for each of the plurality of encoded scan lines.
6. The apparatus of claim 1, wherein the raster encoder and the
network controller manage a transmitting of the plurality of
encoded scan lines as a stream of image data synchronized to the
raster video signal.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates generally to the methods and
apparatus for remotely accessing and operating the data processing
component of a computer system. The invention has particular
application to separating a user interface from a data processing
component, placing the data processor in a different location and
providing a low latency connection between them using a standard
network.
2. Description of the Related Art
Historic advances in computer technology have made it economical
for individual users to have their own computing system, which
caused the proliferation of the personal computer (PC). Continued
advances of this computer technology have made PCs very powerful
but also complex and difficult to manage. For this and other
reasons, there is a desire in many workplace environments to
separate user interface devices, including the display and
keyboard, from the application processing parts of the computing
system. In this preferred configuration, user interface devices are
physically located at the desktop, while processing and storage
components of the computer are placed in a central location. The
user interface devices are then connected to the processor and
storage components over a computer network.
A number of methods and devices have been developed to physically
separate the user interface from the data processor using either
proprietary transmission links (e.g. fiber) or dedicated digital
links (e.g. standard CATS data cabling independent from the
corporate LAN cabling). Examples of these methods include
disclosures by Green et al. in U.S. Patent Application 20030208779
and Thornton in U.S. Pat. No. 6,633,934. Furthermore, commercial
keyboard video and mouse (KVM) systems offered by Avocent and
others provide similar capabilities. All of these physical layer
extension methods are incompatible with existing workplace networks
and therefore do not fulfill the primary objective of lowering
infrastructure and maintenance costs by supporting remote users
over the LAN infrastructure.
In order to understand the impact that a separated user interface
may have on human perception, it is helpful to understand the
system behavior for each existing method from the perspectives of
system latency and quality of the visual experience. Shortcomings
in prior art techniques are identified through an explanation of
how these methods manage the communications of the graphics data to
the remote display.
FIG. 1 illustrates the graphics path of a traditional PC. A
graphical image is displayed after a sequence of events occurs.
First, a software application running on data processor 10 sends
graphic or drawing commands to operating system (O/S) 11. O/S 11, a
graphics driver and graphics hardware then process these commands
and generate an image that is stored in frame buffer 12. Display
controller 13 monitors frame buffer 12 and sends the image to
display 14 as a raster sequence. This raster sequence is
periodically refreshed. Whenever the contents of frame buffer 12
are changed by the application, the modified image is sent to
display 14 the next time display controller 13 reads that part of
frame buffer 12. To create smoother image transitions, an
application or O/S 11 may synchronize its image updates with the
raster timing of display controller 13.
There are three basic methods of communicating the display image
from a data processor across a standard network to a remote
display. The first method is referred to herein as the graphics
command transfer method. Rather than being drawn on the data
processor, the graphics commands from an application are
transferred over a network to a user interface served by a remote
computing system. The remote computing system's O/S, graphics
driver and hardware execute the graphics commands to create an
image that is stored in a remote frame buffer. The remote display
controller reads the image from the remote frame buffer and sends
it as a raster to the remote display. There are a number of
variations on this technique. The X Windows.TM. interface is one
example of an application that acquires graphics commands at a high
level and transfers them to a remote user interface. A second
example is Remote Desktop Protocol (RDP) that converts most of the
graphics commands to simple low-level primitives before
transferring them to the remote user interface.
In the case of a simple remote computing system, the graphics
command transfer method works adequately for transferring simple
images. A few simple graphics commands are communicated across the
network and the resultant network traffic is low. However, for
complex images, the number of commands needed increases
significantly. This increases network traffic and system latency.
Additionally, the image commands also become complex. This
increases the required complexity of the remote computing system
(i.e. O/S, graphics driver and hardware). This in turn increases
the cost, maintenance and support requirements for the remote user
interface, in direct conflict with the original motivation for
centralization--reduced support requirements for remote
computers.
To accommodate low-complexity remote user interfaces, an alternate
graphic command transfer method that may be used. This method
converts the graphics commands to simple commands before
transferring them. One problem with this method is that overall
graphics capabilities are severely constrained by the
low-complexity graphics capabilities of the remote system. This is
due to high-level graphic commands that leverage graphics hardware
acceleration functions in typical computing platforms no longer
available in the simplified command set.
A second problem is that converting commands to simple commands is
performed by the data processor and is a processing intensive
function. The result is that the conversion process slows down the
data processor and reduces the performance of the system.
A variation on the graphic command transfer method is disclosed by
Duursma et al. in U.S. Patent Application 20030177172 entitled
"Method and System for Generating a Graphical Display For a Remote
Terminal Session." In this approach, an application is capable of
recognizing screen images components as either being graphic
commands or bitmaps. Graphic commands are handled similarly to the
method described above. However, when a bitmap is identified, a
compressed data format of the bitmap is retrieved and transmitted
to the remote terminal session in place of the original bitmap.
While this feature adds bitmap capabilities to the command transfer
method, the command processing overheads persist so little overall
improvement to the graphics command processing is realized.
The second method for separating the user interface from the data
processor referred herein as the frame buffer copy method. This
method solves the drawing performance problem described above by
using the O/S, graphics driver and hardware acceleration to draw
the image into the frame buffer on the data processor side of the
network. The image is then copied to a remote frame buffer at the
remote user interface. This frame buffer is then read by the remote
display controller and sent as a raster to the remote display.
Given that there is no direct network connection to the frame
buffer, various methods exist to overcome the problem of
transferring the image from the source frame buffer to the remote
frame buffer. For example, virtual network computing (VNC) provides
a solution that uses a software application at each end. An
application on the data processor side reads the frame buffer,
encodes the image and then sends it to the decoder application at
the remote user interface where it is decoded by the VNC
application and written into the remote frame buffer.
To reduce latency associated with updating image changes, the
encoder continuously monitors the frame buffer and immediately
sends any updates to the decoder. The biggest problem with this
technique arises during times of complex image generation. Given
the encoder software runs on the same processor as the drawing
application, the processor becomes loaded with both encoding and
drawing operations, which slow down the drawing speed and degrades
the user experience.
A second problem with this method arises as a result of multiple,
asynchronous frame buffers. When areas of the remote frame buffer
are updated out of synchronization with the source frame buffer,
the image viewed at the remote display is different from the
intended image in the case of the display being connected directly
to the source frame buffer.
A variation on the VNC software method is a server management
product disclosed under U.S. Pat. No. 6,664,969 to Emerson, et al.,
entitled "Operating System Independent Method and Apparatus for
Graphical Remote Access." Emerson uses a separate hardware module
to read the frame buffer, compress and send the image to an
application at the remote user interface. This variation removes
the encoding software load, but consumes the system bus of the data
processing sub-system each time the frame buffer is read. In cases
where real-time frame updates are required, the load on the system
bus directly compromises the performance of the data processor and
slows down the application. As with the VNC software method, this
method has display continuity problems associated with
synchronizing multiple frame buffers or pointers.
To provide a positive user experience, PC architecture has been
designed for a well-timed image interface at the display controller
output. Therefore, the best existing techniques are derived from
the image being captured at this point. By obtaining the image at
the display controller output, these implementations solve the
problem of loading the processor, but also introduce additional
problems.
A third method for separating the user interface from the data
processor is referred to herein as the frame capture and transfer
method. In this approach, the display controller of the data
processor outputs a standard analog video signal. A frame capture
circuit samples the video signal and captures the image into a
capture frame buffer, one frame at a time. The image in the capture
frame buffer is then transferred over the network to the remote
frame buffer. The transfer operation is performed by an
image-encoding application that accesses the capture frame buffer
and compresses the image before sending it over the network. An
application at the remote end decompresses the image and writes it
into the remote frame buffer. This solution is suitable for
applications where not all frames need to be processed or when the
processing is allowed to take multiple frames.
However, a significant shortcoming of this approach is high delay
and bandwidth consumption that is introduced into the display path
when every frame image is captured and then processed out of the
capture frame buffer memory. Another problem with this method is
that frame-capture circuits lack the ability to detect the image
characteristics, such as sampling frequency, image size and timing.
Rather, these frame capture circuits have predefined capture timing
and do not adapt to the changeable image stream characteristics
defined by the display controller.
Yet another shortcoming of this solution is that frames are dropped
or repeated and images may be torn where the display shows half of
one frame and half of a previous frame. These undesirable effects
are a result of two independent display controllers operating from
different clocks. Even when set to the same frequency, clock
variations result in the display controllers on each side of the
network running at slightly different refresh rates.
Also, current implementations of this approach use analog
connections, which are subject to sampling errors and creates
noise. The noise introduces significantly more data to be
transferred across the network, further increasing system
latency.
A major shortcoming shared by this approach and others described
above is that the image cannot be optimized to the capabilities of
the display. Specifically, given there is no bi-directional
connection between the display controller of the data processing
system and the remote display, the display controller is unable to
detect the display's capabilities.
The disadvantages of the prior art limit the usefulness of the
frame capture and transfer method to low performance support or
administrative computer systems with low-resolution displays.
Finally, another prior approach is to use a video encoder to
capture every frame of a video stream and compresses it for storage
or transmission. The first problem with this video encoder method
is that it does not provide the same spectrum and resolution of a
computer display because of its incompatibility with the RGB
signals used in a computer display. Rather, the video encoder works
with video luminance and chrominance signals.
The second problem with the method is that it uses lossy video
compression techniques, which are not suitable for some of the
images found in a computer display. The third problem with this
method is the compression methods introduce multiple frame delays
whereas the delay introduced by an encoding system must only be a
fraction of a frame period. The fourth problem with video encoders
is they do not support the full range of image sizes available for
a display controller or a computer monitor. Rather they work with
predefined image sizes. The fifth problem with this method is
similar to a problem described for other methods. It is a broadcast
method that captures a defined image size and frame rate and
therefore does not allow the display controller to query the remote
display and adjust its timing, frame rate or size to match the
display.
In addition to the shortcomings described above, the physical
network Impacts the data transfer and consequently the user
experience. Standard corporate LANs use packet-based methods for
data communications, which adds additional performance constraints
when applied to high-data video and graphics applications. There
are two protocols available for data communications over a standard
packet network. The first is the connection-oriented TCP protocol
that guarantees the delivery of the information at the expense of
data transfer performance. The second protocol is the
connectionless UDP protocol that is preferable for real-time
communications due to higher throughput. However UDP does not
guarantee the delivery of data across the network. The frame
transfer methods described above inherently lack real-time
performance and therefore can afford the additional latency
associated with using TCP protocol for graphic transfers.
The communication of real-time images over packet networks using
connectionless protocols such as UDP is common practice in video
communications including video conferencing and video content
delivery across the Internet. H.261 and MPEG4 are examples of
standards for supporting these applications. To minimize the
effects of network-induced data loss, these protocols incorporate
forward error correction methods and redundancy mechanisms. These
protocols also incorporate data reduction methods. For example,
rather than transmitting every frame associated with a video
sequence, the MPEG protocol transmits an intra-coded reference
image known as an I-Frame followed by a series of change vectors
based on a future predicted frame. The decoder then builds a series
of sequential frames using much less data than if each frame were
transferred independently across the network.
The greatest shortcoming of this approach is that erroneous data
may be used as the baseline for future frames causing errors to be
propagated into these future frames and resulting in the display of
distracting artifacts. These lossy compression techniques used are
suitable for video but not for high definition graphics of computer
displays. A second problem is that some encoding schemes introduce
further communication latency that further degrades the user
experience for a computer remote user interface.
One approach to limiting residual error propagation involves
transmitting the graphics image as a sequence of sub-frames and
using a feedback command channel to authorize the retransmission of
corrupt or lost sub-frames. Such a technique is disclosed by Ran in
U.S. Pat. No. 5,768,533 entitled "Video Coding Using Segmented
Frames And Retransmission to Overcome Channel Errors." In this
approach, the receiver waits for all of the sub-frames that make up
an image to be correctly received and then displays the frame. This
may be an adequate method for some wireless video applications but
the lack of display synchronization between the original video
signal and the display frames introduces jitter and adds variable
latency which makes it unsuitable for remote computer displays.
SUMMARY OF THE INVENTION
The present invention solves the problems described above and
allows the data processor of a computer system to be separated from
the user interface using a standard shared network while
interaction latencies between the user interface and the data
processor are actively controlled below the threshold of human
perception.
The present invention provides methods and apparatus that provide
structured communications between the remote user interface devices
and the data processor. According to one aspect of the present
invention, a pair of processing modules enable communication across
a standard shared network between the data processing system and
the devices located at the remote user interface where one
processing module is located at the data processor and the other is
located at the remote user interface. The module located on the
data processor end of the network connects to the system bus, I/O
signals and the raster video output signal of the data processor.
The module located on the user interface end of the network
connects to user devices and a display. In the preferred embodiment
of the invention, the two processing modules communicate with each
other across a standard IP/Ethernet network. In this preferred
embodiment, the display interface is a DVI signal and the user
devices include audio devices, USB peripherals and standard PC I/O
signals. By enabling communications over an IP/Ethernet network,
this aspect of the invention overcomes the problems associated with
the non-standard networking approaches described above.
In another aspect, the present invention utilizes a method that
provides a low latency communication bridge for the display signal
from the data processor to the remote graphic user interface. In a
preferred embodiment, a connectionless network protocol is used in
conjunction with a data sequencing method and a transmission timing
method. By monitoring sequence and timing, this aspect of the
present invention minimizes transmission latencies and perceptible
loss in quality of the original display signal based on network
availability. This method of the present invention overcomes the
delay limitations of the low performance command transfer and frame
capture techniques described. Furthermore, a significant
improvement is offered over fixed bandwidth methods described above
such as the video encoding technique that, in the face of network
congestion, drops content, loses I/O functions or stops working
altogether.
In a third aspect, the present invention provides a remote display
synchronization method that controls the timing of the remote
display. In a preferred embodiment, the method involves setting the
timing of the remote display to that of the video source at the
data processor and repeatedly adjusting the phase of the display
controller to maintain the minimum image transfer delay. This
aspect of the present invention overcomes the display jitter and
tearing problems associated with some of the frame buffering
methods discussed in the background section while minimizing
display latency.
In yet another aspect, the present invention provides a data
recovery method that uses sub-frame partitioning and feedback to
minimize the effects of transmission errors. In accordance with the
preferred embodiment, the method uses the sequencing protocol to
determine the newest image compilation for display, establishes and
requests replacement data requirements from the data sequence
information, and displays frame updates based on an optimized blend
of new data and historic information. In this aspect, the present
invention overcomes the display jitter and latency problems
described above.
By providing a single standard network interface between a data
processor and a remote user interface, the invention reduces the
cost over proprietary methods while the low latency methods
discussed ensure a better user experience than other
standards-based techniques.
Additional features and advantages of the present invention will be
realized by reading the following detailed description, when
considered in conjunction with the accompanying drawings, in
which:
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of the graphics pipeline of a traditional
data processor system.
FIG. 2 is a diagram illustrating one embodiment of the invention
incorporating a Blade PC on one end of a network and a remote
client on the other end of the network.
FIG. 3 is a detailed architecture of the blade PC showing the
interaction of standard processing elements with a PC I/O
communications processor
FIG. 4 illustrates a method in which the PC I/O communications
processor and client I/O communications processor are used to
synchronize the digital raster that drives the remote display with
the digital raster generated by the graphics processor.
FIG. 5 is a block diagram showing the DVI encoding and control
circuits inside the PC I/O communications processor.
FIG. 6 is a block diagram showing the DVI decoding and control
circuits inside the client I/O communication processor.
FIG. 7 is a graph showing the impact of total delay on user
experience.
FIG. 8 shows the packet loss effect for a low delay network, a
medium delay network and a high delay network.
FIG. 9 shows scan line counter delay for different network
types.
A preferred embodiment of the invention is illustrated in FIG. 2.
Blade PC 120 located one end of network 121 includes data
processing system 122 connected to PC I/O communication processor
(PCIOCP) 123 using multiple signal connections 124 which include
signals such as DVI, PCM, PCI and others. Remote client system 200
is located at the other end of network 121 and includes client I/O
communication processor (CIOCP) 125 connected to peripheral devices
normally found connected to a desktop PC via peripheral interface
circuitry 126. These peripheral devices include display 127,
keyboard 128, pointing device 129 and may also include other
peripherals.
FIG. 3 shows a block diagram of blade PC 120 with its connection to
network 121. Blade PC 120 has data processing system 122, typically
found in a desktop PC and PCIOCP 123. Data processing system 122
includes processor 130. Processor 130 is connected to the PC
chipset, which includes north bridge 131 and south bridge 132.
North bridge 131 provides a connection from processor 130 to memory
133, south bridge 132 and graphics processor 134. North bridge 131
provides the necessary high bandwidth connection between these
devices. The connection between north bridge 131 and graphics
processor 134 is a high bandwidth interface such as AGP or
PCI-Express 135. South bridge 132 is an I/O hub that provides the
connection to PC I/O including audio, keyboard, mouse, printers and
other I/O. South bridge 132 may also have connections to hard
drives 136 or, alternatively, all hard drive storage may be
accessed over the network.
There are a number of variations on the described PC data
processing circuitry. Some processors include north bridge
functions while other north bridge devices include the graphics
processor. Some PC digital circuitry includes more processors, more
graphics processors or more I/O devices. All of these are valid
configurations of PC digital circuits for a blade PC. The
requirements for the blade PC are that it includes processor 130,
memory 133, graphics processor 134 and expansion bus 136. Graphics
processor 134 may be substituted with a display controller with
equivalent interfaces. In an embodiment, graphics processor 134
includes both a drawing processor that assists in the drawing of
the image and a display controller that sequences the image out the
display output 138. In another embodiment, the drawing processor
may be absent and the image may be drawn entirely by processor 130.
In either case, a display controller provides the image display
sequence that needs to be communicated over the network. Other I/O
137 may be included with the processing chip set or as additional
I/O devices.
The output of graphics processor 134 is DVI signal 138 that would
normally be connected directly to a computer monitor. Instead, in
the preferred embodiment, the DVI signal is connected to PCIOCP
123, which is capable of communicating the DVI signal across the
network to the remote client. Because the DVI signal is digital, it
may be converted into packets of data and sent across a standard
packet network to be reproduced as a matching DVI signal at the
remote client.
FIG. 4 illustrates how an image is created and transferred from
blade PC 120 to a remote client using PCIOCP 123 and CIOCP 125.
Image 140 is created by graphics processor 134 and output on DVI
signal 138 as a digital raster signal. This DVI signal is then fed
into the PCIOCP 123 where it is encoded and put into packets. It is
then transmitted across network 121 to the CIOCP 125 where it is
converted back into DVI signal 141.
FIG. 4 shows graphics processor 134 and CIOCP raster scan based DVI
signals at a moment in time as they scan down the image. Graphics
processor DVI signal 138 is shown ahead or further down image 143
than CIOCP DVI signal raster scan 144. In an embodiment, the two
DVI signals at each end of the network are synchronized so that the
remote client's DVI signal 141 is operating at a constant delay
from blade PC DVI signal 138. PCIOCP 123 detects the frequency and
phase of the graphics processor's DVI signal and communicates it to
CIOCP 125. CIOCP 125 adjusts the frequency and phase of its DVI
signal to be locked to the graphics processor DVI signal. This is
accomplished through a series of acts. As a first act, PCIOCP 123
detects the frequency of the incoming DVI signal (defined by
resolution and refresh rate) by measuring the frame rate of the
inbound DVI signal 138 to the accuracy of the crystal of PCIOCP
123. PCIOCP 123 communicates the frequency and a tunability range
to CIOCP 125. CIOCP 125 uses the data to set its own frequency and
range of required tunability. As a next act, CIOCP 125 receives the
first scan line and initializes a scan line counter. CIOCP 125
starts displaying the image after a predefined number of scan lines
have been counted. The total latency between DVI signal 138 and
display signal 141 is then equivalent to the network latency and a
predetermined jitter latency determined by the CIOCP counter. As a
next act, the frequency of CIOCP 125 is periodically adjusted
within the controllable range in order to lock the frame rates at a
predefined lag. CIOCP 125 determines the difference between DVI
scan line 138 and display scan line 141 and adjusts its frequency
accordingly. If there is insufficient latency between the two scan
lines, the display clock frequency is marginally decreased. In
cases where the display rate cannot be increased quickly enough
(for example to prevent display jitter), incoming packets or frames
may need to be dropped. If the latency is too long, the display
clock frequency is marginally increased. In an alternative
embodiment, CIOCP 125 develops its timing independently.
By synchronizing blade PC DVI signals and the remote client's DVI
signals, the image seen at the remote client can be exactly the
same as the image seen if the display were connected to the DVI
output of the graphics processor. Changes to the image occur in the
same frame sequence at the remote client as they are created and
displayed by the graphics processor. In addition to minimizing the
delay, this prevents "tearing," an image display problem that
occurs when the display frame rate is not synchronized with the
image source. It also prevents the jitter seen in non real-time
system where frames can be dropped or delayed when the image
complexity increases.
With the blade PC and remote client DVI signals synchronized, the
delay or latency caused by the communication processors and the
network is managed more effectively. In a remote client system
configuration where the network delay and packet loss is very low,
the remote client display is delayed by less then one frame, or
only the number of lines necessary to effectively compress the
image. If the packet loss is high, but the latency is low, the
delay is adjusted to allow for retransmission of packets.
The synchronized display structure is a significant improvement
over the other remote user interface systems as it reproduces the
desktop experience with no image or frame sequence change and only
the delay introduced by the network and the encode and decode time.
When compared to the graphics command transfer method described
above, there is no delay introduced by having the data processor
doing the encoding or the remote client having to draw the
image.
The encoded DVI packet stream can be used to regenerate the DVI
signal without loss at the remote client. However, if data is lost
in the network, or for any other reason, the remote client may not
have the ability to recreate the original DVI signal. The
communication system does not guarantee lossless communications,
but it does guarantee detection of lost data and retransmission of
lost data.
The DVI encoding path on PCIOCP 123 and the DVI decoding path on
CIOCP 125 incorporate mechanisms to enable error recovery in the
case of lost data. Referring first to the encoding path on PCIOCP
123 shown in FIG. 5, DVI receiver 150 receives DVI signal 138 from
the graphics processor 134. DVI receiver 150 passes image stream
158 (including data, clock, and control signals) to DVI encoder
151. DVI receiver 150 also passes display control information 159
such as DDC control information to DVI controller 153. DVI encoder
151 converts the DVI stream into a stream of encoded pixel data
154, which is sent to network controller 152. DVI controller 153
manages DVI encoder 151 using control path 156 and communicates
with network controller 152 for network status and as a
communication channel to the remote client DVI controller using
control path 157 as shown. Network controller 152 manages the
stream of image data and sends it over network 155 to remote client
system 200.
DVI encoder 151 partitions each raster frame generated by
partitioning the DVI signal into a series of block or tile
partitions. Sequence information is added to the encoded DVI data,
so that the remote client can detect lost data and request
corrections. DVI controller 153 maintains the sequence information
for each partition of the DVI signal.
Now referring to the decoding path of CIOCP 125 illustrated in FIG.
6, remote network controller 160 receives the packets sent over
network 155 from PCIOCP 123. Encoded DVI data 161 is sent to DVI
decoder 162 where it is queued for decoding. If the data is
received in advance of when the updated image is required and the
queue fills, the queue is extended into frame buffer/memory 163
until needed. The outputs from DVI decoder 162 are clock, data and
control signals 164 that are a reproduction of signals seen at
PCIOCP 123 in FIG. 5. These signals are sent to DVI transmitter 165
to be converted into DVI signal 141 that goes to the display.
In an embodiment, remote network controller 160 monitors the packet
sequences and sends information back to PCIOCP 123 about the state
of the packet and partition sequence. The information is sent using
an acknowledging protocol with retransmission if it fails. When a
replacement for a sequence number is requested by the remote client
or an acknowledgement missed, DVI controller 153 determines which
DVI signal partition was lost and commands the encoder to encode
that partition on the next frame. This is accomplished by negating
the unchanged indication for the lost partition. Only one sequence
number is maintained for each partition. When a lost sequence
number cannot be found in the partition list, it means that the
associated partition has already been retransmitted and no action
is required.
In cases where the network is congested and the network protocol
forces delayed transmission of partition data, DVI controller 153
monitors both the transmission status of partitions in the queue
and the incoming DVI raster signal. Should the partition be delayed
by an entire frame and an updated version of the same partition
information becomes available, DVI controller 153 refreshes the
display queue with the updated partition data.
In cases where data is successfully transmitted and received but
congestion causes the delayed arrival of the partition such that it
cannot be decoded and written to the buffer ahead of access by the
display controller, decoder 162 uses the control channel to signal
the retransmission of that frame in an attempt to receive newer
replacement data in time for the next frame. If the replacement
data does not get there on time the delayed data can be used as it
will be more accurate than leaving the display unchanged. In an
embodiment, through pinging or the passing of control messages, DVI
controller 153 knows how much time it has to encode and transmit
the signal before it will be too late for the current refresh so it
only sends replacement data if is likely to arrive in time.
An alternative to blade PC 120 is a standard desktop computer, a
pedestal computer or a rack mount computer. Using a standard PCI or
PCI-E card that contains PCIOCP 123, similar capabilities can be
obtained by cabling the DVI signal to the card. In an embodiment,
the remote client is implemented as a standard computer platform
with a standard display. As most of the functionality is
implemented at PCIOCP 123, many of the functions performed by CIOCP
125 are implemented as software in a standard computer platform.
This embodiment does not allow the same low latency performance
because the remote computer does not have access or control over
the real-time video signals.
Other embodiments of the present invention include systems that are
not fully software independent. Some of the functions may be
implemented as driver software on the data processor or as software
on a computer at the remote user interface. While this does not
provide the same guaranteed real-time performance, it can be done
to simplify some of the hardware required.
The present invention describes a lossless sub-frame encoding
scheme for real-time graphical data transfer. Other embodiments of
the invention include lossless or lossy data compression and
encoding methods including progressive build methods, color
compression methods and video or image compression methods.
While the invention refers to standard PC interfaces such as DVI,
the method and apparatus works equally well with other display
controller interfaces including analog interfaces and evolving
digital standards such as HDMI, DPVL, DisplayPort, and the
like.
FIG. 7 is a graph showing the impact of total delay on user
experience. Vertical Axis 700 represents a measure of the
multi-modal interactive experience of a remote display system and
horizontal axis 702 represents increasing delay as a sum of network
delay and controlled delay. Plot 704 shows the causal delay effect
for increased delay. As referred to herein, causal delay describes
the system responsiveness of a remote user interface system during
user interaction with respect to user interface lag. For example,
the lag between when a mouse is moved and when the pointer appears
to move is an example of causal delay. As the delay increases, so
the causal delay effect increasingly degrades the user experience
as shown.
Plot 706 shows the packet loss effect for increased network delay.
As the network delay increases, the packet loss decreases and the
user experience is improved due to improved accuracy in display
image reproduction. Optimum user experience 708 may be found by
adjusting the controlled portion of the delay and establishing the
point where packet loss effect 706 is balanced against causal delay
effect 704 (i.e. the intersection of the curves). Measurement of
the causal delay requires knowledge of the round trip response of
the network. One method of determining the nominal round trip
network latency is by pinging the host system from the client
system. An alternative approach where the round trip delay response
time of the network is unknown is to measure the slope of the
packet loss effect curve by measuring the relative packet delay as
described below.
FIG. 8 shows the packet loss effect for three different networks.
Plot 800 shows a low delay network; plot 802 shows a medium delay
network and plot 804 shows a high delay network. As shown, low
delay networks are characterized by steeper loss effect curves than
high delay networks. The impact is that delay change 804 has much
greater interactive user experience impact 810 on low delay network
800, than same delay change 806 causing less impact 812 on network
802 or same delay change 808 causing even less impact 814 on high
delay network 804.
The slope of the packet loss curve is determined by monitoring the
relative arrival times for each packet in a sequence of transmitted
packets and comparing the relative delay histogram for the packet
sequence with a series of predetermined relative delay histograms
stored in local memory. Each predetermined relative delay histogram
correlates with a different packet loss curve and an associated
optimum scan counter value (related to the optimum delay for that
curve) which is also stored in memory. In an embodiment, a database
or list of scan line offset values is maintained in memory. The
list of scan line offset values contains scan line offset values
read from stored relative delay histograms that have matched a
formed relative delay histogram. In an embodiment, the list
includes most-recently-used scan line offset values. In another
embodiment, the list includes most-often-used scan line offset
values.
FIG. 9 shows scan line counter delay 920 for different network
types. In one embodiment, optimum scan line counter delay 900 is
set for network A (reference numeral 800), optimum delay 902 is set
for network B (reference numeral 802) and optimum delay 904 is set
for network C (reference numeral 804).
In an alternative embodiment, scan line counter delay 906 is
conservatively set to a higher value for low latency network A 800.
Delay 906 is longer than optimum delay 900 but still within human
perception limits. This increases the probability of a greater
quantity of packets arriving ahead of display but without degrading
the interactive experience.
While preferred embodiments of the present invention have been
illustrated and described in detail, it is to be understood that
many changes and modifications can be made to embodiments of the
present invention without departing from the spirit thereof.
* * * * *
References