U.S. patent application number 16/530826 was filed with the patent office on 2019-11-21 for method of video transmission and display.
The applicant listed for this patent is TV ONE Limited. Invention is credited to Richard Peter Disney MALLETT.
Application Number | 20190356940 16/530826 |
Document ID | / |
Family ID | 61617042 |
Filed Date | 2019-11-21 |
![](/patent/app/20190356940/US20190356940A1-20191121-D00000.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00001.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00002.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00003.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00004.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00005.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00006.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00007.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00008.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00009.png)
![](/patent/app/20190356940/US20190356940A1-20191121-D00010.png)
View All Diagrams
United States Patent
Application |
20190356940 |
Kind Code |
A1 |
MALLETT; Richard Peter
Disney |
November 21, 2019 |
METHOD OF VIDEO TRANSMISSION AND DISPLAY
Abstract
Aspects of the present disclosure relate to a method, in a video
output device, for acquiring video data for outputting to a
display. The method comprises subscribing to a multicast stream of
a plurality of multicast streams. Each multicast stream is streamed
from a video source and comprises video frame data corresponding to
a portion of a video frame. The multicast stream to which the video
output device subscribes comprises video frame data that is for
display on a display associated with the video output device. The
method then comprises receiving the video frame data that is for
display on the display.
Inventors: |
MALLETT; Richard Peter Disney;
(Margate, GB) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
TV ONE Limited |
Margate |
|
GB |
|
|
Family ID: |
61617042 |
Appl. No.: |
16/530826 |
Filed: |
August 2, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
PCT/GB2018/050318 |
Feb 2, 2018 |
|
|
|
16530826 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G09G 5/026 20130101;
G09G 2340/0464 20130101; G09G 2350/00 20130101; G09G 2370/20
20130101; G09G 2340/12 20130101; G06F 3/1446 20130101; H04N
21/41415 20130101; H04N 21/431 20130101; G09G 2370/02 20130101;
G09G 5/003 20130101; H04N 21/2393 20130101; H04N 21/437 20130101;
H04N 21/4622 20130101; G09G 2300/026 20130101; G09G 5/14 20130101;
H04N 21/4122 20130101; G09G 2360/121 20130101 |
International
Class: |
H04N 21/41 20060101
H04N021/41; G06F 3/14 20060101 G06F003/14; H04N 21/239 20060101
H04N021/239; H04N 21/437 20060101 H04N021/437; H04N 21/462 20060101
H04N021/462; H04N 21/414 20060101 H04N021/414 |
Foreign Application Data
Date |
Code |
Application Number |
Feb 3, 2017 |
GB |
1701859.9 |
Feb 3, 2017 |
GB |
1701860.7 |
Claims
1. A method, in a video client, for acquiring video data for output
to a display of a video wall, the method comprising: transmitting,
to a video server, a request for video frame data for a portion of
a video frame, the portion of the video frame being associated with
the display of the video wall; receiving video frame data from the
video server, the received video frame data comprising data for a
portion of the video frame corresponding to the request;
transmitting, to the video server, one or more further requests for
video frame data for different portions of the video frame, the
different portions of the video frame being associated with the
display of the video wall; and receiving further video frame data
from the video server, the received further video frame data
comprising data for further portions of the video frame
corresponding to the one or more further requests.
2. A method according to claim 1, wherein: the request defines the
portion of the video frame based on a location of the portion of
the video frame within the video frame; and the further request
defines the different portion of the video frame based on a
different location of the different portion of the video frame
within the video frame.
3. A method according to claim 2, the method comprising outputting
display data, based on the received video frame data, to the
display.
4. A method according to claim 3, wherein the request or further
request for video frame data comprises a request for a block of
pixels.
5. A method according to claim 4, wherein the request for a block
of pixels comprises a location for the block of pixels.
6. A method according to claim 4, wherein the request for a block
of pixels comprises a location of a pixel in the block of
pixels.
7. A method according to claim 4, wherein the request for a block
of pixels comprises an identifier of the block of pixels.
8. A method according to claim 4, wherein the method comprises
determining a mapping of at least one pixel of the display to a
location in the block of pixels.
9. A method according to claim 8, wherein the or each block of
pixels has predefined spatial dimensions.
10. A method according to claim 8, wherein the mapping is such that
a requested block of pixels is for display on a block of pixels of
the display, the block of pixels of the display having the same
dimensions as the requested block of pixels.
11. A method according to claim 8, wherein: the mapping is such
that a requested block of pixels is for display on a block of
pixels of the display; and the method comprises determining pixel
values of the block of pixels of the display by interpolating
between pixel values of the requested block of pixels.
12. A method according to claim 1 wherein: the video server is one
of a plurality of video servers, each video server of the plurality
being configured to provide corresponding video frame data for
display on the video wall; and the method comprises selecting the
server to which a request is sent based on which video frame data
is to be displayed on the display.
13. A method according to claim 1, the method comprising:
transmitting, to a second video server, a second request for video
frame data for a portion of a second video frame, the portion of
the second video frame being associated with the display of the
video wall; and receiving second video frame data from the second
video server, the received second video frame data comprising data
for a portion of the second video frame corresponding to the second
request.
14. A method according to claim 1, comprising receiving the video
frame data as streamed video data.
15. A method according to claim 1, wherein: the or each request
comprises an indication of a desired video quality; and the
received video frame data comprises video frame data at the desired
video quality.
16. A method according to claim 15, wherein the desired video
quality comprises a desired video frame resolution.
17. A method according to claim 15, wherein the desired video
quality comprises a desired degree of video compression.
18. A non-transitory computer-readable storage medium comprising a
set of computer-readable instructions stored thereon which, when
executed by at least one processor, cause the at least one
processor to perform a method according to claim 1.
19. A video client associated with a display of a video wall, the
video client comprising: a network interface; and a processor
coupled to the network interface and configured to: transmit, to a
video server via the network interface, a request for video frame
data for a portion of a video frame, the portion of the video frame
being associated with the display; receive video frame data, via
the network interface, from the video server, the received video
frame data comprising data for a portion of the video frame
corresponding to the request; transmit, to the video server, one or
more further requests for video frame data for different portions
of the video frame, the different portions of the video frame being
associated with the display of the video wall; and receive further
video frame data from the video server, the received further video
frame data comprising data for further portions of the video frame
corresponding to the one or more further requests.
20. A video wall system comprising: a video wall comprising a
plurality of display devices; at least one video server, the or
each video server being configured to provide video frame data; and
a plurality of video clients, each video client being associated
with a corresponding display device of the video wall and each
video client being configured to: transmit to a video server of the
at least one video server, a request for video frame data for a
portion of a video frame, the portion of the video frame being
associated with the display of the video wall; and transmit to the
video server of the at least one video server, one or more further
requests for video frame data for different portions of the video
frame, the different portions of the video frame being associated
with the display of the video wall; the one or more video servers
each being configured to store video frame data and, in response
to: receiving a request from one of said plurality of video
clients, transmit video frame data to the corresponding video
client, the video frame data comprising data for a portion of the
video frame corresponding to the request; and receiving one or more
further requests from one or said plurality of video clients,
transmit further video frame data to the corresponding video
client, the further video frame data comprising data for further
portions of the video frame corresponding to the one or more
further requests.
21. A video wall system according to claim 20, wherein: the
plurality of video clients and the at least one video server
comprise nodes of a network; the network comprises a network switch
which couples each video client to each video server; each video
client is configured to transmit requests for video data via the
network switch; and the or each video server is configured to
transmit video frame data via the network switch.
22. A method of video frame transmission in a video wall system,
the system comprising a plurality of video clients wherein each
video client is associated with a corresponding display of the
video wall, the method comprising: transmitting, from a video
client of the plurality of video clients and to a video server, a
request for video frame data for a portion of a video frame, the
portion of the video frame being associated with the corresponding
display of the video wall; responsive to receiving a said request
from a video client, transmitting, from the video server and to the
video client, video frame data comprising data for a portion of
video frame corresponding to the request; transmitting, from the
video client of the plurality of video clients and to the video
server, one or more further requests for video frame data for
different portions of the video frame, the different portions of
the video frame being associated with the corresponding display of
the video wall; and responsive to receiving the one or more further
requests from the video client, transmitting, from the video server
and to the video client, further video frame data comprising data
for further portions of the video frame corresponding to the one or
more further requests.
23. A method, in a video server, for providing video data for
display on a display of a video wall, the method comprising:
receiving, from a video client, a request for video frame data for
a portion of a video frame; transmitting video frame data to the
video client, the transmitted video frame data comprising data for
a portion of the video frame corresponding to the request;
receiving, from the video client, one or more further requests for
video frame data for different portions of the video frame; and
transmitting further video frame data to the video client, the
transmitted further video frame data comprising data for further
portions of the video frame corresponding to the one or more
further requests.
24. A video server comprising: a network interface; and a processor
coupled to the network interface and configured to: receive, via
the network interface and from a video client, a request for video
frame data for a portion of a video frame, the portion of the video
frame being associated with the display; transmit video frame data,
via the network interface, to the video client, wherein the
transmitted video frame data comprises data for a portion of the
video frame corresponding to the request; receive, via the network
interface and from the video client, one or more further requests
for video frame data for different portions of the video frame, the
different portions of the video frame being associated with the
display; and transmit further video frame data, via the network
interface, to the video client, wherein the transmitted further
video frame data comprises data for further portions of the video
frame corresponding to the one or more further requests.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of International
Application No. PCT/GB2018/050318, filed Feb. 2, 2018, which claims
priority to UK Application No. GB1701860.7, filed Feb. 3, 2017, and
UK Application No. GB1701859.9, filed Feb. 3, 2017, under 35 U.S.C.
.sctn. 119(a). Each of the above-referenced patent applications is
incorporated by reference in its entirety.
BACKGROUND OF THE INVENTION
Technical Field
[0002] The present invention relates to methods, apparatus and
systems for requesting, transmitting and/or receiving video data
for a display.
Background
[0003] A video wall is a video system in which a number of video
monitors or displays are arranged in an array formation to create a
relatively larger overall display screen. A typical video wall
system includes a video processing unit which comprises a number of
video input modules for providing a variety of video inputs, a
central processing module for processing video data from the input
modules, and a number of video output modules for providing video
outputs for connection via standard video cables to the video
monitors. The number of video outputs can be increased by adding
further output modules to the video processing unit. However, the
number of video outputs, and hence the number of monitors supported
by the system, is limited by bandwidth capability of the central
processing module. Increasing resolution of the video inputs places
further load on the central processing module which can result in
further limitations on the number of supported video outputs.
SUMMARY
[0004] According to a first aspect of the present invention, there
is provided a method, in a video output device, for acquiring video
data for output to a display, the method comprising: subscribing to
a multicast stream from a plurality of multicast streams, wherein
each multicast stream of the plurality of multicast streams is
streamed from a video source and comprises video frame data
corresponding to a portion of a video frame; and receiving video
frame data from the subscribed multicast stream for display on the
display.
[0005] A method in accordance with the first aspect of the
invention may enable a video output device to subscribe to
multicast streams corresponding to less than a full frame of video
data from a video source. For example, where the display is a
display of a video wall, the method enables the video output device
to subscribe to streams corresponding to just the video frame data
that is needed for a particular display of the video wall.
Therefore, in comparison to a video output device that receives a
full frame of video data, a video output device operating in
accordance with the first aspect of the invention can receive
relatively less video data but still provide a full set of video
frame data to an associated display. This can have the benefit of
enabling a reduction in the bandwidth requirements for the
reception of video frame data by a video output device.
[0006] In accordance with the invention, the method may further
comprise: subscribing to one or more further multicast streams of
the plurality of multicast streams, and receiving video frame data
from the further subscribed multicast streams for display on the
display.
[0007] A video output device operating in accordance with the
method can subscribe to a set of multicast streams. The set of
multicast streams may contain video data which when output to a
display may fill the entire frame of the display.
[0008] Suitably, subscribing to a particular multicast stream
comprises transmitting a subscription request to a network switch,
the network switch being coupled to the video output device and to
the video source, wherein the subscription request identifies the
particular multicast stream.
[0009] In some embodiments, each of the plurality of multicast
streams comprises video frame data at a given video quality. In
this case, the subscription request may comprise an indication of a
desired video quality, and the received video frame data may
comprise video frame data at the desired video quality.
[0010] This allows a reduction in network bandwidth, as video is
not transmitted to the video output device at a higher quality than
required. The desired video quality may comprise one or more of a
desired video frame resolution, a desired degree of video
compression, a desired video frame bit depth and a desired chroma
subsampling scheme.
[0011] In embodiments, the method comprises determining the or each
multicast stream to request based on a position of the video frame
on the display.
[0012] In some embodiments, determining the position of the video
frame on the display comprises determining a mapping of at least
one pixel of the display to a location in the video frame.
[0013] In embodiments, the method may comprise: determining a
change of the position of the video frame on the display; based on
the determined change in position, determining a new multicast
stream to subscribe to; and transmitting a further subscription
request to the network switch, the further subscription request
identifying the new multicast stream.
[0014] The video output device can thus update the streams to which
it subscribes on the fly, based on changes in the location at which
a given video frame is displayed.
[0015] The or each portion may have predefined spatial
dimensions.
[0016] In some examples, the method comprises outputting display
data, based on the received video frame data, to the display.
[0017] In an embodiment, the method comprises: subscribing to a
second multicast stream from a second plurality of multicast
streams, wherein each multicast stream of the second plurality of
multicast streams is streamed from a second video source and
comprises video frame data corresponding to a portion of a second
video frame; and receiving video frame data from the subscribed
second multicast stream for display on the display.
[0018] A given video output device can thus subscribe to streams
originating from multiple video sources. Because a video output
device only subscribes to streams corresponding to video frame
portions that are for display on its corresponding display, a video
output device may receive video data from multiple video sources
without a significant increase in required bandwidth. Additionally,
this allows a single video to be divided across multiple sources,
which allows higher resolution video to be supported since each
source only needs to provide a lower resolution video.
[0019] The display may be a display of a video wall.
[0020] According to an alternative first aspect of the present
invention, there is provided a non-transitory computer-readable
storage medium comprising a set of computer-readable instructions
stored thereon which, when executed by at least one processor,
cause the at least one processor to perform a method as described
above.
[0021] According to another alternative first aspect of the present
invention, there is provided a video output device comprising: a
network interface; a processor coupled to the network interface and
configured to: subscribe, via the network interface, to a multicast
stream from a plurality of multicast streams, wherein each
multicast stream of the plurality of multicast streams is streamed
from a video source and comprises video frame data corresponding to
a portion of a video frame; and receive, via the network interface,
video frame data from the subscribed multicast stream.
[0022] According to a further embodiment, there is provided a video
system comprising: a plurality of displays; a network switch; at
least one video source coupled to the network switch and configured
to provide a plurality of multicast streams via the network switch,
each multicast stream of the plurality of multicast streams
comprising video frame data corresponding to a portion of a
corresponding video frame; a plurality of video output devices
coupled to the network switch, each video output device being
coupled to a corresponding display and being configured to:
transmit a subscription request to the network switch, the
subscription request being for at least one of the plurality of
multicast streams; the network switch being configured to, in
response to receiving a request from one of the plurality of video
output devices, transmit each requested multicast stream to the
corresponding video output device.
[0023] In some examples, each video source of the least one video
sources provides its corresponding plurality of multicast streams
such that, for a single video frame, a video source transmits video
frame data of each multicast stream in series to the network
switch; and
[0024] the series is such that video frame data corresponding to
adjacent portions of a single video frame is transmitted
nonconsecutively.
[0025] A consequence of the non-consecutive transmission of
adjacent portions is a reduction in the incidence of spikes in
required bandwidth, for example where a given video output device
subscribes to streams corresponding to equivalent portions of
multiple videos.
[0026] The series may be such that multicast streams corresponding
to adjacent portions of a single video frame are transmitted in a
pseudorandom order.
[0027] In some examples, each video source of the least one video
sources provides its corresponding plurality of multicast streams
such that, for a single video frame, a video source transmits video
frame data of each multicast stream in series to the network
switch; and the series is such that video frame data corresponding
to a portion of a single video frame is split into multiple data
packets; and the multiple data packets are transmitted spread out
in time over the entirety of the single video frame.
[0028] In some examples, the network switch is configured to
receive a multicast stream comprising video frame data
corresponding to a portion of a corresponding video frame, and to
provide the video frame data as a sequence of data packets at an
output of the switch. Preferably, the sequence of data packets are
provided at an output of the switch spread out in time over one or
more video frames.
[0029] A consequence of the spreading out in time of the
transmission of the data packets is a reduction in the incidence of
spikes in required bandwidth, for example where a given video
output device subscribes to streams corresponding to equivalent
portions of multiple videos.
[0030] The series may be such that multicast streams corresponding
to adjacent portions of a single video frame are transmitted in
data packets that are spread out in time over a single video
frame.
[0031] The plurality of displays may be displays of a video
wall.
[0032] According to a further embodiment, there is provided a
method of video frame transmission in a video system, the method
comprising: transmitting, from a video source and to a network
switch, a plurality of multicast streams, each multicast stream of
the plurality of multicast streams comprising video frame data
corresponding to a portion of a corresponding video frame;
transmitting, from a video output device and to the network switch,
a subscription request for at least one of the plurality of
multicast streams; and in response to receiving a said subscription
request, transmitting each requested multicast stream from the
network switch to the video output device.
[0033] According to another alternative first aspect of the present
invention, there is provided a method, in a video source, for
providing streamed video data. The method comprises: retrieving
video frame data from a video input; and providing to a network a
plurality of multicast streams, wherein each multicast stream of
the plurality of multicast streams comprises video frame data
corresponding to a portion of a video frame.
[0034] According to another alternative first aspect of the present
invention, there is provided a video source comprising: a video
input; a network interface; and a processor, the processor being
configured to: retrieve video frame data from the video input; and
provide, via the network interface, a plurality of multicast
streams, wherein each multicast stream of the plurality of
multicast streams comprises video frame data corresponding to a
portion of a video frame.
[0035] According to a second aspect of the present invention, there
is provided a method, in a video client, for acquiring video data
for output to a display of a video wall. The method comprises:
transmitting, to a video server, a request for video frame data for
a portion of a video frame, the portion of the video frame being
associated with the display of the video wall; and receiving video
frame data from the video server, the received video frame data
comprising data for a portion of the video frame corresponding to
the request.
[0036] A method in accordance with the second aspect of the
invention may enable a video client to request and receive less
than a full frame of video data from a video server. This in turn
enables the video client to request just the video frame data that
is needed for a particular display of the video wall. Therefore, in
comparison to a video client that receives a full frame of video
data, a video client operating in accordance with the second aspect
of the invention can receive relatively less video data but still
provide a full set of video frame data to an associated display.
This can have the benefit of enabling a reduction in the bandwidth
requirements for the reception of video frame data in a video
system, and has beneficial application in video wall systems
comprising multiple displays.
[0037] In an embodiment, the method comprises: transmitting, to the
video server, one or more further requests for video frame data for
different portions of the video frame, the different portions of
the video frame being associated with the display of the video
wall; and receiving further video frame data from the video server,
the received further video frame data comprising data for further
portions of the video frame corresponding to the one or more
further requests.
[0038] The video client thus receives a series of portions of video
frame data, thereby allowing smooth network traffic.
[0039] In some examples, the method comprises outputting display
data, based on the received video frame data, to the display.
[0040] In some examples, the request or further request for video
frame data comprises a request for a block of pixels. The request
for a block of pixels may comprise a location for the block of
pixels, a location of a pixel in the block of pixels, or an
identifier of the block of pixels. The video client can thus
efficiently request particular blocks of pixels.
[0041] In an embodiment, the method comprises determining a mapping
of at least one pixel of the display to a location in the block of
pixels.
[0042] According to one embodiment, the or each block of pixels has
predefined spatial dimensions. Video data may thus be stored, for
example at the video server, in a block-based format. This allows
efficient requesting and receiving of video frame data.
[0043] In some examples, the mapping is such that a requested block
of pixels is for display on a block of pixels of the display, the
block of pixels of the display having the same dimensions as the
requested block of pixels.
[0044] In other examples, the mapping is such that: a requested
block of pixels is for display on a block of pixels of the display;
and the method comprises determining pixel values of the block of
pixels of the display by interpolating between pixel values of the
requested block of pixels.
[0045] In an embodiment the video server is one of a plurality of
video servers, each video server of the plurality being configured
to provide corresponding video frame data for display on the video
wall; and the method comprises selecting the server to which a
request is sent based on which video frame data is to be displayed
on the display.
[0046] Video from various video servers may thus be provided to a
single video client.
[0047] In some embodiments, the method comprises: transmitting, to
a second video server, a second request for video frame data for a
portion of a second video frame, the portion of the second video
frame being associated with the display of the video wall; and
receiving second video frame data from the second video server, the
received second video frame data comprising data for a portion of
the second video frame corresponding to the second request.
[0048] A video client can thus receive and output video frame data
from multiple video servers. This allows a single video to be
divided across multiple servers, which allows higher resolution
video to be supported since each server only needs to support a
lower resolution video. Additionally, because a video client only
requests portions that are for display on its corresponding
display, a video client may receive video data from multiple video
servers without a significant increase in required bandwidth.
[0049] In some examples, the method comprises receiving the video
frame data as streamed video data.
[0050] In an embodiment, the or each request comprises an
indication of a desired video quality; and the received video frame
data comprises video frame data at the desired video quality.
[0051] This allows a reduction in network bandwidth, as video is
not transmitted at a higher quality than required. The desired
video quality may comprise a desired video frame resolution and/or
a desired degree of video compression.
[0052] According to an alternative second aspect of the present
invention, there is provided a non-transitory computer-readable
storage medium comprising a set of computer-readable instructions
stored thereon which, when executed by at least one processor,
cause the at least one processor to perform a method as described
above.
[0053] According to another alternative second aspect of the
invention, there is provided a method, in a video client, for
acquiring video data for output to a display, the method comprising
transmitting a series of video frame data requests to a video
server, wherein each request is for a different portion of a video
frame; and receiving, in a series of responses to the series of
video frame data requests, video frame data from the video server
for output to the display.
[0054] The different portions of the video frame may be different
areas or regions within the video frame. Furthermore, the video
frame data may be a subset of the video frame data for the complete
video frame.
[0055] According to an embodiment, there is provided a video client
associated with a display of a video wall, the video client
comprising: a network interface; and a processor coupled to the
network interface and configured to: transmit, to a video server
via the network interface, a request for video frame data for a
portion of a video frame, the portion of the video frame being
associated with the display; receive video frame data, via the
network interface, from the video server, the received video frame
data comprising data for a portion of the video frame corresponding
to the request.
[0056] According to a further embodiment, there is provided a video
wall system comprising: a video wall comprising a plurality of
display devices; at least one video server, the or each video
server being configured to provide video frame data; and a
plurality of video clients, each video client being associated with
a corresponding display device of the video wall and each video
client being configured to: transmit to a video server of the at
least one video server, a request for video frame data for a
portion of a video frame, the portion of the video frame being
associated with the display of the video wall; the one or more
video servers each being configured to store video frame data and,
in response to receiving a request from one of said plurality of
video clients, transmit video frame data to the corresponding video
client, the video frame data comprising data for a portion of the
video frame corresponding to the request.
[0057] In some examples, the plurality of video clients and the at
least one video server comprise nodes of a network; the network
comprises a network switch which couples each video client to each
video server; each video client is configured to transmit requests
for video data via the network switch; and the or each video server
is configured to transmit video frame data via the network
switch.
[0058] According to an alternative second aspect of the present
disclosure, there is provided a method of video frame transmission
in a video wall system, the system comprising a plurality of video
clients wherein each video client is associated with a
corresponding display of the video wall, the method comprising:
transmitting, from a given video client of the plurality of video
clients and to a video server, a request for video frame data for a
portion of a video frame, the portion of the video frame being
associated with the corresponding display of the video wall; and
responsive to receiving a said request from a given video client,
transmitting, from the video server and to the given video client,
video frame data comprising data for a portion of video frame
corresponding to the request.
[0059] According to an embodiment, there is provided a method, in a
video server, for providing video data for display on a display of
a video wall, the method comprising: receiving, from a video
client, a request for video frame data for a portion of a video
frame; and transmitting video frame data to the video client, the
transmitted video frame data comprising data for a portion of the
video frame corresponding to the request.
[0060] In a further embodiment, there is provided a video server
comprising: a network interface; and a processor coupled to the
network interface and configured to: receive, via the network
interface and from a video client, a request for video frame data
for a portion of a video frame, the portion of the video frame
being associated with the display; and transmit video frame data,
via the network interface, to the video client, wherein the
transmitted video frame data comprises data for a portion of the
video frame corresponding to the request.
[0061] Further features and advantages of the invention will become
apparent from the following description of preferred embodiments of
the first and second aspects of the invention, given by way of
example only, which is made with reference to the accompanying
drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0062] FIG. 1 shows a video wall system according to an embodiment
of the first aspect of the invention.
[0063] FIG. 2 shows a schematic representation of a method
according to an embodiment of the first aspect of the
invention.
[0064] FIG. 3 shows a schematic representation of a method
according to an embodiment of the first aspect of the
invention.
[0065] FIG. 4 shows an example of data flow in a video system
according to an embodiment of the first aspect of the
invention.
[0066] FIG. 5 shows a schematic representation of a subscription
request data packet according to an embodiment of the first aspect
of the invention.
[0067] FIG. 6 shows a schematic representation of a video source
according to embodiments of the first and second aspects of the
invention.
[0068] FIG. 7 shows a schematic representation of a network switch
according to an embodiment of the first aspect of the
invention.
[0069] FIG. 8 shows a schematic representation of a video output
device according to embodiments of the first and second aspects of
the invention.
[0070] FIG. 9 shows a schematic representation of a video output
device according to embodiments of the first and second aspects of
the invention.
[0071] FIG. 10 shows a schematic representation of a video wall
according to an embodiment of the first aspect of the
invention.
[0072] FIG. 11 shows a schematic representation of a layout of a
display, a canvas and two video windows according to embodiments of
the first and second aspects of the invention.
[0073] FIG. 12 shows an example of transmission, receipt and
display of streamed video data on displays of a video wall
according to an embodiment of the first aspect of the
invention.
[0074] FIG. 13 shows a table of example bit rates required to
transmit video data at various resolutions and chroma subsampling
schemes according to some embodiments of the first and second
aspects of the invention.
[0075] FIG. 14 shows an example procedure for a video output device
to subscribe to streams according to an embodiment of the first
aspect of the invention.
[0076] FIG. 15 shows an ordering of blocks of video frame data for
transmission according to an embodiment of the first aspect of the
invention.
[0077] FIG. 16 shows a division of video frame data for
transmission and an example of a display layout according to an
embodiment of the first aspect of the invention.
[0078] FIG. 17 shows a schematic representation of transmission of
packets of video frame data according to an embodiment of the first
aspect of the invention.
[0079] FIG. 18 shows a video wall system according to an embodiment
of the second aspect of the invention.
[0080] FIG. 19 shows a video wall system according to an embodiment
of the second aspect of the invention.
[0081] FIG. 20 shows a schematic representation of a method in a
video client according to an embodiment of the second aspect of the
invention.
[0082] FIG. 21 shows a schematic representation of a method of
video frame data transmission in a video wall system according to
an embodiment of the second aspect of the invention.
[0083] FIG. 22 shows a schematic representation of a request data
packet according to an embodiment of the second aspect of the
invention.
[0084] FIG. 23 shows an example of data flow in a video wall system
according to an embodiment of the second aspect of the
invention.
[0085] FIG. 24 shows an example of data flow in a video wall system
according to an embodiment of the second aspect of the
invention.
[0086] FIG. 25 shows a schematic representation of a video server
according to an embodiment of the second aspect of the
invention.
[0087] FIG. 26 shows an example process for mapping between a
display and a video window according to an embodiment of the second
aspect of the invention.
[0088] FIGS. 27 to 30 show example methods of downscaling video
data.
[0089] FIGS. 31 to 34 show a method of transmitting downscaled
video data, according to an embodiment.
[0090] FIG. 35 shows a table of example efficiencies for
transmitting downscaled video data.
DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS
[0091] In the following description, reference is made to videos
and video data. In general, videos that are processed by computers
and video processing devices can be thought of as a sequence of
individual still images which are often referred to as video
frames. Each image or video frame consists of a number of pixels
which are typically arranged in a grid of horizontal rows and
vertical columns. The number of horizontal rows (or lines) and
vertical columns (or lines) determines a resolution characteristic
of the image or frame as well as the corresponding video.
Multicast Video Wall
[0092] FIGS. 1 to 17 show examples and representations relating
variously to a multicast aspect of the present invention. Referring
to FIG. 1, there is shown an example video system 100. Links
between entities of the system 100 may for example comprise 1G, 10G
or 25G copper or fibre links.
[0093] The system 100 comprises a video wall 105 comprising an
array of four display devices 105a-d arranged in a two-by-two
layout. Although the video system 100 is shown as a video wall
system, it will be appreciated that embodiments of the present
disclosure are also applicable to other video systems including,
for example, systems where a plurality of displays do not
necessarily form a video wall. Examples video systems include
systems used in courtrooms, government assemblies, boardrooms,
command & control centers, simulators, operating theatres,
television studios, live stage events, casinos, and sports
stadiums.
[0094] Each display device may, for example, be a computer monitor,
a video projector or a television screen. The displays may be
general purpose display devices or, alternatively, they may be
display devices designed specifically for use in a video wall
105.
[0095] The system 100 comprises a network switch 107, such as a
non-blocking network switch. A network switch includes a number of
bidirection input/output ports, and a non-blocking network switch
is one where each port can typically pass data to every other port,
and typically the internal bandwidth of the switch can handle all
the port bandwidths, at the same time, at full capacity. The switch
107 is configured to handle multicast subscription requests. The
switch 107 may be utilised to provide a master Dynamic Host
Configuration Protocol (DHCP) server and/or time server for the
system 100.
[0096] The video system 100 comprises one or more video sources
110. The video source 110 is coupled to the network switch 107 and
together they are configured to provide a plurality of multicast
streams via the network switch 107. Each multicast stream comprises
video frame data corresponding to a portion of a video frame. For
example, a video frame with a particular resolution may be divided
into fixed-size lower-resolution portions as described in more
detail below. Preferably, the video frame is divided in both the
horizontal and vertical direction so that each portion of the video
frame data corresponds to a block of pixels. Each block may be a
rectangular block having the same aspect ratio as the original
video frame. Or the blocks could be square blocks of pixels. Each
portion of the video frame would then be transmitted in a
respective multicast stream.
[0097] The portion of the video frame may be a region within the
video frame, or an area within the video frame. If the video frame
is available in different resolutions, the portion of a video frame
may be a portion of a video frame at a particular resolution. If
the video frame at a particular resolution has particular number of
pixels then the portion of the video frame may be a portion, region
or area of the video frame covering a subset of the full set of
pixels. If the video frame at a particular resolution is
represented by a particular set of video frame data then the
portion of the video frame may be a portion, region or area of the
video frame incorporating a subset of video frame data.
[0098] In some embodiments, a single video source 110 provides a
single video for display on the video wall 105. In a further
embodiment, the system 100 comprises a plurality of video sources
110, each of which are configured to provide respective video frame
data for display on the video wall 105. Some or all of the
plurality of sources may comprise separate hardware. Alternatively
or additionally, some or all of the plurality of sources may
comprise logically separate sources within the same source
hardware. In this manner, a single physical video source 110 can
provide multiple videos for simultaneous display on the video wall
105. In some embodiments wherein sources comprise separate
hardware, multiple video sources 110 provide the same video frame
data. Such an arrangement provides redundancy in the system so that
a loss of video from one video source 110 can be compensated for by
the provision of the same video from another video source.
[0099] In some embodiments, for example where provision of video
frames would require particularly high bandwidth or processing
power, a single source video can be divided, for example spatially,
into separate videos, each corresponding to a region of the single
source video. For example, video frames may be divided into a
number of evenly spaced vertical strips. The separate videos can
then be provided by physically separate video sources. This
approach is useful for relatively high resolution source
videos.
[0100] In the example shown in FIG. 1, the video source 110
provides a single video for display across the entire video wall
105 such that the top-left quarter of a given video frame is
displayed on display device 105a, the top-right quarter is
displayed on video device 105b, the bottom-left quarter is
displayed on display device 105c and the bottom-right quarter is
displayed on video device 105d.
[0101] The video source 110 may include a storage medium such as a
hard disk or flash memory for storing a video file, which can be
retrieved when a video needs to be output by the video source. The
video could also originate from a live source such as a video from
a video camera. Such a storage medium or live source of video can
be local to the video source 110 hardware. For example, a video may
be provided from a flash memory in the same cabinet as the video
source 110. Alternatively, the storage medium or live source of
video can be remote from the video source 110 hardware. For
example, a live source may comprise a video camera coupled via a
wireless link to the video source 110.
[0102] The video system 100 further comprises a plurality of video
output devices 115a-d. Each video output device 115a-d is coupled
to the network switch 107 and is also coupled to a corresponding
display device 105a-d of the video wall 105. In some embodiments, a
given video output device 115a-d is incorporated within the
functionality of its corresponding display device 105a-d. For
example, a display device 105a-d may comprise a dedicated video
output device 115a-d hardware. Alternatively, a general purpose
processor of a display device 105a-d may implement the
functionality of a video output device 115a-d. Each video output
device 115a-d is configured to transmit a subscription request to
the network switch 107. The subscription request is for at least
one of the plurality of multicast streams provided by the video
source 110. The requested stream comprises video frame data that is
for display on the display device 105a-d to which that video output
device 115a-d is coupled. As an example, video output device 115a,
associated with the top-left display 105a, would transmit a
subscription request for at least one multicast stream
corresponding to the top-left quarter of the video frame.
[0103] In an embodiment, a video output device 115a-d is configured
to subscribe to at least one further stream of the plurality. Each
of the at least one further streams comprises video frame data that
is for display on the corresponding display 105a-d such that the
combined requested streams together provide the entirety of video
frame data that is for display on the display device 105a-d. In an
example, a video output device 115a-d may submit separate
subscription requests for each such stream. Alternatively, a video
output device 115a-d may submit a single subscription request for
multiple streams.
[0104] In some embodiments, the multicast streams are managed by a
stream controller 120. Each video source 110 transmits to the
stream controller 120 a description of the streams that it
provides. The description may for example include a unique
identifier of each stream and details of the specific portion of
the video frame to which each stream relates. The stream controller
120 then transmits such a description to each video output device
115a-d. Each video output device 115a-d can thus determine which
streams it should subscribe to, as described in more detail
below.
[0105] In other embodiments, the system 100 does not comprise a
stream controller 120. In such embodiments, each video source 110
broadcasts a description of the streams that it provides to each
video output device 115a-d. As above, the description may include a
unique identifier of each stream and details of the specific
portion of the video frame to which each stream relates. Each video
output device 115a-d can thus receive the broadcast stream
descriptions and hence determine which streams it should subscribe
to.
[0106] The network switch 107 is configured to receive subscription
requests. In response to receiving a subscription request from a
video output device 115a-d, the network switch transmits each
requested stream to that video output device 115a-d. A given video
output device 115a-d thus only receives video frame data based on
the portion or portions to which it subscribes. Hence, the system
100 avoids the need to transmit the entirety of the video frame
data to each video output device. As such, the overall bandwidth
requirements of the system 100 can be reduced.
[0107] By using a network switch to distribute video streams to
different video output devices, as well as manage streaming
requests, the number of components of the system 100 can be kept
low. For example, the system 100 may be configured with the network
switch, the video input(s) 110, and the video output devices but
without using any separate video servers to receive video frame
data from the video source 110 and provide the data to the video
output devices 105a-d.
[0108] It has been observed that, in systems in which the entire
video frame is sent to each video output device 105a-d, the
required network bandwidth increases with the number of displays.
Similarly, the required processing power of a video source 110
increases with the number of displays. This can increase to the
point that no further displays can be added to the video wall. The
present system 100 enables pixel data of a video frame, provided by
a video source 110, to be transmitted only to a subset of the video
output device 115a-d that require that pixel data for their
respective displays 105a-d. The system 100 can also enable a
multicast stream relating to a portion of the video frame which is
to be partially displayed on one display and partially displayed on
one or more further displays to be subscribed to by the video
output devices of those displays. Furthermore, in the case of edge
blending, where two or more projectors acting as display devices
provide partially overlapping video output, the system 100 enables
the output of overlapping video by providing pixel data for the
overlapping portion to the two or more projectors.
[0109] As such, the system 100 enables the bandwidth of video frame
data transmitted from the video source 110 to each video output
devices 115a-d, via the network switch 107, to be significantly
lower than the bandwidth required for transmitting a complete
version of the video frame data from the video source 110 to each
display device 115a-d, regardless of how many displays 105a-d the
video frame is spread across. Additional displays, with
corresponding video output devices, can thus be added to the system
without requiring additional or upgraded video source hardware.
This remains true if such an additional display is physically
remote from the rest of the network hardware, but it may be
necessary to connect the additional display via upgraded connection
hardware capable of transmission over an extended distance.
[0110] Suitably, the number of videos and/or the resolution of
video available in the system 100 can be increased by upgrading
individual video source 110 devices so that they are capable of
handling increased bandwidth. Additionally or alternatively, the
number of videos and/or the resolution of video available in the
system 100 can be increased by adding additional video source 110
devices.
[0111] Typically, a video output device 115a-d will be capable of
outputting video frame data at the maximum resolution of its
corresponding display 105a-d. Therefore, even if the number of
videos and/or the resolution of video available in the system 100
is increased, this would typically not require an upgrade the video
output devices 115a-d.
[0112] After receiving the video frame data from the network switch
107, each video output device 115a-d then outputs display data,
based on the received video frame data, to its corresponding
display 105a-d. For example, the display data may comprise data in
accordance with the HDMI standard, the DVI standard, or the VGA
standard. In some examples, a video output device 115a-d may
produce the display data by identifying a part of the received
video frame data as not being for display, and producing the
display data so as to exclude the identified part of the received
video frame data. For example, where each multicast stream
corresponds to a fixed-size block of a video frame, a given video
output device 115a-d subscribes to streams such that the blocks
corresponding to each requested stream together include the portion
of the video frame that is for display on its corresponding display
105a-d. The received blocks may also cover further portions of the
video frame, adjoining the portion that is for display. The video
output device 115a-d may thus identify such further portions as not
for display, and produce the display data omitting such further
portions.
[0113] In some embodiments, when a video output device 115a-d does
not receive a given block, for example due to network errors, the
video output device 115a-d can interpolate the missing data from
surrounding pixel blocks. Alternatively, the video output device
115a-d may replace the missing block with a corresponding block
from the previous frame, for example held in a buffer. However,
provided that a large number of blocks are not missing, such errors
are typically not very noticeable on a display.
[0114] In some embodiments, multiple switches 107 can be used. In
an example video wall 105 in which the total required bandwidth,
for all displayed videos, does not exceed 10 Gb, the one or more
video sources 110 may be linked to a single 10 Gb switch. This is
then linked via a single 10 Gb link to a similar switch, to which
all video output devices 115a-d are connected. For large video
walls 105, multiple switches could be configured, with each switch
being connected to a subset of the video output devices 115a-d. In
some embodiments in which multiple switches are used, each switch
is located physically near the video output devices 115a-d and/or
video sources 110 to which it is connected. The switches may then
be connected to each other via a fast interconnection, such as a
fibre interconnection. In such examples, it is preferable for the
interconnection to be capable of supporting non-blocking usage over
all connected ports. The interconnection should preferably be
capable of handling the greater of the total video output device
115a-d bandwidth and the total video source 110 bandwidth.
[0115] As a further example of network hardware, two or more
connections could be used to send video frame data from the network
switch 107 to a given video output device 115a-d. Such a
configuration is particularly useful in embodiments such as control
rooms, wherein multiple lower resolution sources are displayed on a
higher resolution display, as this configuration allows a higher
resolution display to be used since more video frame data can be
transmitted from the network switch 107 to a given video output
device 115a-d. Alternatively, a single high-bandwidth connection
could be used instead of two or more lower-bandwidth
connections.
[0116] Alternatively or additionally, two or more connections could
be used to transmit multicast streams from a video source 110 to
the network switch 107. This configuration is particularly
advantageous in embodiments where the resolution of the video from
the video source 110 is higher than the display resolution, as it
allows a higher resolution video frame to be displayed on the video
wall 205. Alternatively, a single high-bandwidth connection could
be used instead of two or more lower-bandwidth connections.
[0117] A further advantage of the use of two connections to video
output devices 115a-d and/or video sources 110 is that such a
system could, in response for example to a network issue, use only
one of the two connections. Such a system thus provides redundancy
in case of network faults. As an example, when two links from a
video source 110 are in use, video frame data could be streamed at
a lower compression ratio and when a single link is in use, video
frame data could be streamed at a higher compression ratio.
Switching between these modes of operation may be triggered by
automated sensing of a network issue, for example by further
network hardware such as a network controller.
[0118] FIG. 2 shows a schematic representation of a method 200, in
a video output device, for acquiring video data for outputting to a
display according to an aspect of the present disclosure.
[0119] The method 200 comprises a subscribing step 205 of
subscribing to a multicast stream of a plurality of multicast
streams. Each multicast stream is streamed from a video source and
comprises video frame data corresponding to a portion of a video
frame. The multicast stream to which the video output device
subscribes comprises video frame data that is for display on a
display associated with the video output device.
[0120] The method then comprises a receiving step 210 of receiving
the video frame data that is for display on the display.
[0121] FIG. 3 shows a schematic representation of a similar method
300 of video frame transmission in a video system. The system, as
described above in connection with FIG. 1, comprises at least one
video source 305, a network switch 310 and at least one video
output device 315. The method comprises data flow between a given
video source 305 and a given video output device 315 via the
network switch 310.
[0122] The video source 305 transmits to the network switch 310 a
plurality of multicast streams 325. As set out above, each
multicast stream comprises video frame data corresponding to a
portion of a corresponding video frame.
[0123] The video output device 315 transmits to the network switch
310 a subscription request 330 for at least one of the plurality of
streams. Each requested stream comprises video frame data that is
for display on a display device associated with the video output
device.
[0124] In response to receiving a said subscription request, the
network switch 310 transmits each requested stream 335 to the video
output device 315.
[0125] FIG. 4 shows an example of data flow in a video system. The
system comprises video sources 405, 410 and video output devices
415, 420, connected to each other via a network switch 425. Video
source 405 provides multicast streams 405a-c to the network switch
425, each of which relate to portions of a first video. Video
source 410 provides multicast streams 410a-c to the network switch
425, each of which relate to portions of a second video. Multicast
streams 405a and 405b correspond to portions of the first video
that are for display on the display associated with video output
device 415. Multicast streams 405b and 405c correspond to portions
of the first video that are for display on the display associated
with video output device 420. Multicast streams 410a and 410c
correspond to portions of the second video that are for display on
the display associated with video output device 420. Multicast
stream 410b corresponds to a portion of the second video that is
not for display on either output device 415, 420.
[0126] Video output device 415 transmits to the network switch 425
one or more subscription requests 430 identifying streams 405a and
405b. A single subscription request 430 may be transmitted,
identifying both streams 405a and 405b. Alternatively, video output
device 415 may transmit separate subscription requests 430 for each
stream 405a, 405b.
[0127] Similarly, video output device 420 transmits to the network
switch 425 one or more subscription requests 435 identifying
streams 405b, 405c, 410a, 410c. A single subscription request 435
may be transmitted, identifying all four streams 405b, 405c, 410a,
410c. Alternatively, video output device 420 may transmit one
request 435 for streams 405b, 405c from video source 405, and a
second request for streams 410a, 410c from video source 410. As a
further example, separate subscription requests 435 may be
transmitted for each stream 405b, 405c, 410a, 410c.
[0128] Dashed lines within the switch 425 indicate data flow from
video sources 405, 410 to video output devices 415, 420 via the
switch 425. In response to receiving the above-described
subscription requests, the switch 425 transmits each requested
stream to the corresponding video output devices 415, 420.
Specifically, the switch 425 transmits streams 405a, 405b to video
output device 415, and transmits streams 405b, 405c, 410a, 410c to
video output device 420.
[0129] FIG. 5 shows a schematic representation of a subscription
request data packet 500 that is transmitted from a video output
device to a network switch according to an embodiment. In analogous
embodiments, the packet may be structured according to a standard
multicast protocol, for example Internet Group Management Protocol
(IGMP).
[0130] The packet 500 comprises a network address 515, for example
a multicast group IP address, of the source of the video data to
which the subscription request relates. In some embodiments, a
stream controller receives details of each available stream,
including the corresponding multicast group addresses, from each
video source. The stream controller transmits these details to each
video output device. In other embodiments, video sources broadcast
details of available streams to each video output device. Each
video output device can thus maintain a record of the available
streams, including the multicast group address of each stream.
[0131] The packet 500 comprises a network address 520 of the
display device from which the subscription request originates.
[0132] In some embodiments, the packet 500 further comprises a
field 525 identifying the packet as a subscription request. In some
examples, the network address 515 is sufficient to identify the
packet 500 as a subscription request. In such examples, the packet
500 does not include the identification field 525.
[0133] The packet 500 then comprises a checksum 530 to enable the
network switch to detect errors in the packet 500. For example, the
checksum may comprise the 16-bit one's complement of the one's
complement sum of the entire packet 500.
[0134] A network switch is configured to identify the packet 500 as
a subscription request, for example by identifying the network
address 515 as a multicast group address or by way of the
identification field 525 identifying the packet as a subscription
request. The network switch is configured to maintain a record of
subscriptions. The network switch receives each multicast stream
from each video source and, based on the record of subscriptions,
forwards each stream to the video output devices that have
subscribed to that stream.
[0135] In some examples, the packet 500 comprises an identification
of multiple streams to which the subscription request relates. For
example, the packet 500 may identify multiple multicast group
addresses. In one such example, the packet 500 comprises a field
identifying the bit length of the packet and/or the number of
streams to which the request relates.
[0136] FIG. 6 shows a schematic representation of a video source
600 according to one embodiment. The source 600 comprises various
modules. Any or all of these modules could, either alone or in any
combination, be implemented by dedicated circuitry, for example one
or more field-programmable gate arrays, or be implemented by
routines in one or more general processors.
[0137] The source 600 comprises an input 605 to receive video data,
for example from a storage medium such as a hard disk or
solid-state memory, or from a live video feed.
[0138] In some examples, video frame data is stored as
block-by-block video data. The video frame data is then received at
the input as block-by-block data, with each block of a given frame
to be sent in a separate multicast stream. Alternatively, video
frame data may be stored as line-by-line video frame data and
converted by a module of the video source 600 to block-by-block
video frame data, with each block corresponding to a separate
multicast stream. As an example of such a conversion, 135 lines of
video frame data may be cached and, from this, blocks of size
240.times.135 pixels may be determined. A 1920.times.1080 pixel
frame may thus be converted into an 8.times.8 grid of blocks. This
allows the block size to be modified on the fly, for example in
response to a command from a stream controller.
[0139] In some examples, block-by-block video frame data is stored
such that a given block is stored in successive memory locations of
a bank of a memory, with adjacent blocks being stored in different
banks of the memory. A given block can thus be rapidly read out
from a bank of the memory. In embodiments wherein adjacent blocks
are streamed in succession, this arrangement facilitates rapid
access to adjacent blocks as one bank of the memory can be read
while another is being opened or closed. An example of such
block-based storage is described in more detail in European patent
EP2446413B1, the contents of which is incorporated herein by
reference.
[0140] The source comprises a sampling rate and colour conversion
module 610, configured to convert the received video to a data rate
and/or colour space that is suitable for providing via the network
switch to video output devices.
[0141] The source 600 comprises a scaling module 615 for
downscaling the video data. The scaling module 615 may for example
perform block-based downscaling, downscaling each block of video
frame data to provide multiple streams for each block, each having
a different overall downscaling factor and/or chroma subsampling
scheme. The downscaled block data is stored in a block memory
620.
[0142] The source 600 comprises an encryption module 625,
configured to encrypt the downscaled video frame data into a format
suitable for transmission via the network switch to the video
output devices.
[0143] The source 600 comprises a control processor 630 configured
to control an ethernet controller 635 to transmit the encrypted
video frame data to the network switch.
[0144] FIG. 7 shows a schematic representation of a network switch
700 according to an embodiment. The switch 700 comprises various
modules. Any or all of these modules could, either alone or in any
combination, be implemented by dedicated circuitry, for example one
or more field-programmable gate arrays, or be implemented by
routines in one or more general processors.
[0145] The switch 700 comprises ports 705 for receiving multicast
streams from one or more video sources. For example, if the switch
700 is an ethernet switch, the ports 705 are ethernet ports.
[0146] The switch 700 further comprises ports 710 for receiving
subscription requests from a plurality of video display devices. It
should be noted that although the ports 705 and the ports 710 are
for ease of explanation shown as distinct in FIG. 7, ports 705 are
not specifically optimised for connection to video sources and
ports 710 are not specifically optimised for connection to display
devices. A video source could be connected to a port 710 and/or a
video display device could be connected to a port 705.
[0147] One or more of the ports 705 and the ports 710 may include
buffers for buffering data going to the port (an input buffer) or
going from the port (an output buffer). The buffers may be provided
by dedicated memory for each port or by logical allocations from a
common memory in the switch.
[0148] In one embodiment of the present invention, the video frame
data is split into multiple data packets, whereby the data packet
may be in the form of a wavelet. A wavelet results from a wavelet
transformation of the data. In wavelet compression, a type of image
data compression, each successive wavelet transformation provides a
successively lower resolution video data wavelet which can be sent
as a different multicast group. Thus, the video frame data
comprises of multiple wavelets of decreasing resolution. Combining
all wavelets that constitute the video frame data reconstructs the
original resolution, or fewer (lower) wavelet levels can be
combined to reconstruct a lower resolution version. Therefore, the
final received and reconstructed image can be varied in resolution
by subscribing to different wavelet multicast groups. An example of
such data compression is described in more detail in U.S. Pat. No.
5,682,441A.
[0149] The switch comprises a processor 715 and switching hardware
720. The processor 715 is configured to process subscription
requests received via ports 710. In response to receiving such a
request, the processor 715 controls the switching hardware 720 to
transmit each requested stream to the video display device from
which the request was received.
[0150] The switch 700 may be configured to have `Quality of
Service` capabilities, whereby transmission of certain data packets
are prioritized. For example, a data packet containing lower
resolution video frame data may be prioritized ahead of other data
packets waiting to be transmitted from the network switch. In
another example, the switch 700 may prioritize the transmission of
subscription requests received from a plurality of video display
devices for data packets containing lower resolution video frame
data.
[0151] Alternatively, or in addition to, the `Quality of Service`
capabilities may prevent the loss of a data packet containing lower
resolution video data should an output buffer overrun occur. In a
comparable example, the `Quality of Service` capabilities may
actively discard a data packet containing higher resolution video
data, particularly when the output buffer is overloading or is
about to overload. Therefore if data packet loss occurs, a lower
resolution set of pixels will still be available in the buffer, and
can be transmitted for view on the video display device.
[0152] In one embodiment of the present invention, the video frame
data packet may be in the form of a wavelet. The switch 700 will
give a higher prioritization to the lowest resolution wavelet, and
a lower prioritization given to higher resolution wavelets.
[0153] FIG. 8 shows a schematic representation of a video output
device 800 according to some embodiments.
[0154] The video output device 800 comprises a network interface
805 and a processor 810 coupled to the network interface. The
processor 810 is configured to subscribe, via the network interface
805, to a multicast stream of a plurality of multicast streams.
Each multicast stream is streamed from a video source and comprises
video frame data corresponding to a portion of a video frame. The
multicast stream to which the processor 810 subscribes comprises
video frame data that is for display on a display associated with
the video output device 800.
[0155] The processor 810 is further configured to receive, via the
network interface 805, the video frame data that is for
display.
[0156] FIG. 9 shows a schematic representation of a video output
device 900 according to an embodiment. The video output device 900
comprises various modules. Any or all of these modules could,
either alone or in any combination, be implemented by dedicated
circuitry, for example one or more field-programmable gate arrays,
or be implemented by routines in one or more general
processors.
[0157] The video output device 900 comprises an ethernet controller
905 controlled by a processor 910. The processor 910 is configured
to transmit a subscription request for a multicast stream, via the
ethernet controller 905 to a network switch, for example as
described above.
[0158] The ethernet controller 905 is further configured to receive
the video frame data of the requested stream from the network
switch.
[0159] The video output device 900 comprises a decryption module
915, which is configured to decrypt the received video frame data
into a format suitable for further processing and display.
[0160] The video output device 900 comprises a pixel cache 920 in
which decrypted video frame data is stored and from which video
frame data is output to a display of the video wall.
[0161] The video output device 900 comprises a transform module
925, configured to apply transforms to the stored video frame data
to produce video frame data suitable for display. For example, the
transforms may comprise warping, rotating, up-scaling,
down-scaling, de-interlacing and/or edge blending, depending on the
desired display configuration of the video frame. Additionally, the
transform module 925 may determine the particular multicast streams
for subscription, as described in more detail below, and transmit
an identification of those streams to the processor 910.
[0162] The video output device 900 comprises a sync pulse generator
930 configured to provide a synchronisation signal for
synchronising the outputting of video frame data with the refresh
rate of the display.
[0163] FIG. 10 shows a schematic representation of a video wall
1000. The video wall comprises an array of four displays 1005a-d in
a two-by-two layout. The video wall 1000 is configured to display
three windows 1010, 1015, 1020. Each window 1010, 1015, 1020
represents a video served by a video server, and by analogy the
window also represents a video frame of the video. Each window
1010, 1015, 1020 has a corresponding layer number, with window 1015
having a higher number than window 1010 and window 1020 having a
higher number than window 1015. In a region where two windows 1010,
1015, 1020 overlap, the window with the highest layer number is
displayed.
[0164] Window 1010 (shown in horizontally hatched shading) is
displayed at a size equal to the combined area of all four displays
1005a-d.
[0165] Window 1015 (shown in diamond shading) is displayed at a
size equal to a single display, and located in the geometric centre
of the video wall. As such, the top-left quarter of window 1015 is
displayed on the top-left display 1005a, the top-right quarter of
window 1015 is displayed on the top-right display 1005b and the
bottom-left quarter of window 1015 is displayed on bottom-left
display 1005c.
[0166] Window 1020 (shown in dotted shading) is displayed at a size
equal to a single display, filling the bottom-right display 1005d.
As such, no portion of windows 1010 and 1015 is displayed on the
bottom-right display 1020.
[0167] As set out above, each multicast stream corresponds to a
portion of a video. Each such portion comprises a block of pixels
in a video frame. A video output device determines the relevant
streams to which it should subscribe based on determining a
position of the video frame relative to the associated display. In
some embodiments, determining the location on the display comprises
determining a mapping of at least one pixel of the display to a
location in the video frame. For example, pixels in the
bottom-right quarter of the top-left display 1005a map to positions
on the top-left quarter of the video frame in window 1015. Based on
the mapping, it is thus determined that the video output device
should subscribe to streams corresponding to the top-left corner of
the video of window 1015.
[0168] Furthermore, pixels in the remaining three quarters of the
top-left display 1005a map to positions in the top-left three
quarters of the video frame in window 1010. Based on the mapping,
it is thus determined that the video output device should subscribe
to streams corresponding to the top-left corner of the video of
window 1010. The "positions" referred to above may be defined by
horizontal and vertical co-ordinates on the display, the canvas
and/or the video frame.
[0169] In some examples, the mapping is such that a block of pixels
received from a given stream is displayed on a block of pixels of
the display, the block of pixels of the display having the same
dimensions as the received block of pixels. In other examples, the
mapping is such that a received block of pixels from a given stream
is displayed on a block of pixels of the display, with pixel values
of the block of pixels of the display being determined by
interpolating between pixel values of the received block of pixels.
For example, the video frame data may be displayed with a rotation
angle with respect to the display such that a single pixel of the
display does not correspond to a single pixel of received video
frame data. In such a case, the pixel values of the display are
determined by interpolating between pixel values of the received
video frame data.
[0170] FIG. 11 shows a schematic representation of a mapping as
described above, according to an example. A canvas 1105 is defined.
The canvas 1105 is a virtual area in which windows 1110, 1115 are
positioned. Each window 1110, 1115 displays video frame data. An
area 1120 of the canvas 1105 corresponds to a display, such that
portions of windows 1110, 1115 that fall within the display area
1120 are displayed at pixels in corresponding portions of the
display.
[0171] The video output device maps each pixel of the display 1120
to a corresponding position on the canvas 1105. From this, the
video output device maps each such position on the canvas to a
position in each relevant window. For example, a pixel 1123 of the
display maps onto a corresponding position in window 1115, and a
pixel 1125 of the display maps onto corresponding positions in both
window 1110 and window 1115. A pixel 1130 does not map onto a
corresponding position in any window.
[0172] As outlined above, each multicast stream corresponds to a
portion of a video, for example a fixed-size 240.times.135 pixel
block. Examples of such blocks are represented in FIG. 11 as dotted
squares within each window 1110, 1115. The video output device
determines blocks that include the aforementioned positions in each
window. The video output device then subscribes to streams
corresponding to each determined block. For example, pixel 1123
maps to a position within block 1135 of window 1115; the video
output device would thus subscribe to the stream corresponding to
this block. Pixel 1130 does not correspond to a position within any
window; thus no stream would be subscribed to based on this
pixel.
[0173] Pixel 1125 maps to a position within window 1115 and also to
a position within window 1110. The video output device determines
which window 1110, 1115 should be displayed at this position. As
described above, each window 1110, 1115 has an associated layer
number and the window 1110, 1115 with the highest layer number is
displayed in any such overlapping regions. In the present example,
window 1110 has a higher layer number than window 1115. Pixel 1125
maps to block 1140 within window 1110, and thus the video output
device would subscribe to a stream corresponding to this block.
Window 1110 and thus block 1140 is rotated at an angle with respect
to the display 1120. As such, once video frame data relating to
this block has been received, the video output device may
interpolate between pixel values of the block to determine
corresponding display pixel values, based on the rotation
angle.
[0174] FIG. 12 shows an example of transmission, receipt and
display of streamed video data on displays 1205a-d of a video
wall.
[0175] Three videos 1210, 1215, 1220 are to be displayed on
displays 1205a-d. Each display 1205a-d has a resolution of
1920.times.1080 pixels. The videos 1210, 1215, 1220 are to be
displayed in the example arrangement shown in FIG. 10, such that
video 1210 is displayed at a size equal to the combined area of all
four displays 1205a-d, video 1215 is displayed at a size equal to a
single display and located in the geometric centre of the video
wall, and video 1220 is displayed at a size equal to a single
display, filling the bottom-right display 1205d. Video 1220 has a
higher layer number than video 1215, and video 1215 has a higher
layer number than 1210.
[0176] Each video 1210, 1215, 1220 is divided into a four-by-four
grid of pixel blocks. Each video 1210 has a resolution of
3840.times.2160 pixels. Each block therefore has a size of
960.times.540 pixels. Large blocks are presented in FIG. 12 for
ease of explanation, but blocks of other dimensions may
alternatively be used. For example, a 1920.times.1080 pixel video
frame may be divided into an 8.times.8 grid of blocks, each with a
size of 240.times.135 pixels.
[0177] A larger block size requires a smaller number of multicast
streams for each video, but requires a higher bandwidth for each
stream. A given video output device would need to subscribe to such
a high-bandwidth stream even if only a small number of pixels were
required from that stream. This would increase the required network
bandwidth. Conversely, a smaller block size decreases the required
bandwidth for each block, but increases the total number of streams
that must be handled by the network switch. The size of the blocks
may be optimised based on a balance between these factors.
[0178] The size of a given block may be predetermined. In other
examples, the size of a given block is varied, for example by a
stream controller connected to the network. This allows the block
size to be varied based on network conditions, for example
available bandwidth and/or network switch processor load.
[0179] In FIG. 12, the blocks of video 1210 are identified
sequentially as blocks a0-a15, the blocks of video 1215 are
identified sequentially as blocks b0-b15, and the blocks of video
1220 are identified sequentially as blocks c0-c15. A multicast
stream corresponding to each such block is transmitted to a network
switch 1225.
[0180] A video output device associated with each display 1205a-d
determines which blocks of each video are to be displayed on its
corresponding display 1205a-d. The video output device then
subscribes to the streams corresponding to those blocks.
[0181] As such, the video output device corresponding to display
1205a subscribes to streams corresponding to blocks a0, a1 and a4
of video 1210, and blocks b0, b1, b4 and b5 of video 1215. It does
not subscribe to the stream corresponding to block a5 of video
1210, because video 1215 overlaps video 1210 in that region and has
a higher layer number. As such, block a5 is not displayed.
[0182] Similarly, the video output device corresponding to display
1205b subscribes to streams corresponding to blocks a2, a3 and a7
of video 1210, and blocks b2, b3, b6 and b7 of video 1215. It does
not subscribe to the stream corresponding to block a6 of video
1210, because video 1215 overlaps video 1210 in that region and has
a higher layer number. As such, block a6 is not displayed.
[0183] The video output device corresponding to display 1205c
subscribes to streams corresponding to blocks a8, a12 and a13 of
video 1210, and blocks b8, b9, b12 and b13 of video 1215. It does
not subscribe to the stream corresponding to block a9 of video
1210, because video 1215 overlaps video 1210 in that region and has
a higher layer number. As such, block a9 is not displayed.
[0184] Finally, the video output device corresponding to display
1205d subscribes to streams corresponding to all blocks c0-c15 of
video 1220. It does not subscribe to any of the streams of videos
1210 and 1215, because video 1220 is arranged to fill display 1205d
and has the highest layer number. As such, no block of videos 1210
and 1215 is displayed on display 1205d.
[0185] As noted above, the displays 1205a-d each have a resolution
of 1920.times.1080 and the videos 1210, 1215, 1220 each have a
resolution of 3840.times.2160. As such, video 1210 is displayed at
full resolution and videos 1215 and 1220 are displayed at half
resolution. In some embodiments, videos 1215 and 1220 are received
at full resolution and downscaled to half resolution by the video
output device. In other embodiments, each video source stores
multiple downscaled versions of each block, with each version being
streamed as a separate multicast stream. For example, streams may
be provided with downscaling factors of 100%, 75%, 50% and 25%, or
100%, 50% and 25%, or 100%, 50%, 25% and 12.5%. This increases the
required bandwidth from the video source to the network switch
1225. This also increases the number of multicast groups, which
correspondingly increases the processing load on the network switch
1225. However, this reduces the required bandwidth from the network
switch 1225 to a video display device that subscribes to a
downscaled stream. Furthermore, this eliminates the need for the
video output device to downscale received video from the full
resolution, which reduces the degree of processing that must be
performed by the video output device.
[0186] The provision of multiple scaled streams from the video
sources may be provided using different approaches. Two such
approaches are described in the section of the description entitled
`Multiple Scaled Streams` with reference to FIGS. 27 to 35.
[0187] In addition to resolution, other examples of video quality
include bit depth and the degree of compression applied to each
block. Separate multicast streams may be provided from a given
video source based on any of these, or other examples of video
quality. For example, different chroma subsampling schemes such as
4:2:2 and 4:2:0 may be provided in different streams. In such
embodiments, a subscription request comprises an indication of a
desired video quality. For example, each stream of a particular
quality may have a separate multicast group address. The
subscription request would then identify the specific multicast
group address corresponding to the desired quality. The received
video frame data then comprises video frame data at that desired
video quality. In an embodiment, the available video qualities are
determined on the fly by a stream controller, and communicated from
the stream controller to each video output device and video source.
For example, the maximum available video quality may be altered to
take into account changing network conditions.
[0188] Although the blocks may be compressed using for example JPEG
compression, such a compression algorithm would potentially cause
spikes in the data rate over the network for example where a series
of detailed blocks is transmitted in succession, as efficient JPEG
compression relies on some areas of a compressed image having
little detail and thus being highly compressed. A discrete cosine
transform compression algorithm with a fixed compressed size per
compressed block of data is particularly advantageous in the
present system, because it ensures minimal variability in the data
rate over the network. Such an algorithm may for example be based
on a modification of the discrete cosine transform algorithm that
is used in JPEG compression, to remove higher frequencies more
harshly and to give more colour depth to less detailed areas. A
constant compressed size per block ensures that the network data
rate is more easily monitored to ensure that network capacity is
not exceeded.
[0189] FIG. 13 shows a table of example pixel rates of video data
of various bit depths at various resolutions with corresponding
data rates required to transmit uncompressed video data at each
resolution with various subsampling schemes. For example, assuming
for simplicity that a given source provides a single multicast
stream for each block of a given video frame, a data rate of 0.69
Gbit/s would be required for a video source to provide uncompressed
720p60 video frame data to network switch. Similarly, a data rate
of 0.69 Gbit/s would be required for the network switch to provide
such 720p60 video frame data to a given video output device. Such
data transmissions could thus be handled by a 1 Gb non-blocking
network switch. Because the video source provides a multicast
stream corresponding to each block of a video frame, the bandwidth
of video frame data streamed from a video source will not change,
regardless of how many displays the video frame is spread across.
Similarly, depending on the number of blocks into which each video
stream is broken, the bandwidth of video frame data streamed from
the network switch to an video output device may not be
significantly higher than the bandwidth required to stream a single
video frame to that video output device. The aforementioned 1 Gb
non-blocking network switch would thus be usable regardless of how
many video sources or video output devices operate on the network,
and regardless of how many different videos are simultaneously
displayed on the video wall.
[0190] Conversely, if a video source could provide video at 4k60
resolution and each video output device could output 1080p60 video
to its corresponding display, the system may be configured such
that the video source compresses video using a 4:1 compression
scheme. A network switch with a 10 Gb connection for the or each
video source, and a 1 Gb connection for each video output device,
could then be used. From the table in FIG. 7 it may be seen that
uncompressed 4k60 video, using a 4:4:4 sampling scheme with a bit
depth of 8 bits, would require a data rate of 12.4 Gbit/s. If
compressed at a 4:1 ratio this would require a quarter of that,
i.e. 3.1 Gbit/s into the network switch. Given four video output
devices, each video output device would then extract 1080p60 video
data at a quarter of 3.1 Gbit/s i.e. at around 0.78 Gbit/s.
[0191] In an embodiment, a video output device determines a change
of the location, on its corresponding display, at which a video
frame is displayed. Such a change in location may mean that a
greater or lesser extent of that video frame is displayed. For
example, a video frame may be moved partially on to the display or
partially off the display. As another example, a video frame may be
moved such that a previously-overlapped portion of another video
frame is exposed. Based on the determined change in location, the
video output device determines a particular further multicast
stream to request. The video output device then transmits a further
subscription request to the network switch, the further
subscription request identifying the particular further multicast
stream. In embodiments, this process is performed on a
frame-by-frame basis such that streams are subscribed to in real
time based on such changes in locations. The window locations may
change faster than the time required for a video output device to
subscribe to a new stream. In some embodiments the video output
device does not render such a change in window location until video
frame data has been received from the new stream. For example,
there may be a delay of one frame in rendering the location update.
In order for such changes in window location to be promptly
rendered, it is preferable for the network switch to be capable of
responding promptly to a subscription request, for example within a
time corresponding to a single frame.
[0192] FIG. 14 shows an example procedure 1400 for a video output
device to subscribe to streams, based on the mapping approach
discussed above in relation to FIG. 11. The procedure takes into
account changes in locations at which a given video is to be
displayed.
[0193] At step 1405, the display coordinates are set to (0, 0)
which may for example represent the top-left pixel of the
display.
[0194] At step 1410, the display coordinates are mapped to the
window coordinates as outlined above in relation to FIG. 11. In
other words, this step determines which particular window is to be
displayed at that pixel of the display (if any) and, for that
particular window, what coordinates of the window are relevant for
that pixel of the display.
[0195] If no window is to be displayed at the (0, 0) pixel, the
flow proceeds to determining whether the end of the frame has been
reached at step 1430.
[0196] If a window is to be displayed, the procedure determines at
step 1415 whether the determined window coordinate corresponds to a
stream to which the video output device is subscribed.
[0197] If the video output device is already subscribed to that
stream, video frame data corresponding to the window coordinates is
retrieved, for example from a cache of the video output device
which has earlier received video data from the corresponding
stream, at step 1420. The corresponding display pixel value is then
determined and stored in a display buffer in preparation for
outputting to the display. Determining the display pixel value may
comprise interpolating between window pixel values as described
above in relation to FIG. 11.
[0198] If the video output device is not already subscribed to that
stream, the video output device subscribes to that stream at step
1425. For example, the video output device may maintain a record of
available streams and the video frame blocks to which each stream
relates. In some embodiments, updates to such a record are
communicated to the video output device by a stream controller. For
example, a stream controller may transmit such an update in
response to receiving an indication of a change in available
streams from a particular video source. Video frame data from that
stream is thus received and cached. The corresponding display pixel
value is then determined and stored in the display buffer.
[0199] The procedure then determines at step 1430 if all pixels of
the display have been accounted for i.e. whether the end of the
display frame has been reached.
[0200] If the end of the frame has not been reached, the procedure
moves to step 1435 in which display coordinates are updated to the
next coordinates of the display, and the flow returns to mapping
the new display coordinates to window coordinates at step 1410. In
this manner, the procedure can be repeated pixel-by-pixel across
the display, determining a mapping of each display pixel to
corresponding window coordinates, and subscribing to streams as
required. The repeat loop for cycling through the pixels of the
display may, for example, correspond to a scan of pixels across a
first line of the display, followed by a scan of pixels across a
second line of the display and so on. Hence, the procedure can
perform a raster-like scan across the pixels of the display.
[0201] If the end of the display frame has been reached, the
procedure moves to step 1440. At step 1440 it is determined whether
the video output device is subscribed to any subscriptions that
correspond to video frame regions that will not be displayed. For
example, window locations may have changed such that a
previously-displayed region is now overlapped by another window. As
another example, window locations may have changed such that a
previously-displayed region is now displayed on a different display
of the video wall. In one embodiment, at step 1420 a flag is set
corresponding to each subscription from which data is retrieved.
Similarly, at step 1425 a flag is set corresponding to each new
subscription. The flags thus identify necessary subscriptions. As
such, any remaining unflagged subscriptions correspond to video
frame regions that will not be displayed.
[0202] If such unnecessary subscriptions are determined, the video
output device unsubscribes from these subscriptions at step 1445.
In some embodiments, unsubscribing comprises transmitting an
unsubscribe request to the network switch. In response to receiving
such a request, the network switch stops transmitting that stream
to the video output device.
[0203] The procedure then moves to step 1450. At step 1450, the
buffered display pixel values for the complete frame are output to
the display. In doing so, the display buffer is emptied in
preparation for the next frame.
[0204] The positions of the window(s) on the canvas are then
updated at step 1455, preferably within a vertical blank of the
canvas frame. The positions of the windows may, for example, be
updated based on a time-based movement of the windows across the
display and/or a sudden appearance, disappearance or
reconfiguration of a window or windows on the video wall.
[0205] Finally, the flow proceeds at step 1460 to the next video
frame, whereby the display coordinates are reset to (0, 0) at 1405,
and the aforementioned steps of the procedure repeated for the next
frame and so on.
[0206] In an alternative procedure to FIG. 14, the display of
buffered display pixel values may occur before the end of the
frame, for example, on a line-by-line or pixel-by-pixel basis.
[0207] In a further alternative, the procedure of FIG. 14 may be
performed without retrieving or displaying data but just
determining which streams to subscribe to or unsubscribe from.
[0208] FIG. 15 shows an ordering of blocks of video frame data for
transmission, according to an embodiment. A video frame 1500 is
divided into an 8-by-8 grid of blocks. The blocks are numbered
sequentially from 0 to 63. Video frame data of each block is
transmitted in a separate multicast stream. Data of each multicast
stream is transmitted from the video source in series.
[0209] In some embodiments, transmissions from each video source to
the network switch are synchronised, for example with a display
refresh rate. For example, each video source could transmit blocks
in series from block 0 to block 63. Streams corresponding to
equivalent areas of different videos would thus arrive at the
network switch at the same time. For example, video frame data
corresponding to the top-left corner of each video, i.e. block 0,
would be transmitted from corresponding video sources
simultaneously.
[0210] With some window arrangements, a given video output device
may subscribe to streams corresponding to equivalent areas of
multiple videos. For example, an output box may display the top
portion of each of many videos. As such, in embodiments wherein
transmissions from each video source are synchronised, a large
number of streams would simultaneously arrive at the network switch
and require transmitting to the output video device. This would
require a high peak network bandwidth. In one example, such streams
are buffered by the network switch and transmitted spread out in
time. This would maintain the same average bandwidth but reduce the
required peak bandwidth. However, the network switch may have
insufficient memory for buffering such a quantity of video frame
data.
[0211] Two examples of embodiments will now be described which
address possible size limits of memory of the output buffer of the
network switch. In one embodiment, the blocks of the video frame
data are pseudorandomly re-ordered, such that consecutive blocks
are not transmitted sequentially. In another embodiment, the blocks
of the video frame data are further divided into multiple packets.
The packets are transmitted spread out in time over the video
frame. This avoids an overrun of the buffer of the network switch,
even in the case where several multicast blocks are subscribed
to.
[0212] In the first example according to some embodiments, the
blocks 0 to 63 of video frame 1500 are reordered for transmission
such that video frame data corresponding to adjacent portions of a
single video frame 1500 is transmitted nonconsecutively. For
example, video frame data corresponding to adjacent portions of a
single video frame 1500 may be transmitted in a pseudorandom order.
The order may be optimised to avoid overloading the network switch
as outlined above. FIG. 15 shows the blocks of video frame 1500
re-ordered in such a manner 1505. Each column of the re-ordering
1505 comprises a cyclic permutation of the corresponding column of
video frame 1500. As an illustrative example, the re-ordering 1505
is such that blocks 0-7 corresponding to the top edge of video
frame 1500 are not adjacent to each other. Similarly, blocks
corresponding to the left, right and bottom edges of video frame
1500 are not adjacent to each other.
[0213] Adjacent blocks of the video frame 1500 are thus transmitted
nonconsecutively. Each video source 1500 may re-order video frames
based on a different re-ordering, such that for example top-left
block 0 from one video source is not transmitted at the same time
as top-left block 0 from another video source. This reduces the
peak bandwidth required when a given output box subscribes to
equivalent blocks, for example the top-left corner, from multiple
video sources.
[0214] A consequence of such re-ordering is that delays are caused
in providing streams from a video source. For example, a video
source may retrieve a given frame from a video store, in
preparation for re-ordering and transmitting, in a line-by-line or
block-by-block fashion. The source would thus retrieve blocks 0-7
first, followed by blocks 8-15, and so on. However, in the
re-ordering 1505, block 0 is transmitted first and followed by
block 57. The video source would thus need to wait until it had
retrieved block 57, which is located on the last line of the
re-ordering 1505. This adds up to one frame to the system
latency.
[0215] FIG. 16 shows a window arrangement, according to an
embodiment. A first video source 1610 is divided into a 3-by-4 grid
of blocks 1615. The blocks are sequentially numbered in hexadecimal
from 10 to 1B. A second video source 1620 is divided into a 3-by-4
grid of blocks 1625, whereby the blocks are also hexadecimally
numbered from 20 to 2B. Video frame data of each block is
transmitted in a separate multicast stream.
[0216] In some embodiments, the first video source 1610 and second
video source 1620 are arranged on a display 1640, whereby the first
video source 1610 may be partially obscured by the second video
source 1620. The arrangement of the grid of blocks 1615 from the
first video source 1610 and the grid of blocks 1625 from the second
video source 1620 are displayed in the corresponding display
1645.
[0217] In such an embodiment, transmission from each video source
to the network switch may be synchronised, for example with a
display refresh rate. For example, the first video source 1610
could transmit blocks in series from blocks 10 to 14 simultaneously
to the second video source 1620 transmitting blocks in series from
blocks 20 to 24. Streams corresponding to equivalent areas of
different video sources would thus arrive at the network switch at
the same time. Simultaneous arrival of multiple blocks risks
overrunning the buffer at the output of the network switch.
[0218] FIG. 17 shows three timing diagrams for the transmission of
the blocks from the network switch for different video sources 1610
and 1620. The blocks originate from the first video source 1610 and
the second video source 1620 for the display arrangement 1645 of
FIG. 16 in which the first video source 1610 is partially obscured
by the second video source 1620. Each block (or multicast stream)
of a video frame is split into multiple data packets. For
simplicity, FIG. 17 shows 4 data packets per block but in a typical
system, the blocks may be divided into many more packets.
[0219] In the display arrangement 1645 shown in FIG. 16, the
transmission of data packets, originating from the first video
source 1610 and second video source 1620, would ordinarily follow
the timing diagram 1700. Two sequential-in-time frames, Frame n and
Frame n+1, are displayed. The number of packets transmitted to the
network switch fluctuates from 8 to 4, wherein the higher
transmission rate of 8 packets risks an output buffer overrun. In
the timing diagram 1700, all 4 of the data packets that constitute
block 10 are sent to the network switch at the same time. Note that
although unsubscribed blocks are not shown for simplicity in the
timing diagram,--they would still be sent from the source to the
switch. However, since they are unsubscribed, these blocks would
not be sent from the switch to the output and thus would not cause
a buffer overrun at the switch's output.
[0220] In the second example in one embodiment 1710, the data
packets of Frame n, originating from the first video source 1610
and the second video source 1620, can be spread across the
neighbouring frames. The data packets that constitute Frame n are
interleaved in time with the preceding frame, Frame n-1 (lighter
shaded area), and the proceeding frame, Frame n+1 (darker shaded
area). In the timing diagram 1710, the 4 data packets that
constitute block 10 are sent to the network switch at different
times, whereby each successive packet in the sequence is sent at a
later point in time (i.e. there is a delay between each packet). In
such an example, at any specific time, there may be transmission of
data packets from multiple frames to the network switch. The total
number of packets transmitted to the network switch remains
constant in time (i.e. the overall packet rate remains constant),
and a risk of an output buffer overrun is reduced.
[0221] In the second example in a separate embodiment 1720, the
data packets, originating from the first video source 1610 and the
second video source 1620, can be spread across a single frame,
Frame n. The data packets corresponding to each frame are spread in
time across that, and only that, frame. In the timing diagram 1720,
the 4 data packets that constitute block 10 are sent to the network
switch at different times. However in contrast to the example
above, the data packets corresponding to consecutive blocks may not
be sent in order. For example for Frame n, one data packet from
block 13 is transmitted before a data packet from block 11. In such
an example, at any specific time, there will only be transmission
of data packets from a single frame to the network switch. In such
an embodiment, the number of packets transmitted to the network
switch remains constant in time, but introduces a one frame latency
in the system. Transmission of all data packets from each
corresponding block from the video source is required to have
completed before transmission to the network switch.
[0222] The splitting of the blocks (or multicast steams) into
multiple packets and the distribution of those packets in time
across a frame is described above as taking place in the video
source(s). However, similar benefits can be achieved by the network
switch receiving a block and spreading the multiple packets out in
time at the output of the switch. For example, the switch can
receive the data for a multicast stream at an input buffer
according to the timing diagram 1700 but provide that data as a
sequence of data packets at an output buffer according to the
timing diagrams 1710 or 1720. A general processing unit in the
switch can be responsible for selectively retrieving the packets
from the input buffer at the appropriate time.
[0223] The transmission of data packets across a single frame
allows for a simultaneous transmission of a plurality of
subscribe/unsubscribe requests. In such an example, when the
transmission of all data packets has completed at the end of the
frame period, a plurality of subscribe and unsubscribe requests may
be transmitted from the video display devices. Simultaneous
transmission of requests provides a simpler solution when the
plurality of video display devices have different frame rates and
synchronisations.
Distributed Video Wall
[0224] FIGS. 18 to 26 show examples and representations relating to
the distributed aspect of the present invention.
[0225] When describing embodiments of the multicast aspect of the
present invention previously, the term `video output device` was
utilized. In the following descriptions of the distributed aspect
of the present invention, the term `video client` will instead be
used.
[0226] Referring to FIG. 18, there is shown an example video system
1800. The system 1800 comprises a video wall 1805 comprising an
array of four display devices 1805a-d arranged in a two-by-two
layout. Although the video system 1800 is shown as a video wall
system, it will be appreciated that embodiments of the present
disclosure are also applicable to other video systems including,
for example, systems where a plurality of displays do not
necessarily form a video wall. Example video systems include
systems used in courtrooms, government assemblies, boardrooms,
command & control centers, simulators, operating theatres,
television studios, live stage events, casinos and sports
stadiums.
[0227] Each display device may, for example, be a computer monitor,
a video projector or a television screen. The displays may be
general-purpose display devices or, alternatively, they may be
display devices designed specifically for use in a video wall
1805.
[0228] The system 1800 comprises at least one video server 1810.
The video server 1810 is configured to provide video frame data for
display on the video wall 1805. In some embodiments, a single video
server provides a single identifiable video for display on the
video wall 1805. In another embodiment, a single video server 1810
provides multiple identifiable videos for simultaneous display on
the video wall. In a further embodiment, the system 1800 comprises
a plurality of video servers 1810, each of which are configured to
provide respective video frame data for display on the video wall
1805. In the case where there are multiple video servers 1810, some
video servers may provide single videos while other video servers
provide multiple videos for simultaneous display on the video wall.
In some examples, multiple video servers 1810 provide the same
video frame data. Such an arrangement provides redundancy in the
system so that a loss of video from one video server 1810 can be
compensated for by the provision of the same video from another
video server.
[0229] In some embodiments, for example where provision of video
frames would require particularly high bandwidth or processing
power, a single source video can be divided, for example spatially,
into separate videos, each corresponding to a region of the single
source video. For example, video frames may be divided into a
number of evenly spaced vertical strips. The system then handles
these vertical strips as separately identifiable videos. The
separate videos can then be provided by separate video servers.
This approach is useful for relatively high resolution source
videos.
[0230] A video served by the video server 1810 may be identifiable
or addressable in the system by a video identifier. The video may
comprise a sequence of video frames, and individual video frames
may be identifiable or addressable in the system, for example, by a
video frame identifier or a timing identifier.
[0231] In the example shown in FIG. 18, the video server 1810
provides a single video for display across the entire video wall
1805 such that the top-left quarter of a given video frame is
displayed on display device 1805a, the top-right quarter is
displayed on video device 1805b, the bottom-left quarter is
displayed on display device 1805c and the bottom-right quarter is
displayed on video device 1805a. The video server 1810 is
communicatively coupled with a video source, from which it receives
the video frame data. The video source may, for example, comprise a
storage medium such as a hard disk or flash memory for storing a
video file, which can be retrieved when a video needs to be output
by the video source. The video could also originate from a live
source such as video from a video camera. The video source can be
local to the video server such as a video on a flash memory in the
same cabinet as the video server. Alternatively, the video source
can be remote from the video server such as a video camera coupled
via a wireless link to the video server. Preferably, the video
source is provided by one or more dedicated video source devices as
will be explained in more detail below.
[0232] The system further comprises a plurality of video clients
1815a-d. Each video client 1815a-d is associated with a
corresponding display device 1805a-d of the video wall. Each video
client 1815a-d is configured to transmit a video frame data request
to a video server 1810, wherein the request is for a portion of a
video frame, the portion being associated with the display of the
video wall.
[0233] In some embodiments, a given video client 1815a-d is
incorporated within the functionality of its corresponding display
device 1805a-d. For example, a display device 1805a-d may comprise
a dedicated video client 1815a-d hardware. Alternatively, a general
purpose processor of a display device 1805a-d may implement the
functionality of a video client 1815a-d.
[0234] The portion of the video frame may be a region within the
video frame, or an area within the video frame. If the video frame
is available in different resolutions, the portion of a video frame
may be a portion of a video frame at a particular resolution. If
the video frame at a particular resolution has particular number of
pixels then the portion of the video frame may be a portion, region
or area of the video frame covering a subset of the full set of
pixels. If the video frame at a particular resolution is
represented by a particular set of video frame data then the
portion of the video frame may be a portion, region or area of the
video frame incorporating a subset of video frame data.
[0235] In some examples, the requested portion may comprise a
portion that, when output to the display, fills the entire frame of
the display on the display device 105a-d. Each request can define
the requested portion based on a location of the requested portion
within the video frame. For example video client 1815a, associated
with display device 1805a, may transmit a request for the entire
top-left quarter of the given video frame.
[0236] In embodiments where a single video is divided into strips,
each strip will be treated as a separate video with separate video
frames, and may be served from multiple video servers. A video
client 1815a-d may, for example, transmit a request for a portion
of a video frame from one of the separate videos.
[0237] The video server 1810 is configured to receive a request for
a portion of a video frame from one of the video clients 1815a-d,
and to respond by transmitting to that video client 1815a-d
corresponding video frame data. The video frame data comprises a
portion of video frame data based on the requested portion of the
video frame. A given video client 1815a-d thus only receives video
frame data based on its respective requested portion or portions.
Hence, the system avoids the need to transmit the video frame data
for the entire video frame to each video client. As such, the
overall bandwidth requirements of the system can be reduced.
[0238] It has been observed that, in systems in which the full or
entire video frame is sent to each video client, the required
network bandwidth increases with the number of displays. Similarly,
the required processing power of a video server increases with the
number of displays. This can increase to the point that no further
displays can be added to the video wall. In the present system,
each pixel or group of pixels of a video frame provided by a video
server 1810 is generally transmitted only to the video client
1815a-d that will display that pixel on its corresponding display
1805a-d. As such, the overall bandwidth of video frame data
requested from the video server 1810 will not be significantly
higher than the bandwidth required for transmitting a single
version of the video frame data, regardless of how many displays
1805a-d the video frame is spread across. Additional displays, with
corresponding video clients, can thus be added to the system
without requiring additional or upgraded video servers 1810 or
corresponding video sources. This remains true if such an
additional display is physically remote from the rest of the
network hardware, but it may be necessary to connect the additional
display via upgraded connection hardware capable of transmission
over an extended distance.
[0239] Analogously, the number of video sources and/or the
resolution of such sources can be increased by way of an updated
media server 1810 or servers capable of handling increased
bandwidth, or by increasing the number of media servers 1810.
Provided that each video client 1815a-d is capable of outputting
video frame data at the maximum resolution of its corresponding
display 1805a-d, it may not be necessary to upgrade the video
clients 1815a-d. This remains true if an additional source and/or
media server is physically remote from the rest of the network
hardware, but it may be necessary to connect the additional source
and/or media server via upgraded connection hardware capable of
transmission over an extended distance.
[0240] After receiving the video frame data from the video server
1810, each video client 1815a-d then outputs display data, based on
the received video frame data, to its corresponding display
1805a-d. For example, the display data may comprise video data
compatible with the HDMI standard for transmission over an HDMI
interface to the display. In some embodiments, received portions of
video frame data each comprise the portion requested in the
corresponding request. For example, a given video client 1815a-d
may request and receive only the portion of video frame data that
is for display on its corresponding display 1805a-d. In other
examples, a video client 1815a-d may produce the display data by
identifying a part of the received video frame data as not being
for display, and producing the display data as not including the
identified part of the received video frame data.
[0241] In some embodiments, video frame data is requestable in
fixed-size blocks. The blocks may be a group of pixels, and the
group of pixels may be defined by a number of pixels in the
horizontal direction in a video frame, and a number of pixels in
the vertical direction in the video frame. In some embodiments, a
given video client 1815a-d transmits a request for blocks such that
the requested blocks together include the region of the video frame
that is for display on its corresponding display 105a-d. The
requested blocks may also cover further regions of the video frame,
adjoining the region that is for display. The video client 1815a-b
may thus identify such further regions as not for display, and
produce the display data so as to omit the data from the blocks
relating to the further regions.
[0242] In an embodiment, a video client 1815a-d is configured to
transmit at least one further video frame data request to the video
server 1810, wherein the or each further video frame data request
is for respective further portions of the video frame associated
with the display of the video wall. For example, if video frame
data is requestable in fixed-size blocks, a video client 1815a-d
may transmit multiple requests, each request relating to a separate
block. The video client 1815a-d then receives further video frame
data from the video server, the received further video frame data
each comprising a portion of video frame data based on the
respective requested further video frame data. The combined
requested portions can together comprise the entirety of video
frame data that is for display on the video device. In such
embodiments, a video client 1815a-d might typically be expected to
request a series of adjoining portions. Based on this, the video
server 1810 may, upon receiving a request or requests for one or
more portions, predict further portions which are likely to be
requested and prepare in advance for transmitting these further
portions, for example by caching them in a memory.
[0243] In some embodiments, when a video client 1815a-d does not
receive a given block, for example due to network errors, the video
client 1815a-d can interpolate the missing data from surrounding
pixel blocks. Alternatively, the video client 1815a-d may replace
the missing block with a corresponding block from the previous
frame, for example held in a buffer. A missing block may further be
remedied by re-requesting that block, provided that enough time is
available between the initial request and the time at which the
pixels must be displayed. However, provided that a large number of
blocks are not missing, such errors are typically not very
noticeable on a display.
[0244] FIG. 19 shows a video wall system 1900 according to an
embodiment. The system 1900 comprises a video wall 1905 comprising
display devices 1905a-d, one or more video servers 1910 and video
clients 1915a-d as described above in relation to FIG. 18. In this
example, the plurality of video clients 1915a-d and the at least
one video server 1910 comprise nodes of a network 1920. Each video
client 1915a-d is configured to transmit its requests for video
data via the network 1920. The or each video server 1910 is
configured to transmit video frame data via the network 1920.
[0245] Links between network nodes may for example comprise 1G, 10G
or 25G copper or fibre links. The links may also be ethernet
links.
[0246] The network 1920 may comprise further components. For
example, the network 1920 may comprise a single network switch,
such as non-blocking network switch, to link the video clients
1915a-d with the or each video server 1910. A network switch
includes a number of bidirection input/output ports. A non-blocking
network switch is one where each port can typically pass data to
every other port, and typically the internal bandwidth of the
switch can handle all the port bandwidths, at the same time, at
full capacity. A switch may be utilised to provide a master Dynamic
Host Configuration Protocol (DHCP) server and/or time server for
the network 1920. In some embodiments, multiple switches can be
used. In an example video wall 205 in which the total required
bandwidth, for all displayed videos, does not exceed 10 Gb, the one
or more video servers 1910 may be linked to a single 10 Gb switch.
This is then linked via a single 10 Gb link to a similar switch, to
which all video clients 1915a-d are connected. For large or complex
video walls 1905, multiple switches could be configured, with each
switch being connected to a subset of the displays 1905a-d. In some
embodiments in which multiple switches are used, each switch is
located physically near the video clients 1915a-d and/or video
servers 1910 to which it is connected. The switches may then be
connected to each other via a fast interconnection, such as a fibre
interconnection. In such examples, it is preferable for the
interconnection to be capable of supporting non-blocking usage over
all connected ports. The interconnection should preferably be
capable of handling the greater of the total video client 1915a-d
bandwidth and the total video server 1910 bandwidth.
[0247] As a further example of network hardware, in embodiments in
which the required bandwidth is beyond that of a single network
cable, two or more cables could be configured to send and/or
receive data from/to the video server 1910. As an example a given
video client 1915a-d, using two connections, could alternate
requests across both connections. The video server 1910 would
receive requests on either port and perform the same steps
regardless which port a pixel block request arrived on. This
configuration is particularly advantageous in embodiments where the
resolution of the source image frame data is higher than the
display resolution, as it allows a higher resolution video frame to
be displayed on the video wall 1905.
[0248] Alternatively or additionally, two or more connections could
be used to send video frame data to a given video client 1915a-d.
Such a configuration is particularly useful in embodiments such as
control rooms, wherein multiple lower resolution videos are
displayed on a single display, as this configuration allows a
higher resolution display to be used since more pixel block data
can be retrieved by a given video client 1915a-d from the video
servers 1910.
[0249] A further advantage of the use of two connections to video
clients 1915a-d and/or video servers 1910 is that such a system
could, in response for example to a network issue, use only one of
the two connections. Such a system thus provides redundancy in case
of network faults. As an example, when both links from a video
server 1910 are in use, video frame data could be transmitted at a
lower compression ratio and when a single link is in use, video
frame data could be transmitted at a higher compression ratio.
Switching between these modes of operation may be triggered by
automated sensing of a network issue, for example by further
network hardware such as a network controller.
[0250] FIG. 20 shows a schematic representation of a method 2000,
in a video client, for acquiring video data for display on a
display of a video wall according to a first aspect of the present
disclosure. The method 2000 comprises a transmitting step 2005 of
transmitting a video frame data request to a video server. The
request is for a portion of a video frame, and the portion is
associated with the particular display of the video wall.
[0251] The method then comprises a receiving step 2010 of receiving
video frame data from the video server. The received video frame
data comprises a portion of video frame data based on the requested
video frame data.
[0252] FIG. 21 shows a schematic representation of a similar method
2100 of video frame data transmission in a video wall system. The
system, as described above in connection with FIGS. 20 and 21,
comprises a plurality of video clients wherein each video client is
associated with a corresponding display of the video wall. The
method comprises data flow between a given video client 2105 of the
plurality of video clients, and a video server 2110.
[0253] The video client 2105 transmits to the video server 2110 a
video frame data request 2115. The request, as set out above, is
for a portion of a video frame, where the portion is associated
with the corresponding display of the video wall.
[0254] Responsive to receiving a said request from a given video
client 2105, the video server 2110 transmits, to the video client
2105, video frame data 2120. As set out above, the video frame data
comprises a portion of video frame data based on the requested
portion.
[0255] FIG. 22 shows a schematic representation of a request data
packet 2205 that is transmitted from a video client to a network
switch according to an embodiment.
[0256] The packet 2205 comprises a server address 2215, for example
an IP address, of the source of the video data, for example a video
server, to which the request relates.
[0257] The packet 2205 comprises a display device address 2220 of
the video client from which the request originates.
[0258] The packet 2205 comprises a checksum 2225 to enable the
network switch, or the video source, to detect errors in the packet
2205. For example, as described in relation to FIG. 5, the checksum
may comprise the 16-bit one's complement of the one's complement
sum of the entire packet 2205.
[0259] The packet 2205 comprises a required video source 2230
information, for example the video frame data requested by the
video client.
[0260] The packet 2205 then comprises a required video frame and
(X,Y) block location 2235 of the requested video frame data.
[0261] The packet 2205 may comprise a field 2240 specifying the
required quality of the video frame, for example a particular
resolution depth, or the wavelet level of the video frame.
[0262] The packet 2205 finally comprises a checksum 2245 to enable
the network switch, or the video source, to detect errors in the
packet 2205.
[0263] A network switch is configured to route the request data
packet 2205. The network switch receives each video frame data from
each video source and forwards each video frame data to the video
client that has requested the video frame data.
[0264] FIG. 23 shows an example of data flow in a video wall
system. The system comprises video sources 2305a,b, video servers
2310a,b and video clients 2315a-d, connected to each other by a
network switch 2320. Video frame data from video source 2305b is
for display on displays associated with video clients 2315b-d, and
video frame data from video source 2305a is for display on displays
associated with video clients 2315a, 2315c and 2315d.
[0265] Dotted lines within the switch 2320 indicate data flow from
video sources 2305a,b. Video source 2305a provides multicast video
frame data to video server 2310a, and video source 2305b provides
multicast video frame data to video server 2310b. Additionally,
video source 2305a is configured to provide video frame data
directly to video client 2315a without passing via a video server.
As such, video client 2315a is not required to transmit any request
for video frame data from video source 2305a.
[0266] Dashed lines within the switch 2320 indicate data flow from
video clients 2315b-d to video servers 2310a,b in the form of
requests for data. Specifically, video clients 2315b-d transmit
requests for video frame data to video server 2310b, and video
clients 2315c,d transmit requests for video frame data to video
server 2310a.
[0267] Solid lines within the switch indicate data flow from video
servers 2310a,b to video clients 2315b-d. Responsive to receiving
the aforementioned requests, video server 2310b transmits different
unicast video frame data, based on each request, to the
corresponding video client 2315b-d. Similarly, responsive to
receiving its requests, video server 2310a transmits different
unicast video frame data, based on each request, to the
corresponding video client 2315c,d.
[0268] FIG. 24 shows an example of data flow in a video wall
system, according to an embodiment. Steps 2405 are performed by a
video client, and steps 2410 are performed by a video server. The
client and server are connected by a network switch 2415. Further
video clients 2417 and video servers 2418 may also be connected via
the network switch 2415, as set out in FIG. 23 by way of
example.
[0269] At 2425, the video client produces a video frame data
request, the request being for a portion of a video frame, wherein
the portion is associated with its corresponding display as
described above. A network packet, for example an Internet Protocol
(IP) packet is then formed, at 2430, including the request. The
packet is transmitted via a network interface at 2435, over the
network 2415, to a network interface of the video server at 2440.
The request is then decoded at 2445, so as to identify the
requested video frame data. Based on the decoded request a
controller 2450 of the server retrieves the requested video frame
data from a video store at 2455. The video store may for example be
a memory or other storage that holds video data. The controller
2450 then forms at 2457 a network packet or packets including the
retrieved video frame data. In some examples, forming the packet at
step 2457 comprises compressing the retrieved video frame data. In
other examples, the video is stored as compressed video frame data,
such as a compressed version of each video frame. In such examples,
the controller at 2450 may retrieve the compressed video frame data
and form the packet or packets directly from this.
[0270] In some embodiments wherein image frame data is requestable
in fixed-size blocks, each block may be stored separately
compressed.
[0271] Although the blocks may be compressed using, for example,
JPEG compression, such a compression algorithm would potentially
cause spikes in the data rate over the network for example where a
series of detailed blocks is transmitted in succession, as
efficient JPEG compression relies on some areas of a compressed
image having little detail and thus being highly compressed. A
discrete cosine transform compression algorithm with a fixed
compressed size per compressed block of data is particularly
advantageous in the present system, because it ensures minimal
variability in the data rate over the network. Such an algorithm
may for example be based on a modification of the discrete cosine
transform algorithm that is used in JPEG compression, to remove
higher frequencies more harshly and to give more colour depth to
less detailed areas. A constant compressed size per block ensures
that the network data rate is more easily monitored to ensure that
network capacity is not exceeded.
[0272] The packet is transmitted via the network interface at 2440,
over the network 2415, to the network interface of the video client
at 2435. The client decodes at 2460 the received video frame data.
Where the video frame data is received as compressed video frame
data, decoding the data at 2460 will also comprise decompressing
the received data. Following decoding, the client caches 2465 the
decoded data for display.
[0273] In some embodiments, requests for video frame data comprise
an indication of desired video quality. The video frame data
transmitted from the server to the client then comprises video
frame data at the desired video quality. The desired video quality
may comprise a desired degree of video compression. Alternatively
or additionally, the desired video quality may comprise a desired
video frame resolution and/or a desired bit depth. Where provision
of video frame data at the desired resolution would exceed the
available network bandwidth, the degree of compression may be
increased.
[0274] In some embodiments where video data is to be displayed at a
resolution below the resolution of the source video, the video
client may request video frame data at that lower resolution, and
the video server can respond with video data at the corresponding
lower resolution as will be described below in more detail. This
can occur when a relatively high resolution video needs to be
displayed on a relatively small region of a display, or the display
itself is low resolution. The benefit of sending just the lower
resolution video data is a reduction in the required network
bandwidth. Another benefit is that it eliminates the need for the
video client to downscale received video from the full resolution,
which reduces the degree of processing that must be performed by
the video client. In some examples, the video quality is defined by
additional network components, such as a network controller, for
example to control the utilisation of network bandwidth.
[0275] In some examples, where downscaled video frame data is
required, the video server performs this downscaling as and when
video frame data is requested, when forming network packets at
2457. In an alternative example, video data is stored in the video
store 2455 at different downscaling ratios, for example 100%, 75%,
50% and 25%, or 100%, 50% and 25%, or 100%, 50%, 25% and 12.5%.
Video data at each downscaling ratio is then available from the
video store 2455 for access by the video server. This may be
referred to as pre-scaling since the data is available in the
downscaled form even before a request is made by the video client.
The controller 2450 thus retrieves the appropriately downscaled
video frame data based on the desired degree of downscaling, and
transmits that video frame data to a video client as required.
[0276] The provision of multiple scaled streams from the video
sources may be provided using different approaches. Two such
approaches are described with reference to FIGS. 25 to 34 in the
section of the description below entitled `Multiple Scaled
Streams`.
[0277] As an example, video data may be stored in the video store
2455 as luma-chroma (YCrCb) colour space data, with each component
transmitted separately to a given video server. Different chroma
subsampling/downscaling schemes may be applied to the YCrCb data,
such as 4:2:2 and 4:2:0. The video client may request video frame
data with a particular subsampling scheme.
[0278] For continuity, FIG. 13 is now described in relation to the
distributed aspect of the present invention according to one
embodiment. FIG. 13 shows a table of example pixel rates of video
data of various bit depths at various resolutions with
corresponding data rates required to transmit uncompressed video
data at each resolution with various subsampling schemes. For
example, a data rate of 0.69 Gbit/s may be required for a video
server to provide uncompressed 720p60 video frame data to a video
client. Such data transmission could thus be performed via a 1 Gb
non-blocking network switch. As noted above, the overall bandwidth
of video frame data requested from a video server will not be
significantly higher than the bandwidth required for transmitting a
single version of the video frame data, regardless of how many
displays the video frame is spread across. The aforementioned 1 Gb
non-blocking network switch would thus be usable regardless of how
many video servers or video clients operate on the network, and
regardless of how many different videos are simultaneously
displayed on the video wall.
[0279] Conversely, if a video server could provide video at 4k60
resolution and each video client could output 1080p60 video to its
corresponding display, the system may be configured such that the
video server compresses video using a 4:1 compression scheme. A
network switch with a 10 Gb connection for the or each video
server, and a 1 Gb connection for each video client, may be used.
From the table in FIG. 13 it may be seen that uncompressed 4k60
video, using a 4:4:4 sampling scheme with a bit depth of 8 bits,
would require a data rate of 12.4 Gbit/s. If compressed at a 4:1
ratio this would require a quarter of that, i.e. 3.1 Gbit/s into
the network switch. Given four video client, each video client
would then extract 1080p60 video data at a quarter of 3.1 Gbit/s
i.e at around 0.78 Gbit/s.
[0280] Previously described above according to one embodiment of
the multicast aspect of the present invention, FIG. 6 is now
described according to one embodiment of the distributed aspect of
the invention. FIG. 6 shows a schematic representation of a video
source 600 according to one embodiment, from which a video server
can receive video frame data. The source 600 comprises various
modules. Any or all of these modules could, either alone or in any
combination, be implemented by dedicated circuitry, for example one
or more field-programmable gate arrays, or be implemented by
routines in one or more general processors. The source 600 may be a
dedicated video source device. Alternatively, the source 600 may be
implemented within another device such as a general-purpose
computer.
[0281] The source 600 comprises an input 605 to receive video data,
for example from a storage medium such as a hard disk or
solid-state memory, or from a live video feed. The input may for
example comprise an HDMI or VGA interface.
[0282] Video frame data may be received, and subsequently processed
and transmitted to a video server, as line-by-line video frame
data. Alternatively, for example in embodiments wherein blocks of
video frame data are to be requestable by a video client, received
line-by-line video frame data may be converted by a module of the
video source to block-by-block video frame data. As an example of
such a conversion, four lines of video frame data may be cached
and, from this, 4.times.4 blocks of video frame data may be
determined. Similarly, eight lines of video frame data may be
cached and, from this, 8.times.8 blocks of video frame data may be
determined.
[0283] The source comprises a sampling rate and colour conversion
module 610, configured to convert the received video to a data rate
and/or colour space that is suitable for providing over a network
to a video server.
[0284] The source 600 comprises a scaling module 615 for
downscaling the video data. The scaling module 615 may for example
perform block-based downscaling, downscaling each block of video
frame data to provide multiple streams, each having a different
overall downscaling factor and/or chroma subsampling scheme. The
downscaled block data is stored in a block memory 620.
[0285] The source 600 comprises an encryption module 625,
configured to encrypt the downscaled video frame data into a format
suitable for transmission via the network to the video server.
[0286] The source 600 comprises a control processor 630 configured
to control an ethernet controller 635 to transmit the encrypted
video frame data to the video server.
[0287] FIG. 25 shows a schematic representation of a video server
2500 according to one embodiment, configured to retrieve video
frame data from a video source and transmit video data to one or
more video clients. The server 2500 comprises various modules. Any
or all of these modules could, either alone or in any combination,
be implemented by dedicated circuitry, for example one or more
field-programmable gate arrays, or be implemented by routines in
one or more general processors. For example, the server 2500 may be
a Network Attached Storage (NAS) PC server. Alternatively, the
server 2500 may share storage with an SSD-driven PC.
[0288] The server 2500 comprises an ethernet controller 2505
controlled by a control processor 2510 and configured to receive
video frame data from a video storage and to receive requests for
video frame data from one or more video clients.
[0289] The server 2500 comprises a memory write module 2515
configured to write the received video frame data to a video memory
2520, which may for example be DDR3 SDRAM memory.
[0290] As noted above, in embodiments wherein blocks of video frame
data are to be requestable by a video source, video frame data may
be received from the video source as block-by-block video frame
data. Alternatively, video frame data may be received from the
video source as line-by-line video frame data and converted by the
video server 2500 to block-by-block video frame data. In some
examples, block-by-block video frame data is stored such that a
given block is stored in successive memory locations of a bank of
the memory 2520, with adjacent blocks being stored in different
banks of the memory 2520. A given block can thus be rapidly read
out from a bank of the memory 2520. Adjacent blocks are likely to
be requested in succession, and this arrangement facilitates rapid
access to adjacent blocks as one bank of the memory 2520 can be
read while another is being opened or closed. An example of such
block-based storage is described in more detail in European patent
EP2446413B1, the contents of which is incorporated herein by
reference.
[0291] For example, the received video frame data may comprise
multicast streams of 8.times.8 pixel blocks in varying
resolutions.
[0292] The server 2500 comprises a memory read module configured to
receive a request for video frame data, and to read from the memory
2520 video frame data corresponding to the request. That video
frame data is then provided via the ethernet controller 2505 to the
video client from which the request was sent.
[0293] Previously described above according to one embodiment of
the multicast aspect of the present invention, FIG. 8 is now
described according to one embodiment of the distributed aspect of
the invention. FIG. 8 shows a schematic representation of a video
client 800 according to some embodiments. The video client 800 is
associated with a display of a video wall.
[0294] The video client 800 comprises a network interface 805 and a
processor 810 coupled to the network interface. The processor 810
is configured to transmit a video frame data request, via the
network interface 805, to a video server. The request is for a
portion of a video frame, the portion being associated with the
display.
[0295] The processor 810 is further configured to receive video
frame data, via the network interface 805, from the video server.
The received video frame data comprises a portion of video frame
data based on the requested video frame data.
[0296] Previously described above according to one embodiment of
the multicast aspect of the present invention, FIG. 9 is now
described according to one embodiment of the distributed aspect of
the invention. FIG. 9 shows a schematic representation of a video
client 900 according to an embodiment. The video client 900
comprises various modules. Any or all of these modules could,
either alone or in any combination, be implemented by dedicated
circuitry, for example one or more field-programmable gate arrays,
or be implemented by routines in one or more general
processors.
[0297] The video client 900 comprises an ethernet controller 905
controlled by a processor 910. The processor 910 is configured to
determine a portion of video frame data to request, and to transmit
a request for that video frame data, via the ethernet controller
905 to a video server, for example as described above.
[0298] The ethernet controller 905 is further configured to receive
the requested video frame data from the video server.
[0299] The client 900 comprises a decryption module 915, which is
configured to decrypt the received video frame data and decrypt it
into a format suitable for further processing and display.
[0300] The client 900 comprises a pixel cache 920 in which
decrypted video frame data is stored and from which video frame
data is output to a display of the video wall.
[0301] The client 900 comprises a transform module 925, configured
to apply transforms to the stored video frame data to produce video
frame data suitable for display. For example, the transforms may
comprise warping, rotating, up-scaling, down-scaling,
de-interlacing and/or edge blending, depending on the desired
display configuration of the video frame. Additionally, the
transform module 925 may determine particular blocks to request, as
described in more detail below, and transmit an identification of
those blocks to the processor 910.
[0302] The client comprises a sync pulse generator 930 configured
to provide a synchronisation signal for synchronising the
outputting of video frame data with the refresh rate of the
display.
[0303] FIG. 10 shows a schematic representation of a video wall
1000. Display 1005a of the video wall displays a region of video
frame 1010. In an embodiment, a mapping is determined of at least
one pixel of the display to a location in the video frame.
Requested portions of a video frame each comprise at least one
block of pixels, wherein each block of pixels corresponds to said
location in the video frame. For example, pixels in the
bottom-right quarter of the display 1005a map to positions on the
top-left quarter of the video frame 1010. Based on the mapping, it
is thus determined that the top-left quarter of the video frame
1010 should be requested.
[0304] In some examples, the mapping is such that a given requested
block of pixels is displayed on a block of pixels of the display,
the block of pixels of the display having the same dimensions as
the requested block of pixels. In other examples, the mapping is
such that a given requested block of pixels is displayed on a block
of pixels of the display, with pixel values of the block of pixels
of the display being determined by interpolating between pixel
values of the requested block of pixels. For example, the video
frame data may be displayed with a rotation angle with respect to
the display such that a single pixel of the display does not
correspond to a single pixel of received video frame data. In such
a case, the pixel values of the display are determined by
interpolating between pixel values of the received video frame
data.
[0305] Each request may define a requested portion based on a
position of a block of pixels within the video frame. In one
example, as mentioned above, video frame data is requestable in
fixed-size blocks of predefined spatial dimensions such as
8.times.8 pixel blocks. A sufficient number of these blocks are
requested such that the combined requested blocks include at least
the required top-left corner of the video frame 1010. The request
may define the requested portion based on, for example, horizontal
and vertical co-ordinates of each block and/or an index number or
other identifier associated with each block. In alternative
embodiments the request defines the requested portion based on a
specific pixel coordinate or coordinates; the video server then
retrieves video frame data corresponding to the requested pixels
and converts this to block-based video frame data.
[0306] Previously described above according to one embodiment of
the multicast aspect of the present invention, FIG. 11 is now
described according to one embodiment of the distributed aspect of
the invention. FIG. 11 shows a schematic representation of a
mapping as described above, according to an example. A canvas 1105
is defined. The canvas 1105 is a virtual area in which windows
1110, 1115 are positioned. Each window 1110, 1115 represents a
video served by a video server, and by analogy the window also
represents a video frame of the video. An area 1120 of the canvas
1105 corresponds to a display, such that portions of windows 1110,
1115 that fall within the display area 1120 are displayed at pixels
in corresponding regions of the display.
[0307] The video client maps each pixel of the display 1120 to a
corresponding position on the canvas 1105. From this, the video
client maps each of said positions on the canvas to a position in
each relevant window. For example, a pixel 1123 of the display maps
onto a corresponding position in window 1115, and a pixel 1125 of
the display maps onto corresponding positions in both window 1110
and window 1115. A pixel 1130 does not map onto a corresponding
position in any window. The "positions" referred to above may be
defined by horizontal and vertical co-ordinates on the display, the
canvas and/or the video frame.
[0308] As outlined above, video frame data is requestable in blocks
of pixels, for example fixed-size 8.times.8 pixel blocks, examples
of which are represented in FIG. 11 as dotted squares within each
window 1110, 1115. The video client may determine and request
blocks that include the aforementioned positions in each window.
For example, pixel 1123 maps to a position within block 1135 of
window 1115; the video client would thus request this block. Pixel
1130 does not correspond to a position within any window;
therefore, no request would be sent based on this pixel.
[0309] Pixel 1125 maps to a position within window 1115 and also to
a position within window 1110. The video client determines which
window 1110, 1115 should be displayed at this position. In one
example, each window 1110, 1115 has an associated layer number and
the window 1110, 1115 with the highest layer number being displayed
in such overlapping regions. In the present example, window 1110
has a higher layer number than window 1115. Pixel 1125 maps to
block 1140 within window 1110, and thus the video client requests
this block. Window 1110 and thus block 1140 is rotated at an angle
with respect to display. As such, once video frame data relating to
this block has been received, the video client may interpolate
between pixel values of the block to determine corresponding
display pixel values, based on the rotation angle.
[0310] FIG. 26 shows an example procedure 2600 for generating
requests for video data using the mapping approach discussed above,
in accordance with one embodiment.
[0311] At step 2605, the display coordinates are set to (0, 0)
which may for example represent the top-left pixel of the
display.
[0312] At step 2610, the display coordinates are mapped to window
coordinates as outlined above in relation to FIG. 11. In other
words, this step in the procedure determines which particular
window is to be displayed at that pixel of the display (if any),
and for that particular window, what coordinates of the window are
relevant for that pixel of the display.
[0313] If no window is to be displayed at the (0, 0) pixel, the
flow proceeds to determining whether the end of the frame has been
reached.
[0314] If a window is to be displayed, the procedure determines
whether video frame data corresponding to the determined window
coordinates have already been received and cached at the video
client. For example, a block comprising the window coordinates may
already have been requested and received by the video client. If
such video frame data is in the cache, the video frame data is
retrieved from the cache at step 2615.
[0315] If the video frame data is not in the cache, a video frame
data request for that video frame data is transmitted at step 2620.
The requested video frame data is then received and cached. The
pixel value at the display coordinates is then determined and
stored in a display buffer. The step of determining the pixel value
may for example comprise interpolating between window pixel values
as described above in relation to FIG. 11.
[0316] The procedure then determines if all pixels of the display
have been accounted for i.e. whether the end of the display frame
has been reached. If the end of the frame has not been reached, the
procedure moves to step 2625 in which display coordinates are
updated to the next coordinates of the display, and the flow
returns to mapping the new display coordinates to window
coordinates at step 2610. In this manner, the procedure can be
repeated pixel-by-pixel across the display, determining a mapping
of each display pixel to corresponding window coordinates, and
making requests for video frame data accordingly. The repeat loop
for cycling through the pixels of the display may, for example,
correspond to a scan of pixels across a first line of the display,
followed by a scan of pixels across a second line of the display
and so on. Hence, the procedure can perform a raster-like scan
across the pixels of the display.
[0317] If the end of the display frame has been reached, the
procedure moves to step 2630. In step 2630, the buffered display
pixel values for the complete frame are output to the display.
[0318] The positions of the window(s) on the canvas are then
updated at step 2635, preferably within a vertical blank of the
canvas frame. The positions of the windows may, for example, be
updated based on a time-based movement of the windows across the
display and/or a sudden appearance, disappearance or
reconfiguration of a window or windows on the video wall.
[0319] Finally, the flow proceeds at step 2640 to the next video
frame, whereby the display coordinates are reset to (0, 0) at 2605,
and the aforementioned steps of the procedure repeated for the next
frame and so on.
[0320] In an alternative procedure to FIG. 26, the display of
buffered display pixel values may occur before the end of the
frame, for example, on a line-by-line or pixel-by-pixel basis.
[0321] Previously described above according to some embodiments of
the multicast aspect of the present invention, FIG. 10 is now
described according to some embodiments of the distributed aspect
of the invention. Returning to FIG. 10, in some embodiments, a
display 1005b of the video wall 1000 displays portions of two video
frames of different videos 1010, 0215. As mentioned above, video
frame data for the second frame 1015 may be provided from a
different video server to the video frame data for the first frame
1010, or from the same video server. In some of these embodiments,
the video client corresponding to the display 1005b transmits a
second video frame data request to either the same video server as
that of the first video frame 1010, or to a second video frame
server, as appropriate. The second request is for a portion of the
second video frame 1015 associated with the display 1005b. That
video client then receives second video frame data from the video
server to which the second video frame data request was sent, the
received second video frame data comprising a portion of second
video frame data based on the requested second video frame data. In
other words, the process described herein for requesting and
receiving video frame data for a single frame may be analogously
repeated for any number of further portions of frames to be
displayed on a given display.
Multiple Scaled Streams
[0322] FIGS. 27 to 35 relate to the provision of multiple scaled
streams and efficiencies that are described in relation to the
first and second aspects of the present invention.
[0323] As explained above, videos or video steams can be thought of
as a sequence of individual still images which are often referred
to as video frames, and each image or video frame consists of a
number of pixels arranged in a grid of horizontal rows and vertical
columns.
[0324] FIG. 27 shows a simplified video frame 2705 consisting of 4
horizontal rows and 4 vertical columns. The video frame therefore
has a resolution of 4.times.4 with 16 pixels in total. For a colour
video frame, each pixel may have 3 colour component values that
each range from 0 to 255 i.e. an 8-bit value. The range of values
is often referred to as the colour depth, with 8-bit, 10-bit, and
12-bit colour depths being typical colour-depths used in current
video. The component values may represent the colours red (R),
green (G), and blue (B). The 4.times.4 grid of pixels can therefore
be divided into a 4.times.4 grid of red pixels, a 4.times.4 grid of
green pixels, and a 4.times.4 grid of blue pixels. Alternatively,
the component values may represent luminance (Y), blue-difference
chroma (Cb), and red-difference chroma (Cr). In this case, the
4.times.4 grid of pixels can be divided into a 4.times.4 grid of Y
pixels, a 4.times.4 grid of Cb pixels, and a 4.times.4 grid of Cr
pixels.
[0325] Because each pixel is represented by component values, the
more pixels in an image, the larger the number of values, and the
larger the number of bits needed to represent the video frame.
Accordingly, the number of bits needed to represent a video frame
can be reduced by lowering the resolution of the video frame. The
second video frame 2710 in FIG. 27 shows a 2.times.2 video frame
which is half the resolution of the 4.times.4 video frame 2705, and
requires only a quarter of the number of bits to represent it.
[0326] The process of lowering the resolution of a video frame and
the associated video stream is known as downscaling. For example,
the video frame 2705 in FIG. 27 is downscaled to a half-resolution
video frame 2710. The process of downscaling involves combining
component values of pixels in the original video frame to determine
the component values of each pixel in the downscaled video frame.
In the example shown in FIG. 27, the luminance component value for
the 4 pixels in the top left portion of video frame 2705 are shown.
The corresponding pixel in the downscaled image is shown in the top
left corner of video frame 2710, and the luminance component A is
calculated by summing the luminance components 65, 40, 30, 45 to
give a summed value 180 and then dividing by the number of pixels 4
to give the pixel value 45.
[0327] The half resolution video frame 2710 can be further
downscaled by a half, resulting in a quarter resolution image which
will be just a single pixel value.
[0328] Following this principle, a 4K high-definition video stream
with a resolution of 3840 (horizontal).times.2160 (vertical) can be
downscaled to a half resolution version with a resolution
1920.times.1080 (also known as 1080p HD video) which in turn can be
downscaled to a quarter resolution version with a resolution
960.times.540, and an eighth resolution version 480.times.270. The
lower resolution versions of the 4K video stream can be sent and
viewed on equipment that may not be capable of sending or viewing
4K video. Furthermore, the lower resolution video streams can be
used to conserve bandwidth in a video system since the bit-rate for
these lower resolution versions is significantly less than the
original 4K video stream.
[0329] An alternative way of reducing the bit-rate of a video
stream is by selectively reducing the resolution of particular
pixel components of the video frame. In particular, this process is
known as chroma subsampling when the video components consist of
luminance (Y), blue-difference chroma (Cb), and red-difference
chroma (Cr).
[0330] FIG. 28 shows a simple 2.times.2 video frame comprising a
2.times.2 Y pixel grid 2810, a 2.times.2 Cr pixel grid 2820, and a
2.times.2 Cb pixel grid 2830. The pixel values are labeled 1, 2, 3,
and 4 for each grid. The overall 2.times.2 pixel grid 2850
comprises a combination of the values from each of the component
grids so that the sets of pixel values consist of (Y1, Cr1, Cb1),
(Y2, Cr2, Cb2), (Y3, Cr3, Cb3), and (Y4, Cr4, Cb4). When all the
components share the same resolution like this, the video is known
as a YCrCb 4:4:4 video stream.
[0331] FIG. 29 shows a second simple 2.times.2 video frame
comprising a 2.times.2 Y pixel grid 2910, a 1.times.2 Cr pixel grid
2920, and a 1.times.2 Cb pixel grid 2930. The pixel values are
labeled 1, 2, 3, and 4 for the Y grid and just 1 and 2 for the Cr
and Cb grids. The overall 2.times.2 pixel grid 2950 comprises a
combination of the values from each of the component grids so that
the sets of pixel values consist of (Y1, Cr1, Cb1), (Y2, Cr1, Cb1),
(Y3, Cr2, Cb2), and (Y4, Cr2, Cb2). When the Cr and Cb components
have a half resolution in just the horizontal direction, as in this
example, the video is known as a YCrCb 4:2:2 video stream.
[0332] FIG. 30 shows a third simple 2.times.2 video frame
comprising a 2.times.2 Y pixel grid 3010, a single Cr pixel 3020,
and a single Cb pixel 3030. The pixel values are labeled 1, 2, 3,
and 4 for the Y grid and just 1 for the Cr and Cb pixels. The
overall 2.times.2 pixel grid 3050 comprises a combination of the
values from each of the component grids so that the sets of pixel
values consist of (Y1, Cr1, Cb1), (Y2, Cr1, Cb1), (Y3, Cr1, Cb1),
and (Y4, Cr1, Cb1). When the Cr and Cb components have a half
resolution in both the horizontal and vertical direction, as in
this example, the video is known as a YCrCb 4:2:0 video stream.
[0333] When sending video between components of a video system, it
is often useful to provide the video at various downscaled
resolutions and/or with different chroma subsampling versions. In
this way, the sending component can provide video at different
quality and bit-rates, and different destination components can
receive video at their desired quality without having to downscale
themselves.
[0334] According to a first approach, an original video stream can
be downscaled by a component of a video system into half, quarter,
and eighth resolution video streams. Each of the 4 video streams
can then be multicast separately by the sending component.
[0335] According to an alternative novel approach, the video frame
can be sent in multiple parts, including one part which is a true
downscaled version of the video frame, and other parts which when
processed together with the downscaled part can provide other,
gradually higher resolutions of the video frame. By processing all
the parts together, it is possible to reproduce the original video
frame.
[0336] In other words, the novel approach involves sending a video
image in multiple parts, whereby the most basic part is of very low
resolution, then other parts with gradually higher resolution (and
with gradually increasing bandwidth) add more detail until the last
part `fills-in` the last missing parts of the video image. The
parts are then sent as separate video streams. The gradually higher
resolution parts or streams may not repeat information that has
already been sent from the lower-resolution streams. Instead, they
may supplement the information from the lower resolution stream or
streams in order to provide information needed to build a higher
resolution version.
[0337] If the streams according to this novel approach are
multi-cast on a network, then receiving devices can subscribe to
only the streams that they want in order to receive a video of the
desired quality. The approach therefore enables receiving devices
to selectively choose the relevant streams that correspond to the
desired resolution of video for display including, in some cases,
the desired chroma sub-sampling format. By doing so, the system
makes efficient use of bandwidth to the receiving device since the
combined bit-rate of the lower resolution streams may not exceed
the bit-rate of the original video frame.
[0338] The bandwidth required for sending the partial video streams
according to the novel approach over a single connection is equal
to the sum of the bit-rates for each of the partial video streams.
Likewise, the bandwidth required for sending the various standalone
downscaled video streams according to the first approach over a
single connection is equal to the sum of the bit-rates for each of
the downscaled video streams.
[0339] Ideally, the bandwidth requirement for the partial video
streams according to the novel approach will be lower than the
bandwidth requirement for sending the various standalone downscaled
video streams according to the first approach. Also, the bandwidth
requirement for the partial video streams according to the novel
approach will ideally not significantly exceed the bandwidth
requirement for streaming the full resolution video stream as will
be explained in more detail below. However, if no downscaling is
provided from a sending component of the system, i.e. only the full
resolution video stream is multicast, then a relatively high
bandwidth usage on the network connection will result, and the
receiving devices may then be required to downscale the received
full resolution video stream.
[0340] If the video image is broken-up into different colour
components, specifically YCrCb, then the Y, Cr and Cb components
could be sent separately, with each component being sent as
multiple resolution partial video streams. Thus a 4:2:2 YCrCb image
could be received from a full 4:4:4 YCrCb image by only subscribing
to half horizontal resolution streams of the Cr and Cb
components--saving about 20% of the bandwidth normally required for
a 4:4:4 image. Similarly, a 4:2:0 YCrCb image could be received
from a full 4:4:4 YCrCb image by only subscribing to half
resolution (i.e. half horizontal and half vertical) streams of the
Cr and Cb components--saving about 40% of the bandwidth normally
required for a 4:4:4 image.
[0341] For example, a 4k resolution, 60 frame per second, 4:4:4
YCrCb 24-bit video, when placed onto a network, will require around
12.4 Gbits/s of bandwidth. However, a receiving device with an SDI
output might only support a 4:2:2 YCrCb format. Hence, the
receiving device might subscribe to the full Y resolution streams
but just the lower resolution Cr and Cb streams and thereby reduce
the overall bit-rate to around 9.8 Gbits/s (and may permit the
video to be sent on a single 10 Gb ethernet connection instead of
two 10 Gb ethernet connections).
[0342] In another example, the receiving device may include an HDMI
output that only supports 4k resolution, 60 frame per second, 4:2:0
YCrCb 24-bit video. In this case, instead of receiving the whole
4:4:4 resolution video and processing it down to the HDMI output,
the receiving device could subscribe to the full resolution Y
streams and just the half resolution Cr and Cb streams, thereby
reducing the bandwidth requirement from 12.4 Gbits/s to around 7.5
Gbits/s.
[0343] Where a single source is multicasting the aforementioned
streams of data, any outputs can effectively avoid the need to
down-scale the image by only subscribing to the necessary
streams--thus avoiding some image processing and excess bandwidth
in the process. This is especially true of a multi-viewer, where a
single video image on a display is replaced by multiple smaller
resolution video images arranged side-by-side on the display. For
example, a 2.times.2 arrangement of 4k60 images, each having been
down-scaled to half resolution in the horizontal and vertical
directions in order to fit into a single 4k60 output, would
normally require the bandwidth for all four original 4k60 video
frames to be received. By just subscribing to the lower half
resolution streams of all four videos, the resulting bandwidth and
processing is greatly reduced.
[0344] Another benefit is where a temporary network failure results
in the loss of data packets. For a normal image transmission
system, this might result in a section of the image being missing
or badly corrupted. When sent as partial streams, the bad packets
can be ignored and the image partly restored from the good
packets--giving only a reduction in image quality on the affected
area.
[0345] A further benefit is when a cross-fade is required on an
output--where two live video sources are faded between, so that at
the mid-point both are visible as a `dissolve` effect. Normally
this would require twice the bandwidth of a single video feed to be
sent and for the output to product the cross-fade. But by altering
the stream subscription during the fade (reducing the outgoing
image quality/resolution whilst the incoming image
quality/resolution is increased) the bandwidth requirement can be
kept to around the same level as that of a single video feed.
Generally a fade or dissolve is done quite quickly to prevent this
being noticed--and since the outgoing image's quality reduction
occurs as it becomes less visible this should prevent the quality
reduction from being a perceptable problem for a viewer.
[0346] The novel approach for encoding and sending partial video
streams will now be described with reference to the example shown
in FIGS. 31 to 34.
[0347] Referring first to FIG. 31, there is shown an 8.times.8
video frame 3110 with 8 rows and 8 columns containing in total 64
pixel values. This grid of pixels may represent a particular colour
component of the video such as a R, G, B, Y, Cr, and/or Cb
component.
[0348] Each pixel value may range from 0 to 255 which corresponds
to an 8-bit pixel depth. Therefore, 512 bits (8.times.64) would be
required to represent this colour component of the video frame. For
a video frame containing three such colour components, the total
number of bits for the video frame would be 1536 bits
(3.times.512).
[0349] The video frame 3110 can be encoded in parts represented by
the downscaled video frame 3120, and the 6 partial video frames
3210, 3220, 3310, 3320, 3410, and 3420.
[0350] The downscaled video frame 3120 (known as "H3V3") is an
accumulated total of all the pixel values from the original video
frame 3110, and so effectively gives a `pixel` value as if the
pixels were down-scaled to 1/8th horizontally and vertically
(although with greater colour depth). Since the original pixels are
8-bit values, then this accumulated value would need to be 14-bits
in size, since it might need to store a minimum value of 0, and a
maximum value of 64*255=16,320 (note: 2 14=16,384).
[0351] The first partial video frame 3210 (known as "V2") is shown
in FIG. 32, and comprises a single pixel value of 1,617 calculated
by the accumulated total of all the pixels in the upper 4 rows of
the 8.times.8 block of pixels 3110, and needs to be a 13-bit
value.
[0352] In all the partial video frames, the pixel value or values
stored and sent are just the values shown in grey highlighting in
the Figures. The other values are values that would make up the
video frame at a particular resolution (or subsampling format), and
can be derived from a combination of the downscaled video frame
3120 and other partial video frames.
[0353] Taking the video frame 3210, the value of the lower pixel
1108 can be calculated by simply subtracting the upper pixel value
from the pixel value 2725 from the downscaled video frame 2020.
i.e. 2725-1617=1108.
[0354] The next partial video frame 3220 (known as "H2") has two
values stored and sent 616 and 484 (highlighted in grey), each
being 12-bits in size, and each representing the total of values
for a 4.times.4 block of pixels of the original video frame. The 2
other 4.times.4 blocks of pixels can have their total worked out
using simple subtraction e.g. 616 was sent for the upper-left
4.times.4 block, and hence the upper-right block must therefore be
1,617-616=1,001. H2 effectively represents an image than has been
down-scaled to 1/4th in the horizontal and vertical directions.
[0355] The remaining partial video frames 3310 ("V1"), 3320 ("H1"),
3410 ("V0") and finally 3420 ("H0") are all sent in similar ways,
with half the values being sent and the other half derivable from
lower-resolution streams. The bit-depth required also reduces to
11-bits for V1, 10 bits for H1, 9 bits for V0 and finally 8-bits
for the H0 stream.
[0356] The pixel values sent in H2, V1, H1, V0, and H0 are all
arranged in a checkerboard pattern. Alternative patterns may be
used without deviating from the general concept. The use of a
checkerboard pattern is preferred as it helps improve interpolation
of data should lower-resolution streams become corrupted or are
missing.
Efficiency
[0357] Since the lower-resolution accumulated values in the streams
3120, 3210, 3220, 3310, 3320, 3410, and 3420 can be quite large
compared to the values of the original pixels in the video frame
3110, they require sending using more bits than the original 8 bit
single-pixel values.
[0358] The additional number of bits needed to enable the
transmission of lower resolution information can be referred to as
the overhead.
[0359] In the example of FIGS. 31 to 34, the original video frame
is represented by 512 bits (64.times.8).
[0360] The number of bits required for each of the 7 streams 3120,
3210, 3220, 3310, 3320, 3410, and 3420 are as follows: [0361] H3V3
520 requires 1.times.14 =14 bits. [0362] V2 610 requires 1.times.13
=13 bits. [0363] H2 620 requires 2.times.12 =24 bits. [0364] V1 710
requires 4.times.11 =44 bits. [0365] H1 720 requires 8.times.10=80
bits. [0366] V0 810 requires 16.times.9 =144 bits. [0367] H0 820
requires 32.times.8=256 bits.
[0368] The total bits required for all 7 streams is therefore 575
bits (14+13+24+44+80+144+256). This represents an overheard of 63
bits compared to the original video frame, which is around 12% in
addition to the original 512 bits and represents a bandwidth
efficiency of 89%.
[0369] For higher original colour bit-depths, the resulting
bandwidth efficiency improves, as detailed in the table in FIG. 35
which shows the bandwidth efficiencies for 8-bit, 10-bit, 12-bit,
and 16-bit original video frame colour depths.
[0370] These values demonstrate that the ability to encode and send
multiple resolution streams does not increase bandwidth
significantly over sending just the original video frame.
[0371] If the first approach is used for sending the original video
frame together with half, quarter, and eighth standalone downscaled
streams, the total number of bits required would be much greater.
For example, the 8.times.8 video frame 3110 in FIG. 31 would be
represented by 512 bits, while the half resolution version would be
128 bits (16.times.8), the quarter resolution 32 bits (4.times.8),
and the eighth resolution 8 bits (1.times.8). This gives a total
number of 680 bits (512+128+32+8) which corresponds to an overhead
of 168 bits. This overhead is significantly greater than the
overhead provided by the novel approach of 63 bits. At the same
time, the downscaled streams would not provide the half horizontal
resolution video frames that are beneficial for chroma subsampling
format transmissions.
EXAMPLES
[0372] The following are examples of how a video receiving device
can subscribe to the video streams encoded using the novel
approach.
[0373] For a subscribing receiver to receive the chroma subsampling
format 4:2:0 equivalent of an 8-bit video image at full resolution,
the Y component would need to receive all of the 7 Y component
H3V3, V2, H2, V1, H1, V0, H0 streams. However, the Cr and Cb
components do not require the Cr/Cb component H0 or V0 streams,
just the Cr/Cb component H3V3, V2, H2, V1, H1 streams. Hence of the
original 3.times.512=1536 bits of information, only 575+175+175=925
bits need to be received. This is 60.2% of the original
bandwidth.
[0374] To receive a chroma subsampling format 4:2:2 equivalent of
an 8-bit video image at full resolution, the Y component would
again need to receive all of the 7 Y component H3V3, V2, H2, V1,
H1, V0, H0 streams. However, the Cr and Cb components do not
require the Cr/Cb component H0 stream, just the Cr/Cb component
H3V3, V2, H2, V1, H1, V0 streams. Hence of the original
3.times.512=1536 is reduced to 575+319+319=1213 bits. This is 79.0%
of the original bandwidth.
[0375] For a video system to display, on a single monitor, a small
preview window of each video source would normally require a
special (separate) video stream to be sent from each source (since
receiving a full-resolution for all sources, and downscaling them,
would be very difficult). But with the suggested mechanism that
video `preview` window is already available as the H3V3 stream,
which is a single 14-bit value representing a 1/8th scaled block of
8.times.8 pixels, being 2.7% of the original image's bandwidth.
Hence, it would be possible to display around 40 times the number
of images to be sent instead of sending them in their original
resolution.
[0376] The lower-resolution streams to do not need to stop at H3V3
(the lowest resolution above). Further lower resolutions could be
offered when starting from a larger original video frame block such
as 16.times.16.
[0377] An alternative to the above method may involve sending all
the pixel values for the H2 video frame 3220 and omitting the video
frames H3V3 and V2. However, this would only offer very slightly
better efficiency (reducing the 575 bits down to 572, for 512 bits
of data).
[0378] The method described here is most suited to loss-less
transmission of video data, although it may be possible to add
further compression techniques to the individual streams.
[0379] Methods of the present disclosure may be implemented by way
of a non-transitory computer-readable storage medium comprising a
set of computer-readable instructions stored thereon which, when
executed by at least one processor, cause the at least one
processor to perform a method according to the present disclosure.
The computer readable instructions may be retrieved from a
machine-readable media, e.g. any media that can contain, store, or
maintain programs and data for use by or in connection with an
instruction execution system. In this case, machine-readable media
can comprise any one of many physical media such as, for example,
electronic, magnetic, optical, electromagnetic, or semiconductor
media. More specific examples of suitable machine-readable media
include, but are not limited to, a hard drive, a random access
memory (RAM), a read only memory (ROM), an erasable programmable
read-only memory, or a portable disc.
[0380] The above embodiments are to be understood as illustrative
examples of the two aspects of the invention. Further embodiments
of the invention are envisaged. For example, the display devices
may be video projectors used for projection mapping. In such
embodiments, it may be desirable to mask certain areas of a given
video frame such that no video data is displayed in those areas.
Such a mapping may be defined by any of the video source, video
output device (which may comprise a video client), and/or other
networked components such as a projection mapping controller. In
examples wherein the mapping is defined by the video output device
(which may comprise a video client), a given video output device
may subscribe to video frame data based on the mapping such that
unneeded portions are not received.
[0381] For example, the video sources may be file servers. In such
embodiments, the file servers may multicast the video frame data
packet from either a static video image or a motion video
image.
[0382] Further embodiments of the two aspects of the invention may
be implemented with dedicated circuitry, for example discrete
integrated circuits, one or more field-programmable gate arrays,
software running on a central or graphics processing units, or in
any combination of the examples described. In examples wherein such
embodiments comprise of a graphics processing unit, the
implementation may be spread over many multiple parallel processing
units. It is to be understood that any feature described in
relation to any one embodiment may be used alone, or in combination
with other features described, and may also be used in combination
with one or more features of any other of the embodiments, or any
combination of any other of the embodiments. Furthermore,
equivalents and modifications not described above may also be
employed without departing from the scope of the invention, which
is defined in the accompanying claims.
* * * * *