U.S. patent application number 14/813369 was filed with the patent office on 2017-02-02 for adaptive profile switching system and method for media streaming over ip networks.
The applicant listed for this patent is Adi ROZENBERG, Limor Touriel. Invention is credited to Adi ROZENBERG, Limor Touriel.
Application Number | 20170034589 14/813369 |
Document ID | / |
Family ID | 57883583 |
Filed Date | 2017-02-02 |
United States Patent
Application |
20170034589 |
Kind Code |
A1 |
ROZENBERG; Adi ; et
al. |
February 2, 2017 |
ADAPTIVE PROFILE SWITCHING SYSTEM AND METHOD FOR MEDIA STREAMING
OVER IP NETWORKS
Abstract
A system for adaptively streaming video content over an IP
network, comprising devices and non-transitory computer-readable
storage media having executable computer modules, comprising a
sender device interacting with the network, the device configured
to send a video stream to the network and to receive a recovery
packet stream from the network; at least one receiving device
configured to receive the at least one video stream from the
network and to transmit a respective recovery packet stream back to
the sender device through the network; a bandwidth probe configured
to be periodically sent with the video stream to the at least one
receiving device to determine respective instantaneous network
bandwidths; at least one profile configured to be chosen by the
sender device to generate the video stream, based upon respective
instantaneous network bandwidths, wherein the video stream is
adaptively changed, based upon instantaneous network bandwidths and
the respective recovery packet stream.
Inventors: |
ROZENBERG; Adi; (Shoham,
IL) ; Touriel; Limor; (Matam, IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
ROZENBERG; Adi
Touriel; Limor |
Shoham
Matam |
|
IL
IL |
|
|
Family ID: |
57883583 |
Appl. No.: |
14/813369 |
Filed: |
July 30, 2015 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 21/6583 20130101;
H04N 21/64322 20130101; H04N 21/23439 20130101; H04N 21/2402
20130101; H04N 21/8456 20130101; H04N 21/6373 20130101 |
International
Class: |
H04N 21/647 20060101
H04N021/647; H04N 21/6373 20060101 H04N021/6373; H04N 21/24
20060101 H04N021/24; H04N 21/643 20060101 H04N021/643 |
Claims
1.-2. (canceled)
3. A computer-implemented method using at least one profile for
adaptively streaming video content over a public IP network, the
method comprising the steps of: using at least one stream generator
to select at least one profile to send at least one video stream
from a sender device to the network, the sender device employing
User Datagram Protocol (UDP) over a connection chosen from the list
including: unicast and VPN; using a segmentation process to segment
the at least one video stream associated with the at least one
profile into a list of segments containing a GOP a group of
pictures (GOP), to detect a start of the respective GOP, and to
record a corresponding start time code; operating a profile
selection module, included in the at least one stream generator, to
fetch respective groups of pictures (GOP) and to stream respective
GOPs sequentially, based upon a command from an adaptive rate
process module, to provide respective continuous RTP streams to the
network; receiving the at least one video stream from the network
by at least one receiving Automatic Repeat-reQuest (ARQ) device,
the network having an instantaneous network bandwidth for the at
least one video stream, and the at least one ARQ receiving device
transmitting a request for at least one recovery packet back to the
sender device through the network upon at least one lost packet
occurrence; preparing to send a recovery packet stream by the
sender device to the at least one ARQ receiving device in response
to the request for at least one recovery packet; limiting the
recovery packet stream by the at least one stream generator,
serving to give priority to the at least one video stream over the
recovery packet stream, the at least one video stream having an
initial media stream bandwidth; periodically probing the
instantaneous network bandwidth, employing ARQ protection of
probing, to simulate an increase in the instantaneous network
bandwidth by sending an expendable dummy packet stream in parallel
to the least one video stream, the dummy packet stream sent from
the sender device to the at least one receiving device to determine
a resultant profile media stream bandwidth which yields a packet
loss lower than a defined packet loss threshold; calculating and
monitoring a protection potential, the protection potential
calculated from a number of packets requested to be corrected but
currently waiting, a number of packets requested to be corrected
and currently being addressed, and a maximum correction index
characteristic of bandwidth limits; comparing the protection
potential to a protection threshold to determine a resultant
profile media stream bandwidth; and adjusting the initial media
stream bandwidth, relative to the instantaneous network bandwidth
and the protection potential, by reselecting at least one profile
for the sender device to use for sending the at least one video
stream over a connection chosen from the list including: unicast
and VPN.
4.-5. (canceled)
6. The method of claim 3, whereby limiting the recovery packet
stream is performed by a decision process that calculates a budget
of recovery packets to be transmitted during a time period for the
at least one video stream, the process serving to limit a rate of
error recovery requests in the instantaneous network bandwidth.
7. The method of claim 6, whereby a backoff decision process is
performed upon receipt of a new error request, the backoff decision
process further including evaluation of an error request queue, the
queue having at least one error request, regarding a number of
error requests in the queue over time.
8. (canceled)
9. The method of claim 7, whereby the at least one stream generator
interacts with the sender device and the adaptive rate process
module, the adaptive process module serving to perform a rate probe
of the instantaneous network bandwidth and to determine the
resultant profile media stream bandwidth.
10. The method of claim 9, whereby network bandwidth probing is
controlled by a rate probe process calculating a number of dummy
packets in the dummy packet stream.
11. The method of claim 10, whereby the resultant profile media
stream bandwidth has a value higher than the initial media stream
bandwidth when there is no lost dummy packet occurrence.
12. The method of claim 10, whereby the resultant profile media
stream bandwidth has a value lower than the initial media stream
bandwidth when there is lost packet occurrence.
13. A system for adaptively streaming video content over a public
IP network, comprising devices and non-transitory computer-readable
storage media having executable computer modules, comprising: a
sender device interacting with the network, the sender device
configured to send a video stream to the network using at least one
profile selected by at least one stream generator, the sender
device configured to use User Datagram Protocol (UDP) over a
connection chosen from the list including: unicast and VPN; a
segmentation process module configured to segment the at least one
video stream associated with the at least one profile into a list
of segments containing a GOP a group of pictures (GOP), to detect a
start of the respective GOP, and to record a corresponding start
time code; a profile selection module, included in the at least one
stream generator configured to fetch respective groups of pictures
(GOP) and to stream respective GOPs sequentially, based upon a
command from an adaptive rate process module, to provide respective
continuous RTP streams to the network; at least one receiving
Automatic Repeat-reQuest (ARQ) device configured to receive the at
least one video stream from the network, the network having an
instantaneous network bandwidth for the at least one video stream,
and the at least one ARQ receiving device further configured to
transmit a request for at least one recovery packet back to the
sender device through the network upon at least one lost packet
occurrence; the at least one stream generator further configured to
limit the recovery packet stream and to give priority to the at
least one video stream over the recovery packet stream, the at
least one video stream having an initial media stream bandwidth; a
rate probe configured to periodically probe the instantaneous
network bandwidth, employing ARO protection of probing, to simulate
an increase in the instantaneous network bandwidth by sending an
expendable dummy packet stream in parallel to the least one video
stream, the dummy packet stream configured to be sent from the
sender device to the at least one receiving device to determine a
resultant profile media stream bandwidth which yields a packet loss
lower than a defined packet loss threshold; a protection potential
configured to be calculated and monitored, the protection potential
calculated from a number of packets requested to be corrected but
currently waiting, a number of packets requested to be corrected
and currently being addressed, and a maximum correction index
characteristic of bandwidth limits; the protection potential
further configured to be compared to a protection threshold to
determine a resultant profile media stream bandwidth; and the at
least one stream generator, further configured to adjust the
initial media stream bandwidth relative to the instantaneous
network bandwidth and the protection potential, by reselecting at
least one profile for the sender device to use for sending the at
least one video stream over a connection chosen from the list
including: unicast and VPN.
14.-15. (canceled)
16. The system of claim 13, wherein a decision process module is
configured to limit the recovery packet stream in the instantaneous
network bandwidth by a calculation of a budget of recovery packets
to be transmitted during a time period for the at least one video
stream.
17. The system of claim 16, wherein a backoff decision process
module is configured to be operative upon receipt of a new error
request, the backoff decision process module further configured to
evaluate an error request queue, the queue having at least one
error request, regarding a number of error requests in the queue
over time.
18. (canceled)
19. The system of claim 17, wherein the at least one stream
generator is further configured to interact with the sender device
and the adaptive rate process module, the adaptive process module
further configured to perform the rate probe of the instantaneous
network bandwidth and to determine the resultant profile media
stream bandwidth.
20. The system of claim 19, wherein the rate probe is further
configured to be controlled by a rate probe process module
configured to calculate a number of dummy packets in the dummy
packet stream.
21. The system of claim 20, wherein the resultant profile media
stream bandwidth has a value higher than the initial media stream
bandwidth when there is no lost dummy packet occurrence.
22. The system of claim 20, wherein the resultant profile media
stream bandwidth has a value lower than the initial media stream
bandwidth when there is lost packet occurrence.
Description
RELATED APPLICATION
[0001] The present application claims priority from U.S.
Provisional Patent Application No. 62/030,630, filed 30 Jul. 2014,
whose disclosure is incorporated herein by reference.
FIELD AND BACKGROUND OF THE INVENTION
[0002] Embodiments of the current invention are related to media
streaming and particularly to a system and method to optimize media
streaming over one or more IP networks.
[0003] In the specification and claims which follow, the expression
"media streaming" or "streaming" is intended to mean the transfer
of video information (and any associated audio information, if
applicable), as known in the art, typically from one or more of
servers to a plurality of devices (typically called "clients")
located at a distance from the respective servers. As such, terms
such as "video content", "content", and "media stream" (or
abbreviated "stream") are used interchangeably in the specification
and claims which follow hereinbelow to mean video content which is
streamed. Typically, a stream comprises a plurality of "packets",
as known in the art and described further hereinbelow.
[0004] Other terms used in the specification hereinbelow, which are
known in the art, include: [0005] "Moving Picture Experts Group
(MPEG)" is intended to mean a working group of experts, formed by
ISO and IEC to set standards for audio and video compression and
transmission; [0006] "MPEG transport stream (TS)" is intended to
mean a standard format for transmission and storage of audio,
video, and program and system information protocol (PSIP) data.
Transport Stream is specified in MPEG-2 Part 1, Systems (formally
known as ISO/IEC standard 13818-1 or ITU-T Rec. H.222.0); [0007]
"TS Packet" is intended to mean the basic unit of data in a
transport stream. "Program Clock Reference (PCR)" is intended to
mean a value transmitted in the adaptation field of an MPEG-2
transport stream packet. PCR, when properly used, is used to
generate a system_timing_clock in a decoder to present synchronized
content, such as audio tracks matching the associated video, at
least once each 100 ms; [0008] "Presentation timestamp (PTS)" is
intended to mean a timestamp metadata field in an MPEG transport
stream or MPEG program stream that is used to achieve
synchronization of programs separate elementary streams (i.e.,
video, audio, subtitles). Reference:
https://en.wikipedia.org/wiki/Presentation_timestamp#cite_note-teknotes-1
[0009] "Group of Pictures (GOP)" has an intended meaning of a group
of pictures, or GOP structure in video coding, (ref
https://en.wikipedia.org/wiki/Data_compression#Video) and specifies
the order in which intra- and inter-frames are arranged. GOP is a
group of successive pictures within a coded video stream. Each
coded video stream consists of successive GOPs. Visible frames are
generated from the pictures contained in GOP; [0010] "Packetized
Elementary Stream (PES)" is intended to mean a specification in the
MPEG-2 Part 1 (Systems) (ISO/IEC 13818-1) and ITU-T H.222.0 that
defines carrying elementary streams (usually the output of an audio
or video encoder) in packets within MPEG program stream and MPEG
TS. The elementary stream is packetized by encapsulating sequential
data bytes from the elementary stream inside PES packet headers.
"Real-time Transport Protocol (RTP)" is intended to mean a
standardized packet format for delivering audio and video over IP
networks. RTP is used extensively in communication and
entertainment systems that involve streaming media, such as
telephony, video teleconference applications, television services
and web-based push-to-talk features. RTP is used in conjunction
with the RTP Control Protocol (RTCP). While RTP carries media
streams, RTCP is used to monitor transmission statistics and
quality of service (QoS) and aids synchronization of multiple
streams. RTP is originated and received on even port numbers and
the associated RTCP communication uses the next higher odd port
number. RTP was developed by the Audio-Video Transport Working
Group of the Internet Engineering Task Force (IETF) and first
published in 1996 as RFC 1889, superseded by RFC 3550 in 2003;
[0011] "User Datagram Protocol (UDP)" is intended to mean one of
the core members of the Internet Protocol Suite, the set of network
protocols used for the Internet. With UDP, computer applications
can send messages, in this case referred to as datagrams, to other
hosts on an IP network without requiring prior communications to
set up special transmission channels or data paths. UDP uses a
simple transmission model without implicit handshaking dialogues
for providing reliability, ordering, or data integrity. Thus, UDP
provides an unreliable service and datagrams may arrive out of
order, appear duplicated, or go missing without notice. UDP assumes
that error checking and correction is either not necessary or
performed in the application, avoiding the overhead of such
processing at the network interface level. [0012] "Forward Error
Correction (FEC)"is intended to mean a technique to recover partial
or full, packet information based on calculation made on the
information. FEC may be effected by means of XOR between packets or
another mathematical computation; [0013] "Pro-MPEG" is intended to
mean Professional-MPEG Forum--an association of broadcasters,
program makers, equipment manufacturers, and component suppliers
with interests in realizing the interoperability of professional
television equipment, according to the implementation requirements
of broadcasters and other end-users; [0014] "SMPTE 2022" is
intended to mean an FEC standard for video transport, initially
developed by Pro-MPEG Forum and added to by the Video Services
Forum, and describes both a FEC scheme and a way to transport
constant bit rate video over IP networks.
[0015] Media streaming over switching IP networks such as fiber,
leased line, CDN, public IP, wireless data networks, VSAT, and
cellular networks is a challenging technical problem. A media
stream may be impacted by a number of network aberrations (ex:
packet loss, jitter, disorder, and capacity changes, inter alia)
that make it difficult to sustain a constant stream from sender to
receiver. In parallel to data connectivity growing worldwide,
clients want to be able to receive media content to their devices
(mobile phones, tablets, TV, PC and similar playing devices) with
the best quality and the shortest delay.
[0016] Reference is currently made to FIG. 1, which is a prior art
block diagram of a media server 15 (also referred to as a "media
sending device" or a "sender" hereinbelow and in the claims which
follow) connected with a plurality of receiving devices 20 (i.e.,
mobile devices, smart TVs, inter alia) over a plurality of networks
25 (i.e., mobile and wireless networks, inter alia). Each network
and/or device may experience different network impairments and
network capacities. For example, a cellular network may be more
prone to capacity problems while a wireless network is more prone
to packet loss.
[0017] There are three main approaches known in the art which
address the problem of media streaming over switching IP networks,
as described hereinbelow. [0018] 1. Well managed networks, have
UDP/RTP and redundant protection information in the form of forward
error correction (FEC), which is sent with the media stream and
consumes 30-50% extra bandwidth in one direction. This solution has
a low time delay; however it may not tolerate high packet loss nor
network capacity drop-off. [0019] 2. For small scale operation,
streaming with retransmission protection, also called Automatic
Repeat-reQuest (ARQ) may be used. However ARQ is not useful for
large-scale operations. ARQ has modest time delays, it may tolerate
high packet loss, but it cannot tolerate network capacity drop.
[0020] 3. For large distribution over multiple networks, HTTP
adaptive bit rate (HTTP ABR) streaming has become a de facto
standard for most over-the-top systems. ABR has a large time delay.
Packet loss and network capacity drops are managed by reducing bit
rate to a lower value--inferring large time delays.
[0021] Each of the three main approaches listed above is addressed
hereinbelow:
UDP/RTP
[0022] Media streaming with UDP/RTP is not suited for mobile or
mass distribution application as these larger-scale networks are
not considered "managed".
ARQ
[0023] Another solution, ARQ, is currently offered by several
vendors to address 100% recovery of lost packets. ARQ has been
found to offer superior performance at lower overhead compared with
existing packet loss recovery solutions.
[0024] Reference is currently made to FIG. 2, which a prior art
block diagram, similar to that of FIG. 1, showing an ARQ
configuration of media streaming based on a method called
protection streaming, having a sender 35, connected with a
plurality of ARQ receiving devices 40 (also referred to hereinbelow
individually as "ARQ receiver") over a plurality of networks 45
(indicated as Network 1, . . . Network N-1, Network N in the
figure), with a respective receiver being fed from a respective
network.
[0025] Protection streaming performed with the configuration shown
in FIG. 2 allows each receiver to be addressed differently, thus
allowing for precise packet recovery to be applied to each
receiver.
[0026] Prior art ARQ systems work with a sender
sending/transmitting UDP/RTP packets in a stream over an unmanaged
IP-based packet network to several receivers. Packet loss detected
by a receiver is reported to the sender using special RTCP
messages. Each message may contain one or more different requests.
The ARQ packet processing is effective when network capacity is
larger than that of the initial media stream bandwidth. As noted
previously, the ARQ process allows for packet recovery with
retransmission of lost packets. However if the network capacity
(i.e. maximum bandwidth available for the network) drops below that
of the media stream bandwidth, the ARQ method (i.e. protection
streaming) cannot effectively recover lost packets.
[0027] Reference is currently made to FIG. 3, which is a prior art
flow and block diagram showing an exemplary video stream 50 from a
sender 52 to an ARQ receiver 65 and a loss of several packets 55
(indicated as D2, D6, D8, and D9) and subsequent respective request
packets 60 (indicated as R2, R6, R8-9.) In general, a receiver
requests resending packets several times during a time window in
which a packet is in a receiver buffer (not show in figure). In the
figure, sender 52 processes the receiver's request packets (R2, R6,
R8-9) and sends respective recovery packets 62 (D3, D5, D10) back
to the receiver on the main content stream (indicated by the arrows
connecting the sender with the receiver).
[0028] A major shortcoming of such an ARQ system is that sometimes
the IP link (i.e. the bandwidth between the sender and the
receiver) may reach its capacity limit due to either a physical
connection (ex: ADSL/VDSL) or by a capacity limit provided by the
service provider (ex: a mobile network provider). As shown in FIG.
3, ARQ systems can send a burst of recovery packets in response to
a burst of packet loss requests. The burst of recovery packets may
block or interfere with the stream's packet flow, causing
additional lost packets.
[0029] Some ARQ systems limit the link by employing traffic shaping
as known in the art. Traffic shaping can act to impact both the
stream and the recovery packets by limiting bandwidth, effectively
not addressing situations where recovery packets may block the
media stream.
HTTP ABR
[0030] Prior art HTTP ABR (HTTP adaptive bit rate) systems use a
method which employs several encoding profiles of the same video
content, which is split into segments using dedicated logic. The
term "profile" (as in "encoding profile", for example) is intended
to mean in the specification and claims which follow hereinbelow
encoder settings for streaming, as known in the art. Included in a
profile are: encoding method, resolution, bitrate, and additional
information.
[0031] An original media stream is encoded in several profiles;
each media stream is then split to segments. A segment is a portion
of video pictures or frames and in most cases is in a GOP (group of
pictures) boundary. Data relevant to respective segment bitrate and
location are published to a client. The client then decides which
segment to download; each segment having a predefined bitrate. The
client may download several consecutive segments and use a specific
algorithm to determine the order of successive segments. (One
example of a specific algorithm could be that based on
time-to-download a segment.)
[0032] The use of several segments requires a larger buffer (having
a size of at least 2-3 segments), which yields a larger time delay
for media streaming. The bitrate for respective profiles is fixed
at the origin point, as known in the art, and the client must
decide which segment to take from respective profiles. The entire
media streaming process is time consuming (due to the need to
buffer the segments).
[0033] Most ABR solutions employ HTTP protocol as the signaling and
data transfer protocol. HTTP protocol is used to initiate the
connection between client and server and use the HTTP tunneling
capability to pass public internet and firewalls. HTTP adaptive
streaming is widely adapted to tackle network aberrations and
capacity changes. Several standards have been developed which
differ from one another by signaling method or by underlining
encoding, having a similar functionality. The underlying TCP
protocol is very susceptible to network impairments such as packet
loss and jitter.
[0034] Prior art solutions do not optimize available bitrate nor do
they effectively overcome network-related inconsistencies, such as
packet loss and bandwidth fluctuations.
[0035] ABR employs a scheme of several profiles to represent the
same media source. The clients downloads a section of the media and
if the operation is halted or stalled due to network aberrations or
due to a capacity drop, a smaller size profile of that same media
section is downloaded to try to overcome the problem. Profiles are
prepared in advance to `guestimate` which bitrates are more likely
to pass through the network to allow the client to switch between
one profile and another. The client must ensure sufficient buffer
time to allow switching between profiles, storing several profiles,
and to then play them out.
[0036] As noted hereinabove, high buffering requirements and time
delays are experienced between the original media source and the
output to the client. Most client solutions do not include an
attempt to recover packet loss or to overcome large delays, as
these issues are related to limitations of TCP protocol. Generally,
lowering bitrate profiles infers a lower likelihood of suffering
from packet loss or capacity problems.
[0037] Another limitation of such methods is that link capacity
cannot be fully utilized. This is because as a profile segment is
downloaded and the client has little information on the real link
capacity/potential, the only way to switch to a higher capacity
profile is by trial-and-error. Studies have shown that, on average,
effective ABR tends to use only 50-60% of available network
capacity, as higher utilization of network capacity yields packet
loss. Standards in use may be HTTP Live Streaming (HLS), Adobe HTTP
Dynamic Streaming (HDS), Microsoft Smooth Streaming and
MPEG-DASH--all as known in the art, ref
http://en.wikipedia.org/wiki/Adaptive_bitrate_streaming.
Workflow
Prior Art
[0038] An input video feed is encoded by a multi profile encoder,
which creates several profiles of the video stream. Each profile
may have a different encoding bit rate and video resolution. Most
techniques employ either: open a group of pictures (GOP; having no
definite number of frames), or; a closed GOP (fixed number of
frames per GOP)--all as known in the art.
[0039] Profiles are packaged and segmented into blocks. Blocks are
documented in a manifest/playlist list. Respective block
information is published with its properties and location (bitrate
and resolution)--as known in the art. Profiles are stored in HTTP
servers and may be downloaded by a remote client based on the
information in a manifest/playlist.
[0040] Every client is responsible to pull (download) segments to
its local memory/buffer and to assure a full segment download. For
this purpose, the client monitors each new segment download. As a
download is performed over HTTP, the download is susceptible to
packet loss and to network jitter, which may contribute to increase
segment download time. This is because the TCP protocol--which is
the heart of the HTTP--contributes a time delay until all the data
has been downloaded.
[0041] During a download, the client can sense download speed
and/or a network artifact adversely impacting the download. The
client can decide to stop the current download and/or to select a
different profile for the current or the next segment download.
This approach assures a continuous download of segments, with each
adaptive client using at least a triple-buffer approach to buffer
several consecutive segments.
[0042] For segment N, the buffer will be constructed as: [0043]
Segment (N-2)--waiting to be played; [0044] Segment (N-1)--last
downloaded; and [0045] partial Segment (N)--downloaded.
[0046] For each standard (HLS-HDS, SMOOTH, etc.), a client should
determine the amount of buffering and segments stored prior to play
out. A common practice in the art is to allocate sufficient
buffering to store at least 3 segments. However, most solutions
tend to have more buffering capacity than 3 segments to allow
switching the last downloaded profile with a lower resolution in
case of streaming problems.
[0047] As several full segments are downloaded, the client starts
to read the stream in an orderly fashion, as the segments are
GOP-aligned (based upon a start and stop of the GOP boundary). This
approach allows seamless stitching of the stream and smooth
decoding, as known in the art.
[0048] Most multi-profile solutions known in the art do not
maintain the same resolution across different profiles nor is it
mandatory to keep the same GOP structure. Most solutions tend to
use MPEG2 transport for profiles and have PCR/PTS/DTS information.
Most solutions have adopted the H.264 encoding standard. Some
multi-profile solutions have adopted newer encoding options such as
H.265, with the condition that encoding remains the same between
profiles. A profile may share a common PCR counter to sync DTS and
PTS.
[0049] Because an adaptive streaming viewing experience may be
different from one vendor and another, or from one location to
another, adaptive streaming does not provide the same viewing
experience characteristic of linear/conventional TV, which gives
all viewers the same delay and viewing experience.
[0050] There is therefore a need to have a media streaming system
that can operate over challenging network impairments, and which
can provide the highest media bandwidth and shortest time delay to
the receiver.
SUMMARY OF THE INVENTION
[0051] According to the teachings of the present invention there is
provided a system for adaptively streaming video content over an IP
network, comprising devices and non-transitory computer-readable
storage media having executable computer modules, comprising: a
sender device interacting with the network, the device configured
to send a video stream to the network and to receive a recovery
packet stream from the network; at least one receiving device
configured to receive the at least one video stream from the
network and to transmit a respective recovery packet stream back to
the sender device through the network; a bandwidth probe configured
to be periodically sent with the video stream to the at least one
receiving device to determine respective instantaneous network
bandwidths; at least one profile configured to be chosen by the
sender device to generate the video stream, based upon respective
instantaneous network bandwidths; wherein the video stream is
adaptively changed, based upon instantaneous network bandwidths and
the respective recovery packet stream.
[0052] According to the teachings of the present invention there is
further provided a computer-implemented method for adaptively
streaming video content over an IP network, comprising the steps of
sending at least one video stream from a sender device to the
network; receiving the at least one video stream from the network
by at least one receiving device and the at least one receiving
device transmitting a respective recovery packet stream back to the
sender device through the network; periodically sending a bandwidth
probe with the video stream to the at least one receiving device to
determine respective instantaneous network bandwidths; choosing at
least one profile by the sender device to generate the at least one
video stream based upon respective instantaneous network
bandwidths; whereby the video stream is adaptively changed, based
upon instantaneous network bandwidths and the respective recovery
packet stream.
BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The invention is herein described, by way of example only,
with reference to the accompanying drawings, wherein:
[0054] FIG. 1 is a prior art block diagram of a media server
connected with a plurality of receiving devices over a plurality of
networks;
[0055] FIG. 2 is prior art block diagram, similar to that of FIG.
1, showing an ARQ configuration of media streaming based on a
method called protection streaming, having a sender, connected with
a plurality of ARQ receiving devices, with a respective receiver
being fed from a respective network;
[0056] FIG. 3 is a prior art flow and block diagram showing an
exemplary stream from a sender an ARQ receiver and a loss of
several packets and subsequent respective request packets;
[0057] FIG. 4 is schematic process diagram showing the use of five
profiles and a local storage to switch between the profiles to form
a single video stream, in accordance with embodiments of the
current invention;
[0058] FIG. 5 is a schematic process block diagram similar to FIG.
4, illustrating the use of five profiles and using a local storage
switching between the profiles to form two exemplary streams, in
accordance with embodiments of the current invention;
[0059] FIG. 6 is a schematic process block diagram of a sender
device and its interaction with a common packet buffer pointer
pool, a bi-directional network interface, and a stream generator,
in accordance with an embodiment of the current invention;
[0060] FIG. 7 is a schematic process block diagram detailing
elements/functions of the stream generator FIG. 6, in accordance
with an embodiment of the current invention;
[0061] FIG. 8 is a schematic process block diagram detailing
elements/functions of the sender device of FIG. 6, in accordance
with an embodiment of the current invention;
[0062] FIG. 9 is a flowchart indicating a rate probe decision flow,
in accordance with an embodiment of the current invention;
[0063] FIG. 10 is a process flowchart showing a backoff decision
flow, in accordance with an embodiment of the current
invention;
[0064] FIG. 11 is a flowchart of an array mapping algorithm, in
accordance with an embodiment of the current invention;
[0065] FIG. 12 is a flowchart indicating a process of reading
request entries, in accordance with an embodiment of the current
invention;
[0066] FIG. 13 is a flowchart showing a controlled adaptive rate
decision process to increase bitrate, in accordance with an
embodiment of the current invention;
[0067] FIG. 14 is flowchart indicating a manifest reading the
general profile selection process in accordance with an embodiment
of the current invention;
[0068] FIG. 15 is stream rate versus time schematic diagram showing
a stream rate axis and a time axis, an initial media stream,
network bandwidth capacities, probes tests, and resultant profile
media stream adjustments (i.e. "stream rates"), in accordance with
an embodiment of the current invention;
[0069] FIG. 16 is a schematic time versus stream rate "snapshot",
showing how profile change impacts stream bandwidth, in accordance
with an embodiment of the current invention;
[0070] FIG. 17 is a schematic process block diagram of a receiver
device (also "ARQ client"), in accordance with an embodiment of the
current invention; and
[0071] FIG. 18 is a schematic block diagram of a media flow from a
server to a plurality of ARQ clients, showing the main process
blocks involved in the media process, in accordance with an
embodiment of the current invention.
DETAILED DESCRIPTION OF EMBODIMENTS
[0072] Embodiments of the current invention include a novel
solution to distribute media content to various receivers in
parallel over a plurality of networks and impairments. Embodiments
of the current invention address the need for high bitrate delivery
while providing low time delay and error correction handling
capability. Embodiments of the current invention address
limitations of the prior art solutions noted hereinabove.
[0073] An embodiment of the current invention uses adaptive profile
switching and protected ARQ over UDP protocol to adapt stream rate
to network condition for each client (i.e., recipient). The
solution is based on a combination of addressing both stream
impairments and link capacity.
[0074] The solution, as described in detail hereinbelow, is based
on the following techniques: [0075] ARQ UDP streaming to recover
any packet; [0076] ARQ recovery packet rate limitation; [0077]
calculation of the stream health based on ARQ request numbers;
[0078] profile switching capability; and [0079] network probing
capability.
[0080] The underlying protocol used in embodiments of the current
invention is UDP, which is considered unreliable but is faster
compared to TCP for streaming applications. To accommodate for UDP
unreliable delivery characteristics, embodiments of the current
invention employ Real Time Protocol (RTP) and packet recovery with
ARQ. This approach yields results as reliable as those using the
TCP protocol but with higher bitrate utilization.
[0081] Embodiments of the current invention further address another
aspect of network behavior, namely capacity/bandwidth changes. An
IP network (i.e., public internet, metro internet, home wireless
network, and cellular network) may change its capacity from time to
time due to network element failure, over subscription, and
congestion, inter alia. Capacity changes express themselves as
changes to the available capacity available to respective users at
a given time. As noted hereinabove, bandwidth drop/decrease causes
packet loss in cases where the media stream is higher than given
link/network capacity. One embodiment of the current invention
includes detection and reaction to bandwidth change; to recover
packet loss during a bandwidth drop and to reduce the stream
bitrate, so that the new bit rate will be lower than an initial
bandwidth.
[0082] Another embodiment of the current invention includes a
solution to detect when conditions are favorable to increase stream
bit rate. The solution incorporates sophisticated network bandwidth
probing along with ARQ to determine a new bit rate to be used. This
capability allows utilizing 80-90% of the available network
capacity, which is nearly a 50% increase over adaptive streaming
and which additionally yields increased picture quality (as
increased media bitrate infers higher video quality and
resolution).
[0083] Embodiments of the present invention include profile
switching on the sender side, thereby eliminating the need to
buffer several segments (as is done using ABR). The technique of
profile switching on the sender side serves to reduce receiver
buffering requirements and yields a smaller time delay, in
comparison to linear TV.
[0084] The decision processes related to profile switching are
salient aspects of embodiments of the current invention. For each
client, a specific streamer is created and maintained to be
responsible for pulling a "current" profile from a local storage
device and pushing (or "streaming") it to the client. The streamer
maintains a list that keeps track of packet sequence numbers and
the source profile sequence number for retransmission.
[0085] Embodiments noted hereinabove are discussed in detail in the
figures which follow.
[0086] Reference is currently made to FIG. 4, which is a schematic
diagram showing the use of five profiles (Profiles 1, 2, 3, 4, and
5) and a local storage 80 to switch between the profiles to form a
single video stream 84, in accordance with embodiments of the
current invention. Seven exemplary, respective GOP's (GOP1, 2, 3,
4, 5, 6, and 7) are created upon reception of the initial stream
(not shown in the figure) for each profile, as shown. Creation of
the respective GOP's includes creating a segmentation list, which
includes the GOP number, and respective RTP sequence start and stop
numbers for each GOP. (Segmentation lists are not shown in the
figure.) Note that whereas five exemplary profiles are shown in the
current figure, the number of profiles used in embodiments of the
current invention may be less, but is typically much more than
five.
[0087] Upon a request to change to a new profile, a stream
encapsulator 88 (also referred to as "streamer" hereinbelow) reads
current profile packets from respective profiles and switches to a
new profile, creating video stream 84, which is sent to the client
(not shown in the figure). Exemplary video stream 84 is shown
schematically including GOP's from the respective profiles.
[0088] Reference is currently made to FIG. 5, which is a schematic
diagram similar to FIG. 4, illustrating the use of five profiles
(Profiles 1, 2, 3, 4, and 5) and using a local storage 80 switching
between them to form two exemplary streams 150 and 152, in
accordance with embodiments of the current invention. Apart from
differences described below, local storage 80 and Profiles 1, 2, 3,
4, and 5 are identical in notation, configuration, and
functionality to that shown in FIG. 4, and elements indicated by
the same reference numerals and/or letters are generally identical
in configuration, operation, and functionality as described
hereinabove. In the figure, there are two streamers 90 and 92,
which function similarly to streamer 88 of FIG. 4. Streamers 90 and
92 serve to create streams 94 and 96, respectively.
[0089] Other embodiments of the current invention include a
limitation on the recovery packet stream (refer to recovery packets
62 and video stream 50 of FIG. 3). The streamers interact with
clients (clients are not shown in the figure, but are part of
"network" indicated in the figure) to recover any lost packet(s) by
means of retransmission of the lost packet back to the client. The
streamers work to limit a recovery packet stream by giving priority
to the video stream over the recovery packet stream. Recovery
packets are organized in a priority queue, according to their
presentation time.
[0090] The term "protection potential", as used in the
specification and claims which follow, is intended to mean a
statistic calculated from: the number of packets requested to be
corrected, but currently waiting; plus the number of packets
requested to be corrected and currently being addressed; plus the
maximum correction index (which is a function of a predefined time
delay and a rate limit--either hardware and or software limit of
bandwidth). One example of the relationship of protection potential
versus protection threshold is if the protection potential is
higher than a protection threshold, then a lower profile (lower bit
rate) that meets a new bandwidth is selected for the next segment
transmission.
[0091] Another embodiment of the current invention includes network
bandwidth probing; employing ARQ protection and protection
statistics to determine network capacity--another term intended to
have the same meaning as "network bandwidth". As described further
hereinbelow, a bandwidth probe and a pre-defined bandwidth step are
sent from time to time with the media stream to each client. If the
packet loss is lower than a defined packet loss threshold, then an
algorithm determines a network bandwidth which may allow more data
to be sent.
[0092] It is noted that whereas two exemplary streams are shown in
FIG. 5, the number of streams applicable in embodiments of the
current invention is typically much more than two. Likewise,
although five exemplary profiles are shown in the current figure,
the number of profiles used in embodiments of the current invention
may be less, but is typically much more than five.
[0093] Reference is currently made to FIG. 6, which is a schematic
process block diagram of a sender device 105 and its interaction
with a common packet buffer pointer pool 110, a network interface
112, and a stream generator 114, in accordance with an embodiment
of the current invention. Arrows in the figure connecting exemplary
Profiles 1, 2, 3, and 4 to a common packet buffer pointer pool 110
and to stream generator 114 represent respective video streams.
Segmentation is performed in common packet buffer pointer pool 110,
as described hereinbelow. Note that whereas four exemplary profiles
are shown in the current figure, the number of profiles used in
embodiments of the current invention may be less, but is typically
much more than four.
[0094] A bidirectional Network interface 112 is assigned for each
streamer and the network interface includes a means to support
unicast/multicast/VPN connection types: For VPN (Virtual Private
Network), any type of VPN may be used, as long as it is capable of
supporting UDP/RTP traffic, bi directional communication, stream
encryption, and retransmission of packets. The VPN may also support
unicast/multicast. Embodiments of the present invention are not
limited to one type of VPN technology, as it may be adapted to VPNs
like Generic routing encapsulation, IPSEC, OPENVPN, HTTP tunneling
any many other similar solutions known in the art.
[0095] On the transmitting side the bidirectional Network interface
112 serves to receive a IP media stream, RTCP
communication/control, and auxiliary packets internally forwarded
to it and to send them through a predefined protocol
(unicast/multicast/VPN) to the client.
[0096] On the receiving side the bidirectional Network interface
serves to accept traffic from a unicast/multicast/VPN source and to
convert standard IP packets. The packets are them forwarded to
their designated blocks (RTCP, Auxiliary; NTP, DNS and similar
services). In most SW/HW implementations a common memory space is
reserved to hold incoming and outgoing traffic.
[0097] For each input profile received by Common packet buffer
pointer pool 110, a special segmentation process block in the
common packet buffer pointer pool is assigned to inspect the stream
and to issue a segmentation list (not shown in the figure). One
skilled in the streaming field art will find this function to be
similar to a packaging function of an adaptive bitrate packager,
which maintains a manifest for each profile and GOP. The
segmentation process block checks the media stream headers to
detect the start of each new GOP and its time code. This GOP time
code and number information is then stored in a manifest list
associated to all the profiles. The outcome is a list with several
GOP references for each profile representing the location of each
GOP and GOP encoding alternatives versions. Each profile includes a
sequential list of entries for easy data extraction, with each
entry representing one media unit. Memory may be accessed by
multiple clients to retrieve media packets for streaming or for
recovery retransmission. Each packet is assigned a time/lifetime
value. When the time/lifetime value is exceeded, old entries are
released, allowing new media packets to be stored.
[0098] Reference is currently made to FIG. 7, which is a schematic
process block diagram detailing elements/functions of stream
generator 114 of FIG. 6, in accordance with an embodiment of the
current invention. Apart from differences described below, ABR
client 114 and Profiles 1, 2, 3, and 4, Network interface, and
Adaptive rate logic 124 are identical in notation, configuration,
and functionality to that shown in FIG. 6, and elements indicated
by the same reference numerals and/or letters are generally
identical in configuration, operation, and functionality as
described hereinabove.
[0099] ABR client 114 includes the functions of: a Profile
selection 115; a Convert to fixed rate 116; a New RTP sequence
number 118; and a new profile request 120. The discussion which
follows is directed to a process of selecting and streaming
different profiles to each client.
[0100] ABR client 114 serves to convert data read from the profile
into a continuous RTP stream. As data may come from various
streams, the ABR client process block stitches data from various
streams together and encapsulates them into new RTP packets. The
block is also responsible to document where data pieces came from
(which profile and which physical location in the memory) for the
purpose of fetching data in case a recovery request is made to a
specific RTP packet. Following a profile switch, the current RTP
packet is closed and sent, and a new RTP packet process starts. The
process described hereinabove assures data in each RTP packet
belongs to the same profile.
[0101] Profile Selection 115 has the role of fetching a new
GOP/segment and streaming it after completion of the previous GOP.
A decision as to which new GOP/Segment to read is based on a
command from Adaptive rate logic 124. A new RTP request is received
by New profile request 118 and the request is correlated to a
current manifest list. A new profile is then selected and the
information is transferred to fetch the next GOP in the new
profile. Each new command includes the profile number and GOP
location in the profile.
[0102] New RTP sequence number 118 is a process wherein new RTP
sequence numbers are created and maintained for each client. The
streamer converts media packets to RTP payload with a new sequence
number, a time stamp, and all other necessary RTP header flags and
fields. The streamer pushes the new stream to the "Output" stream
to the Network Interface.
[0103] Reference is currently made to FIG. 8, which is a schematic
process block diagram detailing elements/functions of sender device
105 of FIG. 6, in accordance with an embodiment of the current
invention. Apart from differences described below, sender device
105 and Network interface are identical in notation, configuration,
and functionality to that shown in FIG. 6, and elements indicated
by the same reference numerals and/or letters are generally
identical in configuration, operation, and functionality as
described hereinabove.
[0104] Sender device 105 includes the following process blocks: a
rate limit FIFO 126; a prioritize array queue 128; a resend packet
counter 130; an RTCP ARQ message receiver 132; a sliding window
counter 133; an adaptive rate logic 134; and a rate probe 136--all
of which are described further hereinbelow. Furthermore, sender
device 105 receives recovery packets, a manifest list, and RTCP and
outputs read recovery packets, profile selection, and video stream,
as indicated in the figure.
[0105] Rate limit FIFO 126 receives recovery packets and serves to
report the number of bits waiting to be transmitted for further
processing and its interaction with other processes within sender
device 105 is discussed further hereinbelow.
[0106] Prioritize array queue 128 serves to store incoming ARQ
packet requests in a prioritize data base (based on a request
sequence number). Requests are rearranged in a novel data base that
maintains a small array of requests representing a sequence of
requests or a single request. A request may be of a fixed size or
of a range of values. Each individual request is split into two
entries; request START (RS) and request END (RE) with the request
value being identical. A range request is split to request START
range to request END. When entries are inserted into the data base,
they are organized once within the data base to create sub ranges.
Two consecutive entries are read from the database, and based on
flags in the message the requested packet is pulled from the stream
buffer for retransmission. The two entries may signal a single
packet readout (in the case where the sequence value is the same)
or a range of packets. The packet signals rate limit FIFO 126 for
its readiness to enter the FIFO data base. The packet then waits to
for acknowledgment to enter or a `backoff` signal to wait
further.
[0107] Sliding window counter 133 receives a data flow from rate
limit FIFO 126 and serves to monitor how many bits pass from the
rate limit FIFO. The sliding window counter serves to calculate and
store the number of bits sent during a period of one second.
[0108] The sliding window counter uses so-called "ticks", Each tick
can, for example, represent 10 ms, with each entry having a budget
of bits that may be transmitted during that timeframe. A "window"
can be defined, for example, as 100 ticks, namely 1 second. For
each new tick, a predefined value is added and a value of actual
transmitted bits is subtracted. As part of the monitoring function
of how many bits pass from the rate limit FIFO, every time an old
tick entry is cleared memory for a new entry is made available. The
total value of the entries in the window in the description above
is calculated for the total number of bits that passed during the
last 100 ticks, i.e. last second.
[0109] It is noted that the value of a tick in embodiments of the
current invention may be less or more than 10 ms, although tick
values are typically significantly less than 1 second. Likewise,
the value of 100 ticks to measure a total number of bits that have
passed may be more or less; however a typical total time value of a
plurality of ticks can range approximately from 0.5 to 2
seconds.
[0110] Data from Sliding window counter 133 gives an indication of
the budget of how many recovery packets may be transmitted to the
client during a given time, for example: one second.
[0111] Rate limit FIFO 126 serves to impose a first priority on the
RTP media stream, with protection packets being constrained to a
predefined pre-allocated bitrate. One aspect of embodiments of the
current the invention is to overcome a fundamental constraint of an
ARQ system, which typically causes extra packet loss in case of
protection bursts and/or following requests. Rate limit FIFO 126
serves to not exceed allowed bitrates and acts to smooth recovery
packet flow while keeping a fixed total bitrate. Keeping a constant
and predefine bitrate is crucial for applications making use of
limited capacity links such as ADSL/VDSL or satellite data
links
[0112] The rate limit FIFO process takes use of data from sliding
window counter 133 to allow IP packets to be transmitted on time
every tick so that the transmitted number of bits does not exceed a
predefined bandwidth allocation.
[0113] Rate limit FIFO 126 allows packets to be inserted to a data
base if the data base is not full and to wait their turn for
transmission, if the data base is full. If the data base is full,
the rate limit FIFO process serves to issue a `backoff` signal to
halt transmitters from sending packets, until such time that the
rate limit FIFO allows a new entry to be inserted.
[0114] Resend packet counter 130 serves to gather statistics from
various process blocks to assess the protection potential (ref
discussion of FIG. 5) status of the stream. Values gathered from
Prioritize array queue 128, Rate limit FIFO 126, and the Sliding
window counter processes serve to generate a value that is used by
Adaptive rate logic 134 to decided which profile to select next, or
to what Rate probe 136 action to take.
[0115] Stated schematically:
Protection Potential=(Stream buffer size+Sliding window)-(number of
request in Priority queue+Number of Packets in Rate FIFO)
[0116] RTCP ARQ message receiver 132 serves to manage the RTCP
messaging system between sender and receiver units. The RTCP
protocol is a part of the RTP protocol which carries control
information between sender and receiver, and is associated with
each stream. RTCP messages carry control and other information
between the sender and the receiver in accordance with the RTCP
standard. The RTCP client is also responsible for sending packet
request information in a specific format that in includes a missing
packet sequence number range; start range-end range. Several such
requests may be sent with one RTCP packet back to the sender.
[0117] Upon receiving a new request message, RTCP ARQ message
receiver 132 passes the information to Priority array queue 128 for
further processing.
[0118] Adaptive rate logic 134 serves to monitor events and
statistics coming from various process blocks and to decide on the
next profile selection and the Rate probe action. Adaptive rate
logic 134 has three major functions:
[0119] 1. Monitor statistics information and the protection
potential value to decide on profile bitrate drop or profile
bitrate increase;
[0120] 2. Control Rate probe 136 process, while monitoring the
impact of rate probe on the stream; and
[0121] 3. Read the available manifest list to select next
profile.
[0122] The Monitoring function of Adaptive rate logic 134 has two
main tasks:
[0123] 1. Monitor the protection potential to decrease the
bitrate/profile; and
[0124] 2. Monitor if the protection potential is a higher than a
predefined protection potential threshold value and then to attempt
to increase the bitrate/profile after a successful probe test.
[0125] The first task is simply monitor the `Protection Potential`
value, compared to a threshold, if the threshold is passed, then
the Protection potential of the system is low and may not guaranty
proper protection in the near future. Then select a low bitrate
profile for next selection point. This monitoring is essential to
maintain protection capability for the media stream, a capacity
drop will translate to an increase of lost packets and increase of
requests, the Protection Potential will decrease in correlation to
the increase of request in process by the ARQ block. The system
must maintain enough buffering to allow the drop to a new lower bit
rate profile to overcome the capacity drop.
[0126] The second task is to attempt to probe the network to see if
a higher profile may be in use, this action will test the network
with a `dummy` stream sent in parallel to the media stream
simulating an increase of bandwidth (e.g. higher profile). If the
operation is successful and minimal impact is seen on the
protection potential, then a new profile may be selected to the
next transition point.
[0127] Rate probe 136 serves to send a predefined stream of media
packets to be added to the standard media stream to measure
available bandwidth the media stream would experience if the rate
had a higher bandwidth. Rate probe 136 accepts a command to start
transmitting a `step` of bit rate, the step defined as a percentage
of the media stream, corresponding to a difference between the
current stream profile and the next successive stream profile.
[0128] The stream in Rate probe 136 uses `dummy` packets to create
the basis to the stream used for bandwidth evaluation. The stream
is transmitted between the Sender and the receiver for a fixed
duration on time. Dummy packets are selected so that they don't
interfere with the original media stream. Upon completion of the
transmission, the rate probe process block waits for the next
command to perform a probe test.
[0129] Reference is currently made to FIG. 9, which is a flowchart
indicating a rate probe decision flow 145, in accordance with an
embodiment of the current invention. Rate probe decision flow 145
starts with Get rate to test 147 and continues with Calculate
number of packets to send N 149. Then Send next dummy packet 151 is
performed, followed by the test: Is number of packets sent <N?.
If "yes" control is iterated back to send next dummy packet 151 and
the number of packets is again compared with N. If "no", meaning
the number of packets is equal to or greater than N, rate probe
decision flow 145 is complete.
[0130] Reference is currently made to FIG. 10, which is a process
flowchart showing a backoff decision flow 160, in accordance with
an embodiment of the current invention. The backoff decision flow
process is performed upon receipt of a new error request (not shown
in the figure). Read rate limit FIFO & request queue occupancy
level 162 serves to read rate limit FIFO and the request queue
occupancy level and clears the fail flag. A first decision is Is
request queue occupied (i.e. full) and FIFO full? 164. If yes,
control is transferred to Wait time cycle 166. If no, control is
reverted back to Read rate limit FIFO & request queue occupancy
level 162. After Wait time cycle 166 the fail flag is incremented
in Increment fail flag 168, followed by Read rate limit FIFO and
priority queue occupancy 170. Another decision is Is request queue
occupied (i.e. full) and FIFO full? 172. If no, control is reverted
back to Read rate limit FIFO & request queue occupancy level
162. If yes, another check is performed in Fail flag greater than
number of attempts? 174. If no, control is reverted to Wait time
cycle 166. If yes--meaning there currently is no "room" for
additional recovery requests--then the priority queue is cleared in
Clear priority queue 176, then Issue backoff signal 178, and Wait
half for FIFO to reach the half full point 180, then control is
reverted to Read rate limit FIFO & request queue occupancy
level 162.
[0131] Reference is currently made to FIG. 11, which is a flowchart
of an array mapping algorithm 180, in accordance with an embodiment
of the current invention. Array mapping algorithm starts with Wait
for new request 182, followed by New request arrives 184. As
described hereinabove (ref discussion of FIG. 8) a request is split
into two entries; request START (RS) and request END (RE). In RS-RE
contained in stream buffer range? 186, the two entries are
compared. If no, then Drop request 190 and control is reverted to
Wait for new request 182. If yes, Insert new entry and Mark Flags
188 and proceed to Sort array 192. Then control is reverted to Wait
for new request 182.
[0132] In array mapping algorithm 180, if all of the entries in a
range have been sent, pop the two entries and continue to read the
next two. If there is a failure in the middle (i.e. failed to send
a request to the rate limit block) update the entry to the last
index and mark as `start` flag and wait for approval to send
again.
[0133] Reference is currently made to FIG. 12, which is a flowchart
indicating a process of reading request entries 200, in accordance
with an embodiment of the current invention. Process of reading
request entries 200 begins with Wait for array not empty 205. As
long as the array is not empty, control is transferred to step 210,
Request access to rate limit FIFO. The request is acknowledged and
a decision Access Granted ? 215 is made. If yes, Read the last two
entries 220. If no, control reverts to step 210, Request access to
rate limit FIFO. After Read the last two entries 220, control is
transferred to step 225 Try to send range between two entries to
FIFO. Set range values. The next request, step 230 Send next
request, is performed. Decision Send failed? 235 yields either no,
whereupon control is reverted to step 230 Send next request, or
yes, step 240 Replace last entry with current entry index and start
flag. Step 245 Write back last entries to priority queue follows.
Then control reverts to Wait for array not empty 205.
[0134] Reference is currently made to FIG. 13, which is a flowchart
showing a controlled adaptive rate decision process 250 to adjust
bitrate, in accordance with an embodiment of the current
invention.
[0135] Controlled adaptive rate decision process 250 includes a
process for evaluating the action of a bitrate probe (ref FIGS. 8
and 9). If the action creates a number of requests (each request
representing a lost packet) larger than a predefined maximum
threshold value, the current bit rate may be too high for current
network conditions and a lower bandwidth profile is advisable. If
the action creates low amount of requests, i.e. lower than a
predefined minimum threshold value, network conditions can most
probably tolerate a higher bandwidth profile.
[0136] The overall data/process flow involves: [0137] Read the rate
limit FIFO and request queue occupancy levels to calculate the
protection potential); [0138] If the occupancy of either the
request Queue or the Rate limit FIFO is full--start over, else;
[0139] Monitor the request queue over a predefined period of time.
If during this time period the number of requests is higher that a
predefined maximum request threshold; reduce current bitrate by
selecting a lower profile and start over; [0140] Else, if the
number of requests during the time period is below a minimum
request threshold; increase the bitrate by selecting a higher
profile; and [0141] Start over the process.
[0142] A full description of all the process steps of adaptive rate
decision process 250 follows.
[0143] Read rate limit FIFO & request queue occupancy level 255
serves to read rate limit FIFO and the request queue occupancy
level and clears the fail flag. A first decision is Is request
queue occupied (i.e. full) and FIFO full? 258. If yes, control is
transferred back to step 255 Read rate limit FIFO & request
queue occupancy level. If no, a rate probe is sent in step 260 Send
rate probe. Then step 262 Read request queue, set timer is
performed. The question Is no. request>request maximum
threshold? 264 is asked. If yes, the bitrate is reduced in step 266
Set bitrate to minimum video rate level and control is returned to
Read rate limit FIFO & request queue occupancy level 255. If
no, step 268 Time expired? checks total allocated time. If yes, the
question Is no. request<request minimum threshold? 270 is asked.
If no, control is reverted to step 262 Read request queue set
timer.
[0144] If Is no. request<request minimum threshold? 270 is yes,
then step 272 Increment bitrate and revert control to step 255 Read
rate limit FIFO & request queue occupancy level. If no, revert
control to step 255 Read rate limit FIFO & request queue
occupancy (without incrementing bitrate).
[0145] Reference is currently made to FIG. 14, which is a flowchart
indicating a manifest reading the general profile selection process
280 in accordance with an embodiment of the current invention.
Process 280 starts with step 282 Wait for new manifest list. Then
the decision step 284 Manifest entry updated 284 is performed. In
parallel, process 280 also starts in step 288 with Get starting
profile. In step 290 Read profile, the profile is read. If the
answer to step 284 is yes, then control is transferred to Read
profile 290. If not, control is reverted back to step 282 Wait for
new manifest list.
[0146] After the profile is read (step 290) the decision step 292
Is profile not maximum (bitrate) profile? is performed. If the
answer is yes, control is reverted back to Read profile 290. If the
answer is no, Send probe 294 is performed. Then the decision New
bitrate change 296 is performed. If the answer is yes, Select next
profile 298. If no, control is reverted back to step 288 with Get
starting profile.
[0147] Reference is currently made to FIG. 15, which is a stream
rate versus time schematic diagram 300 showing a stream rate axis
302 and a time axis 304, an initial media stream 310, network
bandwidth capacities 315 and 316, probes tests 325, 335, 345, and
355, and resultant profile media stream adjustments (i.e. "stream
rates") 328 and 348, in accordance with an embodiment of the
current invention. An Initial media stream 310 starts with a
profile stream rate adjustment 328. As profile stream rate 328 is
below network bandwidth capacity 315, the Adaptive rate logic
process 134 (identified in FIG. 8) initiates a rate probe 325,
which causes a packet loss 327 and a request 329 to send the packet
loss back to the sender. (Packet requests are responded with ARQ
retransmission packets). As rate probe 325 created lost events, the
Controlled adaptive rate process waits a probe tick 320 to perform
another probe test 335. There is another packet loss 337 and
another request 338 to send it back to the sender. After another
tick, network capacity 316 increases, and a probe test 345 is
successful with no packet loss 347. The successful probe test
causes Controlled adaptive rate process to select a new higher
profile 348. Finally, after another probe tick, another probe test
355 is performed and another packet loss 357 occurs, and so on.
[0148] Because exemplary Network capacities 315 and 316 are
representative of periodical probing, the expression "instantaneous
network bandwidth" is used hereinbelow in the specification and in
the claims to have the same meaning as "Network capacity" described
hereinabove.
[0149] Reference is currently made to FIG. 16, which a schematic
time versus stream rate "snapshot" 340, showing how profile change
impacts stream bandwidth, in accordance with an embodiment of the
current invention. Apart from differences described below, stream
rate axis 302 and time axis 304 are identical in notation and
configuration to that shown in FIG. 15, and elements indicated by
the same reference numerals and/or letters are generally identical
in configuration, operation, and functionality as described
hereinabove. Five exemplary profiles (Profile 1, 2, 3, 4, and 5)
having successively lower rates are shown with respective GOP's,
over time. The exemplary stream for client 1 is shown schematically
with the 5 GOP's that are associated with different
profiles/different stream rates. Stream rates 362, 363,364, 365,
and 366 are shown against stream rate axis 302 and time axis
304.
[0150] In the exemplary stream for client 1, profile switching from
GOP N to GOP N+1 decreases the bandwidth; jumping to GOP N+2
lowering the bandwidth further. As capacity of the network may have
picked up, the next step from GOP N+2 to N+3 to N+4 represents a
profile bandwidth increase.
[0151] Reference is currently made to FIG. 17, which is a schematic
process block diagram of a receiver device 361 (also called "ARQ
client"), in accordance with an embodiment of the current
invention. Receiver device 361 works with a media sender 382
(identified hereinbelow in FIG. 18) to receive media packets and
recover lost packets, if such event occurs. In the case of packet
disorder, the ARQ client uses an RTP sequence number to reorder
packets to their original order, so that any packet disorder or
late arrival due to retransmission is handled by the sequence
number index to the data base. ARQ client 361 monitors the primary
media stream and `dummy` packets associated with the `network
probe` function (ref FIGS. 8, 13, 14, 15, inter alia). The ARQ
client reports any loss event for a `dummy` packet only once, while
media packets are reported several times (according to a
configuration of how may request to send per each loss event). A
media stream enters the receiver device and Network interface 362,
as shown in the figure, and another media stream exits the receiver
device as shown, going to a video/audio decoder--not shown in the
current figure.
[0152] A Network interface 362 is associated with each ARQ
receiver. The network interface includes the means to support
unicast/multicast/VPN connection types. For the VPN capability, any
type of VPN may be used, as long as it is capable of supporting
UDP/RTP traffic, bi directional communication and stream
encryption. The VPN may also support unicast/multicast. Embodiments
of the current invention are not limited to one type of VPN
technology, as they may be adapted to VPNs such as Generic routing
encapsulation, IPSEC, OPENVPN, and HTTP tunneling any many other
similar solutions.
[0153] On the receiving side, Network interface 362 serves to
accept traffic from a unicast/multicast/VPN source and to convert
standard IP packets. The packet are then forwarded to their
designated processes (i.e. to stream classification, RTCP and
auxiliary; NTP, DNS and similar services).
[0154] On the transmitting side the Network interface receives an
IP media stream, RTCP communication /control, and auxiliary packets
internally forwarded to it and sent to it through a predefined
protocol (unicast/multicast/VPN) to the destination.
[0155] In most software/hardware implementations a common memory
space is reserved to hold incoming and outgoing traffic in the
Network interface.
[0156] A Stream classification process 364 blocks any unwanted
traffic and allows and redirects incoming packet for further
processing. The stream classification process may be achieved in
many ways: hash data base; allowed lis; inter alia. Stream
classification process 364 accepts a pointer representing a link to
the actual IP packet in memory. The pointer includes basic IP
header parameters and an RTP header. Each IP packet pointer is
inspected for its IPv4/IPv6 parameters and UDP source/destination
port and redirection information is applied.
[0157] An RTP sequence packet inspector 366 monitors the progress
of incoming RTP media packets pointers prior to their temporary
storage in the pointer buffer. Each pointer RTP sequence number is
extracted for inspection. A new pointer is inspected in the
following steps: [0158] 1. If the new sequence number is out of
range of the buffer head and tail range, then drop the pointer and
free its associated memory for that media packet; [0159] 2. If the
new sequence number is smaller than the previous sequence number
then go to step 5 below; [0160] 3. If the new sequence number is
higher by 1 or more from the previous sequence number, push the
packet into the buffer, update the previous sequence number to be
the new sequence number and set flag `missing packet detected`;
[0161] 4. If `missing packet detected` is set, send the sequence
number notification to the missing packet detector process block;
[0162] 5. Test for wrap around. If wrap around is found set a
wrap_around flag and go to step 3 above. Else drop the packet and
free its associated memory for that media packet; and [0163] 6.
Clear temp flags and go to wait for new pointer.
[0164] A buffer process block 368 holds a sequential array of
pointers based on 16 bits of the RTP sequence number, with each
pointer entered according to the RTP packet sequence number. The
buffer process maintains a head pointer signaling the last entry to
buffer and a tail pointer the next pointer to read from the buffer
(for POP operation). The difference between head and tail is the
amount of packets in the buffer.
[0165] A missing packet detector 370 monitors and generates ARQ
requests for missing packets. Each new `missing packet detected`
event is logged and assigned a test duration timer. The number of
test periods is predefined and should not exceed the buffer delay.
Each time a missing packet duration timer expires, a buffer
inspection logic is invoked to examine if the packet did not
arrive, as follows: [0166] 1. If the packet request is missing,
issue a new one and a new timer is set; [0167] 2. Else, the
operation stops.
[0168] For each new `missing packet detected` event, the process
also checks the sequence number neighbors to determine if they may
also be missing, and then may generate a range of `requests` for
further processing. Each request is then assigned with a timer.
[0169] The buffer inspection is invoked for these two events:
[0170] 1. New `missing packet detected` notification from the RTP
sequence number inspector process block; and [0171] 2. One of the
timers expires.
[0172] For each packet request or range or request a message is
sent to the RTCP ARQ message block for processing.
[0173] An RTCP ARQ message process 372 manages the RTCP messaging
system between sender and receiver units. RTCP protocol is a part
of the RTP protocol to carry control information between sender and
receiver, and it is associated with each stream. RTCP messages
carry control and other information between sender and the receiver
in accordance to the RTCP standard, as known in the art. The RTCP
client sends packet request information in a specific format that
includes the stream, the missing packet sequence number range, and
start range--end range. Several such requests may be sent with one
RTCP packet back to the sender. Upon receiving a new request
message from Missing packet detector 370, RTCP ARQ message process
372 generates an RTCP message which includes the request and any
other requests pending. The message is transmitted to sender RTCP
process block through network interface 362 to media server 382 for
processing.
[0174] A Rate calculation process 374 keeps track of the number of
packet pointers entering and departing the buffer, and calculates
the bitrate based on timing information in the RTP packet or media
internal timing information (such as PCR, if available in the media
stream) or the average arrival of packets. The Rate calculation is
then used by a Playout rate process 376 to read packets out to the
destination receiver/player. The Playout rate process reads packets
from the buffer and frees associated memory back to the SW/HW
memory pool. Playout rate process 376 assures a fixed delay and
constant packet readout to feed the receiver. Any packet that was
has a null packet removal operation conducted on it is
reconstructed back to its original state. The null packet is
reinserted back to its original location within the IP packet.
[0175] Reference is currently made to FIG. 18, which is a schematic
block diagram 380 of a media flow from a server 382 to a plurality
of ARQ clients (indicated as 1, 2 . . . N-1, and N), showing the
main process blocks involved in the media process, in accordance
with an embodiment of the current invention. Server 382 includes:
an ABR client 384 (which is equivalent to common buffer pool 110 of
FIG. 6); exemplary Profiles 1, 2, 3, 4, and 5, a Common packet
buffer pointer pool 386; a Stream generator 388 (equivalent to
Stream generator 114 of FIG. 6); and Adaptive rate ARQ transmitters
for respective clients 390 (equivalent to Sender device 105 of FIG.
6).
[0176] In embodiments of the current invention, the incoming media
stream may come from plurality of sources. The incoming stream is
encoded by the system to create several copies of the same content
in various encoding profiles, received from an external encoder
that streams different profiles simultaneously or from an existing
HTTP adaptive server. If an HTTP adaptive server is used as a
source, then embodiments of the current invention include simple
adaptive client block 384 that communicates with the server and its
manifest list and which reads all available profiles from the
server. It is a straightforward solution for an adaptive client to
read each profile as it becomes available, as known in the art.
Media stream data is then passed to designated profile memory
process block associated with each profile.
[0177] In another embodiment of the current invention, there is the
capability to gather statistics on subscribers (i.e. clients) to
optimize profiles according to existing network conditions.
Information gathered for each client and the bitrate used by each
client may further optimize an ABR system to allocate more adequate
bitrate profiles to provide higher quality to the end user. An ABR
encoding solution can serve to change the encoding bitrate profile
to a new bitrate or to allocate new bit rate profiles to better fit
in the bitrate range used by large clients groups to provide better
visual quality.
[0178] The processes and/or process blocks described hereinabove
are to be understood as generally residing on non-transitory
computer-readable storage media having executable computer modules.
The word "computer" used hereinabove and in the claims which follow
is intended to mean any computing device, such as, but not limited
to: a CPU; a personal computer; a server; and a mobile device,
inter alia.
[0179] It will be appreciated that the above descriptions are
intended only to serve as examples, and that many other embodiments
are possible within the scope of the present invention as defined
in the appended claims.
* * * * *
References