U.S. patent application number 10/838578 was filed with the patent office on 2004-10-21 for program insertion in real time ip multicast.
Invention is credited to Brassil, John Thomas, Garg, Sukesh Kumar, Schulzrinne, Henning Gunther.
Application Number | 20040210944 10/838578 |
Document ID | / |
Family ID | 32771669 |
Filed Date | 2004-10-21 |
United States Patent
Application |
20040210944 |
Kind Code |
A1 |
Brassil, John Thomas ; et
al. |
October 21, 2004 |
Program insertion in real time IP multicast
Abstract
A system for broadcasting television programs over the Internet.
A content provider, such as a broadcaster of a sports event,
packages a television program into digital packets, and transmits
the packets over the Internet to subscribers, from a first
location. The content provider periodically pauses in transmitting
the packets, to allow advertisers to transmit advertising messages
during the pauses, from a second location. Arrangements are
provided to synchornize the content provider and the
advertisers.
Inventors: |
Brassil, John Thomas; (Los
Gatos, CA) ; Garg, Sukesh Kumar; (Sayreville, NJ)
; Schulzrinne, Henning Gunther; (Leonia, NJ) |
Correspondence
Address: |
Gregory A. Welte
806 North County Road 700 West
Frankfort
IN
46041
US
|
Family ID: |
32771669 |
Appl. No.: |
10/838578 |
Filed: |
May 4, 2004 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10838578 |
May 4, 2004 |
|
|
|
09398979 |
Sep 17, 1999 |
|
|
|
6771644 |
|
|
|
|
Current U.S.
Class: |
725/135 ;
715/201; 725/109; 725/110; 725/136; 725/32 |
Current CPC
Class: |
H04L 12/1895 20130101;
H04N 21/23424 20130101 |
Class at
Publication: |
725/135 ;
725/136; 725/032; 715/500.1; 725/109; 725/110 |
International
Class: |
H04N 007/173; H04N
007/10; H04N 007/025; H04N 007/16; H04J 003/16; H04J 003/22 |
Claims
1. A program insertion system, comprising: a) a primary program
provider; b) a first system for i) transmitting a primary program
from said primary program provider to subscribers, and ii)
receiving and scheduling at least one request for insertion of a
secondary program during a pause in said transmission of said
program; c) a secondary program provider for providing said
secondary program; d) a second system for i) transmitting said
secondary program from said secondary program provider to
subscribers, and ii) issuing scheduling requests to said first
system to insert said secondary program during said pause in said
transmission of said primary program; and e) an insertion protocol
governing i) the scheduling of transmission of said primary and
secondary programs, and ii) transfer of control between said first
and said systems.
2. The program insertion system of claim 1, wherein said secondary
program provider is an advertiser and said secondary program
comprises at least one advertisement.
3. The program insertion system of claim 1, wherein said first
system and said primary program provider are implemented in a
single computer.
4. The program insertion system of claim 1, wherein said second
system and said secondary program provider are implemented in a
single computer.
5. A method, comprising: a) receiving program content from a
content provider; b) transmitting the program content over an
Internet to a destination; c) suspending transmission of the
program content, to thereby suspend display of the program content
at the destination, to allow an advertiser to send advertising over
said Internet to said destination; and d) resuming transmission of
the program content to the destination.
6. Method according to claim 5, and further comprising: e)
receiving a message indicating that a suspension should occur; and
f) suspending the transmission in response to the message.
7. Method according to claim 5, and further comprising: e)
scheduling predetermined times and durations of suspension; and g)
suspending transmission of the program content at said
predetermined times, for said durations.
8. Method according to claim 7, wherein no messages are transmitted
which induce the suspensions.
9. Method according to claim 5, wherein the program content
comprises audio, visual, or audio-visual material, and is
transmitted in the form of packets, each containing a sequence
number.
10. Method according to claim 9, wherein sometimes a packet having
a relatively late sequence number arrives at the destination before
a packet having a relatively early sequence number.
11. Method according to claim 9, wherein the advertising is
transmitted in the form of packets, each containing a sequence
number indicating its position in a sequence.
12. Method according to claim 11, wherein, i) prior to a
suspension, a packet of program content having a sequence number N
is transmitted; and then ii) a packet of advertising having a
sequence number N+1 is transmitted.
13. A method, comprising: a) from a first location, transmitting
program content over an Internet to a destination; b) from the
first location, sending a suspension signal to a second location
indicating that transmission of the program content is to be
suspended; c) from the second location, in response to the
suspension signal, transmitting advertising content over the
Internet to the destination; d) from the second location,
transmitting a termination signal to the first location indicating
that transmission of the advertising content is to be terminated;
and e) repeating the processes of paragraphs (a), (b), (c), and (d)
at least once.
14. A method, comprising: a) from a first location, transmitting
program content over an Internet to a destination; b) from a second
location, transmitting advertising content over the Internet to the
destination; c) coordinating the transmissions from the first and
second locations so that the transmissions arrive at the
destination in the approximate chronological order of
program--advertising--program--advertising--progra- m--advertising,
and so on, thereby enabling equipment located at the destination to
play the program content in a chronological sequence, interrupted
periodically by the advertising content.
15. A method, comprising: a) from a first location, transmitting
program content over an Internet to a destination; b) from a second
location, transmitting advertising content over the Internet to the
destination; c) coordinating the transmissions from the first and
second locations so that equipment located at the destination is
able to construct both types of content in a chronological order
following the following pattern:
program--advertising--program--advertising--program--advertising,
and so on.
Description
FIELD OF THE INVENTION
[0001] The present invention relates generally to inserting a
secondary audio and/or video program into a real time multicast of
a primary audio and/or video program. In one specific embodiment
the invention relates to the insertion of advertisements, into real
time multicasts of audio and/or video over the Internet.
BACKGROUND OF THE INVENTION
[0002] With increased popularity and availability of the Internet,
programmers, users and content providers are continuously searching
for new applications of the Internet. One such application is
real-time and near real-time audio and video transmitted over
networks using the Internet Protocol (IP). With continued growth
users will be able to view their television broadcasts and even
listen to their radio programs over the Internet. While presently,
the quality of the multimedia received over the Internet currently
lags the quality that is received over more traditional medium, the
technology is likely to soon rival existing audio/video broadcast
technology.
[0003] Expanding Internet connectivity, network infrastructure
improvements, and more powerful computers are all supporting this
growth. Access technologies already under deployment, such as cable
modems, promise enough downstream bandwidth to support real-time
multimedia transport to residences. The software required to
receive and render low-rate video conferences is now bundled with
new personal computers or freely available (e.g., NetMeeting).
Personal computers are being equipped with inexpensive native
peripherals, such as DVDs, cameras, and frame grabbers, supporting
content creation, capture, editing and transmission. New video
standards such as MPEG-4 are also emerging and are expected to
improve IP video quality. Carrier backbone transmission and
switching equipment is being upgraded to keep up with increasing
traffic demands and quality of service requirements. Thus growing
network and computing capabilities suggest continued growth in
real-time multimedia transport over IP networks.
[0004] Despite the considerable advances in network and end system
technology, use of real-time IP multicast has grown at a relatively
slow rate. Four reasons are typically cited for this limited
growth: 1) limited available transmission bandwidth (particularly
for residential users); 2) the predominance of packet loss at
congested network routers; 3) the existing infrastructure's lack of
a simple, scaleable multicast routing protocol; and 4) the lack of
broadcast content for users to receive. The first two factors yield
what is perceived as poor reception quality--for video in
particular.
[0005] To address the immediate problem of poor reception, most
content providers have steered clear of multicast transport. The
overwhelming majority of audio content is unicast and streamed in
near real time (i.e., up to a few seconds delay). This,
notwithstanding that IP multicast is better suited for reaching
larger audiences, exchanging certain types of personal information,
and satisfying advertisers. In addition, exciting new services and
opportunities are more easily achieved in a multicast setting.
While packet loss recovery mechanisms in streaming transport
protocols assure users of the highest quality reception, it comes
at the expense of unicasting to each individual user to emulate a
broadcast medium.
[0006] But, as discussed above, network infrastructure is being
continuously upgraded, and routers and standards are being
introduced to address existing network service quality
deficiencies. These include the Reservations Protocol ("RSVP") and
Differentiated Services ("DiffServ") which attempt to achieve
quality of service, and Protocol Independent Multicast ("PIM")
which attempts to achieve a scaleable multicast protocol described
in S. Deering, et al., "The PIM Architecture for Wide Area
Multicast Routing,: IEEE/ACM Transactions on Networking vol. 4 no.
2, pp. 153-62 (1996), hereby incorporated by reference as if fully
set forth herein. As for lack of content, while little video is
currently transmitted on the Internet, much content is potentially
available. Much of this content is of interest to smaller audiences
than would be efficiently targeted by other technologies such as
broadcast radio or television. Hence, the technical barriers to
growth in real-time multicast are being addressed, and all
indications are that this trend will continue.
[0007] What limits further growth in IP multicasting, however, is
the absence of a viable operational business model to justify an
investment in broadcasting. Traditionally, advertisers pay for a
time slot and provide a broadcaster with a pre-recorded tape of, or
a disk containing an advertisement. While the content of television
broadcasters can be either pre-recorded or televised live, the
advertisements are broadcast by simply flipping a switch from the
live camera or player feed of the feature presentation, to the
player of the pre-recorded advertisement.
[0008] Streaming audio over the Internet can be expensive and
presently audiences are typically small. While traditional
television broadcasts and cable profit from advertisements, it is
difficult to interest advertisers to pay for programs viewed by
small audiences on the Internet. Clearly, solutions for effective
and efficient insertion of advertisements or other programs during
a real time presentation of a feature program serviced over the
Internet are desired. Furthermore, a viable business model will
lead to further desirable infrastructure improvements.
[0009] Consider a sporting event. The interested audience normally
far exceeds the actual number of attendees. Event organizers
recognize the potential value of broadcasting their content, but
currently are unable to capture that value. Recording content for
deferred, on-demand playback has little promise, since a sporting
event's value diminishes rapidly as results become known. In some
cases, organizers might consider a real-time broadcast on the
multicast backbone (MBone). A significant barrier here is that
multicasting forces the organizers to make an initial and often
substantial, up-front investment. Yet no practical mechanism exists
to enable the content owners to be compensated for their expense.
Event organizers seek assured funding to finance broadcasting and
to ensure programming quality.
SUMMARY OF THE INVENTION
[0010] Accordingly, the present invention is a system and method
for supporting audio/video program insertion in real-time IP
multicasts. Secondary programs are seamlessly inserted into
multicast sessions of primary programs in a decentralized fashion.
Relying on coordination of nodes using the Real Time Protocol (RTP)
and IP multicast, inserted video and/or audio programs are
displayed within the same `window` as the primary program
regardless if the primary and secondary streams originate from
physically distinct sources. If the inserted content is an
advertisement, viewers have the familiar experience of commercial
advertising on broadcast television.
[0011] In accordance with the present invention a primary content
provider transmits its multimedia stream to a first proxy that
transmits the data to a destination multicast session. A secondary
content provider interested in inserting a program into that
destination multicast session sends a request through a second
proxy to the first proxy requesting a time slot to insert its
secondary program. The request includes the duration of the
secondary program. The first proxy responds with a time slot or a
denial of the request depending if a time slot is available.
Assuming an available time slot, at the appropriate time the first
proxy transfers control of the destination multicast session to the
second proxy which transmits the secondary program. Upon completion
of the secondary program the second proxy returns control of the
session to the first proxy. All scheduling and control transfer
occurs through the development of a new protocol to manage the
transfer of control. Smooth transitions occur by manipulation of
the RTP header in the packets and the associated RTCP stream.
[0012] In accordance with the present invention, primary content
providers, secondary content providers (e.g., advertisers) and
viewers are afforded a far richer collection of relationships,
opportunities and features than possible with traditional broadcast
mediums. Thus for example spontaneous meetings and relationships
between primary and secondary content providers may be established;
program insertions may be set at arbitrary times from arbitrary
locations in a multicast tree; verifiable viewer demographics from
Real Time Control Protocol messages may be sent to content
providers; and the system may support multiple, simultaneous
program transmissions with user-selectable reception and viewing.
In addition, operation of the system of the present invention does
not require any modifications to existing software viewers or
players.
[0013] Furthermore, the present invention is not dependent on IP
being used from end-to-end, or that the network be public.
Considerable applications of the system of the present invention
exist outside of the Internet where IP multicast may be used on a
more limited basis, such as with satellite distribution of
audio/video feeds to "headends" for local internet or intranet
distribution.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] FIG. 1 depicts one instance of the general architecture of
the program insertion system of the present invention.
[0015] FIG. 2 illustrates a typical packet header in accordance
with real time protocol version 2.
[0016] FIG. 3 illustrates the interprocess communication in the
s_proxy as operated in accordance with the present invention.
[0017] FIG. 4 illustrates an example of a program insertion
protocol in accordance with the system of the present
invention.
[0018] FIG. 5 illustrates a linear topology of a wide area network
and the mean packet latency between various nodes therein between
which the program insertion system of the present invention is
implemented.
DETAILED DESCRIPTION OF THE INVENTION
[0019] One instance of the architecture of the program insertion
system and method of the present invention is shown in FIG. 1. For
ease of discussion the primary content provider is referred to as
sender 10, while the secondary content provider is known as
advertiser 16. In fact in accordance with the method and system of
the present invention, the denotation of a content provider as
primary and secondary is arbitrary. It merely simplifies the
following discussion to refer to the provider of the principle
program as primary and the provider of the inserted program as
secondary. Since one common and advantageous embodiment of the
present invention is to insert advertisements into a primary
program, we refer to the secondary provider as the advertiser.
Indeed, a system for inserting commercial advertisements sets the
foundation for a viable business model for television and radio
over the Internet. However, it should be clear that the following
discussion is not limited in scope to advertisement insertion.
Further applications include insertion of emergency broadcasts,
video chat sessions, and virtual radio/television stations. Indeed,
the program insertion system of the present invention is operable
with any application capable of sending and receiving RTP packets,
or packets using an equivalent sending and receiving transport
protocol which resides on the User Datgram Protocol ("UDP").
[0020] In accordance with the present invention proxies are
associated with sender 10 and advertiser 16. Each proxy is
responsible for broadcasting the content of their provider to
receivers tuned to the destination multicast session. By using
proxies, the method of the present invention is independent of
multimedia viewers or players. Only one proxy forwards a Real Time
Protocol ("RTP") stream (an Internet standard protocol for
transporting continuous media) at any time. H. Schulzrinne, at al.,
"RTP: A Transport Protocol for Real-Time Applications," Internet
RFC 1889 (January 1996), hereby incorporated by reference as if
fully set forth herein.
[0021] The s_proxy 12 and a_proxy 14, which are servers, form
client-server pairs with their respective providers that coordinate
switching between the providers, the streams being transmitted to
receivers 18 and 20. Each proxy (a_proxy 14 or s_proxy 12) software
can be run on the same system as the content provider, (advertiser
or sender, respectively) if desired. Unless otherwise indicated,
the terms a_proxy 14 and s_proxy 12 as used herein refer to both
the application software and the system running the software. The
systems running the proxy software appear to the user to be the
actual broadcast sources (as opposed to sender 10 and advertiser
16).
[0022] s_proxy 12 provides two basic services: First, when an
advertisement is not in progress, s_proxy 12 will continuously
relay packets received on a private unicast, or a multicast
session, from the sender 10 to the destination (public) multicast
session. As discussed more fully below, RTP header fields are
usually modified as part of the relay function. When an
advertisement is in progress, s_proxy 12 interrupts the packet
relaying process, permitting the advertiser's proxy, a_proxy 14 to
transmit its own stream to the destination multicast session. The
second service performed by s_proxy 12 is receiving and scheduling
incoming requests for future advertisements. This task includes
passing RTP header information to advertisers to permit them to
inject advertisements seamlessly.
[0023] The content of either provider can be any audio and/or
video, pre-recorded or live, encapsulated in RTP. However, if the
program comprises both audio and video, it is advantageous to
transmit them as separate RTP streams, handled by separate
proxies.
[0024] Similar to the interaction of sender 10 with s_proxy 12,
advertiser 16 ordinarily transmits packets to a_proxy 14 on a
private unicast or multicast address and a_proxy 14 transmits the
content of advertiser 16 on the destination multicast session. As
with s_proxy 12, a_proxy 14 also provides two services: First, when
an advertisement is in progress, it relays packets received from
advertiser 16 to the destination multicast session, in which case
RTP header fields are always modified, as explained below with
reference to FIG. 2. When an advertisement is complete, a_proxy 14
terminates the packet relaying process. As part of this service,
a_proxy 14 is also responsible for passing RTP header information
back to the s_proxy 12 to permit a return to the primary content at
the conclusion of the inserted program. Second, a_proxy 14 also
schedules requests to place future advertisements and is
responsible for properly handling denials of service (e.g. server
responds that a desired time slot is unavailable) returned from the
s_proxy 12.
[0025] Receivers 18 and 20 can be any destination capable of
receiving and rendering audio/video in the encoding formats of both
the sender and advertiser. The program insertion method and system
of the present invention is independent of existing software
players such as vic version 2.8, as described in S. McCanne and V.
Jacobson, "vic: A Flexible Framework for Packet Video," Proceedings
of ACM Multimedia '95 (November 1995), hereby incorporated by
reference as if fully set forth herein, and vat version v4.0b2 on
Solaris 2.6; iCast Viewer version 2.0 on NT 4.0, as described in
iCast Corp., iCast Viewer 3.0 Datasheet, http://www.icast.com/
(1998), hereby incorporated by reference as if fully set forth
herein; or Precept Software's IP/TV demonstration version 2.0 on NT
4.0, as described in Precept Software (Cisco Systems), IP/TVViewer,
http://www.cisco.com/warp/public/732/net_enabled/iptv/(1998)- ,
hereby incorporated by reference as if fully set forth herein.
Thus, some viewers may not require any change in order to operate
with the present invention. FIG. 1 reflects one specific instance
where a vic player is used by sender 10 and rtpplay is used by
advertiser 16.
[0026] The final element of the architecture is the insertion
protocol. This protocol comprises a set of messages sent between
the a_proxy 14 and s_proxy 12. The purpose of the protocol is to
coordinate the transfer of a token; only the token holder is
permitted to transmit packets on the destination multicast session.
The protocol has three logical phases. During the scheduling phase
a_proxy 14 communicates with s_proxy 12 to arrange an insertion at
a future time. Control is passed to a_proxy 14 at that scheduled
time during the transfer phase, and control is returned to s_proxy
12 when the insertion is complete during the return phase.
[0027] In one advantageous embodiment of the present invention
sender 10 and advertiser 16 and their respective proxies 12 and 14
can be implemented in any combination of one or more machines. In
another advantageous embodiment of the present invention the proxy
functions could be integrated into the transmitting application of
sender 10 and advertiser 16.
[0028] Referring to the RTP header format shown in FIG. 2, the
first twelve bytes of the header are required by the protocol. The
synchronization source identifier (SSRC) is a random number which
uniquely identifies the source of an RTP packet stream. Packets
from a synchronization source are distinguished by a timestamp and
sequence number. These fields are used by receivers for proper
signal reconstruction and playout timing. The initial sequence
number value is also random, and is incremented for each
consecutively transmitted packet. Packets can and do arrive at
their destination out-of-order.
[0029] The timestamp indicates the time of the sampling instant of
the RTP payload relative to the initial timestamp value, which is
randomly selected. The sampling rate for many audio/video encoding
formats is constant, well known, and registered with the Interent
Assigned Numbers Agency ("IANA"). Other formats can have
time-varying sampling rates. Media formats are specified by the
Payload Type (PT) field. Multiple packets can have the same
timestamp, as in the case where a large video frame is grabbed,
encoded, but then transported in multiple packets.
[0030] A list of contributing source identifiers is present only if
multiple RTP streams have been mixed. In this case, the CSRC count
(CC) field indicates the number of contributors, and the CSRC
contains the original SSRC identifier of each contributing
source.
[0031] Associated with RTP is a monitoring and control protocol
called RTCP. RTCP's primary functions is to provide feedback on
reception quality, and distribute source identification and control
information. This is realized by distributing messages including
Sender Reports (SR), Receiver Reports (RR), and Source Descriptions
(SDES). A session's RTCP and RTP messages are ordinarily
transmitted to separate ports on a shared destination multicast
address.
[0032] To illustrate the operation of the insertion protocol of the
present invention, consider an implementation of the present
invention where the system injects programs on demand, i.e. without
advance reservation. Advertiser 16, wishing to insert a secondary
program contacts sender 10 immediately prior to insertion. Sender
10 may be transmitting an analog video signal captured from a live
camera, encoded, packetized and transmitted as an RTP stream using
vic, as shown in FIG. 1. An associated RTCP stream is transmitted
on a different port. The RTP and RTCP streams are transmitted from
sender 10 to s_proxy 12 and relayed from there to the destination
multicast session.
[0033] s_proxy 12 controls and manages the process of forwarding
the sender's RTP session to receivers 18 and 20 on the destination
multicast session. This function includes receiving, manipulating
and forwarding the RTP and RTCP packets. It also receives and
processes the insertion request by advertiser 16, interrupting the
relaying of the stream from sender 10 for the duration of the
advertisement. Finally, it takes control of the RTP session at the
advertisement termination time, and resumes relaying the stream
from sender 10.
[0034] Referring to FIG. 3, the three separate tasks of s_proxy 12,
relay, scheduler and server processes, are shown. The relay process
30 establishes a software interface for network communication,
called a socket for receiving the sender's RTP stream, and a second
socket for transmitting the modified stream to the receivers. The
process stores the sender's SSRC, as well as the timestamp and
sequence number of the last packet received. It also maintains an
offset timestamp and offset sequence number. These values are
initially zero but change according to the offset values returned
by the advertiser. The offsets are added to the timestamp and
sequence numbers field in the header of each incoming packet to be
forwarded.
[0035] The server process 50 listens for a request from advertiser
16 on an incoming TCP connection. Each accepted connection creates
a child server process. Information contained in a request is
passed to the scheduler 40 using a queue structure. Advantageously
the requests are queued and served on a first-come, first-serve
basis as needed to handle multiple incoming insertion requests.
This can be implemented using UNIX pipes for the interprocess
communication. Each request from advertiser 16 contains the desired
duration of the advertisement in seconds. Relaying the sender
stream is stopped for the desired advertisement duration. At the
end of the advertisement the server process receives current
timestamp and sequence number values from advertiser 16 (on the
still open TCP connection) and passes it to the relay process 30 to
calculate new offsets.
[0036] The scheduling phase is necessarily message based.
Advertiser 16 seeking to insert a program must explicitly contact
s_proxy 12 with a desired program start time and duration. s_proxy
12 must explicitly accept or deny the request, although the
response can optionally contain additional information. For
example, if an insertion request is denied because of a scheduling
conflict, s_proxy 12 could return alternate available time slots.
Reliability demands suggest that this portion of the protocol
should be TCP-based.
[0037] Thus the scheduler process 40 receives the incoming request
from a server process and schedules an interrupt of the relay
process. It is also possible, if desired, to make the scheduler a
null process. This results in each insertion request immediately
becoming an interrupt. The scheduler process receives the value of
the advertisement duration from the server process 50 and informs
the relay process 30 to sleep for the specified duration. At the
end of the secondary program, new timestamp and sequence number
values are received from the advertiser (through the scheduler
process) and are passed to the relay process 30.
[0038] While in many instances the stream from a sender will be
privately unicast to a single s_proxy, the system can support
multicast to multiple s_proxies. As in broadcast television, it is
desirable to permit multiple distributors of the same content, with
each distributor having its own relations with advertisers. A
typical use would be broadcasting an event of international
interest (e.g., World Cup Soccer) with different agents assigned
distribution rights by country. Different distributors could then
cater to their own sets of advertisers, and their respective
audiences.
[0039] By design, the implementation of a_proxy 14 is very similar
to that of s_proxy 12. However, the scaling issues that arise for
the s_proxy 12 (possibly needing to handle frequent incoming
scheduling requests from many a_proxies) are not anticipated for
a_proxy 14, permitting a simpler implementation. a_proxy 14 has a
single process that does three jobs. First, it establishes a TCP
connection to s_proxy 12 and requests an immediate time slot for
program insertion. Only the intended secondary program duration is
specified. For an immediate insertion, the SSRC and RTCP Session
Description (SDES) parameters for sender 10 are returned, as well
as the timestamp and sequence number of the most recently received
sender packet. a_proxy 14 then calculates offsets for both
timestamp and sequence number. That is, for seamless insertion, it
is necessary for the stream transmitted by a_proxy 14 to pick up
where the s_proxy 12 left off. This requires receiving the incoming
advertiser's private stream, modifying RTP headers, and
multicasting the modified stream on public destination multicast
session.
[0040] Relaying the RTP stream from advertiser 16 begins
immediately. For each incoming packet the advertiser's SSRC is
changed to that of the s_proxy 12, and appropriate timestamp and
sequence number offsets are added. At the completion of the
secondary program transmission, a_proxy 14 sends back to s_proxy 12
its last transmitted packet's timestamp and sequence number. At
this time, the s_proxy 12 calculates new timestamp and sequence
number offsets, then begins relaying the signal from sender 10 once
again, picking up where advertiser 16 left off.
[0041] Scheduling and control transfer phases can be implemented as
coincident, or with arbitrary duration intervals between each
phase. Note that in some applications scheduling insertions might
be preceded by regular broadcast announcements of the availability
of open time slots in a program schedule.
[0042] While scheduling requires explicit messaging, control
transfer from s_proxy 12 to a_proxy 14 and control return from
a_proxy 14 to s_proxy 12 can be implemented with implicit, or
explicit messaging. In the implicit messaging model, no messages
are sent between s_proxy 12 and a_proxy 14 during the control
transfer and control return phases. Instead, it is assumed that
both parties are time synchronized by a separate out-of-band
mechanism such as the Network Time Protocol described in D. Mills,
"Network Time Protocol (Version 3) Specification, Implementation
& Analysis," Internet RFC 1305 (March 1992), hereby
incorporated by reference as if fully set forth herein. This
mechanism should be able to realize modestly precise time
synchronization (e.g., 20 ms). This synchronized time is used by
both a_proxy 14 and s_proxy 12 to determine when to start and stop
relaying the respective content of advertiser 16 and sender 10.
Each party monitors RTP packets on the destination session, and
infers from this an appropriate timestamp and sequence number
offset to use at the assumption of control.
[0043] For explicit messaging, several alternatives exist to
transfer control. One option transmits two messages over a single
TCP connection. These messages simply contain the RTP header of the
last packet transmitted to the destination multicast session by the
active proxy. This approach works well where immediate program
insertion occurs upon request and s_proxy 12 and a_proxy 14 are
geographically closely located.
[0044] A second embodiment uses messages based on the Real Time
Streaming Protocol (RTSP) as described in H. Schulzrinne et al.,
"RTSP:A Transport Protocol for Real Time Applications," Internet
Draft (February 1998), hereby incorporated by reference as if fully
set forth herein. Existing RTSP methods (e.g., SETUP, PLAY) could
be used to trigger playout of stored advertisements located on
video servers. In this embodiment, sender's and advertisers'
proxies operate in part as RTSP proxies. Control transfer messages
between these proxies are based on existing (or extensions to
existing) RTSP control messages.
[0045] For example, suppose an advertiser has scheduled a program
insertion. Immediately prior to insertion time, the RTSP SETUP
method could be used to pass transport parameters (e.g. SSRC,
timestamp, sequence number) to the advertiser's proxy. At insertion
time, the PLAY method would be sent by the sender's proxy to the
advertiser's proxy to initiate advertisement playout. In this
embodiment the communication between the sender and it's proxy, as
well as the advertiser and its proxy, could also be performed using
RTSP messages. For example, at a scheduled program insertion time,
the sender's proxy could use the RTSP PAUSE method to interrupt the
sender's communication, thus allowing the advertiser to insert
content.
[0046] A third embodiment of explicit messaging requires the
destination multicast session be monitored by s_proxy 12 and
a_proxy 14. New RTCP application messages represent an explicit way
to exchange necessary information to facilitate control transfer
and return. In this embodiment, extensions to RTCP messages could
serve as control transfer messages between sender and receiver
proxies. Rather than using a separate communication channel (e.g. a
TCP connection) between the proxies, the IP multicast session
itself is used to transfer program control. For example, a new RTCP
message could pass transport parameters (e.g. SSRC, timestamp,
sequence number) from the sender's to the advertiser's proxy. This
approach, however, is neither private (i.e., not unicast) nor
reliable.
[0047] It is also possible to design a robust protocol by combining
aspects of the foregoing three approaches. Deferred, remote playout
could be triggered by an RTSP PLAY message sent in advance (e.g.,
several seconds or minutes) of a scheduled control transfer. Such a
message would contain appropriate session control information (e.g.
timestamp, sequence number and start time). In the event a deferred
trigger message failed to arrive, advertiser 16 could simply
monitor the session and estimate these parameters independently and
begin when scheduled. In the worst case, advertiser 16 would begin
only after detecting that active transmissions have ceased.
[0048] Selecting from these alternatives should be done with an eye
toward the goal of achieving "seamless" switching between sender 10
and advertiser 16. Switching between video sources transmitting
less than 30 frames/sec fortunately affords some leeway in both the
timing of transmitted packets and the calculation of timestamp
offsets.
[0049] However, achieving seamless switching in a wide area setting
requires consideration of the mean packet latency in the network.
Consider the linear topology shown in FIG. 5 where two receivers,
25 and 27, are connected to a_proxy 14 and s_proxy 12 over links
with specified mean packet latency. Suppose a_proxy 14 and s_proxy
12 are perfectly time synchronized, s_proxy 12 is to transfer
control to a_proxy 14 which has the benefit of advance knowledge of
the timestamp (time frame was captured), sequence number of packet
and admission time (time packet was admitted in the network) of the
last packet that s_proxy 12 will transmit.
[0050] If a_proxy 14 transmits its first packet at exactly the time
s_proxy 12 would have transmitted its next packet, that first
packet will nominally arrive late due to the additional delay in
receiver 25. On the other hand, a_proxy's 14 first packet might
arrive at receiver 27 before s_proxy's 12 final packet if the mean
packet latency is not accounted for. If sender 10 and advertiser 16
are not precisely synchronized and a_proxy 14 poorly estimates the
stamp and sequence number, duplicate packets may result, causing
the receivers to discard one or more packets. Poorly estimated
timestamps could easily produce unintended, perceptible `freeze
frame`, if for example the timestamp increases from frame to frame
so that the playback is very slow. Conversely, `fast forward` play
occurs where the timestamp decreases from frame to subsequent
frame.
[0051] RTP requires that the identifiers be unique for each source
in a session, and recommends actions for sources to take in the
unlikely event of collision. Accordingly, the RTP header must be
appropriately handled to assure display of the primary and
secondary programs in the same window.
[0052] It is important to consider the handling of the RTCP
messages which are critical to achieving the desired result. With
respect to the data packet relayed by a proxy from provider to
receiver, these two entities may be jointly owned or individually
owned. Where the provider and proxy are jointly owned and operated
by a single entity, the proxy could simply relay the provider's
SDES and sender report (SR) messages, without modifying any
identifying information (e.g. CNAME, EMAIL, PHONE, TOOL fields in
SDES messages).
[0053] Other applications, however, require that a content provider
and its proxy (or proxies) be distinguished, as for example, where
a content provider and one or more broadcasters are entirely
separate entities. Here, each proxy should transmit its own session
description and sender report messages, rather than relay these
messages received from the provider.
[0054] With respect to the relationship between the a_proxy and
s_proxy, some applications such as advertisement insertion,
editorial integrity and possibly legal and regulatory guidelines
might require program and advertising content to be distinguishable
by viewers. With other applications, however, a single broadcaster
might wish to hide--or at least not call attention to--content
being transmitted from distinct sources.
[0055] In accordance with one embodiment of the present invention
this is accomplished by insuring both content providers share a
single SSRC identifier if they are to share a single window in the
viewer. Other embodiments as explained below relax this
requirement, relying instead on intelligent viewers.
[0056] Similarly, current viewers will automatically open a second
window upon receipt of an RTCP message containing a CNAME which is
different from the CNAME associated with the RTP packet
encapsulating the primary program. Thus proper handling of RTCP
packets is necessary to assure the desired single window display.
For example, it is possible to have the a_proxy modify and relay
SDES messages from the advertiser so that they contain the same
CNAME as any SDES messages issued by s_proxy. Some viewers, for
example, the iCast viewer will accept this modification and perform
the insertion with no difficulty. Other viewers, however, will
recognize from the source IP address that the secondary program
originated from a source different from the primary program and
open a second window to display the secondary program. As
applications such as program insertion become popular, uniform and
proper handling of SSRC sharing among different viewers can be
written into future versions of the RTP standard.
[0057] Meanwhile it is still possible to implement program
insertion in accordance with the present invention with current
viewers by spoofing IP addresses, a technique known in the art,
which can always be performed by a separate software
mixer/translator application program located at any receiver. Such
tools are widely available in the public domain. One such example
is RTP Trans, available at http://www.cs.columbia.edu/.about.hts/-
rtptools.
[0058] In accordance with the RTP specification the data payload
may be of any encoding type that a viewer can decode. Thus content
providers may use any standard format for their audio and/or video.
However, the secondary content provider should ensure that,
independent of any chosen encoding scheme, with respect to video, a
full frame should be sent first (e.g., an I-frame in MPEG-2). This
is a common recommendation for any video mixing system, as
described in Cable Television Laboratories, Inc. "Digital Program
Insertion: Request for Information," (April 1997), hereby
incorporated by reference as if fully set forth herein.
[0059] This recommendation is easily realized in stored programs.
However, achieving this becomes more difficult for "live"
insertions, and also when returning control to "live" programming
content. One possible solution is to have each proxy, upon
receiving session control, discard video data until a full frame is
received. This suffers however, in that switching between sources
will take a longer, more variable period of time (e.g., perhaps 1/2
second before receiving the next I-frame in a typical MPEG
sequence).
[0060] In accordance with the present invention as described above,
several distinct advantages are enjoyed. First, the system is
operable with existing transmitters and viewers. Second, secondary
content providers can efficiently transmit their own content,
obviating any need for first forwarding their programs to event
broadcasters. Moreover, since the secondary content providers are
responsible for transmitting their own content event broadcasters
are relieved from unwanted administration and retransmission tasks.
Third, because the transport medium is a bidirectional multicast
network, there is no need to impose topological restrictions on
program insertion points. Fourth, despite the fact that the source
of packets varies during a session, viewers see seamless
transitions between primary and secondary content.
[0061] Fifth, advance scheduling of programs is not necessary
because the secondary program is directly inserted by its owner (or
proxy). Consequently, negotiating an insertion takes negligible
time. Moreover, primary and secondary content providers should need
no pre-existing relationship nor agreements. Furthermore, secondary
content providers can tailor their insertions to current
programming content, because spontaneous agreements can be
realized. These advantages and others are not available with
program insertion in either radio or television broadcasting.
[0062] A further advantageous embodiment of the present invention
is to provide for the simultaneous transmission of different
secondary programs, such as advertisements, to different viewers.
With multiple secondary programs available it is even possible for
individual users to customize their reception by choosing among the
different transmissions, or even choosing no advertisements at all.
This can be accomplished by allowing viewers to specify an SSRC
mask. The mask is a 32 bit number logically AND'ed with each
incoming packet's 32 bit SSRC value. If the result equals the SSRC
value of the desired session, then the packet is accepted as
belonging to that stream.
[0063] This approach effectively creates a set of SSRCs which can
be used by advertisers to create separate. simulcast advertisement
channels. For example, suppose receiver A has mask Oxffffff0f and
receiver B has mask Oxfffffff0. Both receivers are tuned to a
multicast session with SSRC identifier 0x55555500. Then, if during
a `commercial` break one advertiser begins transmitting with an
SSRC identifier 0x55555501, this content will be accepted and
played out by receiver B as a logical continuation of a programming
sequence. If a second advertiser begins transmitting with an SSRC
identifier 0x55555510, this content will be accepted and played out
by receiver A. In the event of conflicts (i.e., multiple
simultaneous streams with valid SSRCs), a conflict resolution
procedure is invoked (e.g., packets with the lowest valid SSRC are
played out, packets with other SSRCs are discarded).
[0064] It should be noted that masking would require coordination
of SSRCs among content providers. In addition each receiver would
have to learn of and specify its desired SSRC mask. One manner in
which this can be accomplished is through a session directory
listings for a session and its related secondary program
sub-channels.
[0065] It should be obvious from the foregoing that back to back
secondary programs are possible as well, much the same as multiple
advertisements inserted during commercial breaks on radio and
television. As requests for program insertion specify the length of
the secondary program, the s_proxy is in a position to determine
whether more than one request may be satisfied.
[0066] In a further advantageous embodiment of the present
invention, it is possible to minimize the perceived time required
to switch between two sources by properly estimating an observed
RTP stream's time stamp and sequence number at a future time. In
general, conservative overestimates of future sequence numbers are
preferred because receiving an RTP stream with missing sequence
numbers is less harmful than receiving duplicates. Estimating a
future timestamp for an RTP stream with a unknown constant
timestamp rate is trivial. For unknown constant timestamp rates,
acceptable estimates can be readily obtained after observing the
session for just a few seconds.
[0067] Returning to the plight of the sporting event organizers.
With the ability for third parties to inject advertisements in
real-time broadcasts, organizers now have a valuable commodity to
sell--the `dead` time in their programs. As we will see, the effort
required is minimal and often simply an extension of what
organizers are already doing by seeking industry sponsors and
patrons. In the program insertion system of the present invention
advertisers themselves are responsible for their ad production and
insertion, freeing event organizers from involvement in
re-broadcasting tasks. Advertisers would certainly seem to have
considerable willingness-to-pay to place ads in content viewed by
an audience with specific, identifiable and verifiable
demographics. The additional revenues generated by advertising
represent the assured funding required to enable organizers to
broadcast a high quality production. These same revenues can
indirectly contribute to network infrastructure improvement, since
all parties are now interested in ensuring high quality viewer
reception.
[0068] The foregoing merely illustrates the principles of the
present invention. Those skilled in the art will be able to devise
various modifications, which although not explicitly described or
shown herein, embody the principles of the invention and are thus
within its spirit and scope.
* * * * *
References