U.S. patent application number 12/792668 was filed with the patent office on 2011-10-13 for apparatus and method for transmitting media using either network efficient protocol or a loss tolerant transmission protocol.
This patent application is currently assigned to REBELVOX, LLC. Invention is credited to Matthew J. Ranney.
Application Number | 20110252083 12/792668 |
Document ID | / |
Family ID | 44168482 |
Filed Date | 2011-10-13 |
United States Patent
Application |
20110252083 |
Kind Code |
A1 |
Ranney; Matthew J. |
October 13, 2011 |
APPARATUS AND METHOD FOR TRANSMITTING MEDIA USING EITHER NETWORK
EFFICIENT PROTOCOL OR A LOSS TOLERANT TRANSMISSION PROTOCOL
Abstract
A method and apparatus for transmitting media from a node to one
or more recipients over a network using either a network efficient
protocol or a loss tolerant protocol. The method includes
ascertaining at the node if media available for transmission is or
will be consumed in real-time. When the media is or will be
consumed in real-time and the condition on the network is not
sufficient for the consumption of the media in real-time using the
network efficient protocol, then the media is transmitted using the
loss tolerant protocol. Alternatively, the media is transmitted
using the network efficient protocol when the media is not or will
not be consumed in real-time or the condition on the network is
sufficient for the consumption of the media in real-time using the
network efficient protocol. The apparatus, which may be either a
communication device or server, implements the above-describe
method.
Inventors: |
Ranney; Matthew J.;
(Oakland, CA) |
Assignee: |
REBELVOX, LLC
San Francisco
CA
|
Family ID: |
44168482 |
Appl. No.: |
12/792668 |
Filed: |
June 2, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61323609 |
Apr 13, 2010 |
|
|
|
Current U.S.
Class: |
709/203 ;
709/230 |
Current CPC
Class: |
H04L 69/16 20130101;
H04L 69/14 20130101; H04L 51/04 20130101; H04L 45/00 20130101; Y02D
30/50 20200801; H04L 65/80 20130101; H04L 65/4092 20130101; H04L
65/4084 20130101; H04L 69/165 20130101; Y02D 50/30 20180101; H04L
67/24 20130101 |
Class at
Publication: |
709/203 ;
709/230 |
International
Class: |
G06F 15/16 20060101
G06F015/16 |
Claims
1.-20. (canceled)
21. A method for transmitting media over a network comprising:
transmitting the media over the network using a network efficient
protocol when either (i) the media is not being consumed in
real-time or (ii) the media is being consumed in real-time and the
available bandwidth on the network is good enough to support a
transmission rate suitable for the real-time consumption of the
media using the network efficient protocol; or transmitting the
media over the network using a loss tolerant protocol when (iii)
the media is being consumed in the real-time mode and (iv) the
available bandwidth on the network is not good enough to support a
transmission rate suitable for the real-time consumption of the
media using the network efficient protocol.
22. The method of claim 21, further comprising: ascertaining the
available bandwidth on the network; and transmitting the media
using either the network efficient protocol or the loss tolerant
protocol depending on the ascertained available bandwidth on the
network.
23. The method of claim 21, further comprising: identifying the
media transmitted using the loss tolerant protocol; and
re-transmitting at least a portion of the identified media using
the network efficient protocol when the bandwidth on the network is
available.
24. The method of claim 23, wherein re-transmitting at least a
portion of the identified media comprises one of the following: (i)
re-transmitting the identified media in full; or (ii)
re-transmitting only the media lost when transmitting using the
loss tolerant protocol.
25. The method of claim 21, wherein the loss tolerant protocol is
UDP.
26. The method of claim 21, wherein the network efficient protocol
is TCP.
27. The method of claim 21, wherein the network efficient protocol
guarantees the delivery of the media over the network.
28. The method of claim 21, further comprising: ascertaining the
presence status of one or more recipients of the media; and
ascertaining if the transmitted media is to be consumed in
real-time based on the presence status of the one or more
recipients respectively.
29. The method of claim 21, wherein the consumption of the media in
real-time further comprises one of the following: (i) the rendering
of the media "live" as the media is created and transmitted; or
(ii) the rendering of previously created and stored media streamed
from storage.
30. The method of claim 21, further comprising: defining the
network efficient protocol as a default protocol when initially
transmitting the media; and switching the transmission of the media
from the network efficient protocol to the loss tolerant protocol
when: the media is or will be consumed in real-time; and the
available bandwidth on the network is not good enough to support
the transmission of the media at a rate suitable for consumption in
real-time using the network efficient protocol.
31. The method of claim 21, further comprising: defining the loss
tolerant protocol as a default protocol when initially transmitting
the media; and switching the transmission of the media from the
loss tolerant protocol to the network efficient protocol when
either: (i) the media is not being consumed in real-time; or (ii)
the media is being consumed in real-time and the available
bandwidth on the network is good enough to support the transmission
of the media at a rate suitable for consumption in real-time using
the network efficient protocol.
32. The method of claim 21, further comprising ascertaining the
available bandwidth on the network by determining when a
transmission buffer used for buffering the media for transmission
exceeds a predetermined threshold or overflows during the
transmission of the media.
33. The method of claim 21, wherein the transmitted media consists
of voice, video, still pictures, text, GPS or position data, sensor
data, or any combination thereof.
34. A communication device, comprising: a transmission element, the
transmission element configured to: transmit media over a network
using a network efficient protocol when either (i) the media is not
being consumed in real-time or (ii) the media is being consumed in
real-time and the available bandwidth on the network is good enough
to support a transmission rate suitable for the real-time
consumption of the media using the network efficient protocol; or
transmit the media over the network using a loss tolerant protocol
when the media is being consumed in the real-time mode and the
available bandwidth on the network is not good enough to support a
transmission rate suitable for the real-time consumption of the
media using the network efficient protocol.
35. The device of claim 34, wherein the transmission element is
further configured to: ascertain the available bandwidth on the
network; and transmit the media using either the network efficient
protocol or the loss tolerant protocol depending on the ascertained
available bandwidth on the network.
36. The device of claim 34, wherein the transmission element is
further configured to: identify the media transmitted using the
loss tolerant protocol; and re-transmit at least a portion of the
identified media using the network efficient protocol when the
bandwidth on the network is available.
37. The device of claim 36, wherein the transmission element, when
re-transmitting the at least a portion of the identified media, is
further configured to: (i) re-transmit the identified media in
full; or (ii) re-transmit only the media lost when transmitting
using the loss tolerant protocol.
38. The device of claim 34, wherein the loss tolerant protocol is
UDP.
39. The device of claim 34, wherein the network efficient protocol
is TCP.
40. The device of claim 34, wherein the network efficient protocol
guarantees the delivery of the media over the network.
41. The device of claim 34, wherein the transmission element is
further configured to: ascertain the presence status of one or more
recipients of the media; and ascertain if the transmitted media is
to be consumed in real-time based on the presence status of the one
or more recipients respectively.
42. The device of claim 34, wherein the consumption of the media in
real-time further comprises one of the following: (i) the rendering
of the media "live" as the media is created and transmitted; or
(ii) the rendering of previously created and stored media streamed
from storage.
43. The device of claim 34, wherein the transmission element is
further configured to: define the network efficient protocol as a
default protocol when initially transmitting the media; and switch
the transmission of the media from the network efficient protocol
to the loss tolerant protocol when: the media is or will be
consumed in real-time; and the available bandwidth on the network
is not good enough to support the transmission of the media at a
rate suitable for consumption in real-time using the network
efficient protocol.
44. The device of claim 34, wherein the transmission element is
further configured to: define the loss tolerant protocol as a
default protocol when initially transmitting the media; and switch
the transmission of the media from the loss tolerant protocol to
the network efficient protocol when either: (i) the media is not
being consumed in real-time; or (ii) the media is being consumed in
real-time and the available bandwidth on the network is good enough
to support the transmission of the media at a rate suitable for
consumption in real-time using the network efficient protocol.
45. The device of claim 34, wherein the transmission element is
further configured to ascertain the available bandwidth on the
network by determining when a transmission buffer used for
buffering the media for transmission exceeds a predetermined
threshold or overflows during the transmission of the media.
46. The device of claim 34, wherein the transmitted media consists
of voice, video, still pictures, text, GPS or position data, sensor
data, or any combination thereof.
47. The device of claim 34, wherein the device comprises one of the
following: a cellular phone, a mobile phone, a land-line phone, a
PTT radio, a satellite radio, a desktop computer, a mobile
computer, a wired communication device, a wireless communication
device or a server.
48. A communication application embedded in a non-transient
computer readable medium and intended to run on a communication
device, the application comprising: a transmission module, the
transmission module configured to: transmit media over a network
using a network efficient protocol when either (i) the media is not
being consumed in real-time or (ii) the media is being consumed in
real-time and the available bandwidth on the network is good enough
to support a transmission rate suitable for the real-time
consumption of the media using the network efficient protocol; or
transmit the media over the network using a loss tolerant protocol
when (iii) the media is being consumed in the real-time mode and
(iv) the available bandwidth on the network is not good enough to
support a transmission rate suitable for the real-time consumption
of the media using the network efficient protocol.
49. The application of claim 48, wherein the transmission module is
further configured to: ascertain the available bandwidth on the
network; and transmit the media using either the network efficient
protocol or the loss tolerant protocol depending on the ascertained
available bandwidth on the network.
50. The application of claim 48, wherein the transmission module is
further configured to: identify the media transmitted using the
loss tolerant protocol; and re-transmit at least a portion of the
identified media using the network efficient protocol when the
bandwidth on the network is available.
51. The application of claim 50, wherein the transmission module,
when re-transmitting the at least a portion of the identified
media, is further configured to: (i) re-transmit the identified
media in full; or (ii) re-transmit only the media lost when
transmitting using the loss tolerant protocol.
52. The application of claim 48, wherein the loss tolerant protocol
is UDP.
53. The application of claim 48, wherein the network efficient
protocol is TCP.
54. The application of claim 48, wherein the network efficient
protocol guarantees the delivery of the media over the network.
55. The application of claim 48, wherein the transmission module is
further configured to: ascertain the presence status of one or more
recipients of the media; and ascertain if the transmitted media is
to be consumed in real-time based on the presence status of the one
or more recipients respectively.
56. The application of claim 48, wherein the consumption of the
media in real-time further comprises one of the following: (i) the
rendering of the media "live" as the media is created and
transmitted; or (ii) the rendering of previously created and stored
media streamed from storage.
57. The application of claim 48, wherein the transmission module is
further configured to: define the network efficient protocol as a
default protocol when initially transmitting the media; and switch
the transmission of the media from the network efficient protocol
to the loss tolerant protocol when: the media is or will be
consumed in real-time; and the available bandwidth on the network
is not good enough to support the transmission of the media at a
rate suitable for consumption in real-time using the network
efficient protocol.
58. The application of claim 48, wherein the transmission module is
further configured to: define the loss tolerant protocol as a
default protocol when initially transmitting the media; and switch
the transmission of the media from the loss tolerant protocol to
the network efficient protocol when either: (i) the media is not
being consumed in real-time; or (ii) the media is being consumed in
real-time and the available bandwidth on the network is good enough
to support the transmission of the media at a rate suitable for
consumption in real-time using the network efficient protocol.
59. The application of claim 48, wherein the transmission module is
further configured to ascertain the available bandwidth on the
network by determining when a transmission buffer used for
buffering the media for transmission exceeds a predetermined
threshold or overflows during the transmission of the media.
60. The application of claim 48, wherein the transmitted media
consists of voice, video, still pictures, text, GPS or position
data, sensor data, or any combination thereof.
Description
[0001] This application claims the benefit of U.S. Provisional
Patent Application No. 61/323,609, filed Apr. 13, 2010, entitled
"Communication Services Network and Client Enabled Communication
Devices," which is incorporated herein by reference for all
purposes.
BACKGROUND
[0002] 1. Field of the Invention
[0003] This invention relates to communications, and more
particularly, to a communication apparatus and method for
transmitting media between nodes using either a network efficient
transmission protocol or a loss tolerant transmission protocol,
depending on (i) the condition on the network and (ii) if either
the transmitted media is being consumed in real-time or in a
time-shifted mode.
[0004] 2. Description of Related Art
[0005] The Transmission Control Protocol or "TCP" is an example of
a network efficient protocol. TCP guarantees the delivery of
transmitted data between a sender and a recipient at the expense of
speed. For this reason, TCP is currently the most common delivery
protocol used on the Internet. A feature called "flow control" is
the main reason why TCP is able to guarantee the delivery of media.
Flow control determines when data needs to be re-sent and stops the
flow of data until previous packets are successfully transferred.
For example, when a recipient receives a defective packet or a
packet is not received (i.e., a missing packet), a request for
retransmission of the defective and/or missing packet is made and
flow of subsequent packets is stopped until the retransmission
request is satisfied. The guaranteed delivery feature of TCP is
beneficial for certain applications, such as the transfer of the
content of web pages, files or database information. The
possibility that the flow of data may be stopped, however, makes
TCP less than ideal for delivery of time critical media, such as
streaming voice or video.
[0006] The User Datagram Protocol or "UDP" is an example of a loss
tolerant protocol, commonly used on the Internet for streaming
audio, video and other time-based media (i.e., media that changes
over time). UDP is mainly concerned with the delivery of the most
recently available media, as opposed to quality. To achieve the
necessary delivery rate for streaming audio or video, there is no
form of flow control or error correction with UDP. Without any
mechanism for guaranteeing delivery, packets may be received out of
order, defective, or lost altogether, possibly resulting in reduced
quality of the media delivered to the recipient. As a result when
the condition on the network are poor, media may be delivered at a
rate sufficient for real-time consumption using UDP when TCP would
otherwise be inadequate.
[0007] Conventional communication systems are typically either
real-time or time-shifted. Consequently, a protocol like UDP, which
is optimized for "real-time" delivery, is typically used for
real-time systems, while a protocol like TCP, which is optimized
for reliable delivery, is used for time-shifted systems.
SUMMARY OF THE INVENTION
[0008] This invention relates to a method and apparatus for
transmitting media from a node to one or more recipients over a
network using either a network efficient protocol or a loss
tolerant protocol. The method includes ascertaining at the node if
media available for transmission is or will be consumed in
real-time. When the media is or will be consumed in real-time and
the condition on the network is not sufficient for the consumption
of the media in real-time using the network efficient protocol,
then the media is transmitted using the loss tolerant protocol.
Alternatively, the media is transmitted using the network efficient
protocol when the media is not or will not be consumed in real-time
or the condition on the network is sufficient for the consumption
of the media in real-time using the network efficient protocol. The
apparatus, which may be either a communication device or server,
implements the above-describe method.
BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The invention may best be understood by reference to the
following description taken in conjunction with the accompanying
drawings, which illustrate specific embodiments of the
invention.
[0010] FIG. 1 is an architecture diagram of the communication
services network according to the invention.
[0011] FIG. 2 is a block diagram of a client application used in
the communication services network of the present invention.
[0012] FIG. 3 is a media flow diagram on a client device running
the client application of the present invention.
[0013] FIG. 4 is a diagram of a server architecture used in the
communication system of the present invention.
[0014] FIG. 5 is a flow diagram for transmitting media according to
the present invention.
[0015] FIG. 6 is a timing diagram illustrating the transmission of
media between sending and receiving nodes according to the present
invention.
[0016] It should be noted that like reference numbers refer to like
elements in the figures.
DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS
[0017] The invention will now be described in detail with reference
to various embodiments thereof as illustrated in the accompanying
drawings. In the following description, specific details are set
forth in order to provide a thorough understanding of the
invention. It will be apparent, however, to one skilled in the art,
that the invention may be practiced without using some of the
implementation details set forth herein. It should also be
understood that well known operations have not been described in
detail in order to not unnecessarily obscure the invention.
[0018] The term "media" as used herein is intended to broadly mean
virtually any type of media, such as but not limited to, voice,
video, text, still pictures, sensor data, GPS data, or just about
any other type of media, data or information.
[0019] As used herein, the term "conversation" is also broadly
construed. In one embodiment, a conversation is intended to mean a
thread of messages, strung together by some common attribute, such
as a subject matter or topic, by name, by participants, by a user
group, or some other defined criteria. In another embodiment, the
messages of a conversation do not necessarily have to be tied
together by some common attribute. Rather one or more messages may
be arbitrarily assembled into a conversation. Thus a conversation
is intended to mean two or more messages, regardless if they are
tied together by a common attribute or not.
A. System Architecture
[0020] Referring to FIG. 1, a block diagram of the communication
system according to the invention is shown. The system 10 includes
a communication services network 12, a circuit switched (CS)
network 14 such as the Public Switched Telephone Network (PSTN), a
cellular or mobile phone network, a Push to Talk (PTT) network, or
a combination thereof, and a gateway 16 coupling the two networks
12, 14. The communication services network 12 includes one or more
servers 18. The circuit switch network 14 includes, depending on
the type of network or combination of networks, one or more circuit
switches 22 and possibly one or more radio towers 24.
[0021] The communication services network 12 is IP based and
layered over one or more communication networks (not illustrated),
such as the Public Switched Telephone Network (PSTN), a cellular
network based on CDMA or GSM for example, the Internet, an intranet
or private communication network, a tactical radio network, a
satellite radio network, a first responder network, or any other
communication network, or a combination thereof. In various
embodiments, the communication services network 12 is either
heterogeneous or homogeneous.
[0022] One or more legacy communication devices 26 are connected to
the circuit switched network 14 through either wired or wireless
connection(s) 28, as is well known in the art. In various
embodiments, the legacy device(s) may include conventional
land-line telephones, mobile or cellular phones, PTT radios,
satellite phones or radios, desktop or mobile computers, or any
combination thereof.
[0023] One or more client application 30 enabled communication
devices 32 are coupled to the communication services network 12
through an IP-based network connection 34. Depending on the type of
communication device 32, the connection 34 may be wired (e.g.,
Ethernet) or wireless (e.g., Wi-Fi, PTT, radio, satellite, CDMA,
GSM, etc.). In various embodiments, the client application 30
enabled communication devices 32 may be any type of telephone,
including both land-line, cellular or mobile phones, a PTT radio,
satellite based communication device, any type of computer,
including but not limited to desktop, laptop, note pad computers,
or any other type of wired or wireless communication device.
B. Client Application Architecture
[0024] The client application 30 is a messaging application that
operates in a time-shifted mode, a real-time mode, and provides the
ability to seamlessly transition between the two modes. With the
client application 30, both inbound and outgoing media is
simultaneously and progressively stored as it is either (i)
received over a network connection 34 at the communication device
32 or (ii) created on the communication device 32 and transmitted
over the network connection 34. The storage of the media allows the
participants to converse in a time-shifted mode, providing a user
experience similar to conventional messaging systems (e.g., email
or voice Short Messaging Service (SMS)). The simultaneous and
progressive nature of the application 30 enables the participants
to converse "live" in a real-time mode, providing a user experience
similar to a full-duplex telephone conversation.
[0025] The simultaneous and progressive storage of both transmitted
media as it is being created and received media as it is being
received further enables a host of rendering options. Such
rendering options include, but are not limited to: the real-time
rendering of media as the media is received over the network
connection 34, pause, replay, play faster, play slower, jump
backward, jump forward, catch up to the most recently received
media, Catch up to Live (CTL), or jump to the most recently
received media. As described in more detail below, the storage of
media and certain rendering options allow the participants of a
conversation to seamlessly transition the conversation from the
time-shifted mode to the real-time mode and vice versa. In
addition, the client application 30 is capable of supporting
multiple types of media, including but not limited to, voice,
video, text, still pictures, sensor data, GPS data, or just about
any other type of media, data or information.
[0026] Referring to FIG. 2, a logic block diagram of the client
application 30 is shown. The application 30 includes a Multiple
Conversation Management System (MCMS) module 40, a Store and Stream
module 42, and an interface 44 provided between the two modules. In
one embodiment, the MCMS module 40 and the Store and Stream module
42 communicate through interface 44 using an encapsulation format
(e.g., JSON or XML) over a transport header protocol (e.g.,
Hypertext Transfer Protocol or HTTP). The application 30
functionally is similar to the client application described in U.S.
application Ser. Nos. 12/028,400 (U.S. Publication No.
2009/0003558), 12/253,833 (U.S. Publication No. 2009/0168760),
12/253,820 (U.S. Publication No. 20090168759) and 12/253,833 (U.S.
Publication No. 2009/0168760), all of which are incorporated by
reference herein for all purposes. The individual modules are
briefly described below. For more details, the above-listed,
co-pending applications should be reviewed.
[0027] The MCMS module 40 includes a number of modules and services
for creating, managing and conducting multiple conversations. The
MCMS module 40 includes a user interface module 40A for enabling
the user to interface and control the audio and video rendering and
creating functions on the device 32, rendering/encoding module 40B
for performing rendering and encoding tasks, a contacts service 40C
for managing and maintaining information needed for creating and
maintaining contact lists (e.g., telephone numbers and/or email
addresses), a presence status service 40D for both sharing the
online status of the user of the device 32 as well as the online
status of the other users on the communication services network 12.
The MCMS data base 40E stores and manages the meta data for
messages and conversations conducted using the application 30
running on a device 32 as well as contact and presence status
information. In alternative embodiments, the MCMS database 40E may
include, but is not limited to, relational databases, file-based
databases, object databases, document-oriented databases, or any
other type of database and/or database management system that is
capable of storing and retrieving data.
[0028] The Store and Stream module 42 includes a Permanent Infinite
Memory Buffer or PIMB 46 for storing, in an indexed format, the
media of received and sent messages. The Store and Stream module 42
also includes an encode-receive module 42A, net receive module 42B,
transmit module 42C and a render module 42D. The encode-receive
module 42A performs the function of receiving, encoding, indexing
and storing in a time-indexed format in the PIMB 46 media created
using the client application 30 on device 32. The net receive
module 42B performs the function of indexing and storing in the
time-indexed format in the PIMB 46 the media contained in messages
received from other devices 32 or 26 through a gateway client 16.
The transmit module 42C is responsible for both storing in the PIMB
and transmitting to recipients the media of messages created using
the device 32. The render module 42D enables the client application
30 to render on device 32 the media of messages, either in the near
real-time mode as media is received over the network 12 or in the
time-shifted mode by retrieving and rendering the media stored in
the PIMB 46.
[0029] The MCMS module 40 and the Store and Stream module 42 also
each communicate with various hardware components 48 provided on
the device 32, including, but not limited to, encoder/decoder
hardware 48A, network interface 48B for connecting the device 32 to
network connection 34, and media drivers 48C. The encoder/decoder
hardware 48A is provided for encoding the media, such as voice,
text, video or sensor data, generated by a microphone, camera,
keyboard, touch-sensitive display, GPS, sensor, etc. provided on or
associated with the device 32 and decoding similar media before it
is rendered on the device 32. The media drivers 48C are provided
for driving the media generating components, such as speaker and/or
a display (not illustrated) after the media has been decoded. The
network interface 48B is provided for the connecting device 32 to a
network connection 34, either through a wireless or wired
connection. Although not illustrated, the client application 30
runs or is executed by an underlying processor embedded in device
32, such as a microprocessor or microcontroller.
[0030] The transmitted and received media stored in the PIMB 46 is
persistently stored. The term persistent storage as used herein is
intended to have broad meaning. In various embodiments, persistent
storage is intended to mean the storage of media and meta data from
just beyond transient storage needed to either transmit or render
media in real-time to storage for an indefinite period of time. The
term persistent storage therefore may have different meanings,
depending specific implementations or embodiments.
[0031] In addition, "real-time" is intended to mean the consumption
or rendering of time-based media (i.e., media that changes over
time) as the media is being transmitted, regardless if the media is
"live" or not. The real-time consumption of "live" media is
intended to mean the rendering of time-based media as the media is
being created and transmitted. The real-time consumption of
non-live media is intended to mean the consumption of previously
recorded time-based media that is being transmitted out of
storage.
[0032] Referring to FIG. 3, a media flow diagram on a communication
device 32 running the client application 30 is shown. The diagram
illustrates the flow of media for both the transmission and receipt
of media, in either the real-time mode or the time-shifted
mode.
[0033] (i) The Receipt of Media: Media received from the
communication services network 12 is simultaneously and
progressively stored in the PIMB 46 by the net receive module 42B
as the media is over the network connection 34, as designated by
arrow 50, regardless if the media is to be rendered in real-time or
in the time-shifted mode. When in the real-time mode, the media is
also simultaneously and progressively provided by the net receive
module 42B, as designated by arrow 52, to the render module 42D. In
response, the render module 42D simultaneously and progressively
renders the media as it is received over connection 34 on a
media-rendering device (e.g., a speaker and/or display). In the
time-shifted mode, the media is retrieved by the render module 42D,
as designated by arrow 54, from the PIMB 46 at an arbitrary time
after it was persistently stored. The retrieved media is then
rendered on the media rendering devices, such as a speaker and/or
display. In this manner, the recipient of the media may review
persistently stored media at any time after storage in the
time-shifted mode.
[0034] (ii) Transmitting Media: Media created on device 32 by a
media creating device (e.g. a microphone, keyboard, video and/or
still camera, a sensor such as a thermometer or GPS, or any
combination thereof) is progressively stored in the PIMB 46 in a
time-indexed format as it is created, as designated by arrow 58. In
most situations, the media is also provided, as designated by arrow
56, to the transmit module 42C, which simultaneously and
progressively transmits the media as it is created. In other
situations, media may be transmitted by transmit module 42C out of
the PIMB 46 at some arbitrary time after it was created, as
designated by arrow 60. Transmissions out of the PIMB 46 typically
occur when media is created when the device 32 is disconnected from
the network 12. When the device 32 reconnects, the media is read
from the PIMB 46 and transmitted by the transmit module 42C.
[0035] As a clarification, the media creating devices (e.g,
microphone, camera, keyboard, etc.) and media rendering devices
(e.g., speaker and display) as illustrated in FIG. 4 are intended
to be symbolic of such devices. It should be understood such
devices are typically embedded in communication devices 32, such as
mobile or cellular phones, radios, mobile computers, etc. With
other types of communication devices 32, such as desktop computers,
the media rendering or generating devices may be either embedded in
or plug-in accessories.
C. Communication Services Network
[0036] Referring to FIG. 4, a block diagram of the communication
services network 12 is shown. As noted above, the network 12
includes one or more servers 18. In a non-exclusive embodiment,
each server 18 includes one or more routers 20, one or more header
stores 62, and one or more body stores 64.
[0037] The routers 20 communicate with other routers 20, to header
stores 62 for read and/or write operations, to body stores 64 for
read and/or write operations. Routers 20 are further responsible
for updating routing tables and maintaining the presence status
information of users on the network 12. Routers 20 also perform a
number of security functions, including authentication, encryption,
and authorization.
[0038] In a non-exclusive embodiment, the one or more servers 18 on
the network 12 are highly configurable and scalable. For example,
if a large number of users subscribe to the services provided by
the network 12, then a large number of routers 20 may be needed. If
a server 18 routes a high volume of traffic, but the messages tend
to be relatively short in duration (e.g., contain minimal media),
then the number of header stores 62 may be increased relative to
the number of body stores 64. Alternatively, if the traffic handled
by a server tends to have large amounts of media (i.e., the
messages are long in duration), then more body stores 64 may be
needed. Further, the number of servers 18 included on the network
may be increased or reduced as needed.
[0039] On the network 12, each of the server(s) 18 subscribe to all
of the header and body data for a given user and/or group of users.
As a result, if a server 18 that holds the header and/or body
information for a user becomes unavailable, a router 20 may be able
to locate another server 18 to obtain the data. In other
embodiments, one or more users, servers or any other entity may
subscribe based on the domain of user(s), defined sets of users,
media type, codec type used to encode the media, conversation name,
conversation subject or topic, time range, or any other type of
defined criteria.
D. Vox Messages and Media Payloads
[0040] Client application 30 enabled devices 32 communicate with
one another using individual message units, referred to herein as
"Vox messages". By sending Vox messages back and forth over the
communication services network 12, the users of the devices 32 may
communicate with one another, either in the real-time mode or in a
time-shifted messaging mode, and with the ability to seamlessly
transition between the two modes.
[0041] There are two types of Vox messages, including (i) messages
that do not contain media and (ii) messages that do contain media.
Vox messages that do not contain media are generally used for
message meta data, such as media headers and descriptors, contacts
information (telephone numbers or email addresses), presence status
information, etc. Message meta data includes such attributes as a
message identifier or ID, the identification of the message
originator, a recipient list, and a message subject. The identifier
information may be used for a variety of reasons, including, but
not limited to, building contact lists, associating media with
messages, and/or associating messages with conversations. The Vox
messages that contain media are used for the transport of media. In
one embodiment, messages containing text media may include both
meta data and the text media, whereas messages containing
time-based media, such as voice or video, do not contain meta
data.
[0042] In one embodiment, Vox messages are layered on top of the
application layer of whatever transport protocol or protocols are
used on the underlying network infrastructure below the network 12.
As a result, a new transport protocol for Vox messages is not
needed. Rather, Vox messages are transmitted and routed across the
network 12 using current transport protocols running over the
existing telecommunications infrastructure.
[0043] The presence status information contained in Vox messages
may be used to identify the users that are currently authenticated
by the system 10 and/or if a given user is reviewing a message in
real-time or not. The presence data is therefore useful in
determining, in certain embodiments, how messages are delivered
across the network 12. In situations where the presence status
indicates an authenticated user is reviewing a message in real-time
for example, then a transport protocol optimized for timely (i.e.,
real-time) delivery, such as UDP, may be used, whereas a transport
protocol optimized for efficient delivery of messages, such as TCP,
may be used when the presence status indicates the authenticated
user is not reviewing the message in real-time.
E. Transmission Protocols and Complete Copies of Media
[0044] Referring to FIG. 5, a flow diagram 80 for transmitting
media between nodes on the communication services network 12
according to the present invention is shown. The sending and
receiving node pair may be between a communication device 32 and a
server 18, between server 18 hops on the network 12, or between any
two nodes on the network. As described below, either a network
efficient protocol or a loss tolerant protocol may be used,
depending on (i) the condition on the network and (ii) if the media
is being consumed in real-time by the recipient.
[0045] In the initial decision 82, a transmission loop is defined
at the sending node and the sending node determines if there is any
media available for transmission. If not, step 82 is continually
repeated until media becomes available for transmission. When media
is available, it is next determined (decision 84) if the media is
or will be consumed in real-time based on the presence information
of the recipient(s). If the presence information of all the
recipient(s) indicates none are reviewing in real-time, then the
media is transmitted using a network efficient protocol (step 86).
If one or more of the recipients is reviewing the media in
real-time, then the transmitting node determines if the condition
on the network (decision 88) is sufficient for transmitting the
media at a rate sufficient to support real-time consumption using
the network efficient protocol.
[0046] If the condition on the network is sufficient for supporting
real-time communication using the network efficient protocol, then
the transmitting node continues transmitting using the network
efficient protocol (step 86). If the condition is not sufficient,
however, then the transmitting node transmits the media using a
loss tolerant protocol (step 90).
[0047] With media that is transmitted using the loss tolerant
protocol, it is determined in decision 92 if the condition on the
network is sufficiently good enough (i.e., the rate of media loss
is minimal) to support real-time communication. If the condition on
the network is not sufficient, meaning real-time communication is
not possible or practical using even the loss tolerant protocol,
then the transmitting node stops using the loss tolerant protocol.
Instead, the media is transmitted using the network efficient
protocol (step 86) as the condition of the network permits. On the
other hand if the condition on the network is sufficiently good
enough to support real-time communication, then the media is
transmitted using the loss tolerant protocol.
[0048] In decision 94, it is determined if the message has ended or
if the recipient is no longer reviewing in the real-time mode. If
neither condition is met, then another transmission loop is defined
(decision 82) and the above process is repeated. When either
condition is met, the media originally transmitted using the loss
tolerant protocol is retransmitted when network conditions permit
using the network efficient protocol, which guarantees the
eventually delivery of a complete copy of the media as originally
encoded. In most situations, the retransmission occurs when
bandwidth on the network is available beyond what is needed to
support real-time communication. In one embodiment, a complete copy
of the media previous sent using the loss tolerant protocol is
retransmitted. In an alternative embodiment, just the missing media
is transmitted.
[0049] The aforementioned process is continually repeated while
media is available for transmission. With each cycle, a
transmission loop is defined and the above-described process is
repeated.
[0050] The transmission protocol as described above with respect to
FIG. 5 resides in layer above or on top of the underlying network
efficient and loss tolerant protocols. In one embodiment, the
network efficient protocol is used as the default protocol, meaning
when a transmission begins, the network efficient protocol is
initially used and the loss tolerant protocol is used only if the
media is being consumed in real-time and the condition on the
network is not good enough to support real-time consumption. In an
alternative embodiment, the loss tolerant protocol may be the
protocol that is initially used when a transmission begins and the
network efficient protocol is used only when either (i) the media
is not being consumed in real-time and/or (ii) the condition on the
network is good enough to support real-time communication using the
network efficient protocol. The second embodiment is typically used
in situations when it is known that the transmitted media is or
will likely be consumed in real-time and/or the condition on the
network is typically inadequate to support real-time
consumption.
[0051] In a specific embodiment using TCP and UDP, the transmission
protocol as described above with respect to FIG. 5 monitors the
rate at which the TCP protocol transmits media. So long as the
transmit buffer used by the TCP protocol does not back-up beyond a
predetermined threshold or overflow, TCP is used for transmissions,
regardless if the media is being consumed in real-time or not. If
the transmission buffer backs-up beyond the predetermined threshold
level or overflows, indicating that the condition on the network is
inadequate to support the transmission of media at a rate
sufficient to support real-time consumption, then a switch occurs
and UDP is used for media that is being consumed in real-time. The
transmitting node will continue using UDP until (i) the transmitted
media is no longer being consumed in real-time, (ii) the condition
on the network has improved to the point where TCP may be used for
the transmission and consumption of the media in real-time; and/or
(iii) conditions on the network are so poor, it is not practical
for the media to be consumed in real-time even using UDP.
[0052] The transmission protocol as described above with respect to
FIG. 5 resides at both communication devices 32 and at servers 18
on the network 12. In a non-exclusive embodiment, the transmission
protocol is embedded in the client application 30. In an
alternative embodiment, the transmission protocol may be separate
from but cooperate with the application 30. Regardless of how
implemented, the transmission protocol as described herein allows
both client 30 communication devices 32 and server(s) 18 to
communicate with one another using either a loss tolerant or
network efficient protocol, depending on if the media being
transmitted is being consumed in real-time and the condition of the
network.
[0053] Referring to FIG. 6, a timing diagram illustrating the
transmission of media between sending and receiving nodes is
illustrated. Initially at time T1, client 32A begins transmitting
the media of a message (step 82) to recipient client 32B using the
network efficient protocol (e.g., TCP). At time T2, the presence
information of the recipient 32B indicates if the media of the
message is being reviewed in real-time or not. Based on the
presence information (decision 84), the sending node evaluates if
the media is being consumed in real-time or not. In this timing
diagram example, it is assumed that the media is in fact being
consumed in real-time and the network is not good enough (decision
88) to support real-time communication using the network efficient
protocol. As a result at time T3, the media of the message is
transmitted using the loss tolerant protocol (step 90). Eventually
the message ends and/or the recipient transitions from the
real-time to the time-shifted mode. When either event occurs, as
designated by the time T4, the transmitting client 32A retransmits,
according to different embodiments, either just the missing media
or the media previously sent using the loss tolerant protocol. In
either case, the network efficient protocol is used, guaranteeing
the delivery of a complete copy of the media of the message to the
recipient 32B.
[0054] The timing diagram is intended to illustrate the
asynchronous nature of the transmission flow diagram illustrated in
FIG. 5. Many of the steps and/or decisions outlined in the flow
diagram of FIG. 5 occur in parallel to the extent possible. Many of
the events as described overlap with one another to some extent and
do not necessarily occur in "lock-step", meaning it is not
necessary for one step to start only after the previous step is
complete, as is typical with most synchronous "clock-driven"
communication systems. Rather, each step is started as soon as the
conditions necessary to perform that step are completed. This is
particularly true with the transmission of media for example.
Conventional communication systems typically transmit media using
only a single transmission protocol. With the protocol as described
herein, however, the transmission of media may be switched "on the
fly" during the course of a message between either the network
efficient or loss tolerant protocols, depending on the presence
status of the recipient(s) and/or condition on the network.
[0055] The transmission of media between communication devices 32,
as described above, occurs through one or more servers 18 on the
network 12. In an alternative pier-to-pier embodiment however, it
would be possible for two communication devices 32 to communicate
directly with one another. With this embodiment, the flow and
timing diagrams of FIGS. 5 and 6 would still be applicable, except
the communication flow would occur directly over the network 12
between a sending communication device 32 and one or more recipient
communication devices 32.
[0056] It also should be noted that for the sake of simplicity, the
transmission of messages as described above has been "one-way". It
should be understood that the transmissions of media, using either
the network efficient and/or the loss tolerant protocol, is often
bi-directional or "two-way". With two-way communication in the
real-time mode, the user experience is similar to a conventional
full-duplex conversation. In addition, bi-directional communication
can also take place between multiple parties (i.e, more than two),
similar to a multi-party conference call.
[0057] By using either the network efficient or loss tolerant
protocol, depending on if media is being consumed in real-time
and/or on the condition of the network, transmissions are optimized
for real-time when needed or for efficient delivery when real-time
delivery is not critical or network conditions are sufficiently
good to support real-time communication using the network efficient
protocol. On the other hand when the media is being reviewed in
real-time and network conditions are poor, the loss tolerant
protocol is typically used to extend or enhance the ability to
conduct real-time communication. Since any missing or media
transmitted using the loss tolerant protocol is eventually
retransmitted using a network efficient protocol that guarantees
the delivery, the recipient eventually receives a complete copy
(i.e. a full bit rate representation as the media was originally
encoded). The full bit rate representation of the media typically
replaces any missing, defective or out of order representations of
media previously received and stored in the PIMB 46 of the
receiving device 32. In this manner, the recipient may review the
complete, full quality, version of the media in the time-shifted
mode at a later arbitrary time.
[0058] In one embodiment, TCP is used as the network efficient
protocol, while UDP is used as the loss tolerant protocol. In other
embodiments, TCP is used as the network efficient protocol, while
the Cooperative Transmission Protocol (CTP) described in U.S.
application Ser. No. 12/192,890, incorporated by reference herein,
is used as the loss tolerant protocol. In other embodiments, any
network efficient or loss tolerant protocol may be used.
[0059] Although many of the components and processes are described
above in the singular for convenience, it will be appreciated by
one of skill in the art that multiple components and repeated
processes can also be used to practice the techniques of the system
and method described herein. Further, while the invention has been
particularly shown and described with reference to specific
embodiments thereof, it will be understood by those skilled in the
art that changes in the form and details of the disclosed
embodiments may be made without departing from the spirit or scope
of the invention. For example, embodiments of the invention may be
employed with a variety of components and should not be restricted
to the ones mentioned above. It is therefore intended that the
invention be interpreted to include all variations and equivalents
that fall within the true spirit and scope of the invention.
* * * * *