U.S. patent application number 11/519990 was filed with the patent office on 2008-03-13 for security techniques for cooperative file distribution.
Invention is credited to Andrew Hickmott, Laird A. Popkin, Yaar Schnitman.
Application Number | 20080066182 11/519990 |
Document ID | / |
Family ID | 39171325 |
Filed Date | 2008-03-13 |
United States Patent
Application |
20080066182 |
Kind Code |
A1 |
Hickmott; Andrew ; et
al. |
March 13, 2008 |
Security techniques for cooperative file distribution
Abstract
Security techniques are provided for cooperative file
distribution. An encryption key or a nonce (or both) are generated
for a package containing one or more files that are to be sent in a
cooperative file distribution system. Random access encryption
techniques can be employed to encrypt a package containing one or
more files to be sent in a cooperative file distribution system.
One or more storage proxies are allocated to a package to be
transmitted in a cooperative file distribution system, based on
load. Access to trackers in the cooperative file distribution
system is controlled using security tokens. Content can
automatically expire using a defined expiration period when the
content is uploaded into the system. Variable announce intervals
allow the tracker to control how often the tracker will receive a
message, such as an announcement or a heartbeat message, from peers
in the system.
Inventors: |
Hickmott; Andrew; (New York,
NY) ; Popkin; Laird A.; (West Orange, NJ) ;
Schnitman; Yaar; (New York, NY) |
Correspondence
Address: |
Ryan, Mason & Lewis, LLP
Suite 205, 1300 Post Road
Fairfield
CT
06824
US
|
Family ID: |
39171325 |
Appl. No.: |
11/519990 |
Filed: |
September 12, 2006 |
Current U.S.
Class: |
726/26 |
Current CPC
Class: |
H04L 9/3236 20130101;
H04L 2209/60 20130101; H04L 29/06 20130101; H04L 9/3213 20130101;
H04L 2209/76 20130101 |
Class at
Publication: |
726/26 |
International
Class: |
H04L 9/32 20060101
H04L009/32 |
Claims
1. A method for generating one or more of an encryption key and a
nonce for a package containing one or more files to be sent in a
cooperative file distribution system, comprising: obtaining samples
of at least a portion of each of said one or more files; applying a
hash to said samples; and generating one or more of said encryption
key and said nonce from a result of said hash.
2. The method of claim 1, further comprising the step of using one
or more of said encryption key and said nonce to encrypt said
package.
3. The method of claim 1, wherein said samples of at least a
portion of each of said one or more files maintain an order of said
files in said package.
4. The method of claim 1, wherein said hash is a Secure Hash
Algorithm.
5. The method of claim 1, wherein said generating step further
comprises the step of generating said encryption key from a
predefined portion of said result of said hash.
6. The method of claim 1, wherein said generating step further
comprises the step of generating said nonce from a predefined
portion of said result of said hash.
7. The method of claim 1, wherein said encryption key is based on a
content of said files in said package.
8. The method of claim 1, wherein said samples are a predefined
number of evenly spaced samples from each file in said package.
9. A method for encrypting a package containing one or more files
to be sent in a cooperative file distribution system, comprising:
separating said package into pieces of a predefined size; applying
a random access encryption technique to each of said pieces; and
generating said encrypted package comprised of said encrypted
pieces.
10. The method of claim 9, wherein said random access encryption
technique employs an encryption key and a nonce.
11. The method of claim 10, wherein said encryption key is based on
a content of said package.
12. The method of claim 9, wherein said random access encryption
technique adheres to an Advanced Encryption Standard.
13. The method of claim 9, wherein said random access encryption
technique allows any received encrypted piece to be decrypted by an
authorized recipient.
14. The method of claim 9, wherein said random access encryption
technique encrypts a given piece N of said pieces using an
encryption key for said package and applying a result to an
exclusive or (XOR) gate with said given piece to generate said
encrypted piece.
15. The method of claim 9, further comprising the step of
delivering said encrypted pieces to a cooperative file distribution
system for transport.
16. The method of claim 15, wherein said cooperative file
distribution system is a Bit Torrent system.
17. A method for assigning one of a plurality of storage proxies to
a package to be transmitted in a cooperative file distribution
system, comprising: evaluating a load of each of said plurality of
storage proxies; assigning a weight to each of said plurality of
storage proxies based on said evaluated load; and selecting one of
said plurality of storage proxies for said package using one or
more predefined criteria to balance a load among said plurality of
storage proxies.
18. The method of claim 17, wherein said load is based on one or
more of used storage space, a number of connections, and a number
of active and inactive torrents associated with each of said
storage proxies.
19. The method of claim 18, wherein said weight is based on a
multiple of two or more of said used storage space, a number of
connections, and a number of active and inactive torrents
associated with each of said storage proxies and wherein setting
any one of said values to zero results in said storage proxy
becoming unavailable for being assigned to said package.
20. The method of claim 17, wherein subsets of said plurality of
storage proxies are grouped together.
21. The method of claim 17, further comprising the step of
smoothing said weights in order to reduce a tendency to assign said
package to the most underutilized storage proxy.
22. A method for controlling access to a tracker in a cooperative
file distribution system, wherein said tracker allows peers
associated with related content to discover each other, comprising:
receiving a request to upload or download said content; evaluating
an authorization for said request; and providing a security token
to a sender of said request, whereby said security token can be
used to establish an authorization between said sender of said
request and said tracker.
23. The method of claim 22, wherein said security token is provided
by said sender of said request to said tracker in an announce
message.
24. The method of claim 23, wherein said tracker can validate said
sender of said request using said security token.
25. The method of claim 23, wherein said tracker provides a listing
of one or more of said peers to said sender of said request in
response to said announce message.
26. The method of claim 22, wherein said peers associated with
related content are one or more senders and one or more recipients
of said content.
27. The method of claim 22, wherein said security token has a
defined expiration.
28. The method of claim 22, wherein said security token is an
encrypted binary data string.
29. The method of claim 22, wherein said security token contains a
last torrent update time and wherein said tracker obtains torrent
information if a predefined torrent update time has been
exceeded.
30. The method of claim 25, wherein said listing is empty if said
peer is a storage proxy.
31. The method of claim 25, wherein said listing comprises
addresses of leeches if said peer is a seed peer.
32. The method of claim 25, wherein said listing does not identify
peers behind a firewall.
33. The method of claim 25, wherein said listing does not identify
peers behind a different number address translator (NAT) unless
said peers are not behind a firewall.
34. The method of claim 25, wherein said listing does not identify
a storage proxy peer if the number of peers satisfies a predefined
criteria.
35. The method of claim 25, wherein said listing has a predefined
maximum length, X, and said listing includes a storage proxy, if
present, up to X-1 peers behind the same number address translator
as the requested peer and then other peers.
36. A method for processing a package containing one or more files
to be sent in a cooperative file distribution system, comprising:
receiving a request to upload said package; and responsive to said
request, setting an expiration period for said package.
37. The method of claim 36, further comprising the step of deleting
said package after said expiration period has expired.
38. The method of claim 36, further comprising the step of
deallocating a storage proxy associated with said package after
said expiration period has expired.
39. A method performed by a tracker for communicating with one or
more peers in a cooperative file distribution system, comprising:
receiving a request from one of said peers to start a session;
determining an announce interval within which said one of said
peers should provide an announcement; and providing a message to
said one of said peers comprising said announce interval.
40. The method of claim 39, wherein announce interval is determined
based on whether said peer is a storage proxy.
41. The method of claim 39, wherein announce interval is determined
based on a recency of activity for a torrent associated with said
request.
42. The method of claim 39, wherein announce interval is determined
based on a number of peers in a torrent associated with said
request.
43. The method of claim 39, wherein announce interval is determined
based on whether one or more peers in a torrent associated with
said request are behind a firewall.
44. A system for generating one or more of an encryption key and a
nonce for a package containing one or more files to be sent in a
cooperative file distribution system, comprising: a memory; and at
least one processor, coupled to the memory, operative to: obtain
samples of at least a portion of each of said one or more files;
apply a hash to said samples; and generate one or more of said
encryption key and said nonce from a result of said hash.
45. A system for encrypting a package containing one or more files
to be sent in a cooperative file distribution system, comprising: a
memory; and at least one processor, coupled to the memory,
operative to: separate said package into pieces of a predefined
size; apply a random access encryption technique to each of said
pieces; and generate said encrypted package comprised of said
encrypted pieces.
46. A system for assigning one of a plurality of storage proxies to
a package to be transmitted in a cooperative file distribution
system, comprising: a memory; and at least one processor, coupled
to the memory, operative to: evaluate a load of each of said
plurality of storage proxies; assign a weight to each of said
plurality of storage proxies based on said evaluated load; and
select one of said plurality of storage proxies for said package
using one or more predefined criteria to balance a load among said
plurality of storage proxies.
47. A system for controlling access to a tracker in a cooperative
file distribution system, wherein said tracker allows peers
associated with related content to discover each other, comprising:
a memory; and at least one processor, coupled to the memory,
operative to: receive a request to upload or download said content;
evaluate an authorization for said request; and provide a security
token to a sender of said request, whereby said security token can
be used to establish an authorization between said sender of said
request and said tracker.
48. A system for processing a package containing one or more files
to be sent in a cooperative file distribution system, comprising: a
memory; and at least one processor, coupled to the memory,
operative to: receive a request to upload said package; and
responsive to said request, set an expiration period for said
package.
49. A system performed by a tracker for communicating with one or
more peers in a cooperative file distribution system, comprising: a
memory; and at least one processor, coupled to the memory,
operative to: receive a request from one of said peers to start a
session; determine an announce interval within which said one of
said peers should provide an announcement; and provide a message to
said one of said peers comprising said announce interval.
Description
FIELD OF THE INVENTION
[0001] The present invention relates generally to communication
methods and systems, and more particularly, to cooperative and
secure methods and systems for sharing one or more files among
users.
BACKGROUND OF THE INVENTION
[0002] The providers of popular, large digital files, such as
software, music or video files, must keep pace with the ever
increasing bandwidth demands for delivering such files. As the
popularity of a file increases, a larger number of users are
requesting the file and more bandwidth is required to deliver the
file. With conventional Hypertext Transfer Protocol (HTTP) file
delivery techniques, for example, the bandwidth requirements
increase linearly with the number of requesting users, and quickly
becomes prohibitively expensive.
[0003] A number of techniques have been proposed or suggested for
reducing the bandwidth demands of file delivery on the server,
using peer-to-peer content sharing. In a peer-to-peer content
sharing model, often referred to as "cooperative distribution," one
or more users that have downloaded a file from the server can share
the file with other users. A cooperative distribution model allows
a server to deliver large files in a reliable manner that scales
with the number of requesting users. Among other benefits,
cooperative distribution models exploit the underutilized upstream
bandwidth of existing users.
[0004] The BitTorrent.TM. file distribution system, described, for
example, in http://www.bittorrent.com/documentation.html, or Bram
Cohen, "Incentives Build Robustness in BitTorrent,"
http://www.bittorrent.com/bittorrentecon.pdf (May 22, 2003) is an
example of a cooperative distribution technique. When multiple
users are downloading the same file at the same time using the
BitTorrent file distribution system, the various users upload
pieces of the file to each other. In other words, a BitTorrent user
trades pieces of a file that the user has with other required
pieces that other users have until the complete file is obtained.
In this manner, the cost of uploading a file is redistributed to
the users of the file and the cost of hosting a popular file is
more affordable.
[0005] While the BitTorrent file distribution system provides an
effective mechanism for distributing large files in a cost
effective manner, it suffers from a number of limitations, which if
overcome, could further improve the utility and efficiency of
cooperative file distribution. In particular, if a BitTorrent
receiver is offline, then the receiver is unable to obtain files
from other users. U.S. patent application Ser. No. 11/096,193,
filed Mar. 31, 2005, entitled "Method and Apparatus for Offline
Cooperative File Distribution Using Cache Nodes," discloses a
cooperative file distribution system that uses one or more storage
proxies to store the files that are being transferred among
users.
[0006] A need still exists for improved security techniques for a
cooperative file distribution system.
SUMMARY OF THE INVENTION
[0007] Generally, security techniques are provided for cooperative
file distribution. According to one aspect of the invention, a
method and system are provided for generating an encryption key or
a nonce (or both) for a package containing one or more files that
are to be sent in a cooperative file distribution system.
Initially, samples are obtained of at least a portion of each of
the files. Thereafter, a hash is applied to the samples and the
encryption key or nonce (or both) are generated from a result of
the hash.
[0008] According to another aspect of the invention, random access
encryption techniques are employed to encrypt a package containing
one or more files to be sent in a cooperative file distribution
system. The package is first separated into pieces of a predefined
size, and then a random access encryption technique is applied to
each of the pieces. The encrypted package is comprised of the
encrypted pieces.
[0009] According to yet another aspect of the invention, one or
more storage proxies are allocated to a package to be transmitted
in a cooperative file distribution system. The load of each of
storage proxies is evaluated, and a weight is assigned to each
storage proxy based on the evaluated load. Thereafter, a storage
proxy is selected for the package using one or more predefined
criteria to balance a load among the storage proxies.
[0010] Another aspect of the invention controls access to a tracker
in a cooperative file distribution system. The tracker allows peers
associated with related content to discover each other. The tracker
receives a request to upload or download content. Thereafter, the
tracker determines if the sender of the request is authorized. The
tracker will provide a security token to the sender of the request,
whereby the security token can then be used to establish an
authorization between the sender of the request and the
tracker.
[0011] According to further aspects of the invention, content, such
as a package of one or more files, can automatically expire using a
defined expiration period when the content is uploaded into the
system. In addition, a variable announce interval mechanism is
disclosed that allows the tracker to control how often the tracker
will receive a message, such as an announcement or a heartbeat
message, from peers in the system.
[0012] A more complete understanding of the present invention, as
well as further features and advantages of the present invention,
will be obtained by reference to the following detailed description
and drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 is a schematic block diagram illustrating a
conventional BitTorrent file distribution system;
[0014] FIG. 2 is a schematic block diagram of a cooperative file
distribution system incorporating features of the present
invention;
[0015] FIG. 3 illustrates an exemplary key generation process for
generating an encryption key and nonce in accordance with an
exemplary embodiment of the present invention;
[0016] FIG. 4 illustrates an exemplary encryption process for
encrypting a package of files using the encryption key and nonce
generated by the process of FIG. 3;
[0017] FIG. 5 illustrates the transmission of encrypted data in
accordance with the present invention between one or more senders
and one or more recipients;
[0018] FIG. 6 is a communication sequence diagram in accordance
with a Unified Modeling Language (UML) notation, illustrating
exemplary communications and other processing performed by the
various entities of FIG. 2 for uploading (publishing) content into
the cooperative file sharing system;
[0019] FIGS. 7A through 7C, collectively, are a functional
specification for a storage proxy allocation process incorporating
features of the present invention;
[0020] FIGS. 8A and 8B, collectively, are pseudo code for an
exemplary announce interval computation process incorporating
features of the present invention;
[0021] FIG. 9 is a communication sequence diagram in accordance
with a UML notation, illustrating exemplary communications and
other processing performed by the various entities of FIG. 2 for
downloading content in the cooperative file sharing system;
[0022] FIG. 10 is a communication sequence diagram in accordance
with a UML notation, illustrating exemplary communications and
other processing performed by the various entities of FIG. 2 for
various operations in the cooperative file sharing system; and
[0023] FIG. 11 is a sample table identifying an exemplary token
format.
DETAILED DESCRIPTION
[0024] The present invention provides improved security techniques
for a cooperative file distribution system.
BitTorrent Framework
[0025] FIG. 1 is a schematic block diagram illustrating a
conventional BitTorrent file distribution system 100. As shown in
FIG. 1, a sender 110, desiring to send one or more large files 105
to a receiver 120, interacts with a tracker 130 that is part of the
BitTorrent file distribution system 100. For a more detailed
discussion of the BitTorrent file distribution system 100, see, for
example, BitTorrent Protocol,
http://www.bittorrent.com/protocol.html, or BitTorrent Guide,
http://www.bittorrent.com/guide.html, each incorporated by
reference herein.
[0026] Generally, to publish one or more files 105 using the
BitTorrent file distribution system 100, a corresponding static
file 114 with extension .torrent is put on a web server 160. In
particular, as shown in FIG. 1, a BitTorrent client 116 executing
on the sender computing device 110 typically initiates a web
browser 118, for example, via a manual post 140, to place the
torrent file 114 on the BitTorrent web server 160. Alternatively,
the torrent file 114 can be sent by email to the receiver 120. The
web browser 118 communicates with the BitTorrent web server 160,
for example, by means of conventional HTTP communications 170. The
.torrent file 114 contains information about the file, including
its length, name, and hashing information, and the web address
(e.g., a URL) of a tracker 130. Trackers 130 are responsible for
helping users find each other.
[0027] Trackers 130 communicate using a protocol that may be
layered on top of HTTP in which a downloader 110, 120 sends
information regarding the one or more files that the user is
downloading, the port that the user is listening on, and similar
information, and the tracker 130 responds with a list of contact
information for peers that are downloading the same file.
Downloaders 110, 120 then use this information to connect with one
another.
[0028] To make one or more files 105 available, a downloader 110
having the complete file(s) 105 initiates a seed server 112, using
the BitTorrent client116. The bandwidth requirements of the tracker
130 and web server 160 are low, while the seed must send out at
least one complete copy of the original file.
[0029] The responsibilities of the tracker 130 are generally
limited to helping peers (i.e., users) find each other. Typically,
the tracker 130 returns a random list of peers to each user. In
order to keep track of the files and file pieces held by each user
110, 120, the BitTorrent file distribution system 100 divides files
into pieces of fixed size, typically a quarter megabyte. Each
downloader 110, 120 reports to all of its peers via the tracker
130, the pieces held by the respective downloader 110, 120.
Generally, each peer sends bit torrent tracker messages 165 to the
tracker 130. To verify data integrity, a hash of each piece can be
included in the .torrent file 114, and a given peer does not report
that it has a given piece until the corresponding hash has been
validated.
[0030] On the receiver side 120, the receiver 120 reads the web
page on the tracker web site 160 with .torrent file 114 attached
and uses the browser 126 to click on the .torrent file. As a
result, the BitTorrent client 128 is launched on the receiver 120
and the .torrent file 124 is provided to the client process 128. In
addition, the BitTorrent client 128 initiates a "Leech" server 122
that allows the receiver 120 to connect to the public tracker 130.
In this manner, the file 105 is sent from the "seed" 112 to the
"leech" 122 via connection 150, such as an offline peer-to-peer
connection or swarm delivery, in a known manner. The file copy 105
can then be opened by the receiver 120, for example, using an
operating system function.
Cooperative File Distribution Using Storage Proxies
[0031] FIG. 2 illustrates a cooperative file distribution system
200 that employs one or more storage proxies 260. Among other
benefits, the storage proxy 260 allows an offline receiver to
obtain files or pieces thereof when the receiver comes online.
[0032] Storage node 260 can cache communications between two nodes
210, 220. The sender 210 deposits blocks of data into the proxy
node 260 for subsequent retrieval by one or more receivers 220. A
receiver 220 can thereafter retrieve that data from the storage
proxy 260.
[0033] The cooperative file distribution system 200 may be
implemented, for example, using the BitTorrent file distribution
system 100 of FIG. 1, as modified herein to provide the features
and functions of the present invention. As discussed hereinafter,
the cooperative file distribution system 200 includes a tracker 230
that may be implemented using the tracker 130 of the BitTorrent
file distribution system 100 of FIG. 1, as modified herein to
provide the features and functions of the present invention.
[0034] In addition, as discussed further below, the cooperative
file distribution 200 employs a proxy service 250 to identify
potential nodes that are available to serve as storage proxy 260.
The proxy service 250 may be integrated with the tracker 230, as
shown in FIG. 2, or may be a stand-alone device, as would be
apparent to a person of ordinary skill in the art. The proxy
service 250 may employ, for example, a storage proxy database 255
that identifies the nodes that are available to serve as storage
proxy 260. For each potential storage proxy 260, the exemplary
storage proxy database 255 provides a measure of the idleness,
available disk space, available bandwidth, and the likelihood that
the node is online (e.g., a characterization of whether the node is
transient or permanent). In addition, the storage proxy database
255 optionally provides information on the operating system
employed by the node and the current IP address of the node. The
storage proxy database 255 is optionally indexed by a global unique
identifier (GUID), in a known manner.
[0035] The exemplary profile information maintained in the storage
proxy database 255 may be obtained, for example, by a profile
service that can be integrated with, or independent of, the proxy
service 250. For example, the profile service may obtain
information directly from each potential storage proxy 260
regarding the state of the node (e.g., whether the node is behind a
firewall) and its resources. In addition, in a further variation,
after a given receiver 220 has received a file or a portion thereof
from a given storage proxy 260, the receiver 220 can provide a
confirmation report to the profile service. In this manner, the
validating information from the receivers 220 reduces the
likelihood of abuse by the storage proxy 260.
Encryption in a Cooperative File Distribution System
[0036] According to one aspect of the invention files 205 that are
transmitted in the cooperative file distribution system are
encrypted in transit. In this manner, the files 205 are not
compromised by eavesdropping. In one exemplary implementation, an
Advanced Encryption Standard (AES) 256 in Counter (CTR) mode is
employed.
[0037] FIG. 3 illustrates an exemplary key generation process 300
for generating an encryption key 350 and nonce 360 in accordance
with an exemplary embodiment of the present invention. The
technique shown in FIG. 3 can be employed for potentially very
large files. As shown in FIG. 3, the encryption key 350 and nonce
360 for a package 310 comprised of one or more files 320-1 through
320n are generated in an exemplary embodiment by applying a Secure
Hash Algorithm (SHA-512) hash to blocks (i.e., samples) 340 from
the package. The first 256 bits of the hash are used as the key
350, and the next 128 bits of the hash are used as the initial
nonce 360.
[0038] In this manner, the encryption key 350 depends on the
content of the file(s) 320. In the exemplary implementation shown
in FIG. 3, the blocks 340 are five evenly-spaced 4K sampled blocks
from each file 320.
[0039] The process 300 produces the same key 350 and nonce 360 for
the same package 310 of ordered files 320. In this manner, two
users can package the same content (e.g., the same video) and share
a torrent. The duplicate content only needs to be stored once. In
addition, users who independently publish the same data can take
advantage of sharing a P2P torrent without being aware of each
other.
[0040] If a given file 320 is less than 20K, the whole file is
used. The use of the blocks 330 allows the key 350 and nonce 360 to
be generated without reading the entire file(s), which can be long,
in a similar manner to a thumbprint. Otherwise, each file would
have to be scanned twice, once to generate the key and nonce, and
once to hash it for torrent packaging, which would take too
long.
[0041] FIG. 4 illustrates an exemplary encryption process 400 for
encrypting the package 310 using the encryption key 350 and nonce
360 generated by the process 300 of FIG. 3. According to one aspect
of the invention, the encryption process 400 employs a random
access encryption technique that allows any individual encrypted
pieces that are received by a receiver 220 to be decrypted.
[0042] In one exemplary embodiment, the encryption process 400 uses
an AES 256/CTR technique based on the AES encryption scheme using
256-bit keys 350, 128-bit blocks, and a 128-bit nonce 360. As shown
in FIG. 4, for block N in the stream, (nonce+N) is encrypted using
the key 350 and the result is applied to an exclusive or (XOR) gate
(not shown) with the block to be encrypted to generate the
encrypted data 450. As indicated above, this encryption technique
allows the stream to be decrypted in random order. For a detailed
description of CTR, see, for example, Wikipedia, "Block Cipher
Modes of Operation,"
(http://en.wikipedia.org/wiki/Block_cipher_modes_of operation).
[0043] FIG. 5 illustrates the transmission of encrypted data in
accordance with the present invention between one or more senders
510-1 and 520-2 and one or more recipients 520-1 and 520-2. As
shown in FIG. 5, data is transmitted in the exemplary cooperative
file distribution system using a storage proxy 515 and a tracker
525, as discussed above in conjunction with FIGS. 1 and 2.
[0044] According to one aspect of the invention, the data is
delivered through the cooperative file distribution system as
encrypted data. In other words, the clear data is handed off to the
Bit Torrent system as encrypted data. The clear data 310 is
encrypted into encrypted data 450 using the exemplary encryption
process 400 shown in FIG. 4. The storage proxy 515 does not have
the key 350 or nonce 360. As shown in FIG. 5, the senders 510 and
receivers 520 have the pando files appropriate for the transmitted
information, such as the pando files 530-1 for the sender 510-1 and
receiver 520-1. The pando file 530 includes the key 350 and nonce
360 for the encrypted data 450, which allows the recipient 520-1 to
decrypt the encrypted data 450 and access the original clear data
310.
[0045] In this manner, the encrypted data 450 is delivered without
the ability to decrypt the data midstream. The encrypted data 450
is thus delivered with the benefits of Bit Torrent (including piece
by piece integrity checks) without being able to access the
original data. The data is stored by the storage proxy 515 but the
storage proxy 515 has no ability to access the underlying clear
data 310.
[0046] Uploading Content
[0047] FIG. 6 is a communication sequence diagram 600 in accordance
with a Unified Modeling Language (UML) notation, illustrating
exemplary communications and other processing performed by the
various entities of FIG. 2 for uploading (publishing) content into
the cooperative file sharing system. As shown in FIG. 6, the
communication sequence 600 is initiated during step 650 by the
sender 510 attempting to login to obtain permission to send one or
more files. The sender 510 and services processor 630 have a
"session start" message exchange, whereby the sender 510 provides a
node identifier, and the services processor 630 determines if the
sender 510 has the appropriate permissions to send the desired
file(s). If the sender 510 is approved, the sender 510 receives a
"session start result" message containing, for example, a session
identifier, and indication that the sender node was verified.
Generally, the services processor 630 controls authentication and
database access.
[0048] After the sender 510 is validated by the message exchange
650, the sender 510 attempts to start a session using message
exchange 655. Generally, the sender 510 sends a "start" message to
the services processor 630, which executes a storage proxy
allocation process 700, discussed further below in conjunction with
FIGS. 7A through 7C. Upon selecting a storage proxy 515 and tracker
525, the services processor 630 will store the information in the
database 635 and provide the result to the sender 510, with an
identification of the assigned tracker 525 and a token, discussed
further below in a section entitled "Tracker Tokens." Generally,
tracker tokens are used to control access and use of the tracker
525, without requiring further database access. The token is a key
that can be decrypted by the tracker 525. Among other information,
the token contains the last torrent update time.
[0049] After the sender 610 is notified of the tracker 525 assigned
to the bit torrent, the sender 610 announces his or herself to the
tracker 525, during a message exchange 660. As shown in FIG. 6, the
sender 610 sends an announce message to the tracker 525. The
announce message includes the assigned token, which allows the
tracker 525 to validate the sender 610. The announce message will
trigger the tracker 525 to execute an announce interval computation
process 800, discussed further below in conjunction with FIG. 8. In
addition, based on the torrent update time included in the token,
the tracker 525 may be required to communicate with the services
processor 630 to obtain (update) the torrent information, using the
torrent identifier, during a message exchange 665.
[0050] After implementing the announce interval computation process
800, the tracker 525 will send an announce response to the sender
510. The announce response includes a listing of the peers
associated with the bit torrent, discussed further below in a
section entitled "Tracker Peer Listing," as well as the assigned
announce interval (two seconds in this example). If a storage proxy
is required for the communication, message exchange 670 occurs
between the tracker 525 and the assigned storage proxy 515. The
message exchange 670 includes a request for the storage proxy 515
to join the bit torrent. The storage proxy 515 will respond to the
tracker 525 with an announce message, which will trigger the
tracker 525 to execute the announce interval computation process
800.
[0051] After the defined announce interval, the sender 510 will
send another announce message during message exchange 675. During
message exchange 680, the sender 510 publishes the file on the
assigned storage proxy 515. The sender 510 will continue to
announce periodically to the tracker 525 in accordance with the
assigned announce interval. Thereafter, during message exchange
685, the sender 510 notifies the services processor 630 that the
uploading is complete. Finally, the session is terminated during a
message exchange 690 between the sender 510 and the 630.
[0052] Storage Proxy Allocation Process
[0053] FIGS. 7A through 7C, collectively, are a functional
specification for a storage proxy allocation process 700
incorporating features of the present invention. As shown in FIG.
7A, the storage proxy allocation process 700 collects statistics
710 on each storage proxy to help determine the load experienced by
each storage proxy. For example, in an exemplary embodiment, the
collected statistics 710 include the total number of published
torrents currently assigned to the storage proxy, the maximum
number allowed and the amount of available disk space. The
collected statistics 710 are stored in a storage proxies table 720,
for example, along with an allocation factor, such as a user
controlled number between 0 and 1.
[0054] The storage proxy allocation process 700 includes a section
730 for selecting a storage proxy. As shown in FIG. 7B, during
section 730, the storage proxies table 720 is evaluated, and each
storage proxy is assigned a weight based on the load values
indicated in the table. Each storage proxy is selected with a
frequency that matches its weight.
[0055] In one exemplary embodiment, shown in FIG. 7B, the storage
proxy allocation process 700 allows storage proxies to be grouped
in section 740, with weights calculated based on the assigned
group. In this manner, different levels of services can be offered,
such as higher quality services at a premium.
[0056] The weight function 750 of the storage proxy allocation
process 700 is shown in FIGS. 7B and 7C. As shown in FIG. 7B, the
available disk space is first computed in section 760. Thereafter,
in the exemplary embodiment, four types of storage proxy resources
are evaluated in section 770.
[0057] Finally, the weight is computed in statement 780. Since all
of the factors are multiplied in the weight computation 780, any
one factor being zero (e.g., available disk space) can prevent a
storage proxy from being allocated any more torrents. Taking the
weight to a fractional power (e.g., 0.25), for example, smooths the
distribution of weights, reducing the tendency of the equation to
over-allocate for the most underutilized storage proxy. This factor
can be manipulated to make the allocation scheme sufficiently
responsive without being over-responsive.
[0058] Announce Interval Computation Process
[0059] FIGS. 8A and 8B, collectively, are pseudo code for an
exemplary announce interval computation process 800 incorporating
features of the present invention. The announce interval
computation process 800 is initiated whenever the sender 510,
receiver 520 or storage proxy 515 attempt to communicate with the
tracker 525. The announce intervals are periodic heart beats.
According to one aspect of the invention, the announce intervals
are tuned to (i) reduce the load on the tracker 525; (ii) improve
the efficiency of the swarmed transfer; and (iii) provide the
appearance of responsiveness. For example, the announce interval
can be proportional to the number of peers in a torrent. The
announce interval computation process 800 allows the recency of the
data to be balanced with the load. In general, the larger the
number of peers in a swarm, the less time that is required to keep
the information up to date (i.e., there is less impact if a peer
logs off, so can be tracked in longer intervals).
[0060] As shown in FIG. 8A, the announce interval computation
process 800 contains a number of conditional statements for
establishing the announce interval for various situations. A first
section 810 establishes a fixed announce interval for storage
proxies. Being a critical contributor to the torrent transfer,
storage proxies are tracked more closely than other peers. Section
820 establishes the announce interval for the situation where the
storage proxy is being allocated. When a storage proxy 515 is being
assigned to a torrent being uploaded (see storage proxy allocation
process 700 discussed above), it is desired for the storage proxy
and the sender 510 to connect quickly, so the announce interval is
set to a lower value. Section 830 establishes the announce interval
for the situation where a torrent has been inactive for a while.
Generally, the announce interval can be increased for older
torrents that have not been of significant interest for a period of
time.
[0061] As shown in FIG. 8B, section 840 addresses the situation
discussed above, where there is a large number of peers in a
torrent. Section 850 establishes the announce interval for the
situation where the senders are not behind firewalls. When the
seeds are not behind firewalls, other peers can connect to them,
therefore a longer announce interval is appropriate. Likewise, if a
seed is behind a firewall, a shorter announce interval is
appropriate, since other peers cannot connect directly to the seeds
and the peers must be given addresses (for example, of storage
proxies 515 containing the desired content) to connect to.
[0062] Tracker Peer Listing
[0063] As indicated above, the tracker announce response message
660 (FIG. 6) contains a list of the Internet Protocol (IP)
addresses and listening ports of the peers in a torrent. According
to one aspect of the invention, the listing is optimized to be as
useful as possible for the peers. Therefore, not all possible
addresses are listed.
[0064] In one exemplary embodiment, the listing of a peer in the
tracker announce response message 660 is controlled by the
following announce arguments: [0065] NAT/external_ip--the IP
address the announce message arrives from; [0066] IP--the internal
IP address reported in the announce URL; [0067] port--the listening
port reported in the announce URL; [0068] show_seeds=1|0, 0 default
(indicates who has the same content (whole file)); [0069] fw=0 not
firewalled|1 firewalled|-1 don't know yet (default) [0070] left=0
seed|#leech|-1 don't know yet [0071] type=sp|peer (default)
[0072] The response logic for the exemplary embodiment can be
expressed as follows: [0073] An SP peer (type=sp) always gets an
empty list (storage proxies do not make outgoing connections).
[0074] A seed peer (left=0) only gets addresses of leeches, unless
show_seeds=1 (seeds cannot communicate with other seeds). [0075]
FW=1 is not shown to other peers (peers with firewalls are not
shown to other peers), unless both are behind the same NAT. [0076]
Peers behind different NATs don't see each other, unless peer is
fw=0 (not firewalled) [0077] An SP is not listed if a there is a
certain count of seeds or a certain count non-firewalled seeds
(offload delivery from storage proxies to peers to reduce
costs).
[0078] In a further variation, if the list is longer than a
specified length (such as 40-50 peers), the response can be
randomized in the following manner: [0079] The SP is always the
first in the response. [0080] X peers behind the same NAT as the
requested peer are listed next. [0081] The other peers are
uniformly selected from the complete list.
[0082] Downloading Content
[0083] FIG. 9 is a communication sequence diagram 900 in accordance
with a UML notation, illustrating exemplary communications and
other processing performed by the various entities of FIG. 2 for
downloading content in the cooperative file sharing system.
[0084] As shown in FIG. 9, the downloading process 900 is initiated
using a session start message exchange 950 between the receiver 520
and the services processor 630, in a similar manner to the message
exchange 650 of FIG. 6, whereby the receiver 520 provides a node
identifier, and the services processor 630 determines if the
receiver 520 has the appropriate permissions to receive the desired
file(s). If the receiver 520 is approved, the receiver 520 receives
a "session start result" message containing, for example, a session
identifier, and indication that the receiver node was verified. As
indicated above, the services processor 630 controls authentication
and database access.
[0085] After the receiver 520 is validated by the message exchange
950, the receiver 520 attempts to start a session using message
exchange 955. Generally, the receiver 520 sends a "start" message
to the services processor 630, which executes the storage proxy
allocation process 700, discussed above in conjunction with FIGS.
7A through 7C. Upon selecting a storage proxy 515 and tracker 525,
the services processor 630 will store the information in the
database 635 and provide the result to the receiver 520, with an
identification of the assigned tracker 525 and a token, discussed
below in the section entitled "Tracker Tokens." As indicated above,
tracker tokens are used to control access and use of the tracker
525, without requiring further database access. The token is a key
that can be decrypted by the tracker 525. Among other information,
the token contains the last torrent update time.
[0086] After the receiver 520 is notified of the tracker 525
assigned to the bit torrent, the receiver 520 announces his or
herself to the tracker 525, during a message exchange 960. As shown
in FIG. 9, the receiver 520 sends an announce message to the
tracker 525. The announce message includes the assigned token,
which allows the tracker 525 to validate the receiver 520. The
announce message will trigger the tracker 525 to execute the
announce interval computation process 800, discussed above in
conjunction with FIG. 8. In addition, based on the torrent update
time included in the token, the tracker 525 may be required to
communicate with the services processor 630 to obtain (update) the
torrent information, using the torrent identifier, during a message
exchange 965.
[0087] After implementing the announce interval computation process
800, the tracker 525 will send an announce response to the receiver
520. The announce response includes a listing of the storage proxy
515 and sender 510 associated with the file(s), as well as the
assigned announce interval.
[0088] During message exchange 970, the receiver 520 downloads the
file from the assigned storage proxy 515 or sender 510 (or both).
Thereafter, during message exchange 975, the receiver 520 notifies
the services processor 630 that the downloading is complete.
Finally, the session is terminated during a message exchange 980
between the receiver 520 and the 630.
[0089] Maintenance Operations
[0090] FIG. 10 is a communication sequence diagram 1000 in
accordance with a UML notation, illustrating exemplary
communications and other processing performed by the various
entities of FIG. 2 for various operations in the cooperative file
sharing system.
[0091] As shown in FIG. 10, during a tracker registration process
1010, the tracker 525 reports its state to the services processor
630, and the services processor 630 records the state information
in the database 635.
[0092] During a storage proxy registration process 1020, each
storage proxy 515 reports its state, such as its current load
information, to the services processor 630, and the services
processor 630 records the information in the database 635.
[0093] As shown in FIG. 10, for a storage proxy deallocation
process 1030, there are three possible scenarios whereby a given
storage proxy can be deallocated. In a first scenario, the tracker
525 recognizes that the storage proxy is not needed, for example,
because there are no peers remaining in the torrent, or when there
are a sufficient number of seeds and the storage proxy is no longer
needed (to reduce costs).
[0094] In a second scenario, the services processor 630 recognizes
that a given bit torrent has expired. In one exemplary
implementation, bit torrents can be deleted after a defined
expiration period. For example, each time a file is uploaded, the
expiration period can be extended by two weeks. Therefore, a bit
torrent available for two weeks from the last time the BT was
published. (pstart received plus 14 days). The services processor
630 can expire the bit torrent and deallocate the associated
storage proxy 515 after the bit torrent expires.
[0095] In a third scenario, the storage proxy 515 self terminates
by notifying the services processor 630, if the storage proxy
believes that the torrent has expired, based on the expiration
interval that was indicated in the join torrent message 670 (FIG.
6).
[0096] As shown in FIG. 10, for a tracker/services synchronization
process 1040, the tracker 525 and services processor 630 can
synchronize the meta data associated with a bit torrent.
[0097] Tracker Tokens
[0098] As previously indicated, tracker tokens are used to control
access to and use of the tracker 525 and reduce the number of
accesses to the database(s) 635 for authentication purposes. The
tracker tracks all peers who are participating in a torrent and
help these peers to discover each other. Peers announce their
presence to the tracker 525 on regular (announce) intervals, as
discussed above, and are responded to with a listing of the
addresses of other peers.
[0099] When peers upload or download content (package containing
one or more files), as discussed above in conjunction with FIGS. 6
and 9, respectively, the package has an associated a tracker 525.
When the sender 510 or receiver 520 send a start message 655 or
955, which includes the torrent identifier and peer identifier,
they will receive a tracker URL and a tracker token string in the
start result message, bound to the tracker, torrent identifier and
peer identifier. The peer uses the tracker URL for tracker
announcements, and includes the token string in the
announcements.
[0100] In one exemplary implementation, the assigned tokens are
valid for a limited time period. Thus, an announce response message
may include a "token-expired" error. To obtain a new token, a peer
may issue a request for a token from the tracker 525.
[0101] In one preferred embodiment, the token is an encrypted
binary data structure. The tracker 525 and 630 can share a secret
key. In one implementation, 128 bits AES encryption is used.
[0102] FIG. 11 is a sample table identifying an exemplary token
format 1100.
[0103] System and Article of Manufacture Details
[0104] As is known in the art, the methods and apparatus discussed
herein may be distributed as an article of manufacture that itself
comprises a computer readable medium having computer readable code
means embodied thereon. The computer readable program code means is
operable, in conjunction with a computer system, to carry out all
or some of the steps to perform the methods or create the
apparatuses discussed herein. The computer readable medium may be a
recordable medium (e.g., floppy disks, hard drives, compact disks,
or memory cards) or may be a transmission medium (e.g., a network
comprising fiber-optics, the world-wide web, cables, or a wireless
channel using time-division multiple access, code-division multiple
access, or other radio-frequency channel). Any medium known or
developed that can store information suitable for use with a
computer system may be used. The computer-readable code means is
any mechanism for allowing a computer to read instructions and
data, such as magnetic variations on a magnetic media or height
variations on the surface of a compact disk.
[0105] The computer systems and servers described herein each
contain a memory that will configure associated processors to
implement the methods, steps, and functions disclosed herein. The
memories could be distributed or local and the processors could be
distributed or singular. The memories could be implemented as an
electrical, magnetic or optical memory, or any combination of these
or other types of storage devices. Moreover, the term "memory"
should be construed broadly enough to encompass any information
able to be read from or written to an address in the addressable
space accessed by an associated processor. With this definition,
information on a network is still within a memory because the
associated processor can retrieve the information from the
network.
[0106] It is to be understood that the embodiments and variations
shown and described herein are merely illustrative of the
principles of this invention and that various modifications may be
implemented by those skilled in the art without departing from the
scope and spirit of the invention.
* * * * *
References