File Sharing Made Easy & Fast - I
First of a series of two articles (published in Dec '05 and Jan '06) discussing Bit Torrent - the distributed P2P file sharing protocol.
Content
tools
For u & Me
FILE SHARING
MADE EASY AND FAST
When Bram Cohen introduced Bit Torrent to
>> B
the Internet community a couple of years ago, he literally unleashed a torrential storm in the wired world. This simple yet revolutionary file sharing protocol has, since then, helped “swarms” of Internet users and content publishers to break free from the clutches of chronically sluggish FTP servers.
it Torrent is a true P2P, open source, extremely scalable and efficient file sharing protocol. It achieves this efficiency by optimising the bandwidth usage (both upload as well as download bandwidth) of each participating peer. The real beauty of the protocol lies in its working principle, which, as correctly pointed out by its creator, derives from the age-old common-sense dictum, “Tit for Tat”. Here, each peer can choke and ‘unchoke’ other peers dynamically, depending on the feedback it receives about them. The peers which upload the most to others tend to gain the most in terms of downloads. This also ensures that parasitic peers (only willing to download) are choked.
FTP failed to measure up to expectations
Most of the services on the Internet are modelled on either of the two complementary architectural frameworks, namely, the Client-Server model and the Peer to Peer or P2P model. Where centralised control and security are critical (e.g., in most distributed database
www.itmagz.com | ‘i.t.’
DECEMBER 2005
47
CMYK
For u & Me
tools
applications), the Client-Server model comes across as the natural choice. However, there are certain applications for which this model proves to be a workable but inefficient solution. One particularly common case is that of a file server (using FTP) offering a big file, say a video clip, for download. Such a server may receive thousands or even millions of requests within a short span of time, depending on the popularity of the clip. This could cause the server to break down as it tries to send the entire file to each client that requests it. An effective and cost-efficient solution (and the one implemented in Bit Torrent, among others) requires the content publisher to deploy a different architectural approach, transferring the server load to the clients themselves. This is the P2P model where peer entities communicate directly with each other. Information exchange occurs in a decentralised manner so that the problem of the central server being a bottleneck is virtually non-existent. In a truly efficient implementation of a P2P network, the peers participate dynamically to share various bandwidthintensive tasks, normally attributed to the central servers. Such decentralisation brings about scaleability and cost efficiency. Exactly how this is achieved is simple to see—each new peer adds new resources to the system. Thus, demand and supply grow in direct proportion.
means). Though this may seem to be a drawback, it is this feature which makes Bit Torrent immune to copyright law infringement suits, which led to the downfall of Napster. Last, Bit Torrent, unlike others, tries to optimise both the download as well as the upload rates of all the peers, leading to a more efficient bandwidth utilisation on the whole.
How it works
Though Bit Torrent is said to be based on P2P technology, in actual implementation, there’s a central serverlike component called Tracker, which is used to coordinate the activities of the peers and has no role in the sharing of actual files. A Trackerless version of Bit Torrent is under development. The purpose of Tracker is to help downloaders locate each other, coordinate between themselves and maintain certain other statistics. For a file to be shared via Bit Torrent, it must be associated to one or more trackers. Generally, trackers are hosted by the content publishers. The following sequence is the common procedure for sharing a file(s) via Bit Torrent. The content publisher sets up a tracker or uses a reference to an already hosted tracker. For individuals wanting to share files, there are a number of sites that host trackers. The publisher creates and deploys a static “.torrent” meta data file generally on a Web server. This file provides encoded information about the location of the tracker (URL), details of file(s) to be downloaded (size, name, number of pieces, etc). Most Bit Torrent clients offer a utility to automatically create and read .torrent meta information files. In order to make the file(s) available for download, a peer with the complete copy of the file(s) must be started. This peer is called a seeder. Technically, the presence of a possibly distributed but complete copy of the original file is adequate but, in practice, a seeder is generally present. A peer that is not a seeder is called a leecher. A leecher turns a
Why Bit Torrent is so popular
P2P technology has been made popular (and perhaps illicitly so) mostly by the various file sharing programs, from Napster to the more recent, Kazaa, Edonkey and Bit Torrent. But, what’s so special about Bit Torrent alone? Bit Torrent is different from the others in several ways. First, Bit Torrent is an open source protocol and is thus available to the community for criticism and development. Second, most implementations (remember, Bit Torrent is only a protocol) have a much simpler user interface as compared to many other programs. Searching for and locating a file is handled not by the program (or protocol) but by the users themselves (through search engines or by other
seeder when the complete file(s) are obtained and continues to offer uploads to the swarm instead of disconnecting. It is generally considered good manners to continue seeding a file for at least sometime after the download has completed. Another way to seed a file is to simply select the already available file from its .torrent file. This method is followed by publishers who offer a file to be downloaded. A client desiring to download the file(s) must first retrieve the corresponding .torrent file by some means e.g., by clicking on a link provided on the website. The usual approach in most client implementations is to download the .torrent meta information file locally for future use, e.g. to support resuming a partial download or to seed a downloaded file. Using the information present in the .torrent file, the client contacts the tracker and informs it about the file(s) it requires, the port it is listening on, etc. The tracker, in response, provides information about the other peers downloading the same file. This process is repeated periodically to allow the swarm (a “swarm” is a group of peers downloading the same file) to adapt dynamically, e.g. peers entering and leaving the swarm. The client then contacts the peers directly and starts trading pieces of the file(s) with them. I use the word “trading” because each peer both downloads from, as well as uploads to, a number of other peers, simultaneously. A “Piece” refers to a part of the original file(s) that can be verified by SHA1 algorithm.
“Bit Torrent tries to optimise both the download as well as the upload rates of all the peers.”
48
DECEMBER 2005
‘i.t.’ | www.itmagz.com
CMYK
tools
For u & Me
Salient features
Bit Torrent is designed to cater to the sharing of large, popular files and works best for them. The salient features of the protocol that make it extremely efficient and scalable are: Each client downloads different parts of the file(s) from different peers randomly, instead of each client downloading the entire file from a single central server or from a single peer, as in most other file sharing programs. The partially downloaded file(s) is made available (uploaded) simultaneously to other peers by each client. This ensures that both the upload as well as download bandwidths of each peer are used efficiently. Thus the performance of the swarm, as a whole, improves. This, in turn, is reflected as gain for each participating peer. The protocol inherently supports resumed downloads.
For individual users
The above-mentioned procedure is presented from the viewpoint of a content publisher. It may differ slightly for other classes of users. For instance, individual users need not host a Web server or even a tracker in order to share files. A number of “torrent communities” on the Web offer these services so that an end user wanting to share a file need only create a .torrent for that file, and upload it to the
community to be hosted on its Web server (website). The user can then begin seeding the file to those interested in downloads.
A word of caution
It is important to understand that Bit Torrent is not a magic wand that will facilitate lightning fast downloads. It is, at best, a smart idea that facilitates efficient utilisation of a very precious
resource—bandwidth. However, in its present avatar, Bit Torrent has some limitations. The protocol isn’t particularly efficient for sharing small files. In fact, in some cases the overhead involved may make it unsuitable as compared to the relatively simpler old warhorse, FTP. For a downloader to derive adequate benefit from the technology, the size of the swarm must be appreciable. For this reason, BT mostly suits popular and recent files, while the older files tend to become unavailable as and when users lose interest in them. The procedure for publishing file(s) is somewhat complex in comparison to other file sharing technologies like FTP.
AUTHOR TUSHAR MALHOTRA The author has a keen interest in IT. He can be contacted at [email protected].
www.itmagz.com | ‘i.t.’
DECEMBER 2005
49
CMYK
For u & Me
FILE SHARING
MADE EASY AND FAST
When Bram Cohen introduced Bit Torrent to
>> B
the Internet community a couple of years ago, he literally unleashed a torrential storm in the wired world. This simple yet revolutionary file sharing protocol has, since then, helped “swarms” of Internet users and content publishers to break free from the clutches of chronically sluggish FTP servers.
it Torrent is a true P2P, open source, extremely scalable and efficient file sharing protocol. It achieves this efficiency by optimising the bandwidth usage (both upload as well as download bandwidth) of each participating peer. The real beauty of the protocol lies in its working principle, which, as correctly pointed out by its creator, derives from the age-old common-sense dictum, “Tit for Tat”. Here, each peer can choke and ‘unchoke’ other peers dynamically, depending on the feedback it receives about them. The peers which upload the most to others tend to gain the most in terms of downloads. This also ensures that parasitic peers (only willing to download) are choked.
FTP failed to measure up to expectations
Most of the services on the Internet are modelled on either of the two complementary architectural frameworks, namely, the Client-Server model and the Peer to Peer or P2P model. Where centralised control and security are critical (e.g., in most distributed database
www.itmagz.com | ‘i.t.’
DECEMBER 2005
47
CMYK
For u & Me
tools
applications), the Client-Server model comes across as the natural choice. However, there are certain applications for which this model proves to be a workable but inefficient solution. One particularly common case is that of a file server (using FTP) offering a big file, say a video clip, for download. Such a server may receive thousands or even millions of requests within a short span of time, depending on the popularity of the clip. This could cause the server to break down as it tries to send the entire file to each client that requests it. An effective and cost-efficient solution (and the one implemented in Bit Torrent, among others) requires the content publisher to deploy a different architectural approach, transferring the server load to the clients themselves. This is the P2P model where peer entities communicate directly with each other. Information exchange occurs in a decentralised manner so that the problem of the central server being a bottleneck is virtually non-existent. In a truly efficient implementation of a P2P network, the peers participate dynamically to share various bandwidthintensive tasks, normally attributed to the central servers. Such decentralisation brings about scaleability and cost efficiency. Exactly how this is achieved is simple to see—each new peer adds new resources to the system. Thus, demand and supply grow in direct proportion.
means). Though this may seem to be a drawback, it is this feature which makes Bit Torrent immune to copyright law infringement suits, which led to the downfall of Napster. Last, Bit Torrent, unlike others, tries to optimise both the download as well as the upload rates of all the peers, leading to a more efficient bandwidth utilisation on the whole.
How it works
Though Bit Torrent is said to be based on P2P technology, in actual implementation, there’s a central serverlike component called Tracker, which is used to coordinate the activities of the peers and has no role in the sharing of actual files. A Trackerless version of Bit Torrent is under development. The purpose of Tracker is to help downloaders locate each other, coordinate between themselves and maintain certain other statistics. For a file to be shared via Bit Torrent, it must be associated to one or more trackers. Generally, trackers are hosted by the content publishers. The following sequence is the common procedure for sharing a file(s) via Bit Torrent. The content publisher sets up a tracker or uses a reference to an already hosted tracker. For individuals wanting to share files, there are a number of sites that host trackers. The publisher creates and deploys a static “.torrent” meta data file generally on a Web server. This file provides encoded information about the location of the tracker (URL), details of file(s) to be downloaded (size, name, number of pieces, etc). Most Bit Torrent clients offer a utility to automatically create and read .torrent meta information files. In order to make the file(s) available for download, a peer with the complete copy of the file(s) must be started. This peer is called a seeder. Technically, the presence of a possibly distributed but complete copy of the original file is adequate but, in practice, a seeder is generally present. A peer that is not a seeder is called a leecher. A leecher turns a
Why Bit Torrent is so popular
P2P technology has been made popular (and perhaps illicitly so) mostly by the various file sharing programs, from Napster to the more recent, Kazaa, Edonkey and Bit Torrent. But, what’s so special about Bit Torrent alone? Bit Torrent is different from the others in several ways. First, Bit Torrent is an open source protocol and is thus available to the community for criticism and development. Second, most implementations (remember, Bit Torrent is only a protocol) have a much simpler user interface as compared to many other programs. Searching for and locating a file is handled not by the program (or protocol) but by the users themselves (through search engines or by other
seeder when the complete file(s) are obtained and continues to offer uploads to the swarm instead of disconnecting. It is generally considered good manners to continue seeding a file for at least sometime after the download has completed. Another way to seed a file is to simply select the already available file from its .torrent file. This method is followed by publishers who offer a file to be downloaded. A client desiring to download the file(s) must first retrieve the corresponding .torrent file by some means e.g., by clicking on a link provided on the website. The usual approach in most client implementations is to download the .torrent meta information file locally for future use, e.g. to support resuming a partial download or to seed a downloaded file. Using the information present in the .torrent file, the client contacts the tracker and informs it about the file(s) it requires, the port it is listening on, etc. The tracker, in response, provides information about the other peers downloading the same file. This process is repeated periodically to allow the swarm (a “swarm” is a group of peers downloading the same file) to adapt dynamically, e.g. peers entering and leaving the swarm. The client then contacts the peers directly and starts trading pieces of the file(s) with them. I use the word “trading” because each peer both downloads from, as well as uploads to, a number of other peers, simultaneously. A “Piece” refers to a part of the original file(s) that can be verified by SHA1 algorithm.
“Bit Torrent tries to optimise both the download as well as the upload rates of all the peers.”
48
DECEMBER 2005
‘i.t.’ | www.itmagz.com
CMYK
tools
For u & Me
Salient features
Bit Torrent is designed to cater to the sharing of large, popular files and works best for them. The salient features of the protocol that make it extremely efficient and scalable are: Each client downloads different parts of the file(s) from different peers randomly, instead of each client downloading the entire file from a single central server or from a single peer, as in most other file sharing programs. The partially downloaded file(s) is made available (uploaded) simultaneously to other peers by each client. This ensures that both the upload as well as download bandwidths of each peer are used efficiently. Thus the performance of the swarm, as a whole, improves. This, in turn, is reflected as gain for each participating peer. The protocol inherently supports resumed downloads.
For individual users
The above-mentioned procedure is presented from the viewpoint of a content publisher. It may differ slightly for other classes of users. For instance, individual users need not host a Web server or even a tracker in order to share files. A number of “torrent communities” on the Web offer these services so that an end user wanting to share a file need only create a .torrent for that file, and upload it to the
community to be hosted on its Web server (website). The user can then begin seeding the file to those interested in downloads.
A word of caution
It is important to understand that Bit Torrent is not a magic wand that will facilitate lightning fast downloads. It is, at best, a smart idea that facilitates efficient utilisation of a very precious
resource—bandwidth. However, in its present avatar, Bit Torrent has some limitations. The protocol isn’t particularly efficient for sharing small files. In fact, in some cases the overhead involved may make it unsuitable as compared to the relatively simpler old warhorse, FTP. For a downloader to derive adequate benefit from the technology, the size of the swarm must be appreciable. For this reason, BT mostly suits popular and recent files, while the older files tend to become unavailable as and when users lose interest in them. The procedure for publishing file(s) is somewhat complex in comparison to other file sharing technologies like FTP.
AUTHOR TUSHAR MALHOTRA The author has a keen interest in IT. He can be contacted at [email protected].
www.itmagz.com | ‘i.t.’
DECEMBER 2005
49
CMYK
