Twisted as a Framework for BitTorrent Herd

Alice Lin University of California, Los Angeles

[email protected]

ABSTRACT
As upcoming developers learn the trade, many alliances are made with certain software or programs. Emacs and vim as text editors, stacks and queues in data representation, and object oriented and functional as the programming language of choice are all examples of what programmers have to learn and encompass. Some of these items are strictly a matter of preference - for example, a text editor will always be a text editor, and even though I am an emacs buff, there is not much for me to say if someone says that vim can do exactly the same thing emacs does. However, in other areas such as how to represent data and which language to choose, it is much more important to choose whichever data type, language, or other choice that best suits the end product trying to be made, rather than personal preference. Apache is the most widely used web server software on the web, so it seems like an obvious choice when looking for a framework for applications. However, the ubiquitous Apache does have drawbacks when looking to creating applications that require distinct circumstances. In this paper, the benefits and drawbacks of a competitor framework, Twisted, are examined and compared to Apache. In particular, by seeing how each framework holds up to an application that would be integrated with a BitTorrent client, it'll be shown that different software shines under different specifications.
websites. In short, the LAMP software and Apache servers are quite the mammoth of servers.

Introduction
Apache and Twisted are two different, yet equally effective platforms to design application servers on. This paper investigates the pros and cons of using each framework, and aims to show the reader a better understanding of why certain frameworks are chosen to solve specific problems.

Apache and Wikipedia

Wikipedia is, essentially, a giant database that stores information in a few giant servers, and has a network of more servers so users may obtain or edit the information. It is scaled well by separating read and write options, and also compresses edits to wikipedia articles efficiently, up to a ratio of 100x. Wikipedia contains around 55 servers, and has around 1000 HTTP requests per second. At any point in time, there are between fourteen and thirty-two thousand open connections per server. When looking to create Wikipedia, developers needed consistency and security. Thus, the LAMP platform - and subsequently Apache software - was the perfect choice.

Lamp and Apache

A popular software platform is LAMP, which is based on the Linux OS, Apache Server, MYSQL for database software, and PHP. Used in large projects such as Wikimedia architecture, it is best well suited for the creation of application servers that may or may not communicate with other servers, as well as clients. The component handling the client-server side communication, Apache, is extremely well known and professional, in the sense that it has multiple large sponsors such as Google, Microsoft, Yahoo, etc. backing it up. Forums galore exist, and example code as well as in-depth tutorials are just one internet search way. It does not hurt that it is the most popular HTTP server software, serving an estimated 63% of all

Twisted
Twisted is an event-driven networking framework that is written in Python. While many applications have been created from the Twisted

framework, it has not yet handled anything as extensive as Wikipedia. Being less popular, in the strict sense of how many users use and develop with Twisted, a weaker and less extensive documentation exists for its API. However, this weakness is countered by its strong community, as there exists an IRC channel with a small yet enthusiastic group of advocates and developers for Twisted. While chatting with some of the users, the general consensus was that Twisted was simple to code in and easy to pick up. Indeed, I experienced the described phenomenon when creating the prototype of a server that interacted with Twitter. During my research for the prototype application, an estimated 75% of my total coding time was parsing the AT and WHATSAT commands, as well as handling errors and creating the log. Another 20% was spent reading up and understanding a basic concept of the server, as I had no prior networking experience. After the initial hump of realizing what it actually meant to create a server, I spent a mere 5% of my time understanding the few lines of code it took to create a server, and used it in my own code. By reusing a few lines of code I had just finished understanding, the creation of the server was finished and I no longer needed to worry about handling client connections and other derivatives of networking. In using myself as a test subject, it can be seen that Twisted is friendly to pick up as a developer. In addition to Twisteds ease in programming, having it written in Python results in picking up the pros of writing in Python, which thus eliminates a lot of low level coding that developers would otherwise have to deal with. Dynamic type-checking, for example, is built in, and because of the reference-counting style garbage collector Python has, memory allocation and management is already taken care of. Because of this, developers are allowed to spend a larger portion of their time on the idea of their application, instead of coding. My finished prototype, a functional application server that could take multiple clients and parse input, read out error, and create a log of the servers activity while the server was up, was a mere 150 lines of code, including commenting for clarity. I can only imagine how long it would take if I wanted the same program in C++. Python is safe, quick, and intuitive to program in. However, the drawbacks of coding in Python must be addressed as well dynamic type checking can lead to strange and unintended errors that may not be handled. In addition, an automatic memory manager allows for less flexibility, since all memory will always be allocated in Pythons memory manager without any input from the developer.

With Apache being nearly omnipresent and Twisted as the underground underdog, it seems logical for the everyday developer to choose Apache over Twisted when thinking of creating an application. As stated earlier, I experienced minimal problems creating a networking system using Twisted. Nevertheless, Apache may be at the same level of ease to pick up, with a similarly large, and surely enthusiastic, support group. Apache and Twisted have several underlying differences that set them apart from each other. Apache, for instance, abides more toward the traditional clientserver model to pass information when a file wants to be transferred from one place to the next, the request is received and the file is sent. It does this request extremely well, and is one of the reasons Apache was a great choice for creating Wikimedia architecture. On the other hand, Twisted is event-driven, meaning hightraffic requests are handled well. In addition, Apache is not meant to handle other protocols beside the HTTP protocol, while Twisted has this functionality. A look at Figure 1.1 shows a screen shot of only a portion of the protocols supported in Twisted the complete list is slightly longer.

Figure 1.1

Comparisons between Apache and Twisted

BitTorrent and Deluge

BitTorrent is a peer-to-peer file sharing protocol. Unlike traditional client-server models of transferring one large file from one place to the next, BitTorrent splits up the file into many smaller files. These smaller files are then transferred simultaneously through peers and seeds that own the requested file. BitTorrent clients assist in this transfer by giving users an interface and controlling details on the process such as speed, location of seeds, and tracker details. The client needs to be updated constantly in order for downloads to be optimally quick, as seeds may appear or disappear, and trackers may occasionally be invalidated. In comparison with the Wikimedia architecture mentioned earlier, updates happen many times faster per second. Any application created to work in parallel with it would need the functionality of being able to handle updates quickly, as any BitTorrent client would. Deluge, the BitTorrent client in question, is a client that is written in Python and GTK+. In determining what framework should be used for the specification, it should be taken into account that both Deluge and Twisted are written in Python. Due to this common feature, integrating an application written in Twisted with Deluge would be much simpler and much more natural than if an application were written in the Apache software to be integrated with Deluge.

Acknowledgements I would like to thank UCLA and Paul Eggert for enabling me to do this project and introducing me to networking software. I would also like to thank Jerry Weng for clarifying the project specification, and Lucas Ballard for his explanations and directing me towards tutorials and references. Author Information Alice Lin is currently working towards a B.A. in Computer Science at University of California, Los Angeles. She is currently in her third year and is interested in pursuing game development. References [1] Apache HTTP Server Project, http://httpd.apache.org/ [2] About Deluge, http://dev.deluge-torrent.org/wiki/About [3] Wikimedia Architecture, http://www.nedworks.org/~mark/presentations/san /Wikimedia%20architecture.pdf [4] What is A BitTorrent Tracker?, http://compnetworking.about.com/od/BitTorrent/f/ bttracker.htm [5] Twisted Code Examples, http://twistedmatrix.com/documents/current/core/e xamples/ Additional References Used During Prototype Research and Coding [6] Twitter API, https://dev.twitter.com/docs/api [7] Time Access and Conversions, http://docs.python.org/library/time.html [8] Martelli, Alex. Python in a Nutshell. Sebastopol, CA: O'Reilly, 2003..

Conclusion
The project asks for an opinion on the optimal framework to use when creating applications that would integrate with Deluge. Several features from the spec stand out when looking to see which framework to use. Firstly, various protocols would like to be supported and users may tend to be mobile, a feature that Apache does not have but Twisted does. This in itself could lead a developer to choose Twisted over Apache. Secondly, as a variety of applications would like to be potentially created, the ease of developing needed to be taken into account both Apache and Twisted have straight-forward and simple environments to develop in, with strong communities backing them up and sufficient example code to help beginner programmers. Lastly, and most importantly, high-traffic requests would occur due to BitTorrents nature. Twisted clearly wins out in this matter designing an HTTP server to integrate with a BitTorrent client could result in several magnifications of slowdown, which would not be ideal for users. Thus, in this regard, Twisted is the clear winner when choosing a framework, Twisted vs. Apache, to use for creating applications that go hand in hand with Deluge.