FriendFeed, which was recently acquired by Facebook, just released an interesting piece of open source software.
Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google’s webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking infrastructure.
The story so far
This release generated widespread interest among the Python and open source development communities. Rightfully so. There are many reasons to like Tornado. To begin with, it’s fast — and that’s fundamental for a web server. By using nginx as a load balancer and a static file server, and running a few Tornado instances (usually one per core available on the machine) it’s possible to handle thousands upon thousands of concurrent connections on relatively modest hardware; and this isn’t just theory. Tornado has already proven its worth in the field, by allowing FriendFeed to scale graciously.
Tornado is not only a fast web server, it acts as a very lightweight application framework as well. As such, it’s an appealing alternative to well established frameworks to the growing group of developers who’d like to develop “closer to the metal” and avoid the baggage associated with full-fledged web frameworks. The two things combined make Tornado ideal for developing “real time” web services and applications.
The feedback so far hasn’t been all positive though. Criticism of the project has mainly focused on the lack of test coverage and the fact that FriendFeed has opted not to contribute to, and improve on, the existing Twisted Web project (which has similar goals). To make things worse, there were a few nonchalant comments about it as well. Performance issues and lack of ease of use were the reported motivations for starting a new project from scratch.
Dustin Sallings started working on a hybrid solution (henceforth Tornado on Twisted) that would reportedly keep the good parts that Tornado introduced, while using Twisted as its core for networking and HTTP parsing.
At this point I became naturally curious about the speed of these three web servers. Is Tornado really faster than Twisted Web? And what about Tornado on Twisted, would it be faster or slower? Let’s find out.
Benchmark results
I ran a simple Hello World app for all three web servers. All the web servers were run in standalone mode without a load balancer. I stress tested the web servers with httperf using a progressively larger amount of concurrent requests. 100,000 requests were generated for each test. The web servers were run on a desktop machine with an Intel® Core™2 Quad Processor Q6600 (8M Cache, 2.40 GHz, 1066 MHz FSB) processor and 8GB of RAM. The operating system of choice was Ubuntu 9.04 (x86_64).
Without further ado, here are the results:
As you can see Tornado turned out to be faster than the rest of the Python web servers. Handling a peak of almost 3900 req/s with a single front-end and on commodity hardware is nothing to sneer at.
Twisted Web didn’t do too bad either (max. 2703.7 req/s), but the difference in performance is noticeable. Likewise, the performance of Tornado on Twisted was virtually identical to that of Twisted Web.
There you have it. I was curious about the possible outcome and now I know. Remember, this is a report on the numbers I got on my machine, not a research paper. But I hope that you find them interesting nevertheless.
Show me the code
Tornado:
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import logging
from tornado.options import define, options
define("port", default=8888, help="run on the given port", type=int)
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world!")
def main():
tornado.options.parse_command_line()
application = tornado.web.Application([
(r"/", MainHandler),
])
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
Twisted Web:
from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
from twisted.web import server, resource
class Simple(resource.Resource):
isLeaf = True
def render_GET(self, request):
return "Hello, world!"
site = server.Site(Simple())
reactor.listenTCP(8888, site)
reactor.run()
Tornado on Twisted:
from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
import tornado.options
import tornado.twister
import tornado.web
import logging
from tornado.options import define, options
define("port", default=8888, help="run on the given port", type=int)
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world!")
def main():
tornado.options.parse_command_line()
application = tornado.web.Application([
(r"/", MainHandler),
])
site = tornado.twister.TornadoSite(application)
reactor.listenTCP(options.port, site)
reactor.run()
if __name__ == "__main__":
main()
UPDATE (September 14, 2009):
- The original version of this post included Unicorn as well. This wasn’t fair however, since it’s not an asynchronous web server.
- EventMachine HTTP Server was added, but I have since decided to remove it as I prefer to let the article be a fair comparison between asynchronous Python web servers.
- I initially used Apache Benchmark (ab). The results were misleading at best. I re-ran the tests with httperf and updated the results above.
- Stock Tornado couldn’t be tested with httperf because their HTTP Server doesn’t implement getClientIP(). I had to manually modify a method to return the remote ip address. This may introduce a very minimal advantage for Tornado, but it should be negligible in this context.
- I modified the examples for Twisted and Tornado on Twisted, to ensure that both took advantage of the epoll-based reactor.
Get more stuff like this
Subscribe to my mailing list to receive similar updates about programming.
Thank you for subscribing. Please check your email to confirm your subscription.
Something went wrong.
Nice stats, thank you..
Thin would probably still have been the more interesting thing to include here since Unicorn is not async (or is it?) and Thin is built on EM, which is often compared to Twisted.
Thin doesn’t take advantage of async as much as it could, though, either. It processes requests serially and synchronously — you can’t truly have two concurrent requests, which is probably the numbers looked bad. I’d like to see someone take Thin, break the Rack synchronous call style (keeping everything else from Rack), and build a light framework on top of that. This has been ripe for a while now.
Unicorn is not remotely async, it’s very explicitly one full-fledged UNIX
process per client. It’s intended for very different use cases than
Tornado or Twisted (or even Thin).
See the philosophy and design documents here:
http://unicorn.bogomips.org/PHILOSOPHY.html
http://unicorn.bogomips.org/DESIGN.html
P.S. I’m the primary author of Unicorn
Eric, I’m familiar with those documents. I needed a fast Ruby web server for comparison, so I used Unicorn. I have since removed it, as EventMachine HTTP Server provides a more than adequate comparison with the asynchronous web servers from the Python world.
Since Twisted supports more than one reactor, which reactor did you test? The default select reactor will not be as good as the epoll one, etc.
I often forget to change this and get a huge performance boost by changing one commandline option.
Jack, you just addressed one of Twisted issues for me: Code Usability.
Why by default doesn’t it use the best way of doing it? And if it isn’t possible in any way, why isn’t it stated very very very very clear everywhere? 😉
Jack, I used the default reactor. Using the epollreactor did not improve performance.
Hi.
Your post compares a C++ event framework wrapped on a thin Ruby layer with pure Python frameworks. It would have been nice of you to make that clear on the title or the introduction.
Cheers.
Alecco
Could you post the code/scripts you used for the benchmark? The EM results seem strange compared to the others.
http://github.com/facebook/tornado/blob/8ca616088cfb26ff19fcc6f359d654fef905b8da/tornado/epoll.c
Tornado doesn’t seem too “pure Python” to me =)
I agree with Alecco. What possessed you to add EM to this comparison? It only obfuscates the useful comparison you set out to make.
While it’s great to see EM performing so well, I wonder how much of the HTTP spec does it implement? It’s easy to write code that implements just enough of the spec to run the benchmark and have it blow away general purpose servers.
I’m not remotely surprised that tornado on twisted is slower than plain tornado — the code I pushed was the least code possible to get apps to run without API changes. In order to do that, I ended up wrapping existing code (i.e. adding indirection) to get it to go. There’s more work to go.
I do appreciate the benchmark, though. It gives us a starting point. 🙂
@labria
> Tornado doesn’t seem too “pure Python” to me =)
Sure, bad wording from me. Python doesn’t have straightforward calls to system calls. Both Twisted and Tornado seem to have have very tiny C modules to do handle the system call to the OS. But the main code is Python.
EventMachine seems a full blown asynchronous framework in C++.
http://github.com/eventmachine/eventmachine/tree/1250ffcd62bf94be2dbd9060d86d7556e901c20c/ext
Also, in my experience, EM raises some warning flags since it doesn’t use Boost/Asio.
@Ryan, Eric: I removed Unicorn from the benchmark.
@Alecco: The language used to implement an event-based framework that is used by a given web server doesn’t really matter (BTW, 61% of EM’s codebase is written in Ruby). What counts is whether a web server fully implements the HTTP spec and whether it can easily be used along with the language of choice (e.g., Ruby or Python).
@labria: I added the source code for each program.
@Patrick: I added a Python-only section with a chart that doesn’t include EM.
@Dan: I’m not sure. What I do know is that EM is a project that has potential and the Ruby community should pay attention to it.
@All: Thank you for all the feedback received so far. I’m evolving this post by incorporating valid suggestions. Keep them coming. 🙂
@Antonio
> The language used to implement an event-based framework that
> is used by a given web server doesn’t really matter (BTW, 61% of
> EM’s codebase is written in Ruby). What counts is whether a web
> server fully implements the HTTP spec and whether it can easily be
> used along with the language of choice (e.g., Ruby or Python).
I didn’t say your test or analysis was invalid or pointless. Just asked to make it clear it’s apples and oranges, or perhaps oranges and mandarins. My exact phrasing:
>> “It would have been nice of you to make that clear on the title or the introduction.”
@Folletto:
It’s a goal, but it’s not as simple as you might think. If you’d like to help, see
http://twistedmatrix.com/trac/ticket/2234
@Antonio:
Personally, in the past I’ve had difficulties with the ApacheBench tool. Have you tried other HTTP benchmarking tools, like httperf? Also, I don’t know if any existing benchmarking tools do this, but it would be nice to see a scatter plot of how long each server takes to complete each request, rather than condensing everything down into a single number. For those of us interested in making some of these servers faster, that would produce a lot more interesting output, and perhaps illuminate some distinct bottlenecks :).
Thanks for stopping by Glyph. No matter how trivial the benchmark, I’m interested in its accuracy. If ab cannot be trusted, I’ll run the tests with httperf or siege and update my results.
httperf is a better tool if you really care about useful statistics.
@Bob: Indeed. I’m re-running all the benchmarks with httperf and will update this post very soon. Stay tuned.
Anotonio, thanks for the benchmark. But something is just not right here. There is no way one “framework” can beat another by so much. When it comes to massively parallel asynchronous I/O the bulk of the work is done by the kernel i.e. epoll() on Linux.
There are really two things going on:
a) I/O implementation (all of it is in C – for all your contenders)
b) Data marshaling b/w Ruby/Python and C
I just don’t see how one framework can be more performant than another. The differences you’re seeing can only be attributed to:
a) Richness of implementation, i.e. do all these “Hello world” apps do full HTTP headers parsing and forming “request objects”? How about initializing/maintaining other aspects of maintaining a request state? I doubt we’re looking at oranges-to-oranges here.
b) Error of configuration: did you make sure all of them rely on epoll()? This isn’t as trivial as it may appear since I/O libraries have a habit of falling back to slower select() or poll() silently.
@antonio Thanks for this very interesting benchmark. I see that you have pulled down the EventMachine chart. I am aware that this post is focused on evented python servers. However, EventMachine and Twisted Python are both Evented framework for respective language. look forward more content on EventMachine since that EM server is pushing out some really nice numbers, perhaps a seperate post ?
@Eugeny: The updated post ensures that all three web servers use epoll().
@Taylor: Yes, that’s a good idea.
What command-line options did you use with httperf? I’d like to try this test against various Ruby async http servers.
what? I wanted to see EM benchmarks… could you make another post comparing EM with Tornado please?
is cogen competitive?