Meditations on programming, startups, and technology
New Relic

Reflections on the Ruby shootout

Yesterday I published The Great Ruby Shootout and it quickly gathered a fair deal of attention. It was on the front page of Slashdot, Hacker News, Reddit, and so on. More than 15,000 people came by to read about the results of my comparison between Ruby implementations.

Those numbers looked good but something didn’t add up. Ever since I clicked the “Publish” button, I had a very uneasy feeling about the main shootout figures. They just didn’t seem right. I had a chance, particularly during the writing of my book, to extensively use Ruby on Vista and I can guarantee you that it’s visibly slower than on GNU/Linux. The Phusion team had benchmarked their Ruby Enterprise Edition against Ruby 1.8.6 many times, and found it to be about 25% faster. Yet my results were showing it as twice as fast than Ruby 1.8.7, which in turn is already faster than 1.8.6. To makes things worse, I’ve used Ruby 1.9 and found it to be faster than Ruby 1.8.7, but not 5 times as fast. For most programs that I tried Rubinius didn’t seem faster than Ruby 1.8. And the more I pondered it, the more it began to feel like one too many things didn’t add up.

In the comments, Isaac Gouy reported a couple of issues with the Excel formulas, where a few unsuccessful tests were mistakenly added to the totals. This skewed the results slightly, particularly in terms of penalizing JRuby. However, this wasn’t really it. Sure, the totals were inaccurate, but not enough to fundamentally change the main outcome of those results.

As I was discussing this somewhat unexpected result with Hongli Lai (co-author of Ruby Enterprise Edition), he mentioned that he knew what might be causing this anomaly. I had run the initial test against Ruby installed through apt-get, because I’d made a couple of assumptions. The first was that most people would probably be using the Ruby version that was deployed by their OS’ packaging system in both development and production mode. The second was that the performance of this version would be roughly similar to the one built from scratch. This second assumption would turn out to be highly mistaken.

I decided to run a test using Ruby 1.8.7 built from source as the baseline and added a column for Ruby 1.8.7, installed through apt-get, to the tables. In addition I also corrected the issue pointed out by Isaac. I updated the original shootout with the correct data, and what you see below is a bar chart for the geometric mean of the ratios for the successful benchmarks.

Geometric mean bar chart


Notice how everything makes much more sense now. Ruby 1.9 and JRuby are very close, respectively 2.5 and 1.9 faster than Ruby 1.8.7 (from source) on these benchmarks. Less impressive result sure, but I suspect much more realistic. The results for Ruby Enterprise Edition are in line with the 25% speed increase, if we consider that 1.8.7 is a bit faster than 1.8.6. Rubinius is still slower than MRI for most tests, but it’s improving. Ruby on Windows is slow. So slow in fact, that Ruby on GNU/Linux is twice as fast.

The really big, flashing warning though is what happens when you install Ruby through apt-get. Compiling from source gives you double the speed, according to these tests. I expected a 10/20% increase, not 100%. The gist of it is that prepackaged Ruby is compiled using the option –enable-pthreads and there is the whole issue of shared vs static libraries. But whatever the reason, this is a significant difference. For production use, in light of these results, I feel that it would be foolish to use the slower version of Ruby provided by apt-get/aptitude.

I rectified the results as soon as possible because the last thing I wanted was to mislead the Ruby community or worse still, betray its trust. Major kudos to Isaac for spotting the calculation issue, and Hongli for selflessly pointing out that the excellent Ruby Enterprise Edition results were probably due to the low performance of the Ubuntu’s version of Ruby.


If you enjoyed this post, then make sure you subscribe to my Newsletter and/or Feed.

receive my posts by email

20 Responses to “Reflections on the Ruby shootout”

  1. Phil says:

    You should also compare a couple different distros if you’re going to put the apt-get results there. Each one might be different…you should include centos, Opensuse (my fave) and possibly redhat, but it should be similar to CentOS

  2. Jan says:

    interesting result. dunno how much is the difference between source install and pacman install on archlinux.

  3. This may also be a function of which distribution you use. For instance, in the C-Ruby packages we add to OpenSolaris through the Web Stack Project, there is some effort to test and be sure they’re optimized out of the box. It’s a lot of work, but worthwhile to give users the best binaries we know how to (reliably) make.

  4. Gábor Farkas says:

    it would be interesting to find out what exactly is causing the performance-problems with the apt-get version of ruby, and work together with the ubuntu and/or debian developers to fix the problem…

    also, this is probably obvious, but because you didn’t mention it in the article, imho it’s important to mention it:

    while it might be foolish to use a slower version of ruby, when a faster one is also available, the slower (apt-get) version has certain advantages too, most notably security-updates…

  5. mark says:

    No huge surprise, I mean ….

    Debian is so slow that it slows down everything even more. ;)

  6. hosiawak says:

    That’s why source based ditributions like Gentoo rule, you HAVE to choose exactly what you want and not just assume Debian maintainers could read your mind :P

  7. Florian Ebeling says:

    Do you have an idea how pthread support influences performance of Ruby 1.9?

  8. @Phil: Each distro will probably yield different results. Testing a few of them can be the subject of a different post, but it goes beyond the scope of the original shootout.

    @Matt: Thanks for stopping by. This is definitely distro-based and it’s nice to hear that your team cares about performances.

    @Gábor: There are many advantages provided by apt-get. However on Ubuntu, these benefits are pretty much trumped by a twice as slow intepreter.

  9. planetmcd says:

    Thanks for the updated figures. The comparison is very interesting.

    I do think your premise, that people will use the packaged binaries, holds true for most cases. IF it seems to work fine, that’s what people will use. That’s of course different than what they should use, but including the apt-get version in the comparison is not a bad idea.

  10. mbailey says:

    “The gist of it is that prepackaged Ruby is compiled using the option –enable-pthreads and there is the whole issue of shared vs static libraries.”

    Your article said you did a simple ./configure && make && sudo make install.
    The default configure script enables POSIX Threads, and will use shared libs by default, just like the deb. I’m a bit unclear on what the difference between the binaries are.

    -mb

  11. Randy Parker says:

    On Feb 8 2008 Matz posted to Hongli’s blog asking if he could incorporate Hongli’s GC patch into Ruby 1.9. Assuming 1.9.1 now uses a mark-table, does that mean REE and MRI will merge after 1.9.1?

  12. mbailey says:

    Hmm. Now I get it:

    compiled_ruby-1.8.6p287 is what we use in production, and it is build from scratch (simple ./configure && make && make install)
    compiled_ruby-1.8.7p72 is the latest stable version compiled in the same way (./configure && make)
    deb_ruby-1.8.7p72 is the binary from ubuntu 8.10
    rpm_ruby-1.8.5p0 is the latest version in CentOS 5.2 yum repo
    

    $ ls -shw1
    total 5.3M
    2.6M compiled_ruby-1.8.6p287
    8.0K compiled_ruby-1.8.7p72
    2.7M deb_ruby-1.8.7p72
    12K rpm_ruby-1.8.5p0
    
    $ for i in *; do echo -n "$i: "; strings $i | wc -l ; done
    compiled_ruby-1.8.6p287: 7533
    compiled_ruby-1.8.7p72: 27
    deb_ruby-1.8.7p72: 7642
    rpm_ruby-1.8.5p0: 33
    
    $ for i in *; do echo "$i: "; ldd $i ; echo "" ; done
    compiled_ruby-1.8.6p287: 
    	linux-vdso.so.1 =>  (0x00007fff569fe000)
    	libdl.so.2 => /lib/libdl.so.2 (0x00007fe54e58e000)
    	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fe54e356000)
    	libm.so.6 => /lib/libm.so.6 (0x00007fe54e0d1000)
    	libc.so.6 => /lib/libc.so.6 (0x00007fe54dd5f000)
    	/lib64/ld-linux-x86-64.so.2 (0x00007fe54e792000)
    
    compiled_ruby-1.8.7p72: 
    	linux-vdso.so.1 =>  (0x00007fffc5fff000)
    	libruby1.8.so.1.8 => /usr/lib/libruby1.8.so.1.8 (0x00007f3ebd9e5000)
    	libpthread.so.0 => /lib/libpthread.so.0 (0x00007f3ebd7c9000)
    	libdl.so.2 => /lib/libdl.so.2 (0x00007f3ebd5c5000)
    	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007f3ebd38d000)
    	libm.so.6 => /lib/libm.so.6 (0x00007f3ebd108000)
    	libc.so.6 => /lib/libc.so.6 (0x00007f3ebcd96000)
    	/lib64/ld-linux-x86-64.so.2 (0x00007f3ebdceb000)
    
    deb_ruby-1.8.7p72: 
    	linux-vdso.so.1 =>  (0x00007fff309fe000)
    	libdl.so.2 => /lib/libdl.so.2 (0x00007ffd2840a000)
    	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007ffd281d2000)
    	libm.so.6 => /lib/libm.so.6 (0x00007ffd27f4d000)
    	libc.so.6 => /lib/libc.so.6 (0x00007ffd27bdb000)
    	/lib64/ld-linux-x86-64.so.2 (0x00007ffd2860e000)
    
    rpm_ruby-1.8.5p0: 
    	linux-vdso.so.1 =>  (0x00007fffa8bfe000)
    	libruby.so.1.8 => not found
    	libpthread.so.0 => /lib/libpthread.so.0 (0x00007fa7a063f000)
    	libdl.so.2 => /lib/libdl.so.2 (0x00007fa7a043b000)
    	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fa7a0203000)
    	libm.so.6 => /lib/libm.so.6 (0x00007fa79ff7e000)
    	libc.so.6 => /lib/libc.so.6 (0x00007fa79fc0c000)
    	/lib64/ld-linux-x86-64.so.2 (0x00007fa7a085b000)
    
  13. Isaac Gouy says:

    antonio > Ruby 1.9 and JRuby are very close, respectively 2.5 and 1.9 faster than Ruby 1.8.7 (from source) on these benchmarks.

    1) similar to the benchmarks game ruby 1.9.0 :: ruby 1.8.6

    http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=yarv&lang2=ruby

    2) similar to the benchmarks game jruby 1.1.5 :: ruby 1.8.7

    http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=jruby&lang2=ruby

    3) Unlike the Ruby Shootout, the benchmarks game /does/ include startup time and /does/ include the time required to parse and compile classes and method for the first time.

  14. Isaac Gouy says:

    Rubinius shows “Timeout” for several benchmarks – are measurements for the other implementations on those benchmarks where Rubinius shows “Timeout” included in the “Geometric Mean” headline chart?

  15. Isaac, they are not. Only the 101 benchmarks that were successful for all the VMs (in blue within the tables) contribute to the totals.

  16. Isaac Gouy says:

    In that case, let me suggest that you re-measure just those “Timeout” benchmarks, just for Rubinius, using a wildly high cutoff.

    As a practical matter that shouldn’t be too bad – back in the day Rebol nbody program needed 4 hours to finish at the largest input size, but Rubinius should be much faster on the smaller input sizes used in Ruby Shootout.

    It’s such a shame to exclude benchmarks that actually require a tiny amount of work because of Rubinius.

  17. Sam Livingston-Gray says:

    Interesting. With that adjustment, then, it looks as though MagLev is then comparable to 1.8.7 in performance (for the subset of benchmarks it’s able to complete)?

  18. No Sam, MagLev was tested against Ruby 1.8.6 built from source. The only VMs affected were those within the “main shootout”.

  19. roger pack says:

    You should totally test Lavena’s new “mingw for windows” build, in these tests. It helps windows be a faster :)
    How to get:
    git clone http://github.com/luislavena/rubyinstaller/tree/master
    then
    rake # create a 186p286 build
    or
    rake CHECKOUT=1 # to create a 186 SVN trunk build
    these will create mingw build in sandbox/ruby_mingw
    for use

    Also it would be interesting to see the RAM usage per test [as the alioth benchmark shows, along with speed].

    Some other comments:
    testing against 186 [linux] would also be interesting.

    Thanks for doing this!

  20. [...] If you are in the Ruby business (which probably means “in the Ruby on Rails business” nowadays) sooner or later you’ll have to play around with different Ruby versions on the same machine as you might run into crashing ruby processes or performance issues . At least you’ll notice that running the standard Debian/Ubuntu Ruby versions might get you into serious trouble as it is several times slower than a manually compiled version (for reference see this launchpad bug and this blog entry.). [...]

Leave a Reply

I sincerely welcome and appreciate your comments, whether in agreement or dissenting with my article. However, trolling will not be tolerated. Comments are automatically closed 15 days after the publication of each article.

Current ye@r *

Copyright © 2005-2014 Antonio Cangiano. All rights reserved.