Home
Programming
Reflections on the Ruby shootout
Programming

Reflections on the Ruby shootout

20 Comments

Yesterday I published The Great Ruby Shootout and it quickly gathered a fair deal of attention. It was on the front page of Slashdot, Hacker News, Reddit, and so on. More than 15,000 people came by to read about the results of my comparison between Ruby implementations.

Those numbers looked good but something didn’t add up. Ever since I clicked the “Publish” button, I had a very uneasy feeling about the main shootout figures. They just didn’t seem right. I had a chance, particularly during the writing of my book, to extensively use Ruby on Vista and I can guarantee you that it’s visibly slower than on GNU/Linux. The Phusion team had benchmarked their Ruby Enterprise Edition against Ruby 1.8.6 many times, and found it to be about 25% faster. Yet my results were showing it as twice as fast than Ruby 1.8.7, which in turn is already faster than 1.8.6. To makes things worse, I’ve used Ruby 1.9 and found it to be faster than Ruby 1.8.7, but not 5 times as fast. For most programs that I tried Rubinius didn’t seem faster than Ruby 1.8. And the more I pondered it, the more it began to feel like one too many things didn’t add up.

In the comments, Isaac Gouy reported a couple of issues with the Excel formulas, where a few unsuccessful tests were mistakenly added to the totals. This skewed the results slightly, particularly in terms of penalizing JRuby. However, this wasn’t really it. Sure, the totals were inaccurate, but not enough to fundamentally change the main outcome of those results.

As I was discussing this somewhat unexpected result with Hongli Lai (co-author of Ruby Enterprise Edition), he mentioned that he knew what might be causing this anomaly. I had run the initial test against Ruby installed through apt-get, because I’d made a couple of assumptions. The first was that most people would probably be using the Ruby version that was deployed by their OS’ packaging system in both development and production mode. The second was that the performance of this version would be roughly similar to the one built from scratch. This second assumption would turn out to be highly mistaken.

I decided to run a test using Ruby 1.8.7 built from source as the baseline and added a column for Ruby 1.8.7, installed through apt-get, to the tables. In addition I also corrected the issue pointed out by Isaac. I updated the original shootout with the correct data, and what you see below is a bar chart for the geometric mean of the ratios for the successful benchmarks.

Geometric mean bar chart

Notice how everything makes much more sense now. Ruby 1.9 and JRuby are very close, respectively 2.5 and 1.9 faster than Ruby 1.8.7 (from source) on these benchmarks. Less impressive result sure, but I suspect much more realistic. The results for Ruby Enterprise Edition are in line with the 25% speed increase, if we consider that 1.8.7 is a bit faster than 1.8.6. Rubinius is still slower than MRI for most tests, but it’s improving. Ruby on Windows is slow. So slow in fact, that Ruby on GNU/Linux is twice as fast.

The really big, flashing warning though is what happens when you install Ruby through apt-get. Compiling from source gives you double the speed, according to these tests. I expected a 10/20% increase, not 100%. The gist of it is that prepackaged Ruby is compiled using the option –enable-pthreads and there is the whole issue of shared vs static libraries. But whatever the reason, this is a significant difference. For production use, in light of these results, I feel that it would be foolish to use the slower version of Ruby provided by apt-get/aptitude.

I rectified the results as soon as possible because the last thing I wanted was to mislead the Ruby community or worse still, betray its trust. Major kudos to Isaac for spotting the calculation issue, and Hongli for selflessly pointing out that the excellent Ruby Enterprise Edition results were probably due to the low performance of the Ubuntu’s version of Ruby.

Get more stuff like this

Subscribe to my mailing list to receive similar updates about programming.

Thank you for subscribing. Please check your email to confirm your subscription.

Something went wrong.

I respect your privacy and take protecting it seriously. Feel free to review my privacy policy page for further details.

About The Author

Antonio Cangiano
More from this Author

Antonio Cangiano is an Engineering Manager and AI Specialist at IBM. He authored Ruby on Rails for Microsoft Developers (Wrox, 2009) and Technical Blogging (The Pragmatic Bookshelf, 2012, 2019). He is also the Marketing Lead for Cognitive Class, an educational initiative which he helped grow from zero to over 3 Million students. You can follow him on Twitter.

20 Comments

  1. Phil December 11, 2008

    You should also compare a couple different distros if you’re going to put the apt-get results there. Each one might be different…you should include centos, Opensuse (my fave) and possibly redhat, but it should be similar to CentOS

    Reply
  2. Jan December 11, 2008

    interesting result. dunno how much is the difference between source install and pacman install on archlinux.

    Reply
  3. Matt Ingenthron December 11, 2008

    This may also be a function of which distribution you use. For instance, in the C-Ruby packages we add to OpenSolaris through the Web Stack Project, there is some effort to test and be sure they’re optimized out of the box. It’s a lot of work, but worthwhile to give users the best binaries we know how to (reliably) make.

    Reply
  4. Gábor Farkas December 11, 2008

    it would be interesting to find out what exactly is causing the performance-problems with the apt-get version of ruby, and work together with the ubuntu and/or debian developers to fix the problem…

    also, this is probably obvious, but because you didn’t mention it in the article, imho it’s important to mention it:

    while it might be foolish to use a slower version of ruby, when a faster one is also available, the slower (apt-get) version has certain advantages too, most notably security-updates…

    Reply
  5. mark December 11, 2008

    No huge surprise, I mean ….

    Debian is so slow that it slows down everything even more. 😉

    Reply
  6. hosiawak December 11, 2008

    That’s why source based ditributions like Gentoo rule, you HAVE to choose exactly what you want and not just assume Debian maintainers could read your mind 😛

    Reply
  7. Florian Ebeling December 11, 2008

    Do you have an idea how pthread support influences performance of Ruby 1.9?

    Reply
  8. Antonio Cangiano December 11, 2008

    @Phil: Each distro will probably yield different results. Testing a few of them can be the subject of a different post, but it goes beyond the scope of the original shootout.

    @Matt: Thanks for stopping by. This is definitely distro-based and it’s nice to hear that your team cares about performances.

    @Gábor: There are many advantages provided by apt-get. However on Ubuntu, these benefits are pretty much trumped by a twice as slow intepreter.

    Reply
  9. planetmcd December 11, 2008

    Thanks for the updated figures. The comparison is very interesting.

    I do think your premise, that people will use the packaged binaries, holds true for most cases. IF it seems to work fine, that’s what people will use. That’s of course different than what they should use, but including the apt-get version in the comparison is not a bad idea.

    Reply
  10. mbailey December 11, 2008

    “The gist of it is that prepackaged Ruby is compiled using the option –enable-pthreads and there is the whole issue of shared vs static libraries.”

    Your article said you did a simple ./configure && make && sudo make install.
    The default configure script enables POSIX Threads, and will use shared libs by default, just like the deb. I’m a bit unclear on what the difference between the binaries are.

    -mb

    Reply
  11. Randy Parker December 11, 2008

    On Feb 8 2008 Matz posted to Hongli’s blog asking if he could incorporate Hongli’s GC patch into Ruby 1.9. Assuming 1.9.1 now uses a mark-table, does that mean REE and MRI will merge after 1.9.1?

    Reply
  12. mbailey December 11, 2008

    Hmm. Now I get it:

    compiled_ruby-1.8.6p287 is what we use in production, and it is build from scratch (simple ./configure && make && make install)
compiled_ruby-1.8.7p72 is the latest stable version compiled in the same way (./configure && make)
deb_ruby-1.8.7p72 is the binary from ubuntu 8.10
rpm_ruby-1.8.5p0 is the latest version in CentOS 5.2 yum repo


    $ ls -shw1
total 5.3M
2.6M compiled_ruby-1.8.6p287
8.0K compiled_ruby-1.8.7p72
2.7M deb_ruby-1.8.7p72
12K rpm_ruby-1.8.5p0

$ for i in *; do echo -n "$i: "; strings $i | wc -l ; done
compiled_ruby-1.8.6p287: 7533
compiled_ruby-1.8.7p72: 27
deb_ruby-1.8.7p72: 7642
rpm_ruby-1.8.5p0: 33

$ for i in *; do echo "$i: "; ldd $i ; echo "" ; done
compiled_ruby-1.8.6p287: 
	linux-vdso.so.1 =>  (0x00007fff569fe000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007fe54e58e000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fe54e356000)
	libm.so.6 => /lib/libm.so.6 (0x00007fe54e0d1000)
	libc.so.6 => /lib/libc.so.6 (0x00007fe54dd5f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe54e792000)

compiled_ruby-1.8.7p72: 
	linux-vdso.so.1 =>  (0x00007fffc5fff000)
	libruby1.8.so.1.8 => /usr/lib/libruby1.8.so.1.8 (0x00007f3ebd9e5000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00007f3ebd7c9000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f3ebd5c5000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007f3ebd38d000)
	libm.so.6 => /lib/libm.so.6 (0x00007f3ebd108000)
	libc.so.6 => /lib/libc.so.6 (0x00007f3ebcd96000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3ebdceb000)

deb_ruby-1.8.7p72: 
	linux-vdso.so.1 =>  (0x00007fff309fe000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007ffd2840a000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007ffd281d2000)
	libm.so.6 => /lib/libm.so.6 (0x00007ffd27f4d000)
	libc.so.6 => /lib/libc.so.6 (0x00007ffd27bdb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffd2860e000)

rpm_ruby-1.8.5p0: 
	linux-vdso.so.1 =>  (0x00007fffa8bfe000)
	libruby.so.1.8 => not found
	libpthread.so.0 => /lib/libpthread.so.0 (0x00007fa7a063f000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007fa7a043b000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fa7a0203000)
	libm.so.6 => /lib/libm.so.6 (0x00007fa79ff7e000)
	libc.so.6 => /lib/libc.so.6 (0x00007fa79fc0c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa7a085b000)
    Reply
  13. Isaac Gouy December 11, 2008

    antonio > Ruby 1.9 and JRuby are very close, respectively 2.5 and 1.9 faster than Ruby 1.8.7 (from source) on these benchmarks.

    1) similar to the benchmarks game ruby 1.9.0 :: ruby 1.8.6

    http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=yarv&lang2=ruby

    2) similar to the benchmarks game jruby 1.1.5 :: ruby 1.8.7

    http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=jruby&lang2=ruby

    3) Unlike the Ruby Shootout, the benchmarks game /does/ include startup time and /does/ include the time required to parse and compile classes and method for the first time.

    Reply
  14. Isaac Gouy December 11, 2008

    Rubinius shows “Timeout” for several benchmarks – are measurements for the other implementations on those benchmarks where Rubinius shows “Timeout” included in the “Geometric Mean” headline chart?

    Reply
  15. Antonio Cangiano December 11, 2008

    Isaac, they are not. Only the 101 benchmarks that were successful for all the VMs (in blue within the tables) contribute to the totals.

    Reply
  16. Isaac Gouy December 11, 2008

    In that case, let me suggest that you re-measure just those “Timeout” benchmarks, just for Rubinius, using a wildly high cutoff.

    As a practical matter that shouldn’t be too bad – back in the day Rebol nbody program needed 4 hours to finish at the largest input size, but Rubinius should be much faster on the smaller input sizes used in Ruby Shootout.

    It’s such a shame to exclude benchmarks that actually require a tiny amount of work because of Rubinius.

    Reply
  17. Sam Livingston-Gray December 12, 2008

    Interesting. With that adjustment, then, it looks as though MagLev is then comparable to 1.8.7 in performance (for the subset of benchmarks it’s able to complete)?

    Reply
  18. Antonio Cangiano December 12, 2008

    No Sam, MagLev was tested against Ruby 1.8.6 built from source. The only VMs affected were those within the “main shootout”.

    Reply
  19. roger pack December 13, 2008

    You should totally test Lavena’s new “mingw for windows” build, in these tests. It helps windows be a faster 🙂
    How to get:
    git clone http://github.com/luislavena/rubyinstaller/tree/master
    then
    rake # create a 186p286 build
    or
    rake CHECKOUT=1 # to create a 186 SVN trunk build
    these will create mingw build in sandbox/ruby_mingw
    for use

    Also it would be interesting to see the RAM usage per test [as the alioth benchmark shows, along with speed].

    Some other comments:
    testing against 186 [linux] would also be interesting.

    Thanks for doing this!

    Reply
  20. Brian Takita December 23, 2008

    I filed a bug report on Ubuntu’s bug tracking site.

    https://bugs.launchpad.net/ubuntu/+source/ruby-defaults/+bug/307462

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.