The Great Ruby Shootout (July 2010)

The Great Ruby Shootout measures the performance of several Ruby implementations by testing them against a series of synthetic benchmarks. Recently I ran Mac and Windows shootouts as well, which tested a handful of implementations. However this article reports on the results of extensive benchmark testing of eight different Ruby implementations on Linux.

The setup

For this shootout I included a subset of the Ruby Benchmark Suite. I opted to primarily exclude tests that were executed in fractions of a second in most VMs, focusing instead of more substantial benchmarks (several of which came from the Computer Language Benchmarks Game). The best times and least memory allocations out of five runs are reported here for each benchmark.

All tests were run on Ubuntu 10.4 LTS x86_64, on an Intel Core 2 Quad Q6600 2.40 GHz, 8 GB DDR2 RAM, with two 500 GB 7200 rpm disks.

8 implementations

The implementations tested were:

• Ruby 1.8.7 p299
• Ruby 1.9.1 p378
• Ruby 1.9.2 RC2
• IronRuby 1.0 (Mono 2.4.4)
• JRuby 1.5.1 (Java HotSpot(TM) 64-Bit Server VM 1.6.0_20)
• MagLev (rev 23832)
• Ruby Enterprise Edition 2010.02
• Rubinius 1.0.1

JRuby was run with the –fast and –server optimization flags.

Disclaimer

Synthetic benchmarks cannot predict how fast your programs will be when dealing with a particular implementation. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. The values reported here should be assumed to be characteristic of server-side — and long running — processes; they should be taken with a grain of salt.

Time Results

Please find below the execution times for the selected tests. Timeouts indicate that the execution of a single iteration for a given test took more than 300 seconds and had to be interrupted. Bold, green values indicate the best performer out of each test.

Warning: The bm_primes.rb benchmark was originally written to aid the development of the Prime class. As such in 1.9.2 it was rewritten in C, which makes it a poor representation of implementation performance. This benchmark will removed in the future.

If you are not interested in the individual test results, the information presented in the table above is summarized directly below:

  Ruby 1.9.2         JRuby
Min.   : 0.013   Min.   : 0.382
1st Qu.: 3.258   1st Qu.: 3.051
Median : 4.543   Median : 4.997
Mean   : 9.262   Mean   : 9.180
3rd Qu.: 8.573   3rd Qu.: 8.969
Max.   :45.009   Max.   :48.850

    MagLev         Ruby 1.9.1
Min.   : 0.351   Min.   : 0.015
1st Qu.: 2.140   1st Qu.: 3.387
Median : 6.069   Median : 6.205
Mean   : 9.100   Mean   :10.860
3rd Qu.: 9.266   3rd Qu.:11.559
Max.   :51.221   Max.   :46.849

 Ruby 1.8.7         IronRuby
Min.   : 0.708   Min.   :  3.601
1st Qu.: 5.102   1st Qu.: 10.505
Median : 8.380   Median : 12.912
Mean   :18.785   Mean   : 26.539
3rd Qu.:24.793   3rd Qu.: 36.115
Max.   :75.653   Max.   :135.204

   Rubinius           REE
Min.   : 0.484   Min.   : 0.584
1st Qu.: 3.087   1st Qu.: 4.343
Median : 9.636   Median : 6.660
Mean   :13.232   Mean   :15.036
3rd Qu.:17.674   3rd Qu.:21.336
Max.   :73.050   Max.   :61.960

For the sake of convenience, I also produced a box plot from the successful data points:

There are a few considerations based on these results that I feel are worth mentioning:

• As you can see Ruby 1.9, JRuby and MagLev converge towards a similar performance level according to these tests.
• Ruby 1.9.2 manages to squeeze in a bit of extra speed when compared to Ruby 1.9.1 (which is a welcome improvement).
• Ruby 1.9 seems to be either much faster than Ruby 1.8 or roughly as fast, depending on the test. This appears to be in line with what I’ve seen in real world programs. There are programs that will only receive a 10-20% boost from 1.9, while others improve drastically. The results really depends on what a program spends its time doing.
• Performance wise, Rubinius is really starting to be a much more serious contender.
• Ruby Enterprise Edition is slightly faster than Ruby 1.8.7, to the extent where this is clearly visible in almost all of the tests.
• IronRuby running on Mono was the worse of the lot.

Memory Results

The following table shows the approximate memory consumption for each implementation when running each benchmark:

Summarized:

  Ruby 1.9.2        Ruby 1.9.1
Min.   :  4.320   Min.   :  4.580
1st Qu.:  4.378   1st Qu.:  4.695
Median :  6.285   Median :  6.920
Mean   : 20.795   Mean   : 15.669
3rd Qu.: 10.162   3rd Qu.: 11.383
Max.   :171.500   Max.   :100.570

   Ruby 1.8            REE
Min.   :  3.040   Min.   :  8.220
1st Qu.:  4.290   1st Qu.:  9.682
Median :  7.745   Median : 15.565
Mean   : 20.698   Mean   : 27.014
3rd Qu.: 11.273   3rd Qu.: 38.620
Max.   :103.520   Max.   :125.910

  Rubinius           MagLev
Min.   : 37.63   Min.   : 81.74
1st Qu.: 39.78   1st Qu.: 82.52
Median : 45.48   Median : 83.53
Mean   : 65.70   Mean   : 96.29
3rd Qu.: 58.22   3rd Qu.: 98.10
Max.   :215.33   Max.   :175.85   

    JRuby
Min.   : 49.04
1st Qu.: 71.23
Median :176.72
Mean   :169.41
3rd Qu.:209.04
Max.   :404.06

And finally, in graph form:

A few considerations on memory:

• Memory readings for IronRuby were not available, so they were not included.
• Ruby 1.9.2 uses the least amount of memory (as one might expect).
• JRuby was by far the most memory intensive of the group.
• Ruby Enterprise Edition used less memory than 1.8.7 in a few tests, but overall was more memory hungry than 1.8.7. This is really odd and entirely unexpected.

Linux Vs. Windows

This shootout and the Windows one were both performed on the same machine, thus we can compare how the same implementation perform under different operating systems. The only adjustment that’s required is the timeout. Every result longer than 60 seconds from this shootout needs to be considered a timeout, because the previous shootout did so as well.

It is commonly believed that Ruby performs much better on Linux than on Windows (with the exception of IronRuby). Let’s find out if these test results confirm that notion.

Ruby 1.8.7:

Ruby 1.9.2:

JRuby:

Finally, in chart form (yellow entries are on Windows as indicated by the labels containing W):

To use a beloved MythBusters expression, this myth is confirmed.

Note: As requested by a few commenters, here is a comparison of IronRuby as well (.NET 4.0 Vs. Mono 2.4.4):

Conclusion

In conclusion, let me just state that it’s nice to see several implementations getting faster. Ruby 1.9.2, JRuby, MagLev and Rubinius are all becoming serious competitors and working their respective ways closer to a similar performance level. If you think these benchmark shootouts are becoming boring, then the results are becoming more stable and predictable. I suspect that as time goes on, performance will not be the real distinguishing factor when choosing a Ruby implementation, other features will be.

The Great Ruby Shootout (Windows Edition)

This post contains the results of a Ruby shootout on Windows that I recently conducted. You can find the Mac edition, published last month, here. I was planning to have this one ready much sooner, but a couple of serious events in personal life prevented that from happening. Be sure to grab my feed or join the newsletter to avoid missing the upcoming Linux shootout.

The setup

For this shootout I included a subset of the Ruby Benchmark Suite. I opted to primarily exclude tests that were executed in fractions of a second in most VMs, focusing instead of more substantial benchmarks (several of which come from the Computer Language Benchmarks Game). The best times out of five runs are reported here for each benchmark.

All tests were run on Windows 7 x64, on an Intel Core 2 Quad Q6600 2.40 GHz, 8 GB DDR2 RAM, with two 500 GB 7200 rpm disks.

The implementations tested were:

• Ruby 1.8.7 (2010-01-10 patchlevel 249) [i386-mingw32] (RubyInstaller)
• Ruby 1.9.1 p378 (2010-01-10 revision 26273) [i386-mingw32] (RubyInstaller)
• Ruby 1.9.2 dev (2010-05-31) [i386-mingw32] (experimental)
• JRuby 1.5.1 (ruby 1.8.7 patchlevel 249) (2010-06-06 f3a3480) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_20) [amd64-java]
• IronRuby 1.0 x64 for .NET 4.0

JRuby was run with the --fast and --server optimization flags.

Disclaimer

Synthetic benchmarks cannot predict how fast your programs will be when dealing with a particular implementation. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. The values reported here should be assumed to be characteristic of server-side – and long running – processes and should be taken with a grain of salt.

The results

Please find below the execution times for the selected tests. Timeouts indicate that the execution of a single iteration for a given test took more than 60 seconds and had to be interrupted. Bold values indicate the best performance for each test.

Conclusions

Despite a couple of errors and a few timeouts, JRuby was the fastest of the lot, which can be seen as impressive if we consider that this is Windows we are talking about after all.

Ruby 1.9.1 and 1.9.2 were almost as fast as JRuby on these tests. With a few exceptions, the performances of the two 1.9 implementations were, expectedly, very similar.

JRuby, 1.9.1 and 1.9.2 were all faster than the current MRI implementation, which can be seen as a prerequisite as we move, as a community, away from Ruby 1.8. Finally, it’s worth noting that IronRuby’s performance was however in line with that of Ruby 1.8.7.

Update (July 3, 2010): The following box plot compares the various implementations for the tests for which all the implementations were successful. Only times for the largest successful input number were used in those tests where multiple input numbers were tested.

Benchmarking MacRuby 0.6

Recently MacRuby 0.6 was released. The development team put a lot of emphasis on improving compatibility with Ruby 1.9, and the viability of MacRuby as a tool for developing Mac OS X applications. Focus on these aspects took precedence over performance, but I was still curious to see how well it performed when compared to Ruby 1.8.7 and Ruby 1.9, respectively.

This article showcases the results of a small Ruby shootout for Mac. I plan to publish a Windows one by next week, and then a week or two after that, a complete Linux shootout that will have many more implementations. Grab my feed or join the newsletter to avoid missing upcoming shootout posts.

The setup

The tests are a large subset of the Ruby Benchmark Suite. Each test was run 10 times, five to detect the best execution time, and five to detect the minimal memory consumption. All of the tests were run on Mac OS X 10.6.3, on my MacBook Pro 2.66 GHz Intel Core 2 Duo, 4 GB 1067 MHz DDR3 RAM, 320 GB 7200 rpm disk.

Stable implementations tested:

• MacRuby 0.6
• Ruby 1.8.7
• Ruby 1.9.1

Disclaimer

Synthetic benchmarks cannot predict how fast your programs will be with one implementation or another. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. Furthermore, the Ruby Benchmark Suite has many tests that don’t provide much insight when it comes to comparing implementations. They are there for legacy reasons and will probably be removed in the future. For the time being, take them with a grain of salt.

The results

Without further hesitation, here are the execution times for the tests (divided in A-L, M-Z). Timeouts indicate that the execution of a single run took more than 60 seconds and had to be interrupted. Bold values indicate the best performance for each test.

And here is the estimated memory usage:

Conclusions

MacRuby 0.6 is faster than Ruby 1.9.1 at times, but it can also be significantly slower. Overall, as things stand now, its performance appears to be between that of Ruby 1.9.1 and Ruby 1.8.7, with several outliers and a greater variance compared to those two implementations. Memory wise, MacRuby appears to be significantly more “memory hungry” than the other implementations (even though this wasn’t all that much of a surprise to me).

I’m interested in seeing how future releases that will be focused more on performance will affect these preliminary results. For the time being however, don’t let this outcome discourage you from using MacRuby 0.6, which is the first release that’s considered stable for Mac OS X development.

Download: CSV Files

PS: If you are looking for a fun and easy way to get started with MacRuby, check out ThinkCode.TV’s screencast on the subject.

Update (July 3, 2010): The following box plot compares the various implementations for the tests for which all the implementations were successful. Only times for the largest successful input number were used in those tests where multiple input numbers were tested.

How much faster is Ruby on Linux?

In a previous article I compared the performance of Ruby on Windows, built through Microsoft Visual C++ and GCC. The numbers for the MinGW version were very impressive. So the question now becomes, how does its performance compare to that of Ruby on Linux? To quote one person (Alex) who commented on the aforementioned post:

With the new mingw32 substantial speed improvements, think it makes sense now to also test at least the baseline (MRI) on Mac/Linux on the same battery of tests, so we Windows folks could get a better idea of how far behind are we yet and what the different Windows interpreters speed target shall be.

Any sort of performance improvement for something that is notoriously slow on Windows is more than welcome, but would this be enough to fill the gap between Ruby’s performance on Windows and on Linux? How much faster is Ruby on Linux? Let’s find out.

Setup

• As a reminder, the operating systems used were Windows XP SP3 32bit and Ubuntu 9.04 32 bit.
• The Ruby implementations tested were ruby 1.8.6 (2009-03-31 patchlevel 368) [i386-mingw32], ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32], Ruby 1.8.6 (built from source on Linux) and Ruby 1.9.1 (built from source on Linux as well). The same identical versions of Ruby were used for both operating systems. I’m aware that these are not the latest available versions for Linux, but we are trying to compare apples to apples.
• I used the Ruby Benchmark Suite; the times reported are the best out of five runs, with a timeout of 300 seconds per each iteration.

Benchmark results

The following table/image compares the performance of Ruby 1.8.6 on Windows and Linux. A light green background indicates which of the two was faster. The total times exclude tests that raised an error or were not available (N/A) for any of the four implementations, but includes timeouts (they are counted as 300 seconds to provide a lower bound). The ratio column indicates how many times faster Ruby on Linux was:

The second table/image below compares Ruby 1.9.1 on Windows and on Linux, using the same criteria as above.

Note: The totals shown are different from the ones seen in other posts since the subset of benchmarks included in the totals is different.

Conclusion

According to the geometric mean of the ratios for these tests, it appears that on average Ruby 1.8.6 on Linux is about twice as fast as Ruby 1.8.6 on Windows. Conversely, Ruby 1.9.1 on Linux is about 70% faster than the Windows version.

The Windows implementations use GCC 3.4.5 (a four year old compiler) at the moment, while I built the implementations on Ubuntu with GCC 4.3.3 (which is available by default). This helps, at least in part, to justify the performance gap. Luis Lavena, leader of the Windows port, confirmed to me that a switch to GCC 4.4.x is planned for the future. This should significantly increase performance on Windows yet again, and bump Ruby’s speed on Windows a bit closer to the speed that’s obtainable on Linux.

For the time being, switching to Ruby 1.9.1 on Windows will give you a performance that is better than what’s obtained by those who are still using Ruby 1.8.x on Linux. If it’s possible, switch.

A faster Ruby on Windows is possible (benchmarks for 4 implementations inside)

In yesterday’s post I compared IronRuby 0.9, Ruby 1.8.6 (from the One-Click Installer) and Ruby 1.9.1 (downloaded from the official site) against one another. IronRuby did great, but the discussion in the comment section quickly veered towards what version of the One-Click Ruby Installer should have been used.

I justified my choice of using the “old” One-Click Installer, by the fact that I wasn’t aware of official releases of the new installer. As well as that the old One-Click Installer is the most widely downloaded version. Very few people are familiar with the upcoming version of the project. This point is about to change.

Luis Lavena took over the One-Click Installer project and has been working on the next version (RubyInstaller from now on), the aim of which is to replace the One-Click Installer by building Ruby 1.8 and 1.9 with MinGW and GCC. In theory, this brings performance gains on Windows to the table, and gets rid of having to use Visual C++ 6 (a “ten year old compiler”) to build Ruby and other native gems. The project also strives to be lighter by bundling fewer (unnecessary) gems for Windows users.

There’s no doubt that in the long run, this new project will become the de facto standard for Windows, but the questions on everyone’s mind are, should I bother with it now? How much of a performance boost are we talking about here? 10%? 20%? Let’s find out.

In this follow up article I’m going to compare the performance of Ruby 1.8.6 from the One-Click Installer (mswin32), Ruby 1.8.6 from RubyInstaller (mingw32), Ruby 1.9 (mswin32) downloaded from the Ruby-Lang.com site, and Ruby 1.9 from the RubyInstaller project (mingw32) against one another. I’ll copy and paste part of the setup and disclaimer from yesterday’s post, for those who haven’t read it. Feel free to skip this part if you wish.

Setup

• The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
• Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
• The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available or the setup on my machine was lacking the required libraries to execute it (1 test affected).

Disclaimer

• An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how these implementations compare, but cannot be viewed as a guarantee of how they will actually affect your own programs.
• There are lies, damned lies, and statistics.

Benchmark results

The table/image below shows the times for each benchmark, for Ruby 1.8.6 (mswin32), Ruby 1.8.6 (mingw32), Ruby 1.9.1 (mswin32), and Ruby 1.9.1 (mingw32). In the table I used (RI) as shorthand for RubyInstaller to indicate mingw32 versions.

Red values are errors, timeouts and inapplicable tests. Green, bold values indicate better times than what Ruby 1.8.6 (mswin32) delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the run-time for the subset of benchmarks that were successfully executed by all four implementations. Timeouts have been included this time around. Each timeout has been counted as an additional 300 seconds.

The total runtime (including timeouts) is summarized in the chart below:

Conclusion

Wow! Ruby 1.8.6 (mingw32) improves from 3% to 664% (depending on the test), over the current One-Click Installer. The geometric mean of the ratios (read “on average”) tells us that it was about 283% faster. The Ruby 1.9.1 version provided by the RubyInstaller was slower than the mswin32 version in a couple of tests, but faster everywhere else. How much faster? Up to 342% faster, with an average (again calculated through the geometric mean of the ratios) of a 77% increase in speed.

These finds prompt me to ask, what are we waiting for? You know how unresponsive Ruby is on Windows, and how tests take forever to execute? These mingw32-based releases may very well solve this. And incidentally the bar has been raised for IronRuby as well. I formally invite the Ruby on Windows community to embrace these two projects.

Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Windows

In my latest article I discussed the importance of JRuby as a means of introducing Ruby to the Enterprise world. Most of the companies that belong to this ecosystem are Java based, but we cannot forget that a sizable portion of them are Microsoft-centric. Within these companies, Ruby will be far more welcome if a .NET implementation is available. The answer to this need is sufficiently fulfilled by IronRuby (version 0.9 was just released).

IronRuby has been progressing fast lately. First came support for Rails and then, with this release, a great deal of effort has been placed on improving performance. In the past, IronRuby was all but fine-tuned. In fact, it was several times slower than Ruby MRI, as the team worked on improving compatibility with Ruby 1.8 and mostly ignored performance.

In this article I’m going to provide some performance results for IronRuby 0.9 on Windows, which I’m sure will interest readers of this blog as well as of my book. Before revealing all the details, let’s start with the setup and a disclaimer. Please read through it carefully, because the old, trite comments about how “micro-benchmarks are useless” won’t be published. We’ve already been there, folks. Thank you all for your understanding.

Setup

• The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
• Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
• The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available (2 tests) or the setup on my machine was lacking the required libraries to execute it (1 test).
• I realize that there is a MinGW One-Click Installer effort that may improve the performance of Ruby on Windows. Here however, I used the regular One-Click Installer as it is the most common one (it’s been downloaded almost 3.5 million times so far). For Ruby 1.9 I used the binary provided by the download page on the official Ruby site.

Disclaimer

• An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how IronRuby compares to MRI (Ruby 1.8) and KRI (Ruby 1.9) performance-wise, but cannot be viewed as a guarantee of how this will actually affect your own programs.
• I didn’t calculate a RubySpec completeness score for IronRuby. That said, IronRuby 0.9 should be a fairly complete implementation of Ruby 1.8.
• There are lies, damned lies, and statistics.

Benchmark results

The table below shows the times for each benchmark, for IronRuby 0.9, Ruby 1.8.6 (2008-08-11 patchlevel 287) and Ruby 1.9.1p0 (2009-01-30 revision 21907):

Red values are errors, timeouts, inapplicable tests and times that were worse than Ruby 1.8.6. Green, bold values are better times than what Ruby 1.8.6 delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the runtime for the subset of benchmarks that were successfully executed by all three implementations (whose cardinality is 54).

The total runtime is summarized by the chart below:

And let’s compare each of the “macro-benchmarks” on an individual basis:

Conclusions

IronRuby went from being much slower than Ruby MRI to considerably faster across nearly all the tests. That’s major progress for sure, and the team behind the project deserves mad props for it.

One final warning before we get too excited here. IronRuby is not faster than Ruby 1.9.1 at this stage. Don’t let that first chart mislead you. While it’s faster in certain tests, it’s also slower is many others. Currently, it’s situated between Ruby 1.8.6 and Ruby 1.9.1, but much closer to the latter. The reason why this chart is misleading is that it doesn’t take into account any tests that timed out, and several of such timeouts were caused by IronRuby (more than those caused by Ruby 1.9.1). If you were to add, say, 300 seconds to the total, for each timeout for the two implementations, you’d quickly see that Ruby 1.9.1 still has the edge. The second chart that compares macro-benchmarks does a better job at realistically showing how IronRuby sits between Ruby 1.8.6 and Ruby 1.9.1 from a performance standpoint. If you were to plot every single benchmark on a chart, you’d find a similar outcomes for a large percentage of the tests.

Whether it’s faster than Ruby 1.9 or not, now that good performances are staring to show up, it’s easier to see IronRuby delivering on it’s goal of becoming the main implementation choice for those who both develop and deploy on Windows. This, paired with the .NET and possible Visual Studio integration, the great tools available to .NET developers, and the ability to execute Ruby code in the browser client-side thanks to projects like Silverlight/Moonlight and Gestalt, make the project all the more interesting.

What are your thoughts on IronRuby, and how will this dramatic performance gain affect your projects?

Reflections on the Ruby shootout

Yesterday I published The Great Ruby Shootout and it quickly gathered a fair deal of attention. It was on the front page of Slashdot, Hacker News, Reddit, and so on. More than 15,000 people came by to read about the results of my comparison between Ruby implementations.

Those numbers looked good but something didn’t add up. Ever since I clicked the “Publish” button, I had a very uneasy feeling about the main shootout figures. They just didn’t seem right. I had a chance, particularly during the writing of my book, to extensively use Ruby on Vista and I can guarantee you that it’s visibly slower than on GNU/Linux. The Phusion team had benchmarked their Ruby Enterprise Edition against Ruby 1.8.6 many times, and found it to be about 25% faster. Yet my results were showing it as twice as fast than Ruby 1.8.7, which in turn is already faster than 1.8.6. To makes things worse, I’ve used Ruby 1.9 and found it to be faster than Ruby 1.8.7, but not 5 times as fast. For most programs that I tried Rubinius didn’t seem faster than Ruby 1.8. And the more I pondered it, the more it began to feel like one too many things didn’t add up.

In the comments, Isaac Gouy reported a couple of issues with the Excel formulas, where a few unsuccessful tests were mistakenly added to the totals. This skewed the results slightly, particularly in terms of penalizing JRuby. However, this wasn’t really it. Sure, the totals were inaccurate, but not enough to fundamentally change the main outcome of those results.

As I was discussing this somewhat unexpected result with Hongli Lai (co-author of Ruby Enterprise Edition), he mentioned that he knew what might be causing this anomaly. I had run the initial test against Ruby installed through apt-get, because I’d made a couple of assumptions. The first was that most people would probably be using the Ruby version that was deployed by their OS’ packaging system in both development and production mode. The second was that the performance of this version would be roughly similar to the one built from scratch. This second assumption would turn out to be highly mistaken.

I decided to run a test using Ruby 1.8.7 built from source as the baseline and added a column for Ruby 1.8.7, installed through apt-get, to the tables. In addition I also corrected the issue pointed out by Isaac. I updated the original shootout with the correct data, and what you see below is a bar chart for the geometric mean of the ratios for the successful benchmarks.

Notice how everything makes much more sense now. Ruby 1.9 and JRuby are very close, respectively 2.5 and 1.9 faster than Ruby 1.8.7 (from source) on these benchmarks. Less impressive result sure, but I suspect much more realistic. The results for Ruby Enterprise Edition are in line with the 25% speed increase, if we consider that 1.8.7 is a bit faster than 1.8.6. Rubinius is still slower than MRI for most tests, but it’s improving. Ruby on Windows is slow. So slow in fact, that Ruby on GNU/Linux is twice as fast.

The really big, flashing warning though is what happens when you install Ruby through apt-get. Compiling from source gives you double the speed, according to these tests. I expected a 10/20% increase, not 100%. The gist of it is that prepackaged Ruby is compiled using the option –enable-pthreads and there is the whole issue of shared vs static libraries. But whatever the reason, this is a significant difference. For production use, in light of these results, I feel that it would be foolish to use the slower version of Ruby provided by apt-get/aptitude.

I rectified the results as soon as possible because the last thing I wanted was to mislead the Ruby community or worse still, betray its trust. Major kudos to Isaac for spotting the calculation issue, and Hongli for selflessly pointing out that the excellent Ruby Enterprise Edition results were probably due to the low performance of the Ubuntu’s version of Ruby.

The Great Ruby Shootout (December 2008)

The long awaited Ruby virtual machine shootout is here. In this report I’ve compared the performances of several Ruby implementations against a set of synthetic benchmarks. The implementations that I tested were Ruby 1.8 (aka MRI), Ruby 1.9 (aka Yarv), Ruby Enterprise Edition (aka REE), JRuby 1.1.6RC1, Rubinius, MagLev, MacRuby 0.3 and IronRuby.

Disclaimer

Just as with the previous shootout, before proceeding to the results, I urge you to consider the following important points:

• Engine Yard sponsors this website, and also happens to sponsor, to a much greater extent, the Rubinius project. Needless to say, there is no bias in the reporting of the data below concerning Rubinius;
• Don’t read too much into this and don’t draw any final conclusions. Each of these exciting projects has its own reason for being, as well as different pros and cons, which are not considered in this post. They each have a different level of maturity and completeness. Furthermore, not all of them have received the same level of optimization yet. Take this post for what it is: an interesting and fun comparison of Ruby implementations;
• The results here may change entirely in a matter of months. There will be other future shootouts on this blog. If you wish, grab the feed and follow along;
• The scope of the benchmarks is limited because they can’t test every single feature of each implementation nor include every possible program. They’re just a sensible set of micro-benchmarks which give us a general idea of where we are in terms of speed. They aren’t meant to be absolutely accurate when it comes to predicting real world performance;
• Many people are interested in the kind of improvements that the tested VMs can bring to a Ruby on Rails deployment stack. Please do not assume that if VM A is three times faster than VM B, that Rails will serve three times the amount of requests per minute. It won’t. That said, a faster VM is good news and can definitely affect Rails applications positively in production;
• These tests were run on the machines at my disposal, your mileage may vary. Please do test the VMs that interest you on your hardware and against programs you actually need/use;
• In this article, I sometimes blur the distinction between “virtual machine” and “interpreter” by simply calling them “virtual machines” for the sake of simplicity;
• Some of the benchmarks are more interesting for VM implementers than for end users. That said, if you think the benchmarks being tested are silly/inadequate/lame, feel free to contribute code to the Ruby Benchmark Suite and if accepted, they’ll make it into the next shootout;
• Finally, keep in mind that there are three kinds of lies: lies, damned lies, and statistics.

Ruby implementations being tested

All of the Ruby implementations that were able to run the current Ruby Benchmark Suite have been grouped together in one main shootout. This group consists of Ruby 1.8.7 (p72, built from source, and installed through apt-get), Ruby 1.9.1 (from trunk, p5000 revision 20560), Ruby Enterprise Edition (1.8.6-20081205), JRuby 1.1.6RC1 and Rubinius (from trunk), all of them were tested on Ubuntu 8.10 x64, plus Ruby 1.8.6 (p287. from the One-Click Installer) on Windows Vista Ultimate x64. The hardware used for this benchmark was my desktop workstation with an Intel Core 2 Quad Q6600 (2.4 GHz) CPU and 8 GB of RAM. JRuby was run with the -J-server option enabled and by specifying 4 Mb of stack (required to pass certain recursive benchmarks). The best times out of five iterations were reported, and these do not include startup times or the time required to parse and compile classes and method for the first time. Several of these new tests also have variable input sizes.

The MagLev team provided me with an early alpha version of MagLev for the purpose of testing it in this shootout. Since this VM is not mature enough yet to run the Ruby Benchmark Suite, I used custom scripts against an old version of the Ruby Benchmark Suite on Ubuntu 8.10 x64. MagLev was tested, along with Ruby 1.8.6 (p287), on the same machine as that of the main shotoout, though the benchmarks were different (even when they had the same names as the ones in the main shootout).

MacRuby 0.3 and Ruby 1.8.6 (p114) were tested on Mac OS X Leopard using the previous version of the Ruby Benchamrk Suite. Since my MacBook Pro died (sigh), for this benchmark I used a Mac Pro, with two Quad-Core Intel Xeon 2.8 Ghz processors and 18 GB of RAM.

IronRuby (from trunk) and Ruby 1.8.6 (p287) were tested on a previous version of the Ruby Benchmark Suite on Windows Vista x64 on the same quad-core used for the main shootout. The MagLev, MacRuby and IronRuby numbers reported here were the best times out of five iterations, and include startup time. IronRuby on Mono was not tested because I couldn’t get it to work on my machine, despite having tried several IronRuby versions and two different Mono versions. Please also notice that Ruby 1.8.6 (p287) was tested twice on Windows, once for the main shootout against the current Ruby Benchmark Suite, and a second time to compare it with IronRuby, against the old benchmarks.

Note: As tempting as it is, do not compare implementations that belong to different shootouts directly to one another. It would be very disingenuous to directly compare VMs tested with different benchmarks and/or different machines. The only comparisons that make sense are the ones within each of the four groups.

Main shootout

The following table shows the run times for the main implementations. The table is fairly wide, so you’ll have to click on the image to view the data in a new tab.

Green, bold values indicate that the given virtual machine was faster than Ruby 1.8.7 on GNU/Linux (our baseline), whereas a yellow background indicates the absolute fastest implementation for a given benchmark. Values in red are slower than the baseline. Timeout indicates that the script didn’t terminate in a reasonable amount of time and was (automatically) interrupted. The values reported at the bottom are the total amounts of time (in seconds) that it would take to run the common subset of benchmarks which were successfully executed by every virtual machine. When our baseline VM generated an error, others were used, starting with Ruby 1.8.7 on Vista (for color coding purposes only).

The following image shows a bar chart of the total time requested for the common subset of successfully executed benchmarks (those whose names are in blue within the tables):

More interestingly, the following table shows the ratios of each Ruby implementation based on the baseline (MRI):

The baseline time is divided by the time at hand to obtain a number that tells us “how many times faster” an implementation is for a given benchmark. 2.0 means twice as fast, while 0.5 means half the speed (so twice as slow). The geometric mean at the bottom of the table tells us how much faster or slower a virtual machine was when compared to the main Ruby interpreter, on “average”. Just as with the totals above, only those 101 tests, which were successfully run by each VM, where included in the calculation.

More concisely, here is a bar chart showing the geometric mean of the ratios for the various implementations tested:

I prefer to let the data speak for itself, but I’d like to briefly comment on these results. Just a few quick considerations.

Working off of the geometric mean of the ratios for the successful tests, Ruby MRI compiled from source is twice as fast than the Ruby shipped by Ubuntu, and by the One-Click Installer on Vista. The huge performance gap between ./configure && make && sudo make install and sudo apt-get install ruby-full should not be taken lightly when deploying in production. These numbers also reveal what most of us already knew: Ruby is particularly slow on Windows (800-pound gorillas in the room, or not).

Performance-wise Rubinius has more work left to be done to catch up with Ruby 1.8.7 and other faster VMs, particularly if we take into account the number of timeouts. But it has improved in the past year and I think it’s on the right track.

Ruby Enterprise Edition is about as fast as Ruby 1.8.7 compiled from source, which is reasonable considering that it’s a patched version of Ruby 1.8.6 aimed at the reduction of memory consumption (a parameter which wasn’t tested within the current shootout).

Speaking of excellent results, Ruby 1.9.1 and JRuby 1.1.6 both did very well. It looks like we finally have a couple of relatively fast alternatives to what is a slow main interpreter. According to the results above, and with the exception of a few tests, on average they are respectively 2.5 and 2 times faster than Ruby 1.8.7 (from source), and 5 and 4 times faster than Ruby 1.8.7 installed through apt-get on Ubuntu or Ruby 1.8.6 installed through the One-Click installer on Vista. Again, this does not mean than every program (particularly Rails) will gain that kind of speed, but these results are very encouraging nevertheless.

MagLev

There has been a lot of buzz about MagLev since Avi Bryant’s first benchmarks were shown a few months ago. Here we finally see it being put to the test. The table below shows the times obtained by running MagLev and Ruby 1.8.6 (p287) against MagLev’s set of benchmarks based on the old Ruby Benchmark Suite:

And here are the ratios:

You’ll notice how MagLev swings from being much faster than MRI to being much slower. I believe there is much room for improvement, but at almost twice the speed of MRI, these early results are definitely promising.

MacRuby

These are the times for MacRuby 0.3 on Mac OS X 10.5.5:

And of course, the ratios against the MRI baseline:

MacRuby is relatively new, so these are not bad results. More work is required, but it’s a good start.

IronRuby

Finally (I promise these are the last ones), here are the two tables for IronRuby and Ruby 1.8.6:

IronRuby is slower than Ruby 1.8.6 on Windows, which in turn is much slower than Ruby 1.8.7 on GNU/Linux. This is not very surprising. This project has been focusing on integrating with .NET and catching up with the implementation of the language by improving the RSpec pass rate, as opposed to performing any optimizations and/or fine tuning (as per John Lam’s presentation at RubyConf 2008). We’ll measure its improvements in the next shootouts.

Conclusion

Overall I think these are great results. Ruby 1.8 (MRI), with its slowness and memory leaks, belongs to the past. It’s time for the community to move forward and on to something better and faster – and we don’t lack interesting alternatives to do so at this stage.

I hope that for the next shootout, MagLev, MacRuby and IronRuby will be able to run the benchmark suite, so that they can all be tested and directly compared with each other. I also hope to include Tim Bray’s XML benchmark, some sort of “Pet Shop” sample Rails and Merb application and, above all, include memory usage statistics.

You can find the Excel file for the main shootout here. That’s all for now. Feel free to comment, subscribe to my feed, share this link and promote it on Hacker News, Reddit, DZone, StumbleUpon, Twitter, and Co. Putting together this shootout was a lot of work, so I definitely appreciate you spreading the word about it. Until next time…

Update (December 10, 2008): This article has been updated to correct a couple of major issues with yesterday’s results. I adjusted my commentary as well, in light of the corrected figures.

Update (February 7, 2009): Thanks to Makoto Kuwata, a Japanese version of this article was published in the Rubyist Magazine.

Merb, Rails Myths, Language Popularity and other Zenbits

Zenbits are posts which include a variety of interesting subjects that I’d like to talk about briefly, without writing a post for each of them.

Merb: A few days ago Merb 1.0 was released. Congratulations to Ezra Zygmuntowicz on this important milestone, the Merb community and Engine Yard (who finances the project). Merb 1.0 wasn’t even out yet when some people had already started commenting on the fracturing of the Ruby community that this new framework might bring with this, and the impact that this high visibility “competitor” might have on Rails. I believe that having more than one widely adopted web framework will only benefit the Ruby community. Furthermore, it’s important to remember that this is not a zero-sum game. Ruby programmers are perfectly capable of learning two frameworks and using one or the other, depending on the project at hand. This is particularly true if we consider that Merb, for all of its advantages – and disadvantages – when compared to Rails, is not totally different from its forerunner. If you are an expert Rails programmer, you should be able to become proficient in Merb in very little time. To help with this process, the Merb community needs to concentrate on the documentation now, given that the API is finally stable.

Rails Myths: David Heinemeier Hansson began a series of posts about Rails Myths. I like the idea of seeing common myths addressed straight from the horse’s mouth. Over the past two years, Rails has received quite a bit of backslash and old fashion FUD, so it’s important to set the record straight, whether the myths are entirely fabricated or if there is some element of truth to them. Whether you agree with David or not, it’s also nice to hear two sides of the same story. In fact, at the beginning of my book I debunk a few myths, just to set the record straight regarding what some readers may have heard surrounding the framework. It was a fun part to write.

My Book: Speaking of my book, Ruby on Rails for Microsoft Developers, I’m getting closer to the finish line. I’m about to complete Chapter 9 (out of eleven chapters). The initial schedule I was provided with has been extended slightly so that there will be sufficient time to properly review the content and ensure that it’s up to date with the final release of Rails 2.2. Some people wondered what the “Microsoft Developers” part means. Is it for people that work at Microsoft? Is it for .NET programmers? Is it for people who develop on Windows?

The truth is that “Microsoft Developers” is probably just a marketing term that Wrox selected as a catch-all for of the aforementioned categories of programmers. As an author I’m trying to serve all of them well, by providing a guide that sneaks in much of the Rails culture and softens the migration path by using an Operating System, and to a certain extent, tools that they’re already familiar with. In my opinion one of the major obstacles when switching to, or trying, Rails when coming from the Microsoft world, is the culture shock. The documentation and most books assume that you are familiar with *nix systems and tools, and this can be frustrating for those who are forced not only to learn a new language and framework, but also an entirely new set of tools. As it’s targeted at Microsoft developers, the book obviously makes quite a few references and comparisons to the .NET world, where they fit. This is done so that the many .NET programmers amongst the group of so called “Microsoft Developers” will find the book particularly useful. Yet the book remains generic enough so that it can be used by any programmer (particularly Windows users), even those without any knowledge of the Microsoft .NET Framework or ASP.NET.

Python books: While on the subject of books, I wanted to mention that the final version of the Pylons book is available online. Despite the much less fancy UI, the book pretty much does what the Django Book did in the past. And both are available in print as well (The Definitive Guide to Django: Web Development Done Right and The Definitive Guide to Pylons). Pylons is a Python web framework that can be viewed as a Ruby on Rails clone, in a far greater way than Django could ever be considered.

Another thing I want to mention is that I received a copy of Expert Python Programming. I haven’t gotten to far into it yet, but from what I’ve seen so far, things look good. I hope to be able to read it through, over a weekend in the near future and then provide a proper review. Stay tuned.

Language Popularity: If you take a look at the TIOBE Index, you’ll notice a few interesting things: Ruby has dropped two positions since last year, and it’s now the 11th most popular language in the world. This shouldn’t be cause for concern though, as shown by this Ruby graph. Python on the other hand is increasing in popularity and moved from the 7th to the 6th most popular language. Interestingly, according to the index (the results of which are educated guesses only), Python would seem to be more popular than C#. I find this to be true, in terms of online activity within an increasingly vibrant community, but in my opinion, the job market hasn’t caught up yet. In fact, at least in Toronto, when there’s a Python opening it’s pretty much an event that’s worthy of being discussed on the local Python mailing list. C# openings are much more common. This may be different in Silicon Valley, of course. It would also seem that Delphi has experienced a huge come back, moving from the 11th position last year to the 8th one this time around. It’s hard to imagine that Delphi has had a similar level of adoption as C# and thus has become more popular than Perl, JavaScript and Ruby. Delphi is a great solution for Win32 programming, but I don’t quite believe this overly optimistic outlook. And if this is the case, where are all the Delphi jobs and buzz?

DB2: This interview shows a few good reasons why even smaller and medium sized companies are increasingly adopting DB2. And while the video doesn’t mention it, IBM is coming out with an updated version of DB2 Express-C 9.5. This new version, 9.5.2 or 9.5 FixPack 2, is going to introduce exciting new features, including an engine for full text search.

The Great Ruby Shootout These days you hear a lot of talk about parallel programming. Intel promotes it and despite their bias, it’s plausible that parallel programming will become important as the CPU market heads towards an increasingly larger number of cores, as opposed to focusing on the frequency of said CPUs. In the world of Ruby, this translates into multiprocessing, as opposed to multithreading due to the infamous GIL (Global Interpreter Lock). This means that Ruby will most likely approach the problem similarly to how Python 2.6 did with the multiprocessing module, which is a process-based interface. The obvious exceptions are JRuby and IronRuby, which establish a 1 to 1 relationship between green threads and OS threads.

For the shootout, it would be interesting to see some multithreaded code, so as to get a better sense of how well JRuby and IronRuby compare to MRI and 1.9, when more cores are available. In fact, the long-promised shootout will be performed on a quad-core machine with 8GB of RAM. If Charles Nutter, John Lam, or any of their team members would like to contribute some programs that are able to take advantage of “native” multithreading, I’d be very happy to include them in the Ruby Benchmark Suite, to be used for my shootout.

The repository requires some love and refactoring, since it needs to be split in two types of benchmarks. The simpler one will evaluate the execution time minus the startup time, while the more advanced benchmark will also exclude the time required for parsing and loading modules, classes and methods in the AST. It would also be nice to test each program with variable input sizes and report these results accordingly. Right now I’m very busy with the book, but as I become more available, I’ll start working on this.

Finally, I want to point out a very interesting article about performance and UIs. Slow is indeed a very relative concept, and it’s important to understand how to analyze and respond to the user requirements when it comes to the responsiveness of an application as a user interacts with it.

Hardware: I finally bought a Trackball made by Logitech and the Microsoft Ergonomic Keyboard (Microsoft makes great hardware). I don’t have wrist problems, but I’d like to see how these two affect my extensive computer usage. I plan to report my experience as soon as I’ve had a chance to use these input devices for a while, since I know this is a topic that interests lots programmers (many of whom end up being victims of RSI, and some of the IRS ). I also bought a bad-ass color laser printer which is quite handy when you’re a programmer and you are writing a book. I’ll let you know how it goes. What I didn’t buy, but still think is awesome, is the Flip minoHD. It’s the equivalent of an iPod for the world of camcorders. $235 for a camcorder that’s so perfectly compact, and yet that can record in HD, is a pretty sweet deal. I’m considering it for Christmas, assuming it reaches Canada by then.

A status update

More than a month has passed since my last update. Did you miss me? I sure missed you, I truly love you guys. Jokes aside, it was rather uncharacteristic for this blog to go so long without a post, so I feel that a status update is in order before my feed ends up in the Dinosaur reports of your RSS reader. And with humanity facing the imminent threat of the Large Hadron Collider, I didn’t want to be sucked into a black hole without saying a few last words.

At the beginning of August I moved from Markham to Toronto (technically North York). My wife and I were prepared for this move, so it went well. That said, there were negative aspects beyond our control.

The Satellite Dish installer had a few interesting discussions with our property management, and while they eventually agreed to a compromise that would preserve the holiness and virginity of my balcony, they also caused me to miss the opening ceremony of the Olympic Games. I really wanted to see the fake singing by the Chinese kid and the computer generated fireworks, but alas nothing. I consoled myself by eating dumplings a la Kung Fu Panda, minus the Kung Fu part.

Perhaps more devastatingly my ISP took its sweet time to activate my DSL service. And when they did, things worked on an off for all of August. When my ISP had problems connecting me to the tubes, I tried what any respectable geek would do. Despite my impression of the Statue of Liberty, standing near the window with my laptop in hand, my attempts to freeload on Wi-Fi were futile.

If these disruptive happenings weren’t enough, I’ve been very busy with work and the book I’m writing (Ruby on Rails for Microsoft Developers). I’ve written 7 chapters so far and just passed the 300 page mark. The schedule is tight, and there is so much that I want to include, so I have to dedicate almost all of my spare time to the book. On the bright side, I think I’m going to give readers their money’s worth.

The direct consequence of my focus on the book is that This Week in Ruby and This Week in Rails have been interrupted. A few of you enquired about them, bringing to my attention how much you appreciated them. Unfortunately, for the next two months I won’t have time to read all the daily feeds I normally follow and come up with weekly (or bi-weekly) reports of what’s cool and worth mentioning from. My friend Gregg Pollack may take over for the Rails post that gets published on the official blog. Unless someone sends me a complimentary English Bulldog puppy, in which case I promise to do both reports weekly and I’ll even put your picture with a halo over your head on the sidebar of the blog.

Another victim of overcommitment is the (vapor)shootout. Right now I’m at full capacity and need the laptop for both work and the book, so that will have to wait. If you want to lend a hand with the project, feel free to jump right in though.

Finally, while I may not have time for the weekly reports or the shootout right now, I can still spare some time to write posts more frequently. And there is no reason not to indulge in such a rewarding activity. So expect more technical articles, especially now that there are many interesting things to talk about (Google Chrome, Django 1.0, Rails 2.2, etcetera).

Ruby Shootout Status Update

I want to provide those who are waiting for the Ruby shootout with a heads up. The benchmark suite needs some substantial changes in order to ensure accuracy and fairness for all the VMs involved.

This will delay the execution (and reporting) of the shootout further, but it will be worth it. I definitely prefer a shootout that’s published later in July (or heck even August) that is realistic, fair and provides interesting metrics (e.g. CPU time and memory) over an inaccurate one that was put together in a rush just for the sake of publishing it tomorrow.

For those interested in the technical details, we are trying to separate the parsing and “compilation” of definitions from the actual execution of the code (which needs to be timed). I accomplished this by creating a Proc for each benchmark, and then tested the time spent executing its call method. The problem with this approach is that it penalizes VMs that don’t JIT procs, like JRuby for example.

We also thought about defining a method instead of a Proc, but eval won’t accept class definitions or constants within methods. The workaround would be using MyClass.class_eval instead of class MyClass in the benchmarks and Module#const_set for the constants (or changing them to instance variables, for example). But we’re shooting for a cleaner solution in which we divide definitions from their actual execution in separate files, and only time the latter.

And of course, we also need to add cross-platform memory measurement into the picture. It may take a while, but stay tuned.

This Week in Ruby (June 26, 2008)

This is the 11th episode of This Week in Ruby, please consider subscribing to my feed so as to not miss any weekly installments.

This edition begins with some bad news: Several vulnerabilities that affect the main Ruby implementation have been discovered. There is no reason to freak out, but they are serious. An ill-intended person could take advantage of these vulnerability and execute arbitrary code. Matasano has a few practical examples which illustrate the vulnerabilities in question. To learn more head over to the official advisory. Unfortunately, the suggested upgrades (except those for Ruby 1.8.7) are currently not working for many Rails developers, who’re reporting segmentation faults. The Phusion team has created a patch that was reported to be working, but it would be nice to see the Ruby Core Team verify and incorporate it quickly. If you’re running a version of Ruby that shipped with Mac OS X, don’t upgrade yet. Instead wait for Apple’s Software Update.

RubyGems 1.2 was released and it’s much more responsive than previous versions of it were (no more bulk updates just to install a new gem). To upgrade run: sudo gem update --system (without sudo if you are on Windows). After a substantial refactoring, Mocha 0.9 – a framework for mocking and stubbing – was released this week. A new BitNami RubyStack version was released (1.2 beta) as well, which adds a lot of goodies to the package, including but not limited to NGINX, Thin, Rack, EventMachine and so on. Speaking of EventMachine, check out EventMachine: Fast and Scalable Event-Driven I/O Framework published by InfoQ. Last week they also published an interview with yours truly, in regards to the Ruby Benchmark Suite. I regret that the shootout testing hasn’t started yet as promised, but Murphy’s law got in the way.

For those interested in improving their language-fu, there were a numbers of interesting articles: Using select, reject, collect, inject and detect, Enumerating Enumerable, Macros, Hygiene, and Call By Name in Ruby Eliminating code duplication with Metaprogramming. Also noteworthy, this piece on working with Microformats from Ruby.

A Ruby Community Announcements group was started in order to provide a fast ML for announcements only. It’s for those who’d like to stay in the loop, but wish to avoid the high volume of messages in Ruby-Talk.

The erubycon conference about Ruby and the Enterprise will be held between August 15 and 17 (‘08) in Columbus, Ohio. They still have a few seats available, so if this topic is of interest to you, grab a spot while you still can.

Finally, if you’re hiring Ruby talent or plan to look for a Ruby job any time soon, take a peek at these 15 fundamental questions for Ruby interviews. They’re somewhat basic, but the article is a good staring point nevertheless.

ALT.rb

From the world of alternative implementations and frameworks, I found this article on Rubinius FFI, an introduction to MacRuby as a replacement for RubyCocoa, and the announcement of Merb’s run_later” method for backgrounds tasks, all to be informative.

To keep the good times rolling, the second edition of This Week in Rails is available on the official Rails blog.

Heads up regarding the shootout

Call it Murphy’s law if you will, but over the weekend my MacBook Pro’s charger cord developed a nasty rip (see photo). I tried electrical tape and other (potentially) “unsafe” measures, but to no avail. My laptop has no charge and is essentially dead. I’ve ordered a new MagSafe power adapter and it should be here in a few days (I hope).

This means that the shootout testing that was planned for tomorrow (the 24th), will have to wait just a few days. On the bright side, this should give you time to send us more benchmarks.

This Week in Ruby (June 16, 2008)

This is the 10th episode of This Week in Ruby, please consider subscribing to my feed so as to not miss any weekly installments.

As announced a few days ago, This Week in Ruby is being split into two parts: This Week in Ruby and This Week in Rails. The one you are reading is the Ruby edition, while Riding Rails – the official Rails blog – will host the Rails one. Links to and from each post will be provided, in case you don’t follow both blogs.

Ruby

The Ruby community has shown a clear interest in Behavior-Driven Development (BDD), so if you haven’t taken the plunge yet, check out Ben Emson’s introduction to RSpec Stories. Those who’re already well versed with RSpec, will enjoy an article by another Ben, in which RSpec’s DSL internals are explained in detail so as to cover an example of creating macros with RSpec.

The Pragmatic Programmers published a series of screencasts about The Ruby Object Model and Metaprogramming. So far they’ve received glowing reviews, including my own, hence I highly recommend that you evaluate them.

A couple of weeks ago I announced the creation of a Ruby Benchmark Suite project. The next shootout will take place starting from the 24th and I should be able to get the results up on this blog by the 30th of this month.

While working on modifying his RX Ruby Tokenizer to be included in the Ruby Benchmark Suite, Tim Bray reported a few considerations on the sad status of REXML and Ruby 1.9. It’s definitely an interesting read, and it’s important to increase the awareness about the current pains of working with Ruby 1.9 and REXML.

Yesterday, Tim also had a post titled Deletionist Morons about the controversy surrounding the proposed deletion of Why the lucky stiff’s wikipedia entry. The Ruby community at large vouched for Why, who is clearly one of its biggest, and definitely most original, contributors.

Finally, the fun Ruby article award of the week goes to Ilya Grigorik for his Tumblr, RMagick and a Photo Frame!

ALT.rb

RailConf’s presentation regarding MagLev has been an attention grabber in the world of alternative Ruby implementations. A video of Avi Bryant’s demo is now available online, as well as a somewhat older interview with InfoQ. You can read Chad Fowler’s take as well as mine.

MacRuby 0.2 was released about 10 days ago. For those not familiar with this project, it’s an Ojective-C based implementation of Ruby 1.9 for Mac OS X. The general idea is to have a Ruby version that lets you write Mac applications that perform reasonably well. In the upcoming shootout we’ll be testing this early release as well.

Those of you still working in Java, but interested in the possibility of using Ruby’s testing tools and frameworks, should pay attention to the release of JtestR 0.3. As you can imagine, this works thanks to JRuby’s interoperability capabilities with Java. If this alternative Ruby implementation for the JVM appeals to you, you should probably also read Thomas Enebo’s interview about the future of JRuby. Speaking of interoperability, at Tech Ed (that’s a Microsoft event), John Lam demonstrated a cool prototype for the integration of IronRuby and ASP.NET MVC.

Readers interested in contributing to Rubinius, should take a look at this write-up about getting started with hacking on Rubinius.

Web Frameworks

At the latest Toronto Rails night (which I didn’t attend, but I soon will be as I’m moving downtown), Rowan Hick presented Merb, and has now made his presentation
available online. Speaking of Merb, Engine Yard Express is a new free product that lets you try out an Engine Yard “slice” wrapped up in a VMware image, and both Merb and Rails are supported.

The 2008.06 version of Ramaze was released last week. Aside from switching from a numeric release scheme to a date-based one (which I personally like much more), this is a major release that introduces over 450 patches and a few changes to the internal API.

To keep the good times rolling, the first edition of This Week in Rails is available on the official Rails blog.

MagLev handles trees like a monkey

Over the past couple of nights, I tried out MagLev on my Mac laptop (Mac OS X Leopard, 2.2GHz, 2GB Ram). While the shootout will provide us with a detailed comparison between MagLev and the other major Ruby implementations (on many tests) I thought I’d share my first impressions.

MagLev is going to be a fast implementation of Ruby. How many times faster? Well, it depends on the test, and I don’t have systematic numbers yet to come up with a geometric mean of the ratios (the shootout will do just that). What I can tell you now though, is that it’s fast, significantly faster than MRI. You know that slowness that you’ve almost come to expect when running a Ruby script? It’s simply not there with MagLev. And let’s not forget, that MagLev’s added value is not limited to performance, in the same way that JRuby’s integration with Java is a strong selling point.

It’s not mature, of course, and there are a few things which haven’t been implemented yet. But it’s complete enough to run WEBrick and several other small programs I tried. A truly impressive accomplishment if you think about what was achieved in just three months. So let me reassure you that it’s not vaporware, it’s real and it’s fast.

I want to leave you with an example of a non-trivial benchmark. Running the binary-trees test from the Computer Language Benchmarks Game (chosen because it’s the first test), I obtained the following results:

Aside from being much faster than Ruby 1.8.6, MagLev’s speed is pretty darn close to that of the extremely fast C++ (complied with g++ and all the optimizations), in this specific case.

You can’t read too much from a single test, especially when we are not comparing different VMs from the same identical script like we’ll do for the shootout. It’s impressive nevertheless and it should give you a hint about MagLev’s speed even when facing non-trivial, non-ad-hoc benchmarks.

Let’s create a Ruby Benchmark Suite

My previous post about MagLev and the planning of the next Ruby shootout received a lot of attention. MagLev’s speed claims have been subject to a lot of skepticism, and many believe that these impressive figures are due to a combination of clever optimization for trivial tests and incompleteness. The skepticism is understandable. There have been very bright people working on alternative VMs for years, and this new product shows up after only 3 months, and claims to be way faster than anything seen before.

Except, that it’s not entirely new. What makes the fact that they may be onto something and develop a faster implementation credible, is that they are leveraging decades worth of Smalltalk experience, where the Smalltalk VMs underwent similar development challenges. Ruby and Smalltalk are family, there is not inherent reason why Ruby has to be dramatically slower than certain Smalltalk implementations. Parsing and compiling Ruby code into “smalltalk-ish” bytecode is not the hardest thing to do. So, from a certain prospective, MagLev is 30 years old, not 3 months old. I’m enthusiastic about the VM because I think MagLev is promising, but don’t let people tell you that I’m naive. Despite the fact that MagLev is incomplete, I want to challenge it so that we verify and clarify what kind of speed improvements are really offered at this stage. Let’s investigate how.

One very valid point that was raised by several people both in my comment section and on Slashdot, is the fact that many of the benchmarks that have been employed so far are not very useful. Some of them are meaningless, not because of the usual “micro-benchmark must be taken with a grain of salt” sound logic, but rather because they offer an opportunity for VMs to optimize them out. When a lazy/smart VM realizes that a given loop doesn’t produce any results which will be used in some way, it’ll just skip it, giving us the impression of being many, many times faster than the standard Ruby 1.8 implementation by Matz et al.

In the real world, when that loop has to do something meaningful and the results of the computation have to be printed on the screen, that impressive performance is nowhere to be seen. So far this set of tests, which were employed by Yarv for testing its own progress, have also been used to compare different implementations (and these benchmarks can be found in Rubinius’ repository as well). This was the easy thing to do, but if we’re going to get serious about it, we need to produce a better set of benchmarks, especially when the current ones question both Yarv and MagLev’s impressive results.

In the long run, it would be good to come up with some serious benchmarks based on AST nodes in order to test each of Ruby’s features. We can work on that, but let’s get started with some “beefed up” micro-benchmarks for the imminent shootout. In one of my comments I wrongly called the Yarv tests “standard”. That was unfortunate wording, because there are no “standard” benchmarks that we can rely on even minimally in the Ruby community. Let’s fix that.

I created an empty project on GitHub, called Ruby Benchmark Suite. This project will hold a set of benchmarks that VM implementers can use to monitor their own progresses and that I can use to run periodical shootouts between all of the major implementations. I also created a Lighthouse project, so that we can have some support for communication and project management. For on going discussion about the project, I created a public Google Group which I invite you to join, if you’re interested in helping out. I’d like to see VM implementers get involved with this, in order to make it a set of reasonable, standard benchmarks that we can all agree upon.

For the next shootout, I’d like to start my multiple testing within the next week or two. So it’d be great if we could come up with a bunch of new tests and revisit the existing ones. What I’d like to see is the following:

• Eliminate or modify flawed tests from the Yarv collection of benchmarks. That means, remove the chance that a given VM could optimize out the actual computing, yielding surprising yet useless results;
• Focus on a multitude of simple, cross-platform programs that employ the Core and Standard libraries. These small programs should test a variety of operations (e.g. number crunching, text processing, etc…);
• There can be several types of application benchmarks, from simple algorithms to a program that performs statistical analysis of a large web server log file (for example) — and anything in between;
• The code in the project will be released under the MIT license. That means that if you contribute code that it’s not your own, you need to ensure that it’s released under a compatible license before we can include it. You’re free to place your name and copyright note at the top of the file, in a comment, but if you send it to us, you agree to release it under the MIT license.
• If you’re not too familiar with GitHub, send over your programs by email (acangiano (at) gmail.com) or by submitting them to the Google Group (assuming they are small). Please specify whether you’d like a “Submitted by” line and if you want your email to be included.
• Not all contributions that we receive will be included, but the greater the variety, the smaller the gap between the benchmarks and real world performances will be.

I hope I can count on your help for this project.