The Great Ruby Shootout measures the performance of several Ruby implementations by testing them against a series of synthetic benchmarks. Recently I ran Mac and Windows shootouts as well, which tested a handful of implementations. However this article reports on the results of extensive benchmark testing of eight different Ruby implementations on Linux.
For this shootout I included a subset of the Ruby Benchmark Suite. I opted to primarily exclude tests that were executed in fractions of a second in most VMs, focusing instead of more substantial benchmarks (several of which came from the Computer Language Benchmarks Game). The best times and least memory allocations out of five runs are reported here for each benchmark.
All tests were run on Ubuntu 10.4 LTS x86_64, on an Intel Core 2 Quad Q6600 2.40 GHz, 8 GB DDR2 RAM, with two 500 GB 7200 rpm disks.
The implementations tested were:
JRuby was run with the –fast and –server optimization flags.
Synthetic benchmarks cannot predict how fast your programs will be when dealing with a particular implementation. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. The values reported here should be assumed to be characteristic of server-side — and long running — processes; they should be taken with a grain of salt.
Please find below the execution times for the selected tests. Timeouts indicate that the execution of a single iteration for a given test took more than 300 seconds and had to be interrupted. Bold, green values indicate the best performer out of each test.
Warning: The bm_primes.rb benchmark was originally written to aid the development of the Prime class. As such in 1.9.2 it was rewritten in C, which makes it a poor representation of implementation performance. This benchmark will removed in the future.
If you are not interested in the individual test results, the information presented in the table above is summarized directly below:
Ruby 1.9.2 JRuby Min. : 0.013 Min. : 0.382 1st Qu.: 3.258 1st Qu.: 3.051 Median : 4.543 Median : 4.997 Mean : 9.262 Mean : 9.180 3rd Qu.: 8.573 3rd Qu.: 8.969 Max. :45.009 Max. :48.850 MagLev Ruby 1.9.1 Min. : 0.351 Min. : 0.015 1st Qu.: 2.140 1st Qu.: 3.387 Median : 6.069 Median : 6.205 Mean : 9.100 Mean :10.860 3rd Qu.: 9.266 3rd Qu.:11.559 Max. :51.221 Max. :46.849 Ruby 1.8.7 IronRuby Min. : 0.708 Min. : 3.601 1st Qu.: 5.102 1st Qu.: 10.505 Median : 8.380 Median : 12.912 Mean :18.785 Mean : 26.539 3rd Qu.:24.793 3rd Qu.: 36.115 Max. :75.653 Max. :135.204 Rubinius REE Min. : 0.484 Min. : 0.584 1st Qu.: 3.087 1st Qu.: 4.343 Median : 9.636 Median : 6.660 Mean :13.232 Mean :15.036 3rd Qu.:17.674 3rd Qu.:21.336 Max. :73.050 Max. :61.960
For the sake of convenience, I also produced a box plot from the successful data points:
There are a few considerations based on these results that I feel are worth mentioning:
The following table shows the approximate memory consumption for each implementation when running each benchmark:
Ruby 1.9.2 Ruby 1.9.1 Min. : 4.320 Min. : 4.580 1st Qu.: 4.378 1st Qu.: 4.695 Median : 6.285 Median : 6.920 Mean : 20.795 Mean : 15.669 3rd Qu.: 10.162 3rd Qu.: 11.383 Max. :171.500 Max. :100.570 Ruby 1.8 REE Min. : 3.040 Min. : 8.220 1st Qu.: 4.290 1st Qu.: 9.682 Median : 7.745 Median : 15.565 Mean : 20.698 Mean : 27.014 3rd Qu.: 11.273 3rd Qu.: 38.620 Max. :103.520 Max. :125.910 Rubinius MagLev Min. : 37.63 Min. : 81.74 1st Qu.: 39.78 1st Qu.: 82.52 Median : 45.48 Median : 83.53 Mean : 65.70 Mean : 96.29 3rd Qu.: 58.22 3rd Qu.: 98.10 Max. :215.33 Max. :175.85 JRuby Min. : 49.04 1st Qu.: 71.23 Median :176.72 Mean :169.41 3rd Qu.:209.04 Max. :404.06
And finally, in graph form:
A few considerations on memory:
Linux Vs. Windows
This shootout and the Windows one were both performed on the same machine, thus we can compare how the same implementation perform under different operating systems. The only adjustment that’s required is the timeout. Every result longer than 60 seconds from this shootout needs to be considered a timeout, because the previous shootout did so as well.
It is commonly believed that Ruby performs much better on Linux than on Windows (with the exception of IronRuby). Let’s find out if these test results confirm that notion.
Finally, in chart form (yellow entries are on Windows as indicated by the labels containing W):
To use a beloved MythBusters expression, this myth is confirmed.
Note: As requested by a few commenters, here is a comparison of IronRuby as well (.NET 4.0 Vs. Mono 2.4.4):
In conclusion, let me just state that it’s nice to see several implementations getting faster. Ruby 1.9.2, JRuby, MagLev and Rubinius are all becoming serious competitors and working their respective ways closer to a similar performance level. If you think these benchmark shootouts are becoming boring, then the results are becoming more stable and predictable. I suspect that as time goes on, performance will not be the real distinguishing factor when choosing a Ruby implementation, other features will be.
No related posts.
I sincerely welcome and appreciate your comments, whether in agreement or dissenting with my article. However, trolling will not be tolerated. Comments are automatically closed 15 days after the publication of each article.