Many brilliant developers are working on improving the current implementation of Ruby and on creating alternatives. I was curious about their current respective speeds, so I installed and ran some benchmarks for the most popular implementations. In this article, I’m sharing the results for the community to see.
My tests were conducted on an AMD Athlon™ 64 3500+ processor, with 1 GB of RAM.
The tested Ruby implementations were:
The operating system that was used for all – but Ruby.NET – is Ubuntu 6.10 (for x86). Ruby.NET currently runs on Microsoft Windows only, therefore I’ve used Vista with the .NET Framework 2.0 and have also run Ruby 1.8.5-p12 on Windows as a means of having a more direct comparison with Ruby.NET.
Ruby 1.9, JRuby, Rubinius and Cardinal were all installed using their respective latest development versions from trunk.
The 41 tests used to benchmark the various Ruby implementations can be found within the benchmark folder in the repository of Ruby 1.9. The following is a list of the tests with a direct link to the source code for each of them:
The following table shows the execution time expressed in seconds for Ruby 1.8.5 on Linux, Ruby 1.8.5 on Windows, Ruby 1.9 (Yarv/Rite) on Linux, JRuby on Linux, Gardens Point Ruby.NET on Windows, Rubinius on Linux and finally Cardinal on Linux.
Below is a chart which shows the average and median values, visually:
Of course, the bold green values indicate a positive performance, so for example Cardinal was 4 times faster than Ruby 1.8.5 on Linux for the test vm1_swap, but it was also 18 times slower for so_matrix (therefore in red).
I won’t provide too many personal considerations but rather let you enjoy the numbers. Generally speaking though, Ruby on Windows was about 1.5 times slower than on Linux. Yarv (merged in the development version of Ruby) is clearly the fastest by a long shot. This is good news (there are hopes for a fast Ruby 2.0), and it is not an unexpected result.
Ruby.NET and JRuby had similar performances and were able to execute most of the tests. It is clear though that they will need to focus on improving their individual speeds in the coming future, in order to be ready for prime time.
Cardinal wasn’t able to complete most tests, and was extremely slow in some others. However on a few occasions, it also showed decent results (beating Ruby 1.8.5 in 3 tests). Rubinius was extremely slow too but correctly handled a larger amount of tests than Cardinal was able to (and it was significantly faster in executing so_sieve.rb).
I’d like to conclude by saying that all the people involved with these projects are doing an amazing job. And while some implementations show that they are in an early stage of development, it is in no way detrimental of the great effort and work done by their developers, nor attempts to predict their future success or failure. So once again, great job guys, all of this is nothing short of exciting!
UPDATE 02/21/07: Wow, it looks like this article received a lot of attention and naturally I’m glad it did. Slashdot linked to this and traffic sky rocketed, giving major exposure to all these projects.
Most importantly, I initially thought I’d run another batch of tests in 3 months time, but given the amount of feedback that I’ve received, I’ll carry out another test run fairly soon to incorporate many of the insightful suggestions and requests that were received.
By the way, Ruby 1.8.6 is out in preview, and some of you sent me emails asking to test it out. Running the test shows that it’s usually slightly faster than 1.8.5 and it seems to notably speeds up recursion based tests. The next test run will have details for Ruby 1.8.6 as well.