Meditations on programming, startups, and technology
New Relic

Benchmarking MacRuby 0.6

Recently MacRuby 0.6 was released. The development team put a lot of emphasis on improving compatibility with Ruby 1.9, and the viability of MacRuby as a tool for developing Mac OS X applications. Focus on these aspects took precedence over performance, but I was still curious to see how well it performed when compared to Ruby 1.8.7 and Ruby 1.9, respectively.

This article showcases the results of a small Ruby shootout for Mac. I plan to publish a Windows one by next week, and then a week or two after that, a complete Linux shootout that will have many more implementations. Grab my feed or join the newsletter to avoid missing upcoming shootout posts.

The setup

The tests are a large subset of the Ruby Benchmark Suite. Each test was run 10 times, five to detect the best execution time, and five to detect the minimal memory consumption. All of the tests were run on Mac OS X 10.6.3, on my MacBook Pro 2.66 GHz Intel Core 2 Duo, 4 GB 1067 MHz DDR3 RAM, 320 GB 7200 rpm disk.

Stable implementations tested:

  • MacRuby 0.6
  • Ruby 1.8.7
  • Ruby 1.9.1

Disclaimer

Synthetic benchmarks cannot predict how fast your programs will be with one implementation or another. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. Furthermore, the Ruby Benchmark Suite has many tests that don’t provide much insight when it comes to comparing implementations. They are there for legacy reasons and will probably be removed in the future. For the time being, take them with a grain of salt.

The results

Without further hesitation, here are the execution times for the tests (divided in A-L, M-Z). Timeouts indicate that the execution of a single run took more than 60 seconds and had to be interrupted. Bold values indicate the best performance for each test.

Time table A-L

Time table M-Z

And here is the estimated memory usage:

Memory table A-L

Memory table M-Z

Conclusions

MacRuby 0.6 is faster than Ruby 1.9.1 at times, but it can also be significantly slower. Overall, as things stand now, its performance appears to be between that of Ruby 1.9.1 and Ruby 1.8.7, with several outliers and a greater variance compared to those two implementations. Memory wise, MacRuby appears to be significantly more “memory hungry” than the other implementations (even though this wasn’t all that much of a surprise to me).

I’m interested in seeing how future releases that will be focused more on performance will affect these preliminary results. For the time being however, don’t let this outcome discourage you from using MacRuby 0.6, which is the first release that’s considered stable for Mac OS X development.

Download: CSV Files

PS: If you are looking for a fun and easy way to get started with MacRuby, check out ThinkCode.TV’s screencast on the subject.

Update (July 3, 2010): The following box plot compares the various implementations for the tests for which all the implementations were successful. Only times for the largest successful input number were used in those tests where multiple input numbers were tested.

Mac Shootout Boxplot


If you enjoyed this post, then make sure you subscribe to my Newsletter and/or Feed.

receive my posts by email

9 Responses to “Benchmarking MacRuby 0.6”

  1. Bil Kleb says:

    Is there a way to show the results to only two significant digits?

    • Hi Bil,

      I changed the tables so as to show three decimal places. This should be good enough. You may also want to download and play with the CSV files at the bottom of the article.

  2. Joshua says:

    Hi, thanks for the comparison.
    Are there any chances of including JRuby 1.5 and Rubinius 1.0 to the list?

  3. Fred says:

    How were they compiled?

    Are they all 32bit or 64bit?

    Which GCC Version for each ruby version? gcc4.4? gcc4.1?

    I believe it matters on the memory usage.

    • gcc version 4.2.1 (Apple Inc. build 5659)

      Both Ruby versions were 32bit, while MacRuby was 64bit only. This favors MacRuby when it comes to run time, but penalizes it in terms of memory usage.

      I went with this combination, because it’s what most people would use on Mac OS X.

      I thought I indicated the architecture in my post, but I apparently forgot to. Thank you for giving me an opportunity to clarify that.

  4. r kumar says:

    Any links to samples created using macruby ?

    Thanks.

  5. roger says:

    I believe all the code he used were from the RBS http://github.com/acangiano/ruby-benchmark-suite

  6. […] shootout on Windows that I recently conducted. You can find the Mac edition, published last month, here. I was planning to have this one ready much sooner, but a couple of serious events in personal life […]

  7. […] Ruby implementations by testing them against a series of synthetic benchmarks. Recently I ran Mac and Windows shootouts as well, which tested a handful of implementations. However this article […]

Leave a Reply

I sincerely welcome and appreciate your comments, whether in agreement or dissenting with my article. However, trolling will not be tolerated. Comments are automatically closed 15 days after the publication of each article.

Copyright © 2005-2014 Antonio Cangiano. All rights reserved.