Programming Zen

Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Windows

Antonio Cangiano

In my latest article I discussed the importance of JRuby as a means of introducing Ruby to the Enterprise world. Most of the companies that belong to this ecosystem are Java based, but we cannot forget that a sizable portion of them are Microsoft-centric. Within these companies, Ruby will be far more welcome if a .NET implementation is available. The answer to this need is sufficiently fulfilled by IronRuby (version 0.9 was just released).

IronRuby has been progressing fast lately. First came support for Rails and then, with this release, a great deal of effort has been placed on improving performance. In the past, IronRuby was all but fine-tuned. In fact, it was several times slower than Ruby MRI, as the team worked on improving compatibility with Ruby 1.8 and mostly ignored performance.

In this article I’m going to provide some performance results for IronRuby 0.9 on Windows, which I’m sure will interest readers of this blog as well as of my book. Before revealing all the details, let’s start with the setup and a disclaimer. Please read through it carefully, because the old, trite comments about how “micro-benchmarks are useless” won’t be published. We’ve already been there, folks. Thank you all for your understanding.

Setup

  • The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
  • Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
  • The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available (2 tests) or the setup on my machine was lacking the required libraries to execute it (1 test).
  • I realize that there is a MinGW One-Click Installer effort that may improve the performance of Ruby on Windows. Here however, I used the regular One-Click Installer as it is the most common one (it’s been downloaded almost 3.5 million times so far). For Ruby 1.9 I used the binary provided by the download page on the official Ruby site.

Disclaimer

  • An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how IronRuby compares to MRI (Ruby 1.8) and KRI (Ruby 1.9) performance-wise, but cannot be viewed as a guarantee of how this will actually affect your own programs.
  • I didn’t calculate a RubySpec completeness score for IronRuby. That said, IronRuby 0.9 should be a fairly complete implementation of Ruby 1.8.
  • There are lies, damned lies, and statistics.

Benchmark results

The table below shows the times for each benchmark, for IronRuby 0.9, Ruby 1.8.6 (2008-08-11 patchlevel 287) and Ruby 1.9.1p0 (2009-01-30 revision 21907):

Benchmark File # Ruby 1.8.6 IronRuby Ruby 1.9.1
macro-benchmarks/bm_gzip.rb 100 Timeout IOError N/A
macro-benchmarks/bm_hilbert_matrix.rb 20 1.891 0.453 0.125
macro-benchmarks/bm_hilbert_matrix.rb 30 7.422 1.719 0.656
macro-benchmarks/bm_hilbert_matrix.rb 40 21.500 4.625 2.266
macro-benchmarks/bm_hilbert_matrix.rb 50 56.765 10.031 5.109
macro-benchmarks/bm_hilbert_matrix.rb 60 111.859 18.781 11.297
macro-benchmarks/bm_norvig_spelling.rb 50 Timeout 41.313 31.453
macro-benchmarks/bm_sudoku.rb 1 43.734 Timeout 6.313
micro-benchmarks/bm_app_factorial.rb 5000 1.328 0.063 0.266
micro-benchmarks/bm_app_fib.rb 30 6.156 0.594 0.813
micro-benchmarks/bm_app_fib.rb 35 74.125 6.922 9.344
micro-benchmarks/bm_app_mandelbrot.rb 1 11.953 6.922 0.641
micro-benchmarks/bm_app_pentomino.rb 1 Timeout 59.938 75.859
micro-benchmarks/bm_app_strconcat.rb 1.5M 30.469 2.141 4.813
micro-benchmarks/bm_app_tak.rb 7 5.516 0.531 0.578
micro-benchmarks/bm_app_tak.rb 8 15.609 1.484 1.703
micro-benchmarks/bm_app_tak.rb 9 45.843 3.953 4.531
micro-benchmarks/bm_app_tarai.rb 3 19.985 1.844 2.156
micro-benchmarks/bm_app_tarai.rb 4 19.796 2.219 2.656
micro-benchmarks/bm_app_tarai.rb 5 24.235 2.688 3.063
micro-benchmarks/bm_binary_trees.rb 1 Timeout 53.078 37.375
micro-benchmarks/bm_count_multithreaded.rb 16 0.297 0.266 0.328
micro-benchmarks/bm_count_shared_thread.rb 16 0.250 0.188 0.203
micro-benchmarks/bm_fannkuch.rb 8 3.625 0.344 0.563
micro-benchmarks/bm_fannkuch.rb 10 Timeout 40.750 65.438
micro-benchmarks/bm_fasta.rb 1M 192.937 23.703 35.234
micro-benchmarks/bm_fractal.rb 5 43.172 4.672 5.781
micro-benchmarks/bm_gc_array.rb 1 228.672 32.031 59.828
micro-benchmarks/bm_gc_mb.rb 500K 8.109 1.109 0.469
micro-benchmarks/bm_gc_mb.rb 1M 16.172 2.391 1.016
micro-benchmarks/bm_gc_mb.rb 3M 44.953 6.906 2.938
micro-benchmarks/bm_gc_string.rb 1 47.937 25.250 11.938
micro-benchmarks/bm_knucleotide.rb 1 9.625 2.906 2.016
micro-benchmarks/bm_lucas_lehmer.rb 9689 122.672 18.125 36.250
micro-benchmarks/bm_lucas_lehmer.rb 9941 156.750 19.625 39.391
micro-benchmarks/bm_lucas_lehmer.rb 11213 179.915 28.844 61.063
micro-benchmarks/bm_lucas_lehmer.rb 19937 Timeout 159.078 Timeout
micro-benchmarks/bm_mandelbrot.rb 1 Timeout 65.781 81.766
micro-benchmarks/bm_mbari_bogus1.rb 1 0.031 40.406 8.781
micro-benchmarks/bm_mbari_bogus2.rb 1 0.156 Timeout N/A
micro-benchmarks/bm_mergesort_hongli.rb 3000 25.282 3.531 6.031
micro-benchmarks/bm_mergesort.rb 1 24.735 3.906 3.219
micro-benchmarks/bm_meteor_contest.rb 1 147.704 19.713 19.781
micro-benchmarks/bm_monte_carlo_pi.rb 10M 79.406 5.109 20.672
micro-benchmarks/bm_nbody.rb 100K 37.625 8.281 10.938
micro-benchmarks/bm_nsieve_bits.rb 8 69.656 33.156 6.531
micro-benchmarks/bm_nsieve.rb 9 58.344 5.453 N/A
micro-benchmarks/bm_partial_sums.rb 2.5M 93.391 10.797 26.422
micro-benchmarks/bm_pathname.rb 100 Timeout Timeout Timeout
micro-benchmarks/bm_primes.rb 3000 21.359 9.594 0.031
micro-benchmarks/bm_primes.rb 30K Timeout Timeout 0.469
micro-benchmarks/bm_primes.rb 300K Timeout Timeout 5.281
micro-benchmarks/bm_primes.rb 3M Timeout Timeout 100.406
micro-benchmarks/bm_quicksort.rb 1 51.046 11.594 8.703
micro-benchmarks/bm_regex_dna.rb 20 181.172 21.188 11.938
micro-benchmarks/bm_reverse_compliment.rb 1 61.875 48.469 138.047
micro-benchmarks/bm_so_ackermann.rb 7 2.234 0.563 0.484
micro-benchmarks/bm_so_ackermann.rb 9 50.000 14.938 9.281
micro-benchmarks/bm_so_array.rb 9000 26.328 8.984 10.781
micro-benchmarks/bm_so_count_words.rb 100 Timeout 60.688 42.250
micro-benchmarks/bm_so_exception.rb 500K 78.125 Timeout 32.672
micro-benchmarks/bm_so_lists_small.rb 1000 13.906 4.250 3.172
micro-benchmarks/bm_so_lists.rb 1000 64.531 22.266 16.797
micro-benchmarks/bm_so_matrix.rb 60 8.312 2.781 2.125
micro-benchmarks/bm_so_object.rb 500K 16.375 5.313 1.672
micro-benchmarks/bm_so_object.rb 1M 29.312 10.500 2.844
micro-benchmarks/bm_so_object.rb 1.5M 43.312 16.000 4.281
micro-benchmarks/bm_so_sieve.rb 4000 241.922 37.859 35.688
micro-benchmarks/bm_socket_transfer_1mb.rb 10K 13.266 SocketError 3.359
micro-benchmarks/bm_spectral_norm.rb 100 5.110 0.922 0.719
micro-benchmarks/bm_sum_file.rb 100 Timeout 20.406 23.797
micro-benchmarks/bm_word_anagrams.rb 1 70.828 30.188 8.125
TOTAL TIME – 2933.334 607.088 664.094

Red values are errors, timeouts, inapplicable tests and times that were worse than Ruby 1.8.6. Green, bold values are better times than what Ruby 1.8.6 delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the runtime for the subset of benchmarks that were successfully executed by all three implementations (whose cardinality is 54).

The total runtime is summarized by the chart below:

Total runtime

And let’s compare each of the “macro-benchmarks” on an individual basis:

Macro-benchmarks chart

Conclusions

IronRuby went from being much slower than Ruby MRI to considerably faster across nearly all the tests. That’s major progress for sure, and the team behind the project deserves mad props for it.

One final warning before we get too excited here. IronRuby is not faster than Ruby 1.9.1 at this stage. Don’t let that first chart mislead you. While it’s faster in certain tests, it’s also slower is many others. Currently, it’s situated between Ruby 1.8.6 and Ruby 1.9.1, but much closer to the latter. The reason why this chart is misleading is that it doesn’t take into account any tests that timed out, and several of such timeouts were caused by IronRuby (more than those caused by Ruby 1.9.1). If you were to add, say, 300 seconds to the total, for each timeout for the two implementations, you’d quickly see that Ruby 1.9.1 still has the edge. The second chart that compares macro-benchmarks does a better job at realistically showing how IronRuby sits between Ruby 1.8.6 and Ruby 1.9.1 from a performance standpoint. If you were to plot every single benchmark on a chart, you’d find a similar outcomes for a large percentage of the tests.

Whether it’s faster than Ruby 1.9 or not, now that good performances are staring to show up, it’s easier to see IronRuby delivering on it’s goal of becoming the main implementation choice for those who both develop and deploy on Windows. This, paired with the .NET and possible Visual Studio integration, the great tools available to .NET developers, and the ability to execute Ruby code in the browser client-side thanks to projects like Silverlight/Moonlight and Gestalt, make the project all the more interesting.

What are your thoughts on IronRuby, and how will this dramatic performance gain affect your projects?

Antonio Cangiano

Antonio Cangiano is an Engineering Manager and AI Specialist at IBM. He authored Ruby on Rails for Microsoft Developers (Wrox, 2009) and Technical Blogging (The Pragmatic Bookshelf, 2012, 2019). He is also the Marketing Lead for Cognitive Class, an educational initiative which he helped grow from zero to over 3 Million students. You can follow him on Twitter.

19 responses to “Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Windows”

  1. Luis Lavena

    Antonio,

    You’re comparing IronRuby performance with a 12 years old compiler (VC6) which is the one used by official One-Click Installer.

    Some of the test that failed for your benchmark are related to the lack of supporting DLL need by the binaries you’ve downloaded from Ruby-lang website.

    While you decided to base your test on that specific obsolete compiler, it just hide the effort we as RubyInstaler team has been doing for the past year.

    Would really like and thank if you can take the effort to test against it and keep VC6 based on as reference numbers.

    Thank you.

  2. Antonio Cangiano

    Dear Luis,

    > You’re comparing IronRuby performance with a 12 years old compiler (VC6) which is the one used by official One-Click Installer.

    I’m comparing IronRuby with the official One-Click Installer because, today, that’s what most people use.

    > Some of the test that failed for your benchmark are related to the lack of supporting DLL need by the binaries you’ve downloaded from Ruby-lang website.

    Actually, only one test failed because of this and it’s macro-benchmarks/bm_gzip.rb. Ruby 1.9.1 from the Ruby-Lang site was missing the Zlib.dll.

    > While you decided to base your test on that specific obsolete compiler, it just hide the effort we as RubyInstaler team has been doing for the past year.

    I don’t think this is a fair assessment. I mentioned your project within the Setup section. Want to know what’s really hiding your commendable effort?

    I couldn’t find a single place where I could click, download, and install it. Instead all I found was a GitHub repo, that required me to have the “obsolete” One-Click Installer installed anyway, and build this version myself.

    This is not hard to do by any means, but how many people will? Trust me, relatively few. The whole point of the One-Click Installer is that you download an installer, click on it, and you are done.

    > Would really like and thank if you can take the effort to test against it and keep VC6 based on as reference numbers.

    It would be my pleasure to create a followup post comparing the performance of the official One-Click Installer and your version. As an implementer, you are also welcome to join the RBS efforts if you like.

    Thanks,
    Antonio

  3. DotNetKicks.com

    Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Window…

    You’ve been kicked (a good thing) – Trackback from DotNetKicks.com…

  4. Matt Haley

    It’s unfortunate that the newer installer saren’t more widley known. They perform well and are stable.

    Downloads are at http://rubyinstaller.org/downloads/

  5. Antonio Cangiano

    Hi Matt,

    it is unfortunate. Thanks for the link. As I mentioned to Luis, it’s not that they don’t exist, it’s that they are not easily found.

    So as a result very few people use them. More on the importance of technical marketing can be found in one of my recent posts.

    I’d be glad to give these downloads more exposure in my followup post tomorrow, when I compare their performance with that of the official One-Click Installer.

  6. Marco Mastrodonato

    Hi antonio, congratulations for this article, really interesting! I’d like to know where jruby would was.

  7. Antonio Cangiano

    Ciao Marco,

    I couldn’t get JRuby to run the suite last night, so it wasn’t included. I will definitely benchmark it in the future.

  8. Luis Lavena

    Dear Antonio,

    >I don’t think this is a fair assessment. I mentioned your project within the Setup section. Want to know what’s really hiding your commendable effort?

    > I couldn’t find a single place where I could click, download, and install it. Instead all I found was a GitHub repo, that required me to have the “obsolete” One-Click Installer installed anyway, and build this version myself.

    Announcement about One-Click not dead, at RubyForge, visible in the RubyForge page:

    http://rubyforge.org/projects/rubyinstaller/

    http://rubyforge.org/forum/forum.php?forum_id=33233

    Project information at GitHub

    http://wiki.github.com/oneclick/rubyinstaller/

    And even generated a FAQ about it.

    http://wiki.github.com/oneclick/rubyinstaller/faq

    Preview1 release annouced at the blog:

    http://blog.mmediasys.com/2009/06/28/rubyinstaller-preview1-released/

    Which also links to the announcement at ruby-talk.

    > This is not hard to do by any means, but how many people will? Trust me, relatively few. The whole point of the One-Click Installer is that you download an installer, click on it, and you are done.

    Test drive and introduce to the installers:

    http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-sqlite3/
    http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-mysql/

    You’re right, there is no place to click and download.

    I’ve considered the installers in a preview state, mostly because we are ironing several details to improve the transition from current One-Click.

    But you cannot say it wasn’t stated anywhere.

    Yes, we are working in getting it a better place to show the progress:

    http://wiki.github.com/oneclick/rubyinstaller/website-contest

    But again, was not widely exposed as you mention, we are working on that. Apologize wasn’t visible or me wasn’t reachable during your research period to help you out getting the best possible tools to perform this benchmark.

  9. Paolo

    I agree with Antonio.

    One that doesn’t know about the project or doesn’t trusts one of the posts above will never download and install anything from a site like http://rubyinstaller.org/ which just displays a bare “Coming soon” page.

    Technical marketing is really as important as any other marketing. A lot of good development projects (made for money or just for fun) has never produced any result because of bad communication.

    @Antonio: the IronRuby you tested is compatible with Ruby 1.8.6, 1.8.7, 1.9 or is something entirely different?

  10. Ian Dees

    This is not hard to do by any means, but how many people will? Trust me, relatively few.

    Especially when the quirks of individual systems and download mirrors get in the way of this process. There are so many things that can go wrong (and have gone wrong, in my tests) that are beyond the control of the one-click maintainers.

  11. Luis Lavena

    At Paolo: Commented about this but my post get moderated (which is weird since the first one wasn’t).

    If Antonio approves it you will find answers to your concerns.

  12. Antonio Cangiano

    @Paolo: They aim to support 1.8.x. At this stage it should be 1.8.6.

  13. Antonio Cangiano

    @Luis: Your comment was caught by the anti-spam due to the presence of numerous links. However, I promptly approved it as soon as I saw your second comment mentioning the issue.

  14. IronRuby Testing and development » Blog Archive » IronRuby 0.9 Perf gains

    […] Cangiano has reported on the state of IronRuby on Windows, where IronRuby is showing the fruits of Tomas’ labor: […]

  15. A faster Ruby on Windows is possible (benchmarks for 4 implementations inside) | Zen and the Art of Programming

    […] Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Windows […]

  16. Charles Roper

    Antonio, I explicitly mentioned the preview release of the new RubyInstaller in a thread on the book promotion you were running over on RubyLearning… I guess you must have missed that message!

  17. Antonio Cangiano

    Charles, I remember reading your insightful comment on RubyLearning, but somehow I didn’t think about the preview release. Thanks for your help on the forum, btw.

  18. brianoh

    Thanks for posting the benchmarks Antonio. I think it represents a genuine intention to be unbiased, but a follow up if you have the time and inclination would be good at some point (or have you already?).

  19. Guest Post: What’s IronRuby, and how do I put it on Rails? « IUpdateable from Eric Nelson

    […] passes about 98%. It also benchmarks considerably (more than 4x) faster than MRI 1.8, as Antonia Cangiano demonstrates with his suite of Ruby […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.