In yesterday’s post I compared IronRuby 0.9, Ruby 1.8.6 (from the One-Click Installer) and Ruby 1.9.1 (downloaded from the official site) against one another. IronRuby did great, but the discussion in the comment section quickly veered towards what version of the One-Click Ruby Installer should have been used.
I justified my choice of using the “old” One-Click Installer, by the fact that I wasn’t aware of official releases of the new installer. As well as that the old One-Click Installer is the most widely downloaded version. Very few people are familiar with the upcoming version of the project. This point is about to change.
Luis Lavena took over the One-Click Installer project and has been working on the next version (RubyInstaller from now on), the aim of which is to replace the One-Click Installer by building Ruby 1.8 and 1.9 with MinGW and GCC. In theory, this brings performance gains on Windows to the table, and gets rid of having to use Visual C++ 6 (a “ten year old compiler”) to build Ruby and other native gems. The project also strives to be lighter by bundling fewer (unnecessary) gems for Windows users.
There’s no doubt that in the long run, this new project will become the de facto standard for Windows, but the questions on everyone’s mind are, should I bother with it now? How much of a performance boost are we talking about here? 10%? 20%? Let’s find out.
In this follow up article I’m going to compare the performance of Ruby 1.8.6 from the One-Click Installer (mswin32), Ruby 1.8.6 from RubyInstaller (mingw32), Ruby 1.9 (mswin32) downloaded from the Ruby-Lang.com site, and Ruby 1.9 from the RubyInstaller project (mingw32) against one another. I’ll copy and paste part of the setup and disclaimer from yesterday’s post, for those who haven’t read it. Feel free to skip this part if you wish.
Setup
- The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
- Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
- The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available or the setup on my machine was lacking the required libraries to execute it (1 test affected).
Disclaimer
- An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how these implementations compare, but cannot be viewed as a guarantee of how they will actually affect your own programs.
- There are lies, damned lies, and statistics.
Benchmark results
The table/image below shows the times for each benchmark, for Ruby 1.8.6 (mswin32), Ruby 1.8.6 (mingw32), Ruby 1.9.1 (mswin32), and Ruby 1.9.1 (mingw32). In the table I used (RI) as shorthand for RubyInstaller to indicate mingw32 versions.
Red values are errors, timeouts and inapplicable tests. Green, bold values indicate better times than what Ruby 1.8.6 (mswin32) delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the run-time for the subset of benchmarks that were successfully executed by all four implementations. Timeouts have been included this time around. Each timeout has been counted as an additional 300 seconds.
The total runtime (including timeouts) is summarized in the chart below:
Conclusion
Wow! Ruby 1.8.6 (mingw32) improves from 3% to 664% (depending on the test), over the current One-Click Installer. The geometric mean of the ratios (read “on average”) tells us that it was about 283% faster. The Ruby 1.9.1 version provided by the RubyInstaller was slower than the mswin32 version in a couple of tests, but faster everywhere else. How much faster? Up to 342% faster, with an average (again calculated through the geometric mean of the ratios) of a 77% increase in speed.
These finds prompt me to ask, what are we waiting for? You know how unresponsive Ruby is on Windows, and how tests take forever to execute? These mingw32-based releases may very well solve this. And incidentally the bar has been raised for IronRuby as well. I formally invite the Ruby on Windows community to embrace these two projects.
Get more stuff like this
Subscribe to my mailing list to receive similar updates about programming.
Thank you for subscribing. Please check your email to confirm your subscription.
Something went wrong.
I use a script that fetches a year’s worth of data for approximately 3500 stocks. Using Ruby 1.8(mingw32) it takes about 17 minutes to complete, With Ruby 1.9 (mingw32) it takes about 8 minutes. I did not expect such improvement since the script spends a lot of time primarily fetching data from a remote site.
My results for this one specific script essentially agrees with the result you obtained.
I used to have ruby 1.8 (mswin32) but I have uninstalled it.
Luis has been doing an awesome solo work. Unfortunately there are too few programmers investing time in Windows. I mean, the main barrier has been to convince gem developers to use Luis’ SDK for mingw32 and have their native extensions compile to “universal binaries” (yes, his SDK gives that for free). So, what we are waiting for is for gem developers who have native extensions to jump on board, spend a couple of hours and update their gems.
Hi!
There’s something I don’t understand. Why are total times for 1.8 and 1.9 (not RI) different from yesterdays benchmarks?
Thanks!
@Marcos: The total times are different because today’s results include timeouts. For each timeout, 300 seconds were added to the given implementation. It makes totals a bit more realistic and fair.
What, no JRuby benchmarks on Windows??? 🙂
@Vladimir: You are like Oliver Twist, always asking for more. 🙂
and how do these figures compare to ruby on linux?
Hey, if you’re going to add JRuby, add 1.9.1 built with VC++ 9, too. 🙂
Regards,
Dan
It really make me feel like to install mingw ruby again. Last time it has some problem on irb(consume a lot of cpu processing).
@sgwong:
RubyInstaller ships with this:
http://github.com/luislavena/rb-readline
A replacement of Readline CPU usage.
@Daniel:
http://rubyforge.org/pipermail/rubyinstaller-devel/2008-January/000230.html
Unless we turn PGO (which is not in the express edition) VC9 perform worse than GCC, at least on Ruby and they way it uses Win32API.
@Daniel Berger
Isn’t the “1.9” column already compiled with VC2008? 🙂
@Antonio nice writeup. Glad to see the benchmarks–I think the most interesting result is 1.8.6 mingw performance over mswin–using it as a replacement for the current one click makes rails programming on windows almost tolerable because it is quite faster. Anybody still using the old OCI out there, try it!
Here’s how:
http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-mysql/
http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-sqlite3/
In answer to “what are you waiting for” I think most users are waiting for the official one click page to list these as download options–currently they’re basically only “in house” downloads, though hopefully that will change soon, thanks to your recent post.
Also note that for very aggressive users you could still squeeze more performance out by tweaking GC_MALLOC_LIMIT and/or using GCC 4.4 with profile guided optimizations, etc. Watch my blog for eventual releases I hope to make of it.
http://programming-gone-awry.blogspot.com/
Nice work.
=r
Antonio,
Thanks for the answer. So IronRuby still finishes the benchmark faster than Ruby 1.8.6 RI..
This post and the one before were very interesting, I didn’t find any complete comparison of different implementations of Ruby on the web. Would be great to compare what you have measure against JRuby and Ruby on Linux.
Congratulations, really interesting article.
This is a great post that proves 3 points:
1. Antonio is awesome! Happy birthday!
2. Luis Lavena did a phenomenal technical job on Ruby Installer
3. Marketing job on the Ruby Installer is completely opposite to the technical job. Luis needs some help with this.
@Leon: Hahah. You are too good. Thanks!
With the new mingw32 substantial speed improvements, think it makes sense now to also test at least the baseline (MRI) on Mac/Linux on the same battery of tests, so we Windows folks could get a better idea of how far behind are we yet and what the different Windows interpreters speed target shall be.
Note also that cygwin is pretty slow.
For me on a rails test:
1.8.6 mingw: 7.0s
1.8.6 mswin: 7.2s
cygwin 1.8.7 11s
that’s no good 🙂
Hey, can you do comparison of vs2008,vs2010 beta 2 (or newer if avail) with all optimizations and Intel C++ (trial version should do) compiles? In some tests the Intel compiler is 1.5 – 2x faster than 2008 with optimizations.