In a previous article I compared the performance of Ruby on Windows, built through Microsoft Visual C++ and GCC. The numbers for the MinGW version were very impressive. So the question now becomes, how does its performance compare to that of Ruby on Linux? To quote one person (Alex) who commented on the aforementioned post:
With the new mingw32 substantial speed improvements, think it makes sense now to also test at least the baseline (MRI) on Mac/Linux on the same battery of tests, so we Windows folks could get a better idea of how far behind are we yet and what the different Windows interpreters speed target shall be.
Any sort of performance improvement for something that is notoriously slow on Windows is more than welcome, but would this be enough to fill the gap between Ruby’s performance on Windows and on Linux? How much faster is Ruby on Linux? Let’s find out.
Setup
- As a reminder, the operating systems used were Windows XP SP3 32bit and Ubuntu 9.04 32 bit.
- The Ruby implementations tested were ruby 1.8.6 (2009-03-31 patchlevel 368) [i386-mingw32], ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32], Ruby 1.8.6 (built from source on Linux) and Ruby 1.9.1 (built from source on Linux as well). The same identical versions of Ruby were used for both operating systems. I’m aware that these are not the latest available versions for Linux, but we are trying to compare apples to apples.
- I used the Ruby Benchmark Suite; the times reported are the best out of five runs, with a timeout of 300 seconds per each iteration.
Benchmark results
The following table/image compares the performance of Ruby 1.8.6 on Windows and Linux. A light green background indicates which of the two was faster. The total times exclude tests that raised an error or were not available (N/A) for any of the four implementations, but includes timeouts (they are counted as 300 seconds to provide a lower bound). The ratio column indicates how many times faster Ruby on Linux was:
The second table/image below compares Ruby 1.9.1 on Windows and on Linux, using the same criteria as above.
Note: The totals shown are different from the ones seen in other posts since the subset of benchmarks included in the totals is different.
Conclusion
According to the geometric mean of the ratios for these tests, it appears that on average Ruby 1.8.6 on Linux is about twice as fast as Ruby 1.8.6 on Windows. Conversely, Ruby 1.9.1 on Linux is about 70% faster than the Windows version.
The Windows implementations use GCC 3.4.5 (a four year old compiler) at the moment, while I built the implementations on Ubuntu with GCC 4.3.3 (which is available by default). This helps, at least in part, to justify the performance gap. Luis Lavena, leader of the Windows port, confirmed to me that a switch to GCC 4.4.x is planned for the future. This should significantly increase performance on Windows yet again, and bump Ruby’s speed on Windows a bit closer to the speed that’s obtainable on Linux.
For the time being, switching to Ruby 1.9.1 on Windows will give you a performance that is better than what’s obtained by those who are still using Ruby 1.8.x on Linux. If it’s possible, switch.
Get more stuff like this
Subscribe to my mailing list to receive similar updates about programming.
Thank you for subscribing. Please check your email to confirm your subscription.
Something went wrong.
Any comments about why Ruby on Linux appears, from these tests, to be 2x more buggy (1.8.6) or 1.5x more buggy (1.9.1) than the Windows version?
Does this mean that if a lower risk of error or pathological computation time (generating timeouts in the test codes) is a priority, then I should prefer Windows over Linux?
@ARJ: You raise an interesting point. Out of a demanding test suite, only 3 tests did significantly better on Windows than on Linux. Two timed out for certain parameters, and one raised an error.
Considering all the performance improvements shown in other tests, I’d say that for most mixed workloads, Ruby on Linux would still perform much faster.
Regarding the lower risk of error, the only difference appears to be related to the stack size. The Windows implementation of Ruby 1.8.6 managed to go through a very deep recursion with no errors. All the other implementations raised a SystemStackError: stack level too deep.
Chances are that you could tweak the Linux version to allocate more memory for the stack, anyway.
First, thanks for your efforts on performance comparisons and your RBS GitHub project!
I’ve just discovered RBS so I’m not familiar with what each test does. That said, I’m wondering how much IO the tests perform especially in light of the recent 24842, 24839, and 24837 posts on ruby-core. For example:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/24839
FWIW, I’ve been seeing noticeable wall clock differences between 1.9.1 and 1.8.6 on my Windows machines when running rspec. One spec that uses a lot of Kernel.system and Kernel.open takes ~20s on 1.9.1 and ~12s on 1.8.6. There’s also noticeable differences doing simple things like “gem list”.
It’s not clear what the outcome of the ruby-core posts will be, but I’m wondering if a key part of the differences you’re seeing in RBS results are due to an issue with the 1.9.1 MRI implementation of IO on Windows?
Any plans on doing still-too-early 1.9.2 comparisons?
Jon
You probably shouldn’t average the ratios. That makes no mathematical sense at all. Just calculate the average of the ratios at the end.
Steve, the approach taken is simple, but statistically sound. I don’t average the ratios. I calculate the ratio for each test, and then at the end I calculate the geometric mean of these ratios. It’s a valid way of summarizing the outcome of these benchmarks.
MinGW uses either crtdll.dll or msvcrt.dll that ships with windows. crtdll.dll is only available in 32Bit and is fairly old (mine is from 2004) and has problem with threading and is generally considered slow. msvrt.dll is newer but lacking optimisation as well.
Maybe someone can create a Visual C++ build or an Intel C++ build and test it?
Thanks Antonio for the tests and for your kind citation. It is interesting indeed to see that Windows is becoming a worthy platform at least for the dev/stg phases.
(BTW, love your book and your writing style 🙂
Thanks Alex. I appreciate it. 🙂
I think the good news here is that ruby on windows is *only* 70% slower–I guess this means that 1.9 on windows is faster than 1.8 in Linux 🙂
If you compare jruby times on windows and linux, you see similar speed differences–I think so anyway.
And yes, as noted above, I/O reading tests are far slower in windows as currently the I/O is very slow for some reason in 1.9 + windows.
Thanks for the writeup.
-r
That would be great to have the comparison of the same Ruby version but on Windows host and virtualized Linux guest on the same host, could be helpful for those who develop on Windows machines in making decision to switch to Linux/run virtual machine, etc.
@Antonio: FYI, updated preview2 releases of the RubyInstaller for Windows have recently been released at http://rubyinstaller.org/ in case you’re not already aware.
Ruby 1.8.6 updated to p383 and 1.9.1 updated to p243 in addition to new CHM-based documentation for core and stdlib.
Jon
So the computer language benchmarks game has always been showing Ruby in a most favorable light 😉