13 responses

  1. ARJ
    August 10, 2009

    Any comments about why Ruby on Linux appears, from these tests, to be 2x more buggy (1.8.6) or 1.5x more buggy (1.9.1) than the Windows version?

    Does this mean that if a lower risk of error or pathological computation time (generating timeouts in the test codes) is a priority, then I should prefer Windows over Linux?

  2. Antonio Cangiano
    August 10, 2009

    @ARJ: You raise an interesting point. Out of a demanding test suite, only 3 tests did significantly better on Windows than on Linux. Two timed out for certain parameters, and one raised an error.

    Considering all the performance improvements shown in other tests, I’d say that for most mixed workloads, Ruby on Linux would still perform much faster.

    Regarding the lower risk of error, the only difference appears to be related to the stack size. The Windows implementation of Ruby 1.8.6 managed to go through a very deep recursion with no errors. All the other implementations raised a SystemStackError: stack level too deep.

    Chances are that you could tweak the Linux version to allocate more memory for the stack, anyway.

  3. Jon
    August 10, 2009

    First, thanks for your efforts on performance comparisons and your RBS GitHub project!

    I’ve just discovered RBS so I’m not familiar with what each test does. That said, I’m wondering how much IO the tests perform especially in light of the recent 24842, 24839, and 24837 posts on ruby-core. For example:


    FWIW, I’ve been seeing noticeable wall clock differences between 1.9.1 and 1.8.6 on my Windows machines when running rspec. One spec that uses a lot of Kernel.system and Kernel.open takes ~20s on 1.9.1 and ~12s on 1.8.6. There’s also noticeable differences doing simple things like “gem list”.

    It’s not clear what the outcome of the ruby-core posts will be, but I’m wondering if a key part of the differences you’re seeing in RBS results are due to an issue with the 1.9.1 MRI implementation of IO on Windows?

    Any plans on doing still-too-early 1.9.2 comparisons?


  4. Steve Smith
    August 10, 2009

    You probably shouldn’t average the ratios. That makes no mathematical sense at all. Just calculate the average of the ratios at the end.

  5. Antonio Cangiano
    August 10, 2009

    Steve, the approach taken is simple, but statistically sound. I don’t average the ratios. I calculate the ratio for each test, and then at the end I calculate the geometric mean of these ratios. It’s a valid way of summarizing the outcome of these benchmarks.

  6. K159
    August 10, 2009

    MinGW uses either crtdll.dll or msvcrt.dll that ships with windows. crtdll.dll is only available in 32Bit and is fairly old (mine is from 2004) and has problem with threading and is generally considered slow. msvrt.dll is newer but lacking optimisation as well.

    Maybe someone can create a Visual C++ build or an Intel C++ build and test it?

  7. Alex
    August 11, 2009

    Thanks Antonio for the tests and for your kind citation. It is interesting indeed to see that Windows is becoming a worthy platform at least for the dev/stg phases.

    (BTW, love your book and your writing style :-)

  8. Antonio Cangiano
    August 11, 2009

    Thanks Alex. I appreciate it. :)

  9. roger
    August 11, 2009

    I think the good news here is that ruby on windows is *only* 70% slower–I guess this means that 1.9 on windows is faster than 1.8 in Linux :)

    If you compare jruby times on windows and linux, you see similar speed differences–I think so anyway.

    And yes, as noted above, I/O reading tests are far slower in windows as currently the I/O is very slow for some reason in 1.9 + windows.

    Thanks for the writeup.

  10. Evgeniy Dolzhenko
    August 19, 2009

    That would be great to have the comparison of the same Ruby version but on Windows host and virtualized Linux guest on the same host, could be helpful for those who develop on Windows machines in making decision to switch to Linux/run virtual machine, etc.

  11. Jon
    August 19, 2009

    @Antonio: FYI, updated preview2 releases of the RubyInstaller for Windows have recently been released at http://rubyinstaller.org/ in case you’re not already aware.

    Ruby 1.8.6 updated to p383 and 1.9.1 updated to p243 in addition to new CHM-based documentation for core and stdlib.


  12. Isaac Gouy
    September 17, 2009

    So the computer language benchmarks game has always been showing Ruby in a most favorable light 😉

Leave a Reply




Back to top
mobile desktop