Someone on Twitter politely asked me when I was going to run a new Ruby shootout. I replied that I wasn’t planning to. This got me thinking though that he might not be the only reader wondering about this point, so I thought I’d share a few of the reasons behind my decision not to carry on with this series.
My last shootout (over two years ago now) and its predecessors were highly popular posts. In fact, benchmarking in general tends to attract a lot of attention. It wouldn’t really cost me much to prepare a shootout every, say, six to twelve months. The reason I have not done so again is that I’ve come to view these posts as not being of much value to the community.
You can use as many disclaimers as you want in your report, but many readers will still just take a glance at a couple of graphs and draw broad (and not particularly accurate) conclusions.
An implementation could specifically target these benchmarks, optimizing for them, without a real, general performance improvement. Or more often, be so incomplete that doing well in micro-benchmarks becomes almost trivial. (That said you might still see other benchmarks from time-to-time on this blog, when appropriate.)
Even when it comes to complete implementations, a good performance on synthetic micro-benchmarks only gives us a vague idea of how well an application will be run within the given implementation. Why not test your own code with the major implementations you are interested in using instead?
This actually bring us to another point. These days you can easily install multiple versions of Ruby via tools like RVM or rbenv. So if you do intend to test your application with Ruby, JRuby, and Rubinius, for example, you could do so easily and painlessly. And that’s really the best way to choose an implementation (at least as far as performance and ensuring compatibility are concerned).
I understand that there is some value in showing a ballpark comparison, however arbitrary, but I also feel that it tends to separate, rather than unite, the community. All other implementations, but the winning one, end up worse off.
The various Ruby implementers are welcome to continue running these tests if they’re useful to them (keep in mind that the code is mostly a mix and match of very old tests and therefore quite messy). And of course, other bloggers are welcome to run and publish their own comparisons if they feel that doing so is worthwhile. But for all the aforementioned reasons, I don’t plan to continue holding shootouts here.
Get more stuff like this
Subscribe to my mailing list to receive similar updates about programming.
Thank you for subscribing. Please check your email to confirm your subscription.
Something went wrong.
