Ruby Shootout Status Update

I want to provide those who are waiting for the Ruby shootout with a heads up. The benchmark suite needs some substantial changes in order to ensure accuracy and fairness for all the VMs involved.

This will delay the execution (and reporting) of the shootout further, but it will be worth it. I definitely prefer a shootout that’s published later in July (or heck even August) that is realistic, fair and provides interesting metrics (e.g. CPU time and memory) over an inaccurate one that was put together in a rush just for the sake of publishing it tomorrow.

For those interested in the technical details, we are trying to separate the parsing and “compilation” of definitions from the actual execution of the code (which needs to be timed). I accomplished this by creating a Proc for each benchmark, and then tested the time spent executing its call method. The problem with this approach is that it penalizes VMs that don’t JIT procs, like JRuby for example.

We also thought about defining a method instead of a Proc, but eval won’t accept class definitions or constants within methods. The workaround would be using MyClass.class_eval instead of class MyClass in the benchmarks and Module#const_set for the constants (or changing them to instance variables, for example). But we’re shooting for a cleaner solution in which we divide definitions from their actual execution in separate files, and only time the latter.

And of course, we also need to add cross-platform memory measurement into the picture. It may take a while, but stay tuned. 😉

Get more stuff like this

Subscribe to my mailing list to receive similar updates about programming.

Thank you for subscribing. Please check your email to confirm your subscription.

Something went wrong.

11 Comments

  1. M. Edward (Ed) Borasky July 7, 2008
  2. malcontent July 7, 2008
  3. vic July 7, 2008
  4. Antonio Cangiano July 7, 2008
  5. Tim July 7, 2008
  6. Antonio Cangiano July 7, 2008
  7. Greg Donald July 7, 2008
  8. Piergiuliano Bossi July 8, 2008
  9. David Brady July 8, 2008
  10. Ezra July 9, 2008
  11. Charles Oliver Nutter July 11, 2008

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.