Comments on: Ruby Shootout Status Update

By: Charles Oliver Nutter

Charles Oliver Nutter — Fri, 11 Jul 2008 05:49:28 +0000

@Tim and @Greg: You’re wrong. The original proposal for the tests would have been doing repeated loads of a file in a loop, which on all implementations would require extra processing in the form of parsing and possibly compilation. And on optimizing implementations, this is a further penalty because the code is essentially being loaded anew, so all previous optimizations get thrown out. But the larger point is that it’s no longer a benchmark of some algorithm…it’s a benchmark of that algorithm plus load/compile/optimize time. If that’s the goal, so be it…but in this case all involved agreed it would be extra noise unrelated to the actual code under test.

@Greg: JRuby does not have implementation problems, and we were not trying to get Antonio to work around anything. The original benchmarking logic was flawed, and I suggested a way that would not penalize all the implementations based on parse/compile overhead. And the other implementers agreed.

By: Ezra

Ezra — Wed, 09 Jul 2008 07:34:53 +0000

Why not run tests both including and excluding startup/parse/compile time and report both times? This way we can see all the data and it also doesn’t cater to any VM over another.

By: David Brady

David Brady — Tue, 08 Jul 2008 20:01:26 +0000

What happened to agility? Ship it! Release what you have now, annotate the deficiencies, and iterate, iterate, iterate!

What better way to see what really needs to get fixed next?

By: Piergiuliano Bossi

Piergiuliano Bossi — Tue, 08 Jul 2008 12:13:27 +0000

@Greg & @Tim: I don’t think I agree with you. We are talking about very different VMs and runtimes. Reality is not always black & white and I think we need different ways to compare things together, in order to avoid the usual issues of comparing apples with oranges.

On the other side, Antonio, I think you should provide several different measures: don’t exclude an absolute number like what Greg & Tim are talking about.

By: Greg Donald

Greg Donald — Mon, 07 Jul 2008 20:59:51 +0000

I’m with Tim..

Sounds to me like JRuby has some implementation issues and you are being asked to “work around” them.

If you’re not passing the exact same code, byte for byte to each VM, then it’s not a valid benchmark.

By: Antonio Cangiano

Antonio Cangiano — Mon, 07 Jul 2008 19:26:44 +0000

Hi Tim,
my first approach was very direct: load a file and see how long it takes to get a response. However given that the emphasis has been posed on "fairness" for all the VMs involved in order not to misrepresent their speed, a few objections were raised regarding my simple proposal. Somewhat ironically, Charles Nutter was the one who mostly raised the issue of fairness, given that accounting for compiling and parsing at each iteration would somehow penalize JRuby. You can read about it (and join the discussion) in this thread.

By: Tim

Tim — Mon, 07 Jul 2008 17:31:36 +0000

Are you maybe overthinking this? As someone who wonders “How fast will this Ruby run this code?” I really don’t care whether the time goes into parsing or running. They are all part of the time-to-the-answer.

What am I missing?

By: Antonio Cangiano

Antonio Cangiano — Mon, 07 Jul 2008 13:35:03 +0000

@vic: nope. 🙂

By: vic

vic — Mon, 07 Jul 2008 13:08:03 +0000

Will it include php and python?

By: malcontent

malcontent — Mon, 07 Jul 2008 10:33:52 +0000

Too bad. I was really looking forward to it.