Comments on: Let’s create a Ruby Benchmark Suite

By: Enlaces :ruby, :rails => “junio 2008″ | Javier Vidal Postigo

Tue, 01 Jul 2008 09:09:14 +0000

[…] Antonio Cangiano comienza el proyecto Ruby Benchmark Suite. […]

By: This Week in Ruby (June 16, 2008) | Zen and the Art of Programming

This Week in Ruby (June 16, 2008) | Zen and the Art of Programming — Tue, 17 Jun 2008 02:40:51 +0000

[…] couple of weeks ago I announced the creation of a Ruby Benchmark Suite project. The next shootout will take place starting from the […]

By: Ed

Ed — Wed, 04 Jun 2008 16:25:31 +0000

While Ruby has improved by 5x with the newer 1.9 release.
It is still VERY slow comparing to other language like Lua or Java or even Python.

http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=yarv&lang2=psyco

A quick glance sees Ruby 1.9 is still one of the slowest according to the test shown above.

Will this MagLev bring us up to speed with Lua / Java.

By: dave

dave — Mon, 02 Jun 2008 12:30:09 +0000

This kind of thing is why I have a lot of respect for the Ruby community. When they encounter a problem instead of just whining about it they try to solve it with a combination of programming skill and open source.

Some comments I would add:

* I’ve never liked benchmarks that just have a single number result. I’d rather see the same test done with e.g. more and more data and get a nice graph that shows where bottlenecks get hit.
* Similarly if a result takes 1 second half the time and 11 seconds the rest, it shouldn’t be just averaged to 6, the variability should be shown somewhere
* Memory is important as well as speed. I believe alioth already reports on this?
* For VM’s that get faster with use Alioth also has a graph that shows the time taken for the first and subsequent runs. In effect this gives an upper and lower bound on times depending on how much you want to punish/reward systems in how they trade startup speed against ongoing performance. (http://shootout.alioth.debian.org/debian/miscfile.php?file=dynamic&title=Java%20Dynamic%20Compilation)
* Is the software that displays the alioth results open source, as well as the benchmarks themselves? Could be a good way to centrally report the results and give something back by letting other languages (e.g. Iron/J/python) use a similar setup customised for comparing across multiple implementations of a single language.
* I would have thought that also testing on a dual-core machine would be sensible. Don’t modern VMs use the other thread to do VM-stuff in the background? (http://java.sun.com/performance/reference/whitepapers/6_performance.html#2.1.6)

By: Antonio Cangiano

Antonio Cangiano — Mon, 02 Jun 2008 08:40:00 +0000

@Charles L, I’m not sure about their license, but I ‘d like to add the ones that are missing (Yarv already includes a few of them).

By: Antonio Cangiano

Antonio Cangiano — Mon, 02 Jun 2008 07:38:52 +0000

Hi Monthy,

thanks for stopping by. With my 30 year remark, I was mostly thinking about the kind of implementation challenges solved by pioneers like Alan Kay. But it’s clear that the latest advancements made by GemStone will be the real foundation for MagLev. I also want to clarify that I think you have quite a few challenges ahead, but having a fast performing VM created for a similar language, is a huge jump start.

By: Charles L

Charles L — Mon, 02 Jun 2008 07:29:21 +0000

What about using the ones at shootout.alioth.debian.org as a start, they at least all require output i believe, so their shouldn’t be any cases of pointless loops. Also comes with the added bonus of implementations in many languages.

I think they were just added recently to the rubinius benchmark directory, as an additional set of benchmarks to test perf.

By: Monty Williams

Monty Williams — Mon, 02 Jun 2008 07:10:52 +0000

Thanks for inviting us, Antonio. This sounds like a great idea. We’ll be glad to participate.

Since I’ve been at GemStone since it started in 1982, I have a small correction to your timeline. Your readers can choose what age they wish to view our MagLev VM.

The first GemStone VM (32-bit) was released in 1986, and version 6.2.2 of that product was released last Friday. The first 64-bit GemStone VM was released in 2005 and version 2.2.5.2 of that product was also released last Friday. MagLev is *based* on our upcoming 3.x VM. That VM will be released at such time it is feature complete and passes the tests required for current production customers to upgrade their existing (i.e. non-Ruby) applications.

Running Ruby is new functionality for us, and as Charles has pointed out, Ruby compliance is no small task. We haven’t had time to implement anything besides Webrick, the microbenchmarks, and a bit of XML parsing. So I can’t give an estimate of when we will pass enough RubySpecs for people to view MagLev as real.

But we’ll be glad to share our Ruby code and performance benchmarks (good and bad) along the way.

— Monty

By: malcontent

malcontent — Mon, 02 Jun 2008 03:07:41 +0000

May I make a suggestion?

I suggest you create a virtual machine for each of your platforms. Presuming none of your benchmarks are disk io bound you can test all of the platforms on the exact same machine under the same sort of conditions (same amount of memory, same CPU allocation etc).

VMware workstation allows you to create a base image and then create a set of “diffs” so it might be an ideal solution.