MagLev is going to be a fast implementation of Ruby. How many times faster? Well, it depends on the test, and I don’t have systematic numbers yet to come up with a geometric mean of the ratios (the shootout will do just that). What I can tell you now though, is that it’s fast, significantly faster than MRI. You know that slowness that you’ve almost come to expect when running a Ruby script? It’s simply not there with MagLev. And let’s not forget, that MagLev’s added value is not limited to performance, in the same way that JRuby’s integration with Java is a strong selling point.

It’s not mature, of course, and there are a few things which haven’t been implemented yet. But it’s complete enough to run WEBrick and several other small programs I tried. A truly impressive accomplishment if you think about what was achieved in just three months. So let me reassure you that it’s not vaporware, it’s real and it’s fast.

I want to leave you with an example of a non-trivial benchmark. Running the binary-trees test from the Computer Language Benchmarks Game (chosen because it’s the first test), I obtained the following results:

Aside from being much faster than Ruby 1.8.6, MagLev’s speed is pretty darn close to that of the extremely fast C++ (complied with g++ and all the optimizations), in this specific case.

You can’t read too much from a single test, especially when we are not comparing different VMs from the same identical script like we’ll do for the shootout. It’s impressive nevertheless and it should give you a hint about MagLev’s speed even when facing non-trivial, non-ad-hoc benchmarks.

The post MagLev handles trees like a monkey appeared first on Programming Zen.

Except, that it’s not entirely new. What makes the fact that they may be onto something and develop a faster implementation credible, is that they are leveraging decades worth of Smalltalk experience, where the Smalltalk VMs underwent similar development challenges. Ruby and Smalltalk are family, there is not inherent reason why Ruby has to be dramatically slower than certain Smalltalk implementations. Parsing and compiling Ruby code into “smalltalk-ish” bytecode is not the hardest thing to do. So, from a certain prospective, MagLev is 30 years old, not 3 months old. I’m enthusiastic about the VM because I think MagLev is promising, but don’t let people tell you that I’m naive. Despite the fact that MagLev is incomplete, I want to challenge it so that we verify and clarify what kind of speed improvements are really offered at this stage. Let’s investigate how.

One very valid point that was raised by several people both in my comment section and on Slashdot, is the fact that many of the benchmarks that have been employed so far are not very useful. Some of them are meaningless, not because of the usual “micro-benchmark must be taken with a grain of salt” sound logic, but rather because they offer an opportunity for VMs to optimize them out. When a lazy/smart VM realizes that a given loop doesn’t produce any results which will be used in some way, it’ll just skip it, giving us the impression of being many, many times faster than the standard Ruby 1.8 implementation by Matz et al.

In the real world, when that loop has to do something meaningful and the results of the computation have to be printed on the screen, that impressive performance is nowhere to be seen. So far this set of tests, which were employed by Yarv for testing its own progress, have also been used to compare different implementations (and these benchmarks can be found in Rubinius’ repository as well). This was the easy thing to do, but if we’re going to get serious about it, we need to produce a better set of benchmarks, especially when the current ones question both Yarv and MagLev’s impressive results.

In the long run, it would be good to come up with some serious benchmarks based on AST nodes in order to test each of Ruby’s features. We can work on that, but let’s get started with some “beefed up” micro-benchmarks for the imminent shootout. In one of my comments I wrongly called the Yarv tests “standard”. That was unfortunate wording, because there are no “standard” benchmarks that we can rely on even minimally in the Ruby community. Let’s fix that.

I created an empty project on GitHub, called Ruby Benchmark Suite. This project will hold a set of benchmarks that VM implementers can use to monitor their own progresses and that I can use to run periodical shootouts between all of the major implementations. I also created a Lighthouse project, so that we can have some support for communication and project management. For on going discussion about the project, I created a public Google Group which I invite you to join, if you’re interested in helping out. I’d like to see VM implementers get involved with this, in order to make it a set of reasonable, standard benchmarks that we can all agree upon.

For the next shootout, I’d like to start my multiple testing within the next week or two. So it’d be great if we could come up with a bunch of new tests and revisit the existing ones. What I’d like to see is the following:

• Eliminate or modify flawed tests from the Yarv collection of benchmarks. That means, remove the chance that a given VM could optimize out the actual computing, yielding surprising yet useless results;
• Focus on a multitude of simple, cross-platform programs that employ the Core and Standard libraries. These small programs should test a variety of operations (e.g. number crunching, text processing, etc…);
• There can be several types of application benchmarks, from simple algorithms to a program that performs statistical analysis of a large web server log file (for example) — and anything in between;
• The code in the project will be released under the MIT license. That means that if you contribute code that it’s not your own, you need to ensure that it’s released under a compatible license before we can include it. You’re free to place your name and copyright note at the top of the file, in a comment, but if you send it to us, you agree to release it under the MIT license.
• If you’re not too familiar with GitHub, send over your programs by email (acangiano (at) gmail.com) or by submitting them to the Google Group (assuming they are small). Please specify whether you’d like a “Submitted by” line and if you want your email to be included.
• Not all contributions that we receive will be included, but the greater the variety, the smaller the gap between the benchmarks and real world performances will be.

I hope I can count on your help for this project.

The post Let’s create a Ruby Benchmark Suite appeared first on Programming Zen.

I had no doubt that this would be the case. For the past few days, and before RailsConf, I’ve been in touch with the nice people at Gemstone and witnessed, before most, the speed of the current implementation. My first comment truly was “Holy Shit!” and I don’t like to swear. 😉 In a side-by-side comparison, it was like watching a race between an eight week old chubby bulldog (Ruby MRI) and a full grown jaguar (MagLev). Ruby MRI looked cute, I was almost cheering for it (“come on Ruby, you can do it”). It was so spectacular that you couldn’t help but think that they were pulling your leg.

MagLev is not complete yet, but it’s very remarkable that it accomplished so much, both in terms of tests passed and its speed – despite being only three months old. It speaks volume about the capabilities of the Gemstone platform (on which MagLev is based upon) and MagLev’s team work.

Let’s be clear about one point, this announcement changes everything; it has the potential to revolutionize the Ruby community. It’s not just a matter of a substantial speed increase, Ruby’s main weakness, it’s also a matter of scalability, paradigm shift and Enterprise perception.

When you get to use an object persistence model that can hold up to 17 Petabytes (that means 17 Million Gigabytes for you), with the ability to scale by simply adding instances, the whole “Ruby doesn’t scale” FUD starts to fade real fast. Also, aside from all the technical advantages brought by MagLev, selling a Gemstone based solution to environments that are typically harder for Ruby to penetrate into, becomes a no-brainer. It’s a newfound ticket for Ruby and Rails into many sectors of the Enterprise, and an easy entrance in the financial world as well.

MagLev is unconventional, but that’s because most impressive innovations tend to be so. That’s what the future looks like when seen through the eyes of the present. The reality is that MagLev’s OODB paradigm and architecture are quite a good fit for common Web development too, so if properly pulled off, MagLev could change the way we write Rails applications.

A few people objected that it won’t fly, because it’s going to be a commercial product. I don’t think that’s the case. The Rails community adopts Macs, TextMate, and other commercial software without blinking, so why would they have any issues with paying for a truly superior Ruby VM (if it proves to be so)? The terms of release for MagLev have not been finalized yet, but it’s clear that some part of it will be open and others (e.g. the actual VM written in C) closed source. It’s likely that there will be different pricing levels, including a free version with some limits in place. But price will mostly be irrelevant.

Planning the next Great Ruby Shootout

The great news is that the next edition of my shootout will include MagLev, so we’ll get to test it in a fair and controlled environment, against all the other major implementations. The results of the shootout will be published sometime in June. These are the implementations that I’ll most likely test:

1. Ruby MRI
2. Ruby 1.9
3. JRuby
4. Rubinius
5. MagLev
6. Ruby Enterprise Edition
7. IronRuby
8. IronRuby on Mono

I’ll probably test them on Ubuntu 8.04 (32 and 64 bit), Mac OS X Leopard and Windows Vista (32 bit). Please note that not all eight of them can be tested on each of the four platforms, so you won’t see MagLev on Windows or Linux 32 bit, for example. I’d like to test all four operating systems from the same machine (my MacBook Pro Core 2 Duo 2GHz SR, with 2GB of RAM) so that we can also compare how these major implementations behave on different operating systems, but I may have to opt for some native installations and carrying out others from VMware Fusion; I’ve yet to sort all the details out. 😉

A criticism that I’ve often heard is that these micro-benchmarks are well known and specifically targeted for speed by the implementations’ authors. The new shootout will also include a set of tests that no one has seen before (I just started to write/put them together). They may still be synthetic benchmarks, but they’ll help us form a better idea of the actual speed of these VMs.

The post MagLev rocks and the planning of the next Ruby shootout appeared first on Programming Zen.