The Great Ruby Shootout (July 2010)
The Great Ruby Shootout measures the performance of several Ruby implementations by testing them against a series of synthetic benchmarks. Recently I ran Mac and Windows shootouts as well, which tested a handful of implementations. However this article reports on the results of extensive benchmark testing of eight different Ruby implementations on Linux.
The setup
For this shootout I included a subset of the Ruby Benchmark Suite. I opted to primarily exclude tests that were executed in fractions of a second in most VMs, focusing instead of more substantial benchmarks (several of which came from the Computer Language Benchmarks Game). The best times and least memory allocations out of five runs are reported here for each benchmark.
All tests were run on Ubuntu 10.4 LTS x86_64, on an Intel Core 2 Quad Q6600 2.40 GHz, 8 GB DDR2 RAM, with two 500 GB 7200 rpm disks.
8 implementations
The implementations tested were:
- Ruby 1.8.7 p299
- Ruby 1.9.1 p378
- Ruby 1.9.2 RC2
- IronRuby 1.0 (Mono 2.4.4)
- JRuby 1.5.1 (Java HotSpot(TM) 64-Bit Server VM 1.6.0_20)
- MagLev (rev 23832)
- Ruby Enterprise Edition 2010.02
- Rubinius 1.0.1
JRuby was run with the –fast and –server optimization flags.
Disclaimer
Synthetic benchmarks cannot predict how fast your programs will be when dealing with a particular implementation. They provide an (entertaining) educated guess, but you shouldn’t draw overly definitive conclusions from them. The values reported here should be assumed to be characteristic of server-side — and long running — processes; they should be taken with a grain of salt.
Time Results
Please find below the execution times for the selected tests. Timeouts indicate that the execution of a single iteration for a given test took more than 300 seconds and had to be interrupted. Bold, green values indicate the best performer out of each test.
Warning: The bm_primes.rb benchmark was originally written to aid the development of the Prime class. As such in 1.9.2 it was rewritten in C, which makes it a poor representation of implementation performance. This benchmark will removed in the future.
If you are not interested in the individual test results, the information presented in the table above is summarized directly below:
Ruby 1.9.2 JRuby
Min. : 0.013 Min. : 0.382
1st Qu.: 3.258 1st Qu.: 3.051
Median : 4.543 Median : 4.997
Mean : 9.262 Mean : 9.180
3rd Qu.: 8.573 3rd Qu.: 8.969
Max. :45.009 Max. :48.850
MagLev Ruby 1.9.1
Min. : 0.351 Min. : 0.015
1st Qu.: 2.140 1st Qu.: 3.387
Median : 6.069 Median : 6.205
Mean : 9.100 Mean :10.860
3rd Qu.: 9.266 3rd Qu.:11.559
Max. :51.221 Max. :46.849
Ruby 1.8.7 IronRuby
Min. : 0.708 Min. : 3.601
1st Qu.: 5.102 1st Qu.: 10.505
Median : 8.380 Median : 12.912
Mean :18.785 Mean : 26.539
3rd Qu.:24.793 3rd Qu.: 36.115
Max. :75.653 Max. :135.204
Rubinius REE
Min. : 0.484 Min. : 0.584
1st Qu.: 3.087 1st Qu.: 4.343
Median : 9.636 Median : 6.660
Mean :13.232 Mean :15.036
3rd Qu.:17.674 3rd Qu.:21.336
Max. :73.050 Max. :61.960
For the sake of convenience, I also produced a box plot from the successful data points:
There are a few considerations based on these results that I feel are worth mentioning:
- As you can see Ruby 1.9, JRuby and MagLev converge towards a similar performance level according to these tests.
- Ruby 1.9.2 manages to squeeze in a bit of extra speed when compared to Ruby 1.9.1 (which is a welcome improvement).
- Ruby 1.9 seems to be either much faster than Ruby 1.8 or roughly as fast, depending on the test. This appears to be in line with what I’ve seen in real world programs. There are programs that will only receive a 10-20% boost from 1.9, while others improve drastically. The results really depends on what a program spends its time doing.
- Performance wise, Rubinius is really starting to be a much more serious contender.
- Ruby Enterprise Edition is slightly faster than Ruby 1.8.7, to the extent where this is clearly visible in almost all of the tests.
- IronRuby running on Mono was the worse of the lot.
Memory Results
The following table shows the approximate memory consumption for each implementation when running each benchmark:
Summarized:
Ruby 1.9.2 Ruby 1.9.1
Min. : 4.320 Min. : 4.580
1st Qu.: 4.378 1st Qu.: 4.695
Median : 6.285 Median : 6.920
Mean : 20.795 Mean : 15.669
3rd Qu.: 10.162 3rd Qu.: 11.383
Max. :171.500 Max. :100.570
Ruby 1.8 REE
Min. : 3.040 Min. : 8.220
1st Qu.: 4.290 1st Qu.: 9.682
Median : 7.745 Median : 15.565
Mean : 20.698 Mean : 27.014
3rd Qu.: 11.273 3rd Qu.: 38.620
Max. :103.520 Max. :125.910
Rubinius MagLev
Min. : 37.63 Min. : 81.74
1st Qu.: 39.78 1st Qu.: 82.52
Median : 45.48 Median : 83.53
Mean : 65.70 Mean : 96.29
3rd Qu.: 58.22 3rd Qu.: 98.10
Max. :215.33 Max. :175.85
JRuby
Min. : 49.04
1st Qu.: 71.23
Median :176.72
Mean :169.41
3rd Qu.:209.04
Max. :404.06
And finally, in graph form:
A few considerations on memory:
- Memory readings for IronRuby were not available, so they were not included.
- Ruby 1.9.2 uses the least amount of memory (as one might expect).
- JRuby was by far the most memory intensive of the group.
- Ruby Enterprise Edition used less memory than 1.8.7 in a few tests, but overall was more memory hungry than 1.8.7. This is really odd and entirely unexpected.
Linux Vs. Windows
This shootout and the Windows one were both performed on the same machine, thus we can compare how the same implementation perform under different operating systems. The only adjustment that’s required is the timeout. Every result longer than 60 seconds from this shootout needs to be considered a timeout, because the previous shootout did so as well.
It is commonly believed that Ruby performs much better on Linux than on Windows (with the exception of IronRuby). Let’s find out if these test results confirm that notion.
Ruby 1.8.7:
Ruby 1.9.2:
JRuby:
Finally, in chart form (yellow entries are on Windows as indicated by the labels containing W):
To use a beloved MythBusters expression, this myth is confirmed.
Note: As requested by a few commenters, here is a comparison of IronRuby as well (.NET 4.0 Vs. Mono 2.4.4):
Conclusion
In conclusion, let me just state that it’s nice to see several implementations getting faster. Ruby 1.9.2, JRuby, MagLev and Rubinius are all becoming serious competitors and working their respective ways closer to a similar performance level. If you think these benchmark shootouts are becoming boring, then the results are becoming more stable and predictable. I suspect that as time goes on, performance will not be the real distinguishing factor when choosing a Ruby implementation, other features will be.
If you enjoyed this post, then make sure you subscribe to my Newsletter and/or Feed.
81 Responses to “The Great Ruby Shootout (July 2010)”
Leave a Reply
I will only approve civil comments, whether in agreement or dissenting with my article. Trolling will not be tolerated, nor will negative comments be approved if posted through Tor or similar anonymizers (this is to prevent the same person from disrupting the discussion under false names).
Wow, the Great Ruby Shootout memory results are eye opening! http://bit.ly/bnQaRo (from http://bit.ly/dnHkee) Thanks @acangiano!
This comment was originally posted on Twitter
Those Linux vs. Windows comparisons are incredible. Either Ruby on Linux is blazing or the Windows implementation is a joke.
This comment was originally posted on Reddit
Wow! @acangiano just posted a awesome serie of measurements. Seens that 1.9.2 is really on par with Jruby http://bit.ly/ceG7uD
This comment was originally posted on Twitter
JRuby would be the future if the startup times weren’t so horrendous. It makes it very unpalatable for scripting. Nailgun is not robust enough. I have a feeling this is going to be more of a JVM problem than something Nutter and team is going to tackle. So is the future Rubinius and JRuby?
This comment was originally posted on Hacker News
It will be interesting to see what sort of difference Java 7 makes for JRuby.
This comment was originally posted on Hacker News
Ruby 1.9.2, the leader of the pack! RT @yukihiro_matz The Great Ruby Shootout (July 2010): http://goo.gl/6xcj #en
This comment was originally posted on Twitter
For scripting, that’s definitely a problem. But it’s not a big deal for long-running processes (e.g. a web application server). But Ruby definitely benefits from both: a fast Ruby with ~zero startup time, and a fast Ruby on the JVM.3 years ago, Ruby implementation problems became the hot topic at Ruby conferences. Ruby was a beautiful language with an ugly implementation. It’s great to see that the community has executed on this concern.
This comment was originally posted on Hacker News
serious comparison of the Ruby implementation (speed+mem usage): http://tinyurl.com/2vbu5sn
This comment was originally posted on Twitter
The tradeoff is that you get a much more stable runtime, true multithreading, and the ability to play ball with everything in the Java ecosystem.Sure, it takes longer to start up, but we’re talking a few seconds here, which is nothing for a long-running webapp. Plus, the JVM is rock-solid, has very predictable memory usage, and also has tons of monitoring infrastructure to bring to bear against problems.
Yeah, I’m a fan.
This comment was originally posted on Hacker News
I’m much, much more concerned about the memory usage of JRuby than the startup times. For toss-off scripts I can keep using MRI.I’ve read that JRuby memory usage scales better than MRI though, so you pay a higher cost upfront but can actually accommodate more workers on the same hardware. I haven’t gotten far enough in my exploration of JRuby to see if this is the case in my own apps.
Hats off to all the devs working on bringing us a better Ruby runtime. I realize it’s a lot of work.
This comment was originally posted on Hacker News
While I agree that it’s not important for long-running applications, there is still a lot of scripting done in Ruby, and it’s very frustrating to get a a 5-to-10 second wait just to tell someone their arguments were invalid. Waiting that long for tests is also such a drag on iteration.I think once JRuby has stabilized the native gem support and has some sort of Passenger-like deployment option (Glassfish and Warbler isn’t quite there yet), we’ll see some big players built on MRI start to move their Web applications to the platform.
This comment was originally posted on Hacker News
It’s worth noting that startup times with 1.9 are longer than with 1.8 (in my experience anyway). Startup times for my 1.8 instances on Heroku are sub-second. Whereas instances running 1.9 are 6+ seconds!Disclaimer: I haven’t dug into this issue much, so its possible other factors are the cause…
This comment was originally posted on Hacker News
@acangiano nos regala la edición de julio de 2010 de “The Great Ruby Shootout” http://ow.ly/2du2E
This comment was originally posted on Twitter
JRuby seems to have faired pretty well in the latest Ruby Shootout: http://bit.ly/blBGOj – @headius Any analysis/thoughts?
This comment was originally posted on Twitter
IronRuby on Mono gets it’s ass handed to it in a performance comparison to other Ruby implementations @ http://is.gd/dyh6k
This comment was originally posted on Twitter
the great ruby shootout http://bit.ly/9SDNns somehow not surprising to me that ironruby is slow and error out the most.
This comment was originally posted on Twitter
I can comment on the JRuby results.For the base perf numbers, I’m not surprised. We’ve known that we’re roughly on par with 1.9.2 for some time, and many of the benchmarks in question have started to reach a point of irreducible complexity (e.g. you can only slice and dice strings so fast). It’s good to see JRuby remains at or near the front of the pack as far as performance, especially considering we’ve made no major performance-related changes in almost two years.
The Linux versus Windows numbers are a bit surprising to me. If I were to make a guess, I’d guess that the JVMs used were not identical (perhaps like Isaac Gouy mentions, one platform was 64-bit and the other wasn’t) or some other detail altered the performance characteristics of the test. But the performance drop does seem to be in line with other implementations, so perhaps Windows really does suck and there’s not much we can do about it.
On the memory issue, I have a few recommendations.
JRuby by default allows the JVM to use up to a 512MB heap (the default is usually 32-64MB, which is rarely enough for most nontrivial apps). The JVM likes to use as much memory as you’re willing to give it, to keep GC times low (nearly free) and to give it lots of room to breathe. Almost all these benchmarks could run in far less memory (maybe 1/5 as much or lower) if the JVM were choked down to that level. So it’s not surprising to me that the memory sizes for these very object and CPU-intensive benchmarks start to approach that 512MB limit; the JVM is just stretching its legs.
Expect to see a lot more performance work coming in JRuby 1.6. I’ve blogged about it here: http://blog.headius.com/2010/05/kicking-jruby-performance-up-notch.html;
Also expect to see more work on picking a "winner" as far as lightweight servers go. Something that works as seamlessly as Passenger could be the "last mile" we need to get folks to make a move.
And watch our two Ruby Summer of Code projects: Ruboto, bringing JRuby to Android; and C extension support.
We’re working very hard to bring JRuby to everyone and everyone to JRuby. The reasons not to use JRuby are rapidly disappearing.
This comment was originally posted on Hacker News
The same identical version was used: Java HotSpot(TM) 64-Bit Server VM 1.6.0_20.
The leading theory.
This comment was originally posted on Hacker News
InvokeDynamic and other Java 7 features (method handles, NIO2) are definitely going to improve JRuby’s situation, but maybe not in the way you expect. Indy and method handles will largely allow us to delete (or not load) code we currently generate to get the same effect. Smaller runtime, possibly a smaller distribution. NIO2 will give us more direct access to streams, process handling, and so on, so we can delete hacks we’ve written to do all that ourselves.But perhaps the most interesting aspect of any new major Java release is that they’re usually 10-20% faster across the board, due to new and better optimizations. As the saying goes, if your Java app isn’t fast enough, upgrade to a newer JVM.
I think invokedynamic combined with more runtime profile-driven optimizations in JRuby could easily double JRuby’s Ruby-execution performance (or better), and in many cases reduce memory churn too. Lots of good things coming…it’s nice to have an army of VM engineers on your side.
This comment was originally posted on Hacker News
A large part of startup time is out of our hands, but we continue to look for workarounds.Most current startup slowness is due to the JVM itself running slow during the first few seconds of execution. Ruby scripts execute as they boot, which means we have to parse and run them. But until the JVM’s been up for a little while, nothing in JRuby itself has even JITted to native code…and interpreted JVM bytecode runs even slower than Ruby on most of my tests.
The other complicating factor is the fact that Ruby applications run so much code on boot. RubyGems, for example, degrades startup time O(n) based on how many gems you have installed. Hacks like faster_rubygems (gem install faster_rubygems) help, but changes are needed in RubyGems proper.
In any case, we feel the startup pain too. I’ve blogged a few tips about startup time here: http://blog.headius.com/2010/03/jruby-startup-time-tips.html, and I’ve described some of the challenges here: http://blog.headius.com/2010/06/my-short-list-of-key-missing…;
We’re working on it…we really are. We’re just fighting against a decade of JVM folks who never run command-line tools.
This comment was originally posted on Hacker News
If you are satisfied with the performance of C #ruby, you shouldn’t complain that #jruby is slow, http://bit.ly/bSa4zl
This comment was originally posted on Twitter
The results are for the same implementations measured on Windows and on Linux. One could think that maybe CRuby is just slow on Windows, but JRuby seems to suffer from the same problems. I always had the impression that Microsoft wasn’t very eager to support any languages that weren’t invented (t)here. On the other hand the IronRuby is even slower than the others. I guess they have a lot of catching up to do in order to get close to JRuby.
This comment was originally posted on Reddit
Ruby 1.9.2 & JRuby are rocking the linux benchmarks: http://programmingzen.com/2010/07/19/the-great-ruby-shootout-july-2010/
This comment was originally posted on Twitter
The curve I dream of seeing:C++ slower than any good covering flavor of Ruby.
This comment was originally posted on Hacker News
The ruby landscape is very interesting right now.
This comment was originally posted on FriendFeed
I really need to kick my ass out of MRI 1.8.x RT @daneharrigan: The Great Ruby Shootout (July 2010) http://bit.ly/90RdBG (via @ruby_news)
This comment was originally posted on Twitter
Great article, I’m glad to see 1.9.2 getting pretty close to par with JRuby. And as you mentioned Rubinius is coming along. Thanks for the work.
Oh it’s great to see MagLev in this awesome ruby shootout: http://is.gd/dyqxz
This comment was originally posted on Twitter
this is why I abandoned my dream of ruby on windows for more than for just glue scripts
This comment was originally posted on Reddit
It’s also worth pointing out that while a JRuby/Rails instance might take 100-200MB, that’s all you need to scale a site across pretty much any number of cores. MRI and REE both need to spin up multiple processes to handle concurrent requests, so very quickly the JRuby memory size becomes a tremendous win (think of 25-50 MRI instances using 20-50MB of memory each…you get the picture).
This comment was originally posted on Hacker News
I’d like to see that too. In JRuby, we may be able to get Ruby-to-Ruby calls to perform as well as Java calls, which would at least get that bottleneck out of the way. The remaining performance issues, however, are usually the rate at which objects can be allocated. In order to reduce that we may need a little JVM help (escape analysis that works well enough to actually eliminate allocations) and we may start to explore optional static typing, to allow really reducing numeric operations to raw primitive math.At this point, we realize that sometimes you really do need native performance, and we’re not taking any options off the table to get there.
This comment was originally posted on Hacker News
All the 1.8 implementations are seriously outperformed by the 1.9 implementations on bm_primes.rb. This is due to a difference in algorithm between 1.8 and 1.9 standard libraries, not to underlying VM speed.
If the 1.8 standard library were updated to the 1.9 algorithm, the differences between implementations would likely be similar to other math heavy benchmarks.
That’s right, Monty. I wonder why the naive 1.8.7 implementation hasn’t been updated yet.
I have both MRI and JRuby installed on my systems. For quickie scripts I use ‘ruby’ and for anything with a noticeable runtime (Or multi-core processing!) I use ‘jruby’. I’m sure the JRuby fellows will continue working on startup time, but for me at least, it’s not an issue.I think having a Ruby toolbox instead of a single big hammer is a win for everybody.
This comment was originally posted on Hacker News
My takeaway from this Ruby performance shootout is that all of the implementors deserve a lot of credit! http://bit.ly/9vrGss
This comment was originally posted on Twitter
Would you consider adding MacRuby 0.6 to the shootout?
MacRuby was tested on a Mac here: http://programmingzen.com/2010/05/16/benchmarking-macruby-0-6/
Use MRI for short-lived and JRuby for long-lived. Problem solved?
This comment was originally posted on Hacker News
There’s an error in your chart. Rubinius is faster than MagLev on bm_fannkuch.rb, so you need to switch the green in that row.
Fixed. Thanks Monty.
I think the reason REE used so much more memory than 1.8.x is that its garbage collection settings are tweaked to let ruby allocate many times as much memory before it performs garbage collection, and allocates many times as much memory when it needs to grow the heap. The goal was pretty much to optimize it for massive, memory-hungry rails applications.
The benchmarks game measurements are also made on 2.4GHz Q6600 with Ubuntu™ 10.04 – but most Ruby implementations are only measured on x86 rather than x64
Perhaps x86 versus x64 is the explanation for JRuby bettering Ruby 1.9 in the benchmarks game?
Perhaps the much longer run times in the benchmarks game versus “The Great Ruby Shootout” provides the explanation?
http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php?jruby=on&yarv=on&ruby=on&calc=chart
Ruby Performance Shootout http://bit.ly/btiK53 Nice to see so many options. Rubinius surprisingly fast for a 2 month old.
This comment was originally posted on Twitter
You’ll see a huge uptake in 1.9 support once 1.9.2 is released, and Rails 3 comes out. I know I’m waiting to move everything over all at once, rather than doing two migrations.
This comment was originally posted on Reddit
Ruby 1.9 in production? Bueller? The Ruby Shootout makes me want to try… http://bit.ly/dnHkee
This comment was originally posted on Twitter
Reading: The Great Ruby Shootout: http://is.gd/dyxyw /via @proggit /me JRuby is really holding its own against Ruby 1.9.1
This comment was originally posted on Twitter
Wow, JRuby memory consumption is amazingly high compared to other Ruby implementations: http://bit.ly/bhVAbv
This comment was originally posted on Twitter
It is as Burke said, the difference in REE memory usage can be explained by the default GC settings. Another thing that might play a role is the tcmalloc memory allocator, which releases memory back to the OS at a much slower rate than the normal system memory allocator but is generally faster.
Thanks for posting this. If and when you run another shootout, would you consider also testing IronRuby with Microsoft’s .NET runtime? I’m very curious to see how well it performs compared to Mono.
Hi Dave,
IronRuby was tested with .NET 4.0 last month: http://programmingzen.com/2010/06/28/the-great-ruby-shootout-windows-edition/
Since you asked so nicely, here is a direct comparison: http://programmingzen.com/wp-content/uploads/2010/07/lin-win-ir-table.png
The Great Ruby Shootout (July 2010) http://bit.ly/d0GCCj love Ruby and will be using again soon in a new project. #ruby
This comment was originally posted on Twitter
I don’t know that this theory totally holds water (Windows OS developer here, announcing obvious bias) – looking at the results, the one place Windows really gets nailed is the I/O test; I suspect there’s some significant optimizations that could be done there, as I/O on Windows OS in general certainly isn’t 4x slower than Linux.
This comment was originally posted on Hacker News
The prevailing wisdom seems to be that mono lags behind the Microsoft CLR (and the JVM) in various areas, so is the reason IronRuby is slow IronRuby’s fault, or is it just mono?
It seems a bit dubious that you’d test IronRuby only on mono on linux. I initially thought “OK, maybe he’s only got linux boxes” but then you go and include a bunch of windows tests? If you’ve already got a bunch of windows benchmarks set up, why not run IronRuby on the Microsoft CLR?
Orion, the Windows tests were run a few weeks ago and they include IronRuby on .NET 4.0.
The Linux vs Windows section of this post is only a recap that compares the Windows results from a few weeks ago with the new Linux results, for the fastest implementations available.
If you wish to compare IronRuby on Mono/Linux with IronRuby on .NET/Windows, you can read this table I posted in a comment above:
http://programmingzen.com/wp-content/uploads/2010/07/lin-win-ir-table.png
Thanks for posting the benchmarks. I’ve long enjoyed seeing how the different implementations compare. I’m not surprised IronRuby is lagging far behind at the moment. One thing I’d be curious is what difference Mono 2.7 might make. While working on IronRuby, I couldn’t build easily unless I was on 2.7, so I’m really not surprised at the errors. It may not make a difference, but several people did a lot of work to get IR running on Mono. It’d be nice to see if that really paid off.
Cheers!
OK, I see at the bottom of the comments there is a link to a previous benchmark, comparing IronRuby on Mono vs the Microsoft CLR; At a glance it appears that IronRuby on windows is over twice as fast as on mono.
It would be nice if these values were included in the Shootout results – to leave them out gives an unfair impression of IronRuby.
The casual observer is not likely to realize that there is such a difference between the CLR and Mono, and simply infer that IronRuby has terrible performance, when this appears to be the fault of Mono rather than IronRuby
I updated my post to include the CLR vs Mono table.
Thanks to all those who have put in the effort to improve the performance of the various Ruby implementations. I’m eagerly anticipating the release of Ruby 1.9.2!
Great to see Ruby 1.9.2 (we’re upgrading now) w/ such good CPU/mem performance! The Great Ruby Shootout of July 2010: http://bit.ly/b8PidY
This comment was originally posted on Twitter
It’s really excellent to see all these implementations showing such good numbers.
I’m especially impressed when you compare IronRuby on Windows to the numbers for JRuby (even on Linux) – it’s really close!
Rubinius seems to be doing excellently, and 1.9.2 is also still showing progress, it’s all so positive.
There’s some really swift options available for everyone, all the implementors should be congratulated.
Awesome performance evaluation. RT @quakewang: The Great Ruby Shootout (July 2010): http://bit.ly/c7zwuB via @addthis
This comment was originally posted on Twitter
[...] on the heels of his Windows Ruby implementation shootout comes Antonio Cangiano's Great Ruby Shootout of July 2010 where Antonio pits 8 different Ruby implementations against each other in a performance [...]
Best Ruby implementation? Ruby 1.9.2/Linux. Comparison: http://bit.ly/aszCpX
This comment was originally posted on Twitter
Object sizes in JRuby are certainly larger, especially if you run on a 64-bit JVM which necessarily has 64-bit reference fields for all object references (basically everything in Ruby). It doesn’t surprise me at all to see a single-threaded case with a lot of objects use more memory, but it would be interesting to see how much of that was unused heap space and how much was actually live data. MRI’s less-efficient conservative collector can live in a smaller memory space, but you sacrifice performance.
This comment was originally posted on Hacker News
Really interesting box plots of Ruby implementation performance and memory consumption http://bit.ly/9As9Qw
This comment was originally posted on Twitter
Ruby implementations have seriously improved their performance in the past years http://pyxi.is/aUM5r3
This comment was originally posted on Twitter
Ouch – JRuby memory utilization is ugh…. http://programmingzen.com/2010/07/19/the-great-ruby-shootout-july-2010/ #ruby
This comment was originally posted on Twitter
JRuby says nom nom to memory, poops out raw blazing speed http://bit.ly/9SDNns (via @woollyams)
This comment was originally posted on Twitter
Great shootout. The new .NET CLR IronRuby results are amazing. Would it be possible to add this column to the general results/graphs? It looks like it’s the fastest implementation on a number of benchmarks.
Actually, it would also be great if you could post the results in a Google Spreadsheet, then we can do it ourself
#jRuby almost as fast as #ruby 1.9.2? Wow! http://bit.ly/dsWomu
This comment was originally posted on Twitter
Wow, super comprehensive and a great read: The Great Ruby Shootout (July 2010): http://bit.ly/c7zwuB via @addthis
This comment was originally posted on Twitter
No rdoc benchmark?
A couple of notes:
1) even though jruby uses tons of RAM, it manages that RAM efficiently, and avoids slowdown for larger apps. I think I’ll add a benchmark to show this fact a bit better.
2) jruby can start quickly–for me on windows it only takes about 1s if I use the faster_rubygems gem.
Thanks for the nice shootout.
-r
Hello folks,
I am late to this fascinating discussion.
A bug in Mono’s runtime caused IronRuby to enter a code path that calls Debug.Log too often which slows down the code by an order of magnitude.
We did not find out about these errors until recently, due to a mistake on my side. The Microsoft IronRuby team reached out to us months before they released the latest IronRuby, but I never noticed their email until it was too late where they raised a number of bug reports and pointed out some of our limitations.
I only looked at this about two weeks ago while clearing my mail queue and we were able to fix these problems quickly. We have gained the performance back and these fixes will be in our upcoming Mono 2.8, or are available today from GitHub’s Mono (http://github.com/mono/mono).
The Microsoft guys were kind enough to file the bugs that they identified in Mono and we have fixed almost all of them now. There are a couple of them that we are still working on.
That being said, with Mono 2.8, you can expect the IronRuby test suite to run 10 times faster (not sure about this benchmark, but if someone emails me the instructions, we can look at testing it as well).
There is also a nice performance boost from using our new GC (also available on github, and on the upcoming 2.8 release) depending on the test we get a 10% to 40% performance increase (this is from an email from Paolo who tried some of the tests on this page).
We still have a few unoptimized code paths in Mono that will likely impact IronRuby performance on Linux, but they should be fixed eventually. Our goal is to match the .NET performance on equivalent hardware.
Miguel.
will it not benefit ruby’s speed if we use a true compiler rather than use an interpreter ?
Is there any difficulty to make a compiler?
@my22301 yes it would be faster, but Ruby is tricky because it does runtime setup of methods, so it wouldn’t be easy…
The Computer Language Benchmarks Game measurements have now been updated for Ruby 1.9.2 and it still seems that JRuby is a little ahead on both x86 and x64 – so why doesn’t it look like that in “The Great Ruby Shootout (July 2010)” ?
Could it really be that “The Great Ruby Shootout (July 2010)” results are that badly broken from the Ruby 1.9.2 Prime benchmarks being rewritten in C ?
Please re-run the statistics with Prime excluded as a sanity check.
As I mention in the post, prime will be excluded from future tests.
Aren’t you curious to see if that bogus Prime measurement pushed Ruby 1.9.2 ahead of JRuby into first place?
Given the timeouts, I should ask which data made it into the summary – just data from rows which had no timeout or error?
Yes.