The Best DB2 Ever
Over the past few days the spotlight has certainly been on Sun’s acquisition by Oracle, but today IBM made a remarkable announcement of its own. DB2 9.7 (code-named “Cobra”) will be GA (generally available) in June. This new version of DB2 is loaded with interesting features and is undoubtedly the best DB2 ever.
Within IBM we consider this to be the single most important release of DB2 since the days of DB2 UDB V5. DB2 9.5, already the fastest data server in the world, will receive a major upgrade that includes many benefits in terms of self-optimization and self-healing (autonomic computing features), compression and performance. This release will enable even greater space and related savings (including energy), making it the greenest database software available as well.
From reading the official press release, I found two cases particularly interesting (both quoted below). The first, is about a Medical Center that managed to save 50% on space and shave 35% off of the time spent on administrative tasks by switching to DB2 9.7:
UCLA Medical Center is using this technology to keep all patient records online while reducing storage space needed to manage these records by 50%. In addition, UCLA is currently developing three clinical applications using the new DB2 features that allow it to analyze and better understand patterns and trends among patients with similar symptoms or illnesses, to help determine how to better serve their patient community.
The software also includes workload management and autonomic computing features that improve the performance of high priority applications such as closing quarterly financial reports and helping IT staff more efficiently manage their growing data, leading to a time savings of up to 35% on administrative tasks.
And the second is from Openbravo:
To move a non-DB2 application to a previous version of DB2 would have taken an estimated two-year effort. We were thrilled to see it took only one week to move it to the new version of DB2. This represents a terrific opportunity to expand our international community of users, partners and developers. We’re very excited to partner with IBM to make new deployment options available,” said Paolo Juvara, CTO of Openbravo, a Web-based Solution ERP provider and IBM business partner.
Customer testimonials aside, there is a wealth of new features that will appeal to the developer as well, including improved concurrency support, new data types and functions, improved queries and procedures, state of the art XML support and much more. To learn more about the technical details of this fantastic new release, please join us for a DB2 9.7 Overview Webinar (part of the Chat with the Lab series) on May 6 at 11:30 AM ET.
Finally, if you are interested in trying out DB2 9.7 yourself, you can sign up for the DB2 9.7 Early Access Program. And don’t forget that the DB2 Express-C version is production ready and will be available absolutely free of charge.
IBM Cloud Computing Webinar
The IBM Toronto Software Lab periodically runs webinars on information management related topics, which are called DB2 Chat with the Lab. These tend to be highly informative sessions, which wrap up with a Q&A time that enables attendees to openly ask questions on the subjects that were discussed during the talks. Best of all, these sessions are absolutely free to the general public and anybody is welcome to attend.
The next DB2 Chat with the Lab is a particularly important one, because it deals with a subject that is reshaping our industry and is likely to change the way IT business is done. I’m talking of course about Cloud Computing, a flexible technology that makes computational resources available on-demand, at a pay as you go rate, and ready to scale right from the get-go.
When: March 25, 1:00-2:30 PM Eastern / 12:00 PM Central / 10:00 AM Pacific. Please mark your calendar.
What: DB2 and Cloud Computing; a 1.5 hour session that will cover the following topics:
- Introduce Cloud Computing, its benefits, and IBM’s Cloud strategy.
- Cover usage scenarios for running DB2 in a Cloud.
- Describe various Cloud deployment options for DB2, and touches on the subject of Amazon EC2.
- Hear from business partners leveraging Clouds for DB2-based solutions.
Who: There will be several prominent speakers on the bill. Sal Vella (VP Development – Data Servers & Data Warehousing, IBM) will get the ball rolling, followed by Leon Katsnelson (Program Director of Data Management Portfolio, IBM), and Rav Ahuja (Senior Product Manager – Data Management, IBM). Next up will be some of IBM’s business partners, including Mike Oliver (CTO, Corent), Scott Chate (Dir. of Products, Corent) and finally Uri Budnik (Director of Business Development, RightScale).
How: Register here and you’ll receive all the details you need to be able to dial in for the call. At a later stage you’ll also receive an email containing details about how to join the web conference so that you can follow along with the presentation.
You don’t have to be an analyst or manager to join this call. On the contrary, we’d love to see more developers partecipate and getting involved. Even if you are not interested in DB2 per se, you will probably learn something new or find the information that’s being presented interesting.
And if you happen to be at IDUG on May 12, you can attend the session E03 “DB2 in the Cloud”, presented by the aforementioned Leon Katsnelson, if you want to continue learning about our Cloud Computing and Software as a Service (SaaS) efforts.
IBM’s Python driver is out of beta
There’s a new release of the Python driver/wrapper for DB2 and Informix in town. Version 0.7.0 is officially the first stable, production ready release. It includes fixes for a few known bugs and fully supports Unicode.
This driver, and the DB-API 2.0 wrapper it ships with, have been released under the Apache License 2.0. What’s more, the IBM API team is now legally allowed to accept your contributions to the project.
If you use this release in your projects, let us know, we’d love to hear how our releases are getting used in production.
DB2 support for Django is coming
A few weeks after DB2 Express-C for Mac OS X was announced, I’m here to let you in on another great scoop. DB2 support for the Django web framework is going to be available soon to the community, under the permissive Apache 2.0 License. We are presently waiting for clearance from our lawyers, but the code has been written and tested, and Django is finally working with DB2. This comes on the heels of a new release of the Python driver for DB2, version 0.6.0, which adds full support for Unicode.
The Django community will soon be able to use the rock solid database management system which is DB2, and enjoy all the advantages that it provides. Would you like to introduce Django into your enterprise environment, where DB2 is already in use? If so, you’ll now have an easier time with this. Want to use DB2 as a competitive advantage for your startup? Now you can, whether you opt to use Django/Python, Rails/Ruby, Zend Framework/PHP or Perl.
I have been pushing for Django’s ORM support since 2006, and I distinctly remember the initial reactions of some people at IBM, they were along the lines of, “Djan… what?”. Unlike Rails, Django was much less known back then, especially among IT managers, and in all fairness, while powerful and very productive, the inherited Python philosophy that “explicit is better than implicit” made it look more complex – or at least less impressive – than Rails during 10 minute demos. But I insisted that it was important for our DB2 strategy and for the Django community, and now it’s finally a reality, thanks to the hard work of the IBM API team. Just like for Rails and Ruby, IBM will be the first and only vendor to officially support a Python driver, SQLAlchemy and Django’s ORM adapters.
I can’t help but think, what’s next? What language and/or framework truly needs some DB2 love? I’m definitely interested in a few languages and frameworks, and have already advocated for some of these as well, but I’d like to hear your opinions on this topic. I have created a poll that asks you which, among the technologies that we don’t currently support, do you think it would be most beneficial to have DB2 support for. Feel free to express your opinions in the comment section, as well as in the poll.
Disclaimer: The opinions expressed in this post are mine and mine alone, and do not necessarily represents the opinions of my employer, IBM. The poll is not an official IBM survey.
Install DB2 on Mac OS X and give us feedback
A little over a week ago IBM released a new version of the DB2 beta for Mac OS X Leopard, following up on valuable feedback that we’d received in response to earlier versions. This version is important because it resolves a reported issue that was occurring when building drivers for languages like Ruby and PHP. Hence, I highly encourage you to download the latest DB2 for Mac now and save yourself the hassle of this issue.
The installation procedure is fairly straightforward, but I’ll spare you from having to read the “Installing and setting up DB2 for Mac OS X” PDF by providing some easy steps here.
Install DB2 on Mac
Step 1: Increase your system’s parameters by creating or editing /etc/sysctl.conf. Add the following parameters, save the file, and reboot your machine.
kern.sysv.shmmax=1073741824 kern.sysv.shmmin=1 kern.sysv.shmmni=4096 kern.sysv.shmseg=32 kern.sysv.shmall=1179648 kern.maxfilesperproc=65536 kern.maxfiles=65536
Step 2: Download and extract DB2 Express-C for Mac OS X. To extract the file you can run tar -xvzf db2exc_952beta_MAC_x86_64.tar.gz from a terminal. cd into the extracted folder by running cd exp.
Step 3: Run sudo ./db2setup and follow the wizard that appears. (Note that this step assumes that you have Java 1.6 installed.)
If you’d like specific instructions on installing Ruby and Rails, take a look at my previous post about this subject.
Give us your feedback
Try out DB2 on your Mac, experiment with it, and let us know what you think. Tell us what are your impressions, what you like, what you dislike, and what you’d want to see in future releases. Write me at cangiano{{AT}}ca.ibm.com, and I’ll be happy to hear from you. If this release is particularly important to you and you’re able to speak on behalf of your company, do feel free write as well, we are definitely looking for testimonials – and such letters could prove to be free exposure for your company as well. If you use DB2 on Mac in an interesting way, let us know about it. We truly value your opinions and are looking forward to hearing about how DB2 on Mac affects you.
DB2 on Mac officially released
As pre-announced in my two previous posts, DB2 for Mac OS X Leopard is finally available for download. It’s now official, DB2 on Mac is here.
Reflections on DB2 on Mac
Several people, including myself, would happily ditch their virtual machines and start introducing DB2 into their native Mac development stacks. But this milestone represents much more than the immediate implications would have us believe. A few years ago, the idea of giving away DB2 for free would have been met with rejection. Yet, DB2 Express-C came along, and unlike the other “express” databases, it’s a true production-ready DB2 version that can be used free of charge.
Likewise, the idea of having a DB2 version for Mac was unthinkable up to a few years ago. Yet today we finally have a copy of DB2 Express-C for Mac OS X that’s available for download. Aside from this being an acknowledgment of the growing importance of Mac as a development and business platform, I feel it underlines IBM’s ability to change. The desire that a few of us mac addicts had, coupled with reasonable pressure from the community, was sufficient enough to make DB2 on Mac a reality. This matters and appeals to both the developer and the technical evangelist in me.
In the list of downloads, you’ll notice that the Mac download is only 138 MB, versus the 412 MB of Linux’s 64-bit. The reason for this difference is that DB2 Express-C for Mac currently ships in English only, and at this stage it doesn’t include either DB2 Text Search or the Java based tools like the DB2 Control Center. This lighter package is, in my opinion, a welcome side effect of this brand new beta release.
Getting started with DB2 and Rails on Mac OS X
Since the first download went live on Friday, a newer release that includes a guide for installing DB2 on Mac OS X was published and it incorporates a few changes that will make the lives of developers easier, as they approach building and using drivers (e.g. the ibm_db Ruby gem). If you downloaded this beta version over the weekend, do not worry: just grab – and execute – this shell script (e.g. sudo fixlib.sh). If you are downloading DB2 on Mac now, you won’t need this script of course.
Once you’ve downloaded DB2 for Mac OS X Leopard, please proceed to read this PDF guide, which will tell you everything you need to know (and more) about installing DB2 on your Mac, as well as providing extra details. It’s best not to skip over reading this document, as the installation on Mac OS X requires a few more steps than simply running the setup wizard.
With DB2 installed and started (sudo db2start), and the SAMPLE database created (db2sampl), you’re ready to start playing with this power horse. For details about SAMPLE’s structure you can read this article in the InfoCenter.
To run the DB2 console (known as the Command Line Processor or CLP for short), run:
$ db2
To connect to the SAMPLE database, from within the CLP run:
db2 => connect to sample
Unless you get an error, you should now be ready to query the database. For example, run the following query:
db2 => select count(*) from staff
Then to exit from the CLP, simply run:
db2 => quit
If this sanity test worked well you can proceed with installing the ibm_db gem (which includes the Ruby driver and the Rails adapter for DB2). To do so, run the following, adjusting the path to your own username of course:
$ sudo -s $ export IBM_DB_INCLUDE=/Users/acangiano/sqllib/include $ export IBM_DB_LIB=/Users/acangiano/sqllib/lib32 $ export ARCHFLAGS="-arch i386" $ gem update --system $ gem install ibm_db $ exit
The ibm_db gem will be installed on your system and is ready to be used. To verify that this is the case, run a small Ruby program with the following code:
require 'rubygems'
require 'ibm_db.bundle'
conn = IBM_DB.connect("sample","my_username", "my_password")
if conn
stmt = IBM_DB.exec(conn, "select count(*) from staff")
count = IBM_DB.fetch_array(stmt)[0]
puts "The staff table contains #{count} records."
else
puts "Connection error: #{IBM_DB.conn_errormsg}"
end
If everything is fine and dandy, you should see the message “The staff table contains 35 records.”.
Now that Ruby can talk with DB2, we can move on to Rails. Assuming you have Rails 2.2.x installed, run the following to create a sample bookshelf application:
$ rails books -d ibm_db
This generates a Rails application (as usual) with a config/database.yml file customized for DB2. You’ll notice that unlike with MySQL, the database names are not books_development, books_production and books_test. The names are truncated by default due to the fact that DB2 currently only allows for database names that are up to 8 characters long. Feel free to change the development database in database.yml simply to ‘books’.
As a Rails developer you may also be accustomed to running rake db:create to automatically create the development database, yet this feature is not available for DB2 at this point, so instead you can create the database using the db2 command, as follows:
db2 create database books
DB2 allows you to specify all kinds of options for the creation of databases, but in its simplest form, the line above will work just fine.
Once the development database has been created, you should be able to use Rails with DB2 as you normally would with other database management systems. For example, you could scaffold a resource as follows:
$ ruby script/generate scaffold Book title:string author:string isbn:string description:text loaned:boolean
Start the webserver with:
$ ruby script/server
And then visit http://localhost:3000/books to perform CRUD operations on book records.
At this stage, the only caveats are that you’ll have to use the db2 command, rather than ruby script/dbconsole, and that you won’t be able to use the rename_column method in your migrations. On the plus side, you’ll have the XML datatype (t.xml in your sexy migrations) at your disposal, to natively store XML documents and retrieve them through XQuery and SQL/XML.
I really hope that you’ll enjoy DB2 on Mac! Don’t be afraid to ask for help, if you need it, in the DB2 Express-C forum. Oh and we are trying to get the word out there. Your help is highly appreciated. You can promote this story on Twitter, Hacker News, Reddit, DZone, StumbleUpon and Digg.
Disclaimer: The opinions expressed in this post are mine and mine alone, and do not necessarily represents the opinions of my employer, IBM.
Download DB2 on Mac
I’m glad to announce that DB2 Express-C 9.5.2 for Mac OS X Leopard is available for download. Later tonight, I will provide further details. Meanwhile, enjoy! 
DB2 on Mac to ship before Christmas
This is not an official announcement, but I must share the news with you. DB2 Express-C for Mac OS X Leopard will finally be shipping out (before Christmas), in all likelihood it could be as soon as early next week. You may recall how more than a year ago I blogged about how the work on porting DB2 to the Mac had started. It took admittedly longer than expected but DB2 on Mac is coming, and is absolutely free of charge, of course. The team is still playing with the bubble wrap, but DB2 on Mac is a reality.
What took IBM so long? DB2 is a database management system that’s highly optimized for each platform that it’s available for, so that it can take full advantage of the operating system at hand. In other words, porting DB2 from one platform to another, is not so trivial. The task is made more challenging by the extremely high standards set by IBM. You may be familiar with the whole scandal surrounding MySQL 5.1, which was released despite known fatal bugs. Something like that is simply not acceptable to IBM. Each release of DB2 has to go through a huge amount of regression and performance tests – for months. If the product does not pass all these tests and others, then DB2 is not shipped.
On top of this, a few months ago the decision to ship DB2 Express-C 9.5.2 (rather than 9.5) was made, and as you probably know, DB2 Express-C 9.5.2 was only released a little while ago for other supported platforms. So the first piece of good news is that you’ll get the latest version of DB2 on the Mac. It’s going to be a 64 bit version and will require Leopard to work:
$ db2level DB21085I Instance "acangiano" uses "64" bits and DB2 code release "SQL09052" with level identifier "03030107". Informational tokens are "DB2 v9.5.0.2", "s081205", "DARWIN64", and Fix Pack "2". Product is installed at "/Users/acangiano/sqllib".
The second good thing is that unlike MySQL 64 bit, you won’t have to jump though hoops to build the Ruby driver due to the fact that the database is 64 bits and Ruby ships on Leopard as 32 bits. We ensured that gem install ibm_db would work out of the box, so you don’t have to.
According to Apple, my personal Mac is broken for good (the video chip is dead), which is very bad timing. But I installed DB2 and played around with it on a work Mac Pro machine. I had some fun with Ruby and Rails as well. This is great news for many categories of developers, including those who have been trying to convince their managers to get them a MacBook Pro but didn’t have much of a case due to the lack of availability of a DB2 version. Now, you’ll have a good excuse to get yourself a Mac. 
Stay tuned for the official announcement and keep in mind that this is going to be a beta (perfect for development purposes) and extra features and performance improvements will be added in future releases.
Disclaimer: The opinions expressed in this post are mine and mine alone, and do not necessarily represents the opinions of my employer, IBM.
IBM’s XML Challenge (lots of prizes inside)
IBM is holding a series of challenges centered around XML. The whole event is labeled The XML Challenge (subtitle: Search for the XML superstar). Rockstar references aside, this is a pretty cool initiative that can provide you with some freebies as well as high quality prizes if you win any of the available contests.
The Contests
What I say below applies to US and Canada, as the contest is being held worldwide in 30 countries separately, and each of these may have different individual contests and prizes as well. In fact, the first thing you’ll see when you visit the xmlchallenge.com site should be a popup that prompts you to select your country.
For US/Canada there are 5 contests: Video, Gadget, Query, Porting and XML. The Video Contest consists of creating a funny/creative/cool video about XML, XQuery and/or DB2. The Gadget Contest is about developing a downloadable gadget/widget that leverages DB2. The Query Contest requires you to use XQuery to query a database and come up with the answer to five questions. The Ported Application Contest is all about porting an existing application to DB2 or creating a new one that uses DB2. And finally, the XML Contest asks you to build a useful, user-friendly XML application from scratch. The last two contests can be approached as a team or as an individual. The Query, Ported App and XML contests start today!
You don’t have to participate in all of them, of course. But by participating in any of these you gain points, and there are six badges that you can obtain: XML Challenger, XML Rookie, XML Whiz, XML Star, XML Master and XML Grand Master. I feel so nerdy reporting this. The XML Grand Masters will be enrolled in a draw for an additional prize.
The Prizes
Speaking of prizes, let’s see what goodies are up for grab. There will be a few give-aways just for participating. For example, the first 500 participants will receive an XML Challenge T-Shirt as well as a Rubick’s Cube. The first 1000 to complete the Quick Quiz during registration will also receive a free T-Shirt. But let’s move on to the more substantial prizes.
For the sake of awarding prizes, the contestants will be split in two groups, students and professional developers.
The Video Contest: The deadline is December 17, 2008 and a few selected winners will receive an 8GB iPod Video Nano (for both students and developers).
The Gadget Contest: The deadline is December 17, 2008 and the winning students grab Canon Powershot SD870 Cameras, while winning developers get 80GB Zunes.
The Query Contest: After you register, you’ll have 24 hours to submit your answers. The first 50 successful participants for each group (for a combined total of 100) will receive a 1GB USB key, while all the contestants with the right answers will be entered in a draw for a grand prize. This is a Playstation 3 40GB for the students, and a 32GB iPod Touch for the developers;
The Ported Application Contest: The deadline is January 31st, 2009. The winning team or individual amongst the students will receive an HD Pavilion HDX Notebook, while the winning developer will score a Lenovo IdeaPad U110. The second prize for both of the two groups will a Garmin nüvi GPS.
The XML Contest: The deadline is January 31st, 2009. The 1st prize for each group will be a high-end 17″ Alienware Laptop (two laptops will be awarded in total). The second prize for both groups will be a Nintendo Wii (again, two in total).
Finally, two lucky XML Grand Masters, one developer and one student, will receive a Bose Wave Radio II.
I hope you consider enrolling now and best of luck! If you need some help with getting started with DB2 Express-C, you can download the free e-book which is available in several languages. Oh, and finally we have an Italian version as well.
Promote this post
Hey, would you give me a hand in promoting this post? If you are in the US or Canada, and mention & link to this post from your blog, you’ll receive a free XML Challenge T-Shirt and a Rubick’s Cube as well. All you have to do is send me an email (to acangiano at gmail.com) with a link to your blog entry, as well as your shirt size and complete mailing address. Thank you!
Resolving the gray window when running db2setup
You drank the Kool-Aid and downloaded the awesomeness which is DB2 Express-C. Good job! Next you proceed to install it on Linux with sudo ./db2setup and boom, instead of a launchpad all you see is a gray window. Now what?
This problem is a known Java bug (resolved in Java 6) that shows up on Linux distros where Compiz effects are enabled. For example, this problem manifests itself in recent Ubuntu releases, including 8.10, where Compiz is enabled by default.
There are a couple of easy ways to solve this problem though. The first is to temporary disable these effects during the installation and turn them back on when you’ve finished installing. In Ubuntu, you can do this by clicking on the Appearance menu, Visual Effects tab and then selecting None. The second method is to run export AWT_TOOLKIT=MToolkit, before running sudo ./db2setup.
A new setup is in the works to solve this issue, but for the time being, you can use the workarounds above to install DB2 Express-C on Linux.
Merb, Rails Myths, Language Popularity and other Zenbits
Zenbits are posts which include a variety of interesting subjects that I’d like to talk about briefly, without writing a post for each of them.
Merb: A few days ago Merb 1.0 was released. Congratulations to Ezra Zygmuntowicz on this important milestone, the Merb community and Engine Yard (who finances the project). Merb 1.0 wasn’t even out yet when some people had already started commenting on the fracturing of the Ruby community that this new framework might bring with this, and the impact that this high visibility “competitor” might have on Rails. I believe that having more than one widely adopted web framework will only benefit the Ruby community. Furthermore, it’s important to remember that this is not a zero-sum game. Ruby programmers are perfectly capable of learning two frameworks and using one or the other, depending on the project at hand. This is particularly true if we consider that Merb, for all of its advantages – and disadvantages – when compared to Rails, is not totally different from its forerunner. If you are an expert Rails programmer, you should be able to become proficient in Merb in very little time. To help with this process, the Merb community needs to concentrate on the documentation now, given that the API is finally stable.
Rails Myths: David Heinemeier Hansson began a series of posts about Rails Myths. I like the idea of seeing common myths addressed straight from the horse’s mouth. Over the past two years, Rails has received quite a bit of backslash and old fashion FUD, so it’s important to set the record straight, whether the myths are entirely fabricated or if there is some element of truth to them. Whether you agree with David or not, it’s also nice to hear two sides of the same story. In fact, at the beginning of my book I debunk a few myths, just to set the record straight regarding what some readers may have heard surrounding the framework. It was a fun part to write.
My Book: Speaking of my book, Ruby on Rails for Microsoft Developers, I’m getting closer to the finish line. I’m about to complete Chapter 9 (out of eleven chapters). The initial schedule I was provided with has been extended slightly so that there will be sufficient time to properly review the content and ensure that it’s up to date with the final release of Rails 2.2. Some people wondered what the “Microsoft Developers” part means. Is it for people that work at Microsoft? Is it for .NET programmers? Is it for people who develop on Windows?
The truth is that “Microsoft Developers” is probably just a marketing term that Wrox selected as a catch-all for of the aforementioned categories of programmers. As an author I’m trying to serve all of them well, by providing a guide that sneaks in much of the Rails culture and softens the migration path by using an Operating System, and to a certain extent, tools that they’re already familiar with. In my opinion one of the major obstacles when switching to, or trying, Rails when coming from the Microsoft world, is the culture shock. The documentation and most books assume that you are familiar with *nix systems and tools, and this can be frustrating for those who are forced not only to learn a new language and framework, but also an entirely new set of tools. As it’s targeted at Microsoft developers, the book obviously makes quite a few references and comparisons to the .NET world, where they fit. This is done so that the many .NET programmers amongst the group of so called “Microsoft Developers” will find the book particularly useful. Yet the book remains generic enough so that it can be used by any programmer (particularly Windows users), even those without any knowledge of the Microsoft .NET Framework or ASP.NET.
Python books: While on the subject of books, I wanted to mention that the final version of the Pylons book is available online. Despite the much less fancy UI, the book pretty much does what the Django Book did in the past. And both are available in print as well (The Definitive Guide to Django: Web Development Done Right and The Definitive Guide to Pylons). Pylons is a Python web framework that can be viewed as a Ruby on Rails clone, in a far greater way than Django could ever be considered.
Another thing I want to mention is that I received a copy of Expert Python Programming. I haven’t gotten to far into it yet, but from what I’ve seen so far, things look good. I hope to be able to read it through, over a weekend in the near future and then provide a proper review. Stay tuned.
Language Popularity: If you take a look at the TIOBE Index, you’ll notice a few interesting things: Ruby has dropped two positions since last year, and it’s now the 11th most popular language in the world. This shouldn’t be cause for concern though, as shown by this Ruby graph. Python on the other hand is increasing in popularity and moved from the 7th to the 6th most popular language. Interestingly, according to the index (the results of which are educated guesses only), Python would seem to be more popular than C#. I find this to be true, in terms of online activity within an increasingly vibrant community, but in my opinion, the job market hasn’t caught up yet. In fact, at least in Toronto, when there’s a Python opening it’s pretty much an event that’s worthy of being discussed on the local Python mailing list. C# openings are much more common. This may be different in Silicon Valley, of course. It would also seem that Delphi has experienced a huge come back, moving from the 11th position last year to the 8th one this time around. It’s hard to imagine that Delphi has had a similar level of adoption as C# and thus has become more popular than Perl, JavaScript and Ruby. Delphi is a great solution for Win32 programming, but I don’t quite believe this overly optimistic outlook. And if this is the case, where are all the Delphi jobs and buzz?
DB2: This interview shows a few good reasons why even smaller and medium sized companies are increasingly adopting DB2. And while the video doesn’t mention it, IBM is coming out with an updated version of DB2 Express-C 9.5. This new version, 9.5.2 or 9.5 FixPack 2, is going to introduce exciting new features, including an engine for full text search.
The Great Ruby Shootout These days you hear a lot of talk about parallel programming. Intel promotes it and despite their bias, it’s plausible that parallel programming will become important as the CPU market heads towards an increasingly larger number of cores, as opposed to focusing on the frequency of said CPUs. In the world of Ruby, this translates into multiprocessing, as opposed to multithreading due to the infamous GIL (Global Interpreter Lock). This means that Ruby will most likely approach the problem similarly to how Python 2.6 did with the multiprocessing module, which is a process-based interface. The obvious exceptions are JRuby and IronRuby, which establish a 1 to 1 relationship between green threads and OS threads.
For the shootout, it would be interesting to see some multithreaded code, so as to get a better sense of how well JRuby and IronRuby compare to MRI and 1.9, when more cores are available. In fact, the long-promised shootout will be performed on a quad-core machine with 8GB of RAM. If Charles Nutter, John Lam, or any of their team members would like to contribute some programs that are able to take advantage of “native” multithreading, I’d be very happy to include them in the Ruby Benchmark Suite, to be used for my shootout.
The repository requires some love and refactoring, since it needs to be split in two types of benchmarks. The simpler one will evaluate the execution time minus the startup time, while the more advanced benchmark will also exclude the time required for parsing and loading modules, classes and methods in the AST. It would also be nice to test each program with variable input sizes and report these results accordingly. Right now I’m very busy with the book, but as I become more available, I’ll start working on this.
Finally, I want to point out a very interesting article about performance and UIs. Slow is indeed a very relative concept, and it’s important to understand how to analyze and respond to the user requirements when it comes to the responsiveness of an application as a user interacts with it.
Hardware: I finally bought a Trackball made by Logitech and the Microsoft Ergonomic Keyboard (Microsoft makes great hardware). I don’t have wrist problems, but I’d like to see how these two affect my extensive computer usage. I plan to report my experience as soon as I’ve had a chance to use these input devices for a while, since I know this is a topic that interests lots programmers (many of whom end up being victims of RSI, and some of the IRS 
). I also bought a bad-ass color laser printer which is quite handy when you’re a programmer and you are writing a book. I’ll let you know how it goes. What I didn’t buy, but still think is awesome, is the Flip minoHD. It’s the equivalent of an iPod for the world of camcorders. $235 for a camcorder that’s so perfectly compact, and yet that can record in HD, is a pretty sweet deal. I’m considering it for Christmas, assuming it reaches Canada by then.
And the winner is…
A few days ago I announced that I was going to give away a free ticket for the first Professional Ruby Conference, organized by Obie Fernandez and Addison-Wesley, to one of my readers.
Each person who took the survey, received a discount code for the conference valued at $200. More excitingly, every eligible participant in the survey was added to a draw for a free ticket. Many replies came in, but of them only 30 were eligible to participate, as they answered “yes” to the question “will you be able to attend the conference if you win?”.
You might think that I used some sort of script to come up with the random winner, perhaps using the Roo gem (for Google Spreadsheet). But I didn’t. I used a Rhombic Triacontahedron, or in layman’s terms, a 30-sided dice. Without further ado, I’m here to announce the winner.
The lucky winner is: Nick Quaranto, from the US.
Congratulations Nick! I’ll get in contact with the conference organizers this morning and provide them with your info, so that you can claim your conference pass. Should you not be able to attend the conference and therefore claim this prize, please get in touch with me immediately so that a second draw can be made.
I must really thank everyone who provided feedback. It’s incredible what you can learn from a survey like this and I’ll be sure to incorporate many of your suggestions into this blog. In fact, you can still take the survey and provide me with further feedback (as well as take advantage of the $200 discount code). The only difference, of course, is that there won’t be a further draw for another free ticket.
Disappointed that you weren’t the winner? Let me bring a different type of contest with plenty of prizes to your attention (this one is organized by IBM). It’s the XML Challenge, which includes 5 programming related contests/challenges, with prizes like 17″ laptops, Nintendo Wiis, MP3 players and so on. Aside from the chance to score some neat goodies, you also get to show IBM and the world your coding and XML-fu.
Take this survey and win a free ticket for the Professional Ruby Conference
Addison Wesley will hold their first Professional Ruby Conference in Boston, Massachusetts between November 17 and 20, 2008. This conference, for which Obie Fernandez is the Technical Chair, is highly educational and boasts some of the best speakers from the Ruby and Rails communities.
The organizers were kind enough to invite me, offering me a complimentary pass for the Professional Ruby Conference. I won’t be able to attend, so I decided to donate my free admission to one lucky reader. They also provided me with a priority code (like a coupon) for my readers, which entitle you to receive a $200 discount off the regular admission price.
I really value your opinions and I’d appreciate it if you could take this survey, so that I can improve the quality of this blog. At the end of the survey you’ll receive your $200 discount code, and will be entered into my draw for your chance to win one free ticket. I will announce and get in touch with the winner early next week (Monday or Tuesday depending on participation levels).
Benchmarking DB2 pureXML against 1 TB of XML data
Once upon a time there was a Ruby library called Hpricot. Well it’s still here in fact. This library is the de facto standard for parsing HTML in Ruby, and is often used to parse XML as well.
Hpricot is normally considered to be quite fast, as far as Ruby libraries go. Yet Nokogiri recently garnered some buzz thanks to a microbenchmark that emphasized its speed over Hpricot’s, when it comes to parsing XML in a microbenchmark setting. And I can’t stress the “micro” part enough, since this was the file that was tested:
<location>
<refUrl>http://wikitravel.org/en/Singapore</refUrl>
<info>
<b>Singapore</b> is an island-state in Southeast
Asia, connected by bridges to Malaysia. Founded as a British trading colony
in 1819, since independence it has become one of the <b>world's
most prosperous countries</b> and sports the world's busiest
port. Combining the skyscrapers and subways of a <b>modern,
affluent city</b> with a medley of Chinese, Indian and Malay
influences and a <b>tropical climate</b>, with
tasty food, good shopping and a vibrant nightlife scene, this Garden City
makes a great stopover or springboard into the region.
</info>
</location>
Over the weekend, why made a few tweaks to his library et voilà, it was suddenly faster than Nokogiri in terms of parsing an XML document smaller than 18K. It’s nice to see them striving to improve the speed of these libraries. After all, for parsing HTML or even the occasional small XML document, those two Ruby libraries are fine and have their place.
DB2 users do not generally need them for XML though. In fact, DB2 offers a technology called pureXML to help in this area. In short, XML documents can be stored, indexed, cached, queried, updated, validated and compressed within the database in XML columns. This means that the data is securely stored, properly backed up, and easily restored. What’s more, there is no need to parse large strings to obtain an object representation of the XML document(s). Queries and updates require no parsing at all, since the XML is stored in a parsed hierarchical format. You simply ask for the data that you need (or need to update) directly, and DB2 will eagerly oblige. You can use XPath, XQuery, and also integrate SQL and XML queries to retrieve relational and XML data. It’s as easy as it gets, and of course, all this supports Unicode.
And DB2 pureXML is blazingly fast. How fast? Well, at IBM we like benchmarks too, only we don’t use 18K of XML data. With very little tweaking (pretty much letting DB2 automate and self-tune almost everything), DB2 was tested with the help of the good folks over at Intel, on the latest Xeon processors. The TPoX (Transaction Processing over XML Data) Open Source benchmark was used. This is a well balanced and realistic benchmark that proposes a financial transaction processing scenario. The raw XML data was 1 Terabyte and was stored in three very straightforward tables (with XML indexes). DB2 managed to store all the information, including the indexes, in just 440 GB. And the row data was 1 TB without indexes.
Compression aside, we were pretty much blown away by how immensely fast DB2 is. The throughput with 200 concurrent users was stable throughout the 2 hour run, and under a mixed workload, performed about 34 million queries, almost 7 million updates, almost 4 million insertions and deletitions (for a total of almost 48.5 million transactions). To be exact, the average was 6,763.42 transactions per second. The two tables upon which inserts were performed did 4,913 insert per second (4 to 20Kb of data) and 11,904 inserts per second (1 to 2Kb), respectively. Not only is DB2 often benchmarked as the fastest database in the world when it comes to relational data, but it’s also undisputedly state of the art when it comes to XML handling.
As a side node, by switching from Intel Xeon 7300 processors (4 cores) to Intel Xeon 7400 CPUs (6 cores) the number of cores was increased by 50%. DB2 managed to increase its throughput by 48%. You know, we don’t waste anything in this neighborhood. For full details about the results of the benchmark, you can download the slides of the IOD presentation (PDF warning).
Antonio, I hear you say, we don’t have that kind of hardware. That’s true, but you probably don’t need to process 18 Million documents an hour either. What you do have is that kind of software – for free. In fact, DB2 Express-C is a free version of DB2 that doesn’t impose any restrictions on the size of the database, how many users can be connected or how many databases you can have. It has the same code that ran the test above. So you can have a lightening fast XML engine that opens up a world of possibilities, free of charge.
If your startup or established company is serious about XML, DB2 Express-C 9.5 is a godsend. Now you can even try DB2 Express-C 9.5.2 Beta (currently available only on Windows). This version ships with both pureXML and a very fast Text Search technology, so that you don’t have to use fuzzy creatures like ferrets and sphinxes.
Disclaimer: The opinions expressed in this post, and any last minute remarks about how Oracle’s license won’t allow me to publish comparative benchmarks, are mine and mine alone, and do not necessarily represents the opinion of my employer, IBM, or the aforementioned Intel.
Ruby on Rails at IOD 2008
IBM’s Information on Demand 2008 kicks off in a few days. If you intend on participating, don’t forget to enroll now. For those of you who’ll be attending the event in Las Vegas, I recommend that you schedule in time to check out the session “Putting DB2 on Rails”. This session will be presented by my manager, Leon Katsnelson and my colleague Bradley Steinfeld, and will outline the importance of Ruby on Rails and its integration with one of the fastest DBMS in the world (DB2). The number of seats is limited and they’re getting snapped up like free beer, so act now and add this session to your agenda. The session id is TAD-2969A, and it will be held on Wednesday October 29th, between 3:15pm and 4:15pm.
A close look at three Rails 2.1 bugs
Ruby on Rails 2.1 has been out for six weeks now. Let’s take a closer look at three database related bugs that affect this release.
1. SQLite’s db creation generates false warnings
This is an innocuous bug, and if you work with SQLite I’m sure that you encountered and safely ignored it. When you create a Rails application, the default adapter in use is sqlite3, unless you specify otherwise with the -d option. The config/database.yml will reference three databases by default: db/development.sqlite3, db/test.sqlite3 and db/production.sqlite3. Now if you try to create the databases through the db:create or db:create:all rake tasks, the database will be created, but you’ll also get a warning:
$ rake db:create:all
db/development.sqlite3 already exists
db/production.sqlite3 already exists
db/test.sqlite3 already exists
That warning message is not actually true. The three databases didn’t exist before you ran rake, and were created by the task instead, as you’d expect. This is a small annoyance, but one that pops up way too often for my taste. Let’s see where things went wrong.
The db:create and db:create:all tasks invoke the method create_database defined within the file railties/lib/tasks/databases.rake in the repository (or the frozen vendor/rails folder). This is the definition of the method:
def create_database(config)
begin
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection
rescue
case config['adapter']
when 'mysql'
@charset = ENV['CHARSET'] || 'utf8'
@collation = ENV['COLLATION'] || 'utf8_general_ci'
begin
ActiveRecord::Base.establish_connection(config.merge('database' => nil))
ActiveRecord::Base.connection.create_database(config['database'], :charset => (config['charset'] || @charset), :collation => (config['collation'] || @collation))
ActiveRecord::Base.establish_connection(config)
rescue
$stderr.puts "Couldn't create database for #{config.inspect}, charset: #{config['charset'] || @charset}, collation: #{config['collation'] || @collation} (if you set the charset manually, make sure you have a matching collation)"
end
when 'postgresql'
@encoding = config[:encoding] || ENV['CHARSET'] || 'utf8'
begin
ActiveRecord::Base.establish_connection(config.merge('database' => 'postgres', 'schema_search_path' => 'public'))
ActiveRecord::Base.connection.create_database(config['database'], config.merge('encoding' => @encoding))
ActiveRecord::Base.establish_connection(config)
rescue
$stderr.puts $!, *($!.backtrace)
$stderr.puts "Couldn't create database for #{config.inspect}"
end
when 'sqlite'
`sqlite "#{config['database']}"`
when 'sqlite3'
`sqlite3 "#{config['database']}"`
end
else
$stderr.puts "#{config['database']} already exists"
end
end
At first glance, it looks OK: it tries to establish a successful connection, if it succeeds, it executes the else clause and prints a message stating that the database already exists; otherwise Ruby creates it in the rescue clause with the sqlite3 command line tool (or sqlite if using the sqlite adapter).
What truly happens though, is that the rescue clause is never executed (short of Ruby raising an unrelated error). The reason for this is that the two lines of code between begin and rescue, not only attempt to connect to the given database, but will create the database if this doesn’t exist already (well, the SQLite’s Active Record adapter creates it). This means that regardless of whether the database existed or not, those two lines will be successfully run and won’t raise any errors. The final outcome is that the database that didn’t exist, gets created, and the else clause gets executed, wrongly warning us that the database already existed.
I opened a ticket for this and submitted a patch. This is the new create_database method that works correctly:
def create_database(config)
begin
if config['adapter'] =~ /sqlite/
if File.exist?(config['database'])
$stderr.puts "#{config['database']} already exists"
else
begin
# Create the SQLite database
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection
rescue
$stderr.puts $!, *($!.backtrace)
$stderr.puts "Couldn't create database for #{config.inspect}"
end
end
return # Skip the else clause of begin/rescue
else
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection
end
rescue
case config['adapter']
when 'mysql'
@charset = ENV['CHARSET'] || 'utf8'
@collation = ENV['COLLATION'] || 'utf8_general_ci'
begin
ActiveRecord::Base.establish_connection(config.merge('database' => nil))
ActiveRecord::Base.connection.create_database(config['database'], :charset => (config['charset'] || @charset), :collation => (config['collation'] || @collation))
ActiveRecord::Base.establish_connection(config)
rescue
$stderr.puts "Couldn't create database for #{config.inspect}, charset: #{config['charset'] || @charset}, collation: #{config['collation'] || @collation} (if you set the charset manually, make sure you have a matching collation)"
end
when 'postgresql'
@encoding = config[:encoding] || ENV['CHARSET'] || 'utf8'
begin
ActiveRecord::Base.establish_connection(config.merge('database' => 'postgres', 'schema_search_path' => 'public'))
ActiveRecord::Base.connection.create_database(config['database'], config.merge('encoding' => @encoding))
ActiveRecord::Base.establish_connection(config)
rescue
$stderr.puts $!, *($!.backtrace)
$stderr.puts "Couldn't create database for #{config.inspect}"
end
end
else
$stderr.puts "#{config['database']} already exists"
end
end
I removed the creation of the SQLite (2 and 3) database from the case statement inside the rescue clause, but the actual deal is in these lines:
if config['adapter'] =~ /sqlite/
if File.exist?(config['database'])
$stderr.puts "#{config['database']} already exists"
else
begin
# Create the SQLite database
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection
rescue
$stderr.puts $!, *($!.backtrace)
$stderr.puts "Couldn't create database for #{config.inspect}"
end
end
return # Skip the else clause of begin/rescue
else
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection
end
If the specified adapter is sqlite or sqlite3 then Ruby checks if the database file exists or not. If it exists, the warning message is printed; if it doesn’t, Ruby creates the database automatically by invoking the connection method after running establish_connection with the configuration as an argument. If there is an error during the creation of the database — or the connection to the database, if this already exists — the error and its backtrace are printed. Finally, we exit the method so as to not execute the else clause of the begin/rescue statement, which would erroneously print that the database already existed. If the adapter in use is any other adapter, then we just run the two lines as we did in the non-patched version of the method.
Notice also that there was a second problem with the original code. I couldn’t replace those connection lines with `sqlite3 "#{config['database']}"` because they would open an SQLite3 shell which remains open, so the rake task waits forever for that shell to exit (you can actually see this if you use system rather than “).
2. Preloading has_and_belongs_to_many associations generates non-standard SQL
This second bug is quite serious and relates to the generation of non-standard SQL in has_and_belongs_to_many associations. Imagine that you have two models and a finder as follows:
class Project < ActiveRecord::Base
has_and_belongs_to_many :developers
end
class Developer < ActiveRecord::Base
has_and_belongs_to_many :projects
end
p = Project.find(:all, :include => :developers)
That finder will generate a query like:
SELECT developers.*, t0.project_id as _parent_record_id
FROM developers
INNER JOIN developers_projects as t0 ON developers.id = t0.developer_id
WHERE (t0.project_id IN (1,2))
Notice that _parent_record_id? That’s problematic because an SQL identifier cannot begin with an underscore, and this is true for all three versions of the standard (92, 99 and 2003). You can verify the non-conformity of the generated SQL through this validator. That single underscore broke support for DB2 and Oracle, and possibly other standard compliant databases.
I submitted a patch for this some time ago and it was immediately applied by Jeremy Kemper, so Edge Rails and the next version of Rails aren’t affected. Meanwhile the IBM_DB gem worked around the issue and I believe the Oracle enhanced adapter did the same.
3. DEFAULT NULL NULL
The third bug is somewhat similar to the second one and it’s caused once again by non-standard SQL. This too, like the previous bug, breaks support for DB2, Oracle and possibly some other databases. Imagine that we have the following migration file:
class CreatePosts < ActiveRecord::Migration
def self.up
create_table :posts do |t|
t.string :title
t.text :body
t.timestamps
end
end
def self.down
drop_table :posts
end
end
When the up method gets invoked by a rake task, the following query will be generated (this is the SQLite version of it):
CREATE TABLE "posts" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"title" varchar(255) DEFAULT NULL NULL,
"body" text DEFAULT NULL NULL,
"created_at" datetime DEFAULT NULL NULL,
"updated_at" datetime DEFAULT NULL NULL
)
Exclude for a moment the primary key, whose syntax is entirely adapter specific. The rest of the query still shows a problem. DEFAULT NULL NULL is not standard SQL and will not be accepted by some databases. This is caused by the fact that in the column definition we didn’t specify a :default option, so we have the equivalent of :default => nil, which translates into DEFAULT NULL. This precedes the second NULL, which is there to indicate that the column accepts nulls. Again, even the second NULL is not part of the SQL standard. A field is nullable by default, therefore we only need to specify something if it must not accept NULLs (i.e. by appending NOT NULL).
The solution to this problem lies in changing the behavior of ColumnDefinition’s instance method to_sql in activerecord/lib/active_record/connection_adapters/abstract/schema_definitions.rb. This is the current definition:
def to_sql
column_sql = "#{base.quote_column_name(name)} #{sql_type}"
add_column_options!(column_sql, :null => null, :default => default) unless type.to_sym == :primary_key
column_sql
end
Notice how the second argument of add_column_options! is a hash that sets the null and default values in every case. If these values are nil, the value NULL gets pushed into the final SQL query. We need to prevent this from happening by passing the two options only when they are not nil.
def to_sql
column_sql = "#{base.quote_column_name(name)} #{sql_type}"
column_options = {}
column_options[:null] = null unless null.nil?
column.options[:default] = default unless default.nil?
add_column_options!(column_sql, column_options) unless type.to_sym == :primary_key
column_sql
end
The migration file we saw above will now generate the following query (again, in the case of SQLite):
CREATE TABLE "posts" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"title" varchar(255),
"body" text,
"created_at" datetime,
"updated_at" datetime
)
This is standard SQL, if we exclude the primary key syntax which is specific to SQLite. More importantly, this would work and be accepted by other database systems like DB2 and Oracle. Version 0.9.5 of the IBM_DB gem monkey patches the issue in a similar fashion, in order to work around this problem and provide support for Rails 2.1. It is possible that the Oracle enhanced adapter 1.1.1 does the same thing. I didn’t submit a patch for this issue because several tickets regarding it are already open, and Nick Sieger already beat me to it. His patch was submitted more than a month ago but it hasn’t received any attention yet. I hope that this post will contribute towards changing that.
Conclusion
I think the last two bugs highlight a valid lesson: the importance of adhering to the SQL standard, particularly when implementing an Object-Relational mapper that supports so many databases. ActiveRecord takes the approach of generating SQL queries by mixing common SQL bits, defined outside of a particular adapter, with dialect-specific parts which are defined by each adapter. It is therefore crucial to ensure that the bits that end up in the SQL queries, independently from the adapter in use, are based on the SQL standard and not on the behavior of SQLite or MySQL.
The same principle applies when building a CMS system or a popular blog engine. If the project is aimed at being portable from one RDBMS to another with little hassle, attention should be payed to custom queries in order to keep them as close to the SQL standard as possible. Of course, it’s perfectly fine to create projects that are database specific and take advantage of that particular database’s strength. For example, DB2 Express-C offers the ability to handle XML in a native manner through a technology called pureXML. This is a fantastic feature, which can’t really be ported to a different RDBMS, but it’s worth its weight in gold, particularly when working in Ruby.
Lastly, don’t let this reflect negatively on your judgment of Rails’ code quality. A large and ambitious project like Rails is bound to have bugs here and there, no matter how scrupulous its developers are.
IBM’s crazy DB2 videos. What do you think?
IBM is clearly synonymous with the term “Enterprise”, but I can testify that working here means interacting with many cool, down to earth people who are passionate about their profession. They are not the “corporate drones” that you’d expect. I sometimes think that there is a divide between the image of IBM and the reality of a company that’s mostly comprised of young people having fun while working hard.
Reading comments on sites like Slashdot or Reddit I see that IBM is often unfairly treated, despite being a champion of Open Source, with contributions in excess of those of Google and Amazon combined.
So I’m glad to see that IBM is not “taking itself too seriously” with a series of ads, campaigns and initiatives that portray a less rigid image. The silly “servers taking over” ads are examples of this.
More recently IBM has put out a few “unusual” videos about DB2 Express-C:
- The Pitch Meeting – Monday (Part 1)
- The Pitch Meeting – Tuesday (Part 2)
- The Pitch Meeting – Wednesday (part 3)
- Installing IBM DB2 Without Touching the Keyboard or Mouse (not a pro commercial)
So I’d like to ask you to share your opinions about them. Are they “cool” and funny, or are they goofy and just trying too hard?
This Week in Ruby (May 29, 2008)
This is the 9th episode of This Week in Ruby, please consider subscribing to my feed so as to not miss any weekly installments.
Ruby
Two days ago JRuby 1.1.2 was released. Amongst several bug fixes and improvements, this release is characterized by a focus on performances. Startup time, threading, method calling and YAML symbol parsing have all been drastically improved.
Huw Collingbourne of SapphireSteel, has announced that he’ll be releasing a complete book on Ruby, chapter by chapter, free of charge online. After reading the first chapter, I can attest that it’s excellent. Keep an eye on it, as new chapters get added.
The Pragmatic Programmers put out a series of screencasts for sale. The most relevant series for Ruby programmers is Everyday Active Record. The first two episodes (a half an hour long, each) are out and can be purchased for just $5 a piece. The preview — and Ryan Bates’s reputation — lead me to believe that they are entirely worth their very reasonable sticker price. Speaking of screencasts, a new one about merb-slices was released on Merbunity, check it out if you’re into Merb.
There were two important releases last week, Mack 0.5.5 — which features a new rendering engine with support for Haml and Markaby — and DataMapper 0.9, a major reworking of the ORM. A third release, which is perhaps just as welcomed, was launched by _Why who included a few graphical improvements for Shoes, his GUI application toolkit. Definitely neat stuff, which I invite you to take a look at if you’re working on a Mac.
Peter Cooper published 21 Ruby Tricks You Should Be Using In Your Own Code. You probably know already most of the common ones at least, but they’re quick and fun, so if you haven’t checked out the post yet take a moment and do so. Other must-read tutorials and articles were Ruby && DTrace! (really neat results), Ruby EventMachine – The Speed Demon by one of my favorite Ruby bloggers, and Will’s Guide to Mashing-up Remote Databases using Page Scraping.
In a post made a couple of days ago, Robert Fischer opened up a can of worms by bringing up the issue of Ruby and XML libraries. As most of you know REXML is far from being issue-free (performance in primis), and in The Status of Ruby’s libxml Robert uncovers that the author of LibXml Ruby is unable to actively pursue the development of his extension. This issue concerns me, but if I’m working with databases, I prefer to take advantage of DB2 Express-C ’s fantastic pureXML features, which give me the sort of speed, flexibility and stability that I won’t find in a Ruby library anytime soon.
Before highlighting some of the news from Rails-land, I wanted to inform you that a new version of The Great Ruby Shootout will surface in June, as I intend to test a couple of special new entries.
Rails
Today, RailsConf 2008 started and it certainly stands a great chance of being dubbed an exhilarating event. A few people enquired to see if they could meet me there, but unfortunately I couldn’t make it. Chances are that you’re reading this post from RailsConf. If that’s the case, say hi for me and don’t forget to visit the nice fellas from Engine Yard, Morph (my sponsor), Phusion and GemStone. Oh and also, feel free to pass around the url of this entry.
Rails 2.1 RC1 is out, so you’ll find this article on upgrading to Rails 2.1.0_RC1 useful. Fabio Akita released a new version of his popular tutorials, Rolling with Rails 2.1 (part 1 and part 2). And if you are looking for an advanced authentication/authorization system for Rails 2, take a gander at Lockdown on RubyForge.
My friends at SeeSaw implemented a series of Rails Widgets which can easily be installed as a Rails plugin. Feel free to use them and/or contribute, in order to add further support for simplifying and reusing common UI elements. Speaking of shiny things, check out this Ruby on Rails icon pack; very pleasing to the eye, in my opinion.
RubyInside published a list of 28 mod_rails / Passenger Resources To Help You Deploy Rails Applications Faster. As DHH forecasted, “this could definitely become very popular, very fast ”.
New Relic released their RPM solution for monitoring and improving the performances of Rails applications to the general public. You can get it here.
And finally, some great news just came in, IronRuby is running unmodified Rails. “Excellent” (said in Montgomery Burns’ voice, complete with characteristic hand gesture).
Is the Enterprise world Rails ready?
When searching the web for the words “Rails” and “Enterprise” you’ll find countless discussions about whether Rails is Enterprise ready. Some argue that it is, especially thanks to the extendibility offered by its plugin support, while others claim that realistically it’s not. “Is Rails Enterprise ready?” is the wrong question, I’d rather ask if the Enterprise world is Rails ready. Let me clarify this point.
David Heinemeier Hansson gave a brilliant talk at Startup School, in which he didn’t speak about Rails. He spoke about business models, 37signals’ way of charging for subscriptions to web apps, the odds of a startup becoming the next Facebook and so on. He rarely mentioned Rails, but that presentation can tell you more about Rails and the Enterprise, than most of the essays that you’ll read on the topic.
Companies like 37signals love their way of doing business. They solve problems by creating valuable services for the long tail of small business owners, and of course charge them for doing so. They are all about productivity and having fun in the development process. That’s the kind of environment in which Rails was born. It came out of the necessity to increase productivity, while offering an enjoyable experience to the web programmer, and Ruby, as a language, was the perfect choice for that.
Ruby on Rails was created so as to be an almost perfect match for what David and 37signals needed. They weren’t facing the issue of legacy databases, so they were able to choose simple conventions that made sense for them. They had a server to host their applications, so the whole shared hosting issue that many people complained about was not a problem for them. Rails’ cost of scalability was not troubling for 37signals, because they had paying customers. More scalability issues for companies like 37signals, mean more money rolling in from paying customers; so just throwing hardware at it can be done without frowning. It means that business is good. Given the choice of picking development speed over application speed, in these types of contexts development gains are often a much bigger payoff.
Rails is opinionated because it was tailored for the needs of 37signals and their products. If your web application and business model is somewhat similar to that of David’s company, then Rails is hard to beat and there is little to complain about. Rails will evolve and continue to improve, but most of its focus will remain on 37signals and the needs of similar companies. David prefers to work on real world web applications and introduce the most useful lessons back into the framework, not having them be designed by a committee. That’s why there isn’t a Rails, Inc. and why new Rails features are not arbitrarily introduced to satisfy any possible usage of the framework by the thousands of companies who employ Rails.
Aside from its origins, Rails became a sensation in the web development world. It had a deep impact, just think about how it made the MVC paradigm an accepted and almost expected reality in other web frameworks that come out later. So with everyone jumping on the bandwagon, people started to consider Rails for use in environments that it wasn’t really created for. Namely the so-called Enterprise world.
This is why the question “Is Rails Enterprise ready?” is the wrong one to ask. It questions whether Rails has evolved yet to the point of being able to support developers within the Enterprise. Rails has no expectation of being a good match for the current Enterprise world. It doesn’t now and won’t in the future. There is however an expectation that the Enterprise world will become more Agile and embrace simplicity over complexity. In other words, let’s ask if the current Enterprise is ready for Rails. The answer I’m afraid is not a sound “yes”. Rails can be used in the Enterprise world, and a good part of my job is promoting its adoption exactly in this kind of environment, but there is little point in complaining aloud about the fact that Rails doesn’t have great support for certain features out of the box.
When Dave Thomas made his famous keynote at RailsConf, in which he pointed out important issues that concern the most “enterprisey” customers, David did not embrace the idea at all. The reason is simple, in his eyes, it’s the Enterprise world that needs to change, not Rails.
As a matter of fact, Rails can be successfully used for complex scenarios, and it’s probably still much more productive than using an overkill like J2EE, in most cases. But it’s not natural, it forces you to take advantage of (maintained or not) plugins, and the whole ecosystem around Rails is not very supportive of the needs that you may have.
What happens to those developers who “saw the light” and would like to introduce Rails into an environment where it’s still someone else, or a different team, who defines and handles the database? Plugins help, integration with Java through JRuby is a viable alternative too, but overall the experience can be rather frustrating. The Enterprise, especially when dealing with existing data, can be very slow to adapt. And we all know that fighting against bureaucracy within your large company is anything but enjoyable.
It seems to me that most of the Enterprise is not Rails-ready, just like it’s still hardly Agile-ready. The current compromise is meeting at a middle-ground. Developers add enterprisey features into Rails through plugins, or adopt JRuby on Rails, while at the same time trying to simplify as much as possible in order not to go “against” Rails.
But is there a better way? In the Ruby world, I see two viable possibilities but neither is easy or quick. The strongest one is a Ruby framework that takes into account the needs of the Enterprise, from scratch, while still remaining easy to use like Rails. It would be the equivalent of what motivated the creation of Rails, in a different, substantially more complex environment. The second option is a fork of Rails, specifically targeted towards the Enterprise/Corporate sector, with a rewrite and expansion of a few key components in order to make it a superset of Rails.
Whoever creates either of these frameworks well, will be worth their weight in gold.
On Rails and DB2
James Governor wrote an interesting post entitled “On douchebags, DB2 and Rails: DHH not pragmatic enough?”. It’s partially in response to Jeff Atwood’s article “Douchebaggery” which, in turn is a reply to an old post by David Heinemeier Hansson. So please excuse me if I limit the entertainment value of this post by not calling anyone names.
Amongst a few sensible comments, James also argues that David may not be nearly opinionated and pragmatic enough after all:
What’s bothering me though is that perhaps Hansson isn’t opinionated enough. You see Rails targets MySQL. MySQL is the Rails database. Forget all the configuration crap and just use MySQL. There is no need to consider other database platforms. LAMP may or may not be a literal but MySQL is utterly embedded in the Rails way. In Rails database is not a design decision, its a given.
As most of you will already know, Rails was extracted from a real web application called Basecamp. This approach is thought to be the first sign (scientists believe there are 37 of them ) that we are dealing with a pragmatic framework. Its features and design choices weren’t defined a priori by David, but they were implemented to satisfy particular needs (and opinions) which Hansson and his team had, while developing database driven web applications.
When analyzed under a proverbial microscope, one sees that Rails didn’t actually invent anything new. Neither the MVC paradigm nor the Active record pattern were especially innovative. They’d been well known for years in the Computer Science world and even adopted (to a certain extent) within the enterprise realm. Yet, Rails was a revolutionary piece of software. What Rails did was to tie all the loose ends together into a neat, tidy package. On top of that achievement, it clearly favored simplicity, pragmatism and developer productivity over anything else. Forget about repetition and countless XML configuration files, it was opinionated software that made web development, or at least most of it, rather trivial. Less choices for the developer (the Paradox of Choice comes to mind) turned out to be a significant gain in terms of “ease of use”, which in turn bolstered Rails’ adoption.
Rails wasn’t supposed to be the answer to all web development needs everywhere – just to most of them. This means that the making of such a DSL involved a lot of assumptions. In turn, there are people who don’t use Rails due to these assumptions, and projects where adopting Rails wouldn’t probably be the smartest choice either. But Rails worked for David and 37Signals, and in doing so, it also happens to work very well for thousands of other developers around the world with similar development needs, who’re facing analogous challenges. David did his best to create a usable and enjoyable web framework, and succeeded spectacularly in responding to the needs of most web developers.
For this, Rails constantly gets accused of being far too opinionated. How many times have we seen comments about the lack of prepared statement support, that there’s no connection pool, no thread safety, no unicode, about deployment issues and so on. James’ comments are interesting because they’re entirely unusual. He’s arguing that Rails is not opinionated enough, and that in itself, is an opinion worth replying to (and not simply because he explicitly called upon my views on the subject at hand within his post):
Why in the Getting Started section of the book, do the authors even mention DB2 as a potential target database, even if its just an aside (Rails does allow exceptions it just doesn’t like them)? I mean if a DB2 bigot came to ask Hansson for a job at 37Signals he would be shit out luck wouldn’t he? You can imagine the laughs- “this guy wants to back end Rails apps with DB2 – what a ***tard.”
I know Anthony would disagree with me, and arguably Rails abstracts the database decision, so Rails is not tightly coupled to MySQL. But that’s not the point – we’re not talking about JDBC here.
I’m pretty sure that David doesn’t care about shared hosting, prepared statements, support for stored procedures, JRuby, DB2 or the “enterprise world”. It’s not his way of creating and going about web development, plus he really doesn’t have any need of these for his own purposes. Therefore don’t expect support for these points from him. However, that said, David is also being fairly reasonable with those who think differently than he does.
David’s message is: if you need these type of features, then please by all means, go ahead and implement them yourself. Rails’ core team has enabled the development community to extend Rails to fit a different kind of bill. While the policy in terms of accepting patches may be strict and not particularly welcoming of every single contribution, the plugin architecture coupled with Ruby’s flexibility and extensibility, allows the injection of substantial features which in turn significantly affect Rails. There are limits of course, and some people even preferred to create a similar framework from scratch, because they found it easier and neater to do so than to completely restructure and refactor Rails’ core.
David clearly favors opinionated conceptual integrity and the act of defining a set of sensible conventions over abstraction/modularity/extensibility, which would be regarded highly by those who require enterprise features or view web development in a markedly different way. He did not intentionally set out to block these things either though, and that’s worth remembering. He proposed his take on things, but he certainly did not impose it on anyone else. Doing so simply would not make sense (and might have required more work on his part in some cases).
Take ActiveRecord for example: it’s not a perfect abstraction, given that its Ruby implementation makes several MySQL centric assumptions in regards to SQL generation, which makes development of alternative adapters for other databases less than trivial (particularly for so called enterprise databases). But it’s still essentially a database agnostic pattern that would be almost harder not to implement as such. David’s implementation poses a few challenges here and there, but we’re still very much within the realm of possibility.
David might only use Rails with MySQL, and he may only slightly care about the subset of features which are common to all databases (including SQLite, the current default database) and nothing more, but at the same time he clearly realizes that we are not all Davids. He’s not willing to warp his creation in order to make everyone happy. That’s not David’s style. However, if other databases or options are available to Rails developers, why not mention them as a testament of the interest surrounding Rails?
The fact that DB2 is a viable (free of charge) option for Rails developers doesn’t affect the Rails core team in a perceivable way, and it still offers the possibility of using Rails to a whole new segment of programmers. Which bring us to the next point:
Anthony says “an increasing number of Rails developers adopting DB2 as their database of choice”. I would like to know more about this “increasing number” from Anthony [...]
While perhaps not the most vocal bunch in the blogosphere, there are many developers and companies interested in DB2 on Rails. In my experience, I’d say that there are three main likely/common scenarios (of DB2 on Rails users):
- Companies who have already widely adopted DB2 in their infrastructure and are now considering Rails for some of their projects. The developers involved are usually very excited about DB2 on Rails, because it means that they have a rare chance to use Ruby/Rails rather than Java/J2EE or similar technologies, with a data server that they already have a good understanding of. While some web developers choose the framework first and consider the database at a latter stage, most companies don’t work that way.
- Developers who have DB2 skills and are looking for a good framework for developing web applications. They have no reason to use any other database but DB2.
- Companies or developers who understand the technological advantage offered by DB2′s features (e.g. utmost speed, pureXML, compression, replication, high availability, affordable 24/7 support, etc…) and therefore choose it for their Rails based projects.
Rails doesn’t exploit advanced database features, and as a result, it doesn’t provide any further compelling reasons to adopt other databases aside from MySQL. It takes a discerning developer to realize that DB2 has much more to offer and that there are concrete benefits to adopting it – even when developing with Rails.
It’s also true that working with an independent adapter in Rails will generally be less “comfy” than using SQLite/MySQL/PostgreSQL. For example, certain tasks are not implemented for DB2 by the Rails team, and as such won’t be available. But it’s a small price to pay. In order to change this, I decided to start a project in my spare time (not an IBM project), called DB2 Extended. From the RubyForge description: “db2_extended is a Rails plugin that extends support and enhances the user experience for developers who adopt DB2 as their backend for Rails applications”. This project is currently empty, but I hope to perform the first svn commits over the weekend. I’ll start with simple things that close the gap (like making rake db:create work for DB2) and incrementally, I’d love to throw in DB2 specific functionalities and enhancements as well. Being a personal, open source project, I will more than happily accept patches and contributes from the DB2 on Rails community. Hopefully in a short period, we’ll have a plugin that makes web development with Rails and DB2 even more fun and productive.
As usual, these opinions are only mine and don’t necessarily represent my employer’s (IBM) positions, strategies or opinions.
