In this brief tutorial I’ll show you how to create a complete Python and Django setup for DB2 on Ubuntu. By following my step-by-step instructions, you’ll be able to install the following components:

• Python
• easy_install
• Django
• DB2 Express-C 9.7.4
• The official Python driver, dbi module, and Django adapter for DB2

Installing Python and easy_install

Technically Python is already installed on Ubuntu, so we don’t have to install it. However, we need the python-dev package in order to build the DB2 driver from source. We’ll also need easy_install which is included in the python-setuptools Ubuntu .deb:

$ sudo apt-get install python-dev python-setuptools

Installing Django

Now that we have easy_install available, we can easily install Django by running:

$ sudo easy_install django

You can verify the installed version by running:

$ python

And then executing the following snippet in the REPL:

>>> import django
>>> django.VERSION
(1, 3, 0, 'final', 0)

At this time, version 1.3.0 should get installed as shown above.

Installing DB2

We can now download and install DB2 Express-C 9.7.4. Download the .tar.gz file to a location that’s convenient for you. Then proceed to unpack it:

$ cd ~/Downloads/
$ tar xvfz db2exc_974_LNX_x86.tar.gz
$ cd expc

We’ll install one required library and then proceed with the setup:

$ sudo apt-get install libaio1
$ sudo ./db2setup

Follow the GUI wizard on screen to continue with the installation. Pay close attention to two steps:

• When prompted select a custom installation, and when choosing the components, select all of them. We’ll need the Application Development Tools in order to build the Python driver later on (and these are unchecked by default).
• When asked if you’d like to create an instance user, go with that option. It greatly simplifies the setup process.

When the setup of DB2 is completed, you should receive a confirmation message informing you about the successful installation

For good measure, add the following line to your ~/.bashrc file:

. /home/db2inst1/sqllib/db2profile

This ensures that even your regular, non-DB2, user will be able to connect and interact with the database.

Installing the Python driver and Django adapter

The last step we need to take is to install the ibm_db Python driver and the ibm_db_dbi module, as well as the Django adapter for DB2. These are all IBM supported, open source releases.

Open a new shell and run:

$ sudo -s
$ export IBM_DB_DIR=/home/db2inst1/sqllib
$ export IBM_DB_LIB=/home/db2inst1/sqllib/lib
$ . /home/db2inst1/sqllib/db2profile
$ easy_install ibm_db ibm_db_django
$ exit

A quick sanity test

To ensure that all is well with your setup, run the following command:

$ django-admin.py startproject db2test
$ cd db2test

Now edit settings.py so ENGINE is set to ibm_db_django, NAME to a database like db2test, USER to db2inst1, PASSWORD to the same password you specified for the db2inst1 user during the installation of DB2. Use localhost for the HOST, and 50000 as the port number:

DATABASES = {
    'default': {
        'ENGINE': 'ibm_db_django',
        'NAME': 'db2test',
        'USER': 'db2inst1',
        'PASSWORD': 'secret',
        'HOST': 'localhost',
        'PORT': '50000',
    }
}

Create the database db2test by running:

$ su - db2inst1
$ db2start
$ db2 create db db2test
$ exit

Depending on your hardware specs, the infrequent operation of creating a database can take a bit of time (e.g., several minutes).

We will now run the actual test:

$ python manage.py test

Unless exceptions are raised during this last step, you should be all set to use Django with DB2.

If you need help

IBM is the only database vendor to officially provide and support its Python driver and Django adapter. While commercial DB2 support is available and relatively inexpensive, your first line of defense is posing your questions in the support forum over at Google. Alternatively, if the question is DB2-specific and not related to Python/Django, you can use the DB2 Express-C forum over at developerWorks instead.

No related posts.

Much as with human languages, programming languages are used to communicate. Interestingly they still involve communication between people, whether other programmers will end up reading/modifying/enhancing your code or you’ll do at a later point in time. Unlike natural languages however, programming languages are unequivocally understood by computers, thanks to the aid of interpreters, compilers, and similar types of software.

Depending on the kind of program you’re trying to write, and in which environment you’re working, you’ll find that some programming languages are better suited than others for certain tasks, even if most common programming languages are labelled as being “general purpose”.

For example, if you want to create iPhone and iPad applications, your best bet would be to use Objective-C. For Android smartphones and tablets, in most cases, you’ll need to become accustomed with Java. If you want to build web applications, you’ll want to learn client-side technologies such as HTML, CSS and JavaScript (of those three only JavaScript is an actual programming language, but the other two are necessary nevertheless). For the server-side of your web application, you’ll probably want to learn a language like PHP, Ruby, or Python (all three have frameworks that will help make most web programming tasks easier and less time consuming), rather than a system programming favorite like C. (And of course, one has to mention SQL if you’re interfacing with a relational database).

So should you start your foray into the world of programming languages with JavaScript? What about Objective-C? SQL? My advice is to avoid starting your programming journey with a language that is tailored to a specific environment. This early in the game, your main objective is to learn how to program – and how to reason like a programmer – rather than learning a specific language. Once you have mastered the fundamental concepts with the aid of a general purpose programming language, you can ease into other languages as the need arises, depending on the type of software you’d like to develop.

This is a profession where most of your skills are developed and honed by actually “doing”. If you grasp the fundamentals of programming outright, you can easily pick up a specific language and framework when you need to build iPad applications or whatever else you’d like to create.

A well-rounded programmer will end up learning several programming languages and tools over the course of their career. That said, if you are just starting out, focus on one language first. While this might seem a tad dull on the surface, the good news is that unless you pick a truly exotic programming language, a lot of what you learn will be applicable to other programming languages as well. Remember that at this stage, above all else, you are learning how to communicate your thoughts in precise statements for other programmers to understand (and for the computer to execute).

For this reason, you can’t really go too wrong with any of the “somewhat mainstream”, general purpose programming languages (e.g., C, Java, C#, Perl, Ruby, Python, or Scheme). That said, particularly if you are learning on your own, I’d advise that you go with a language that will make your learning trek easier. In other words, opt for a programming language that is widely used, well documented, and beginner-friendly. You’re looking for one that isn’t overly bureaucratic, doesn’t require you to manage memory directly, or have a deep understanding of mathematics to get started.

As such, my suggestion would be to start with Python, and use it as a tool to learn the general craft of programming. Learning Python is fun, easy, and useful. You’ll be able to use it for a wide array of projects in several environments (scripting, web, scientific research, etc…).

There are a variety of free tutorials on the web, but if you want a more rigorous/systematic/academic introduction, I highly recommend “Python Programming: An Introduction to Computer Science (2nd Edition)” (USA | UK | Canada).

Once you have learned the fundamentals of programming, have a decent command of the Python language, and have gained some experience with practical Python projects, you should be better armed to evaluate and pick up other languages and frameworks based on the projects you intend to develop or contribute to in the future (Open Source projects are awesome for this purpose).

No related posts.

It isn’t stretching things too much to say that both Ruby and Python can be seen as slower, easier (for beginners), object-oriented, infix Lisp dialects.

Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people. — Yukihiro “Matz” Matsumoto

Ruby and Python aren’t intimidating and remain very approachable for absolute beginners. Furthermore, their approachability is not confined to the language design itself, but transcends into the community and ecosystem that surrounds them.

I’m not here to discuss how languages like Ruby and Python managed to become more popular than major Lisp dialects nowadays. I’d rather focus on how these gentler introductions to functional programming are acting as gateway drugs to Lisp for many developers.

A community that values metaprogramming and is obsessed with the construction of DSLs (Domain Specific Languages) like the Ruby’s is, will no doubt find in Lisp a valuable ally. Plus, if you know Ruby inside and out, you should find Lisp to be easy enough to learn.

To attract Ruby developers though, Lisp has to offer something more than just a set of powerful features. You could say that Rails is enough of a reason to learn and use Ruby. But what is Lisp able to solve all that better than Ruby? I’ll answer that question by focusing on a specific dialect of Lisp, that I and continually more Ruby developers are getting into: Clojure.

It wouldn’t be fair to characterize the Lisp community as stagnant, but Clojure is definitely a welcomed dose of new blood. Clojure is a JVM-based modern Lisp designed for concurrency, which elegantly includes a set of carefully chosen features that are not easily found in mainstream languages.

In my opinion, Clojure has three main advantages over Ruby:

• It’s much faster than Ruby, which makes it a better choice for intensive processing. (FlightCaster for example, uses both Rails and Clojure. Rails for the “front-end” and Clojure for the heavy lifting/forecasting.)
• It greatly simplifies concurrent programming, making the language more future-proof as hardware manufacturers continue to produce processors with more CPU cores.
• Clojure emphasizes functional programming and tries to minimize side effects.

Clojure’s interoperability with Java resolves the issue of only having a few available libraries, which often affects new languages. It also helps in getting people to use the language within the enterprise world where Java still dominates.

Of all the “new” languages out there, I find Clojure to be the most fun, interesting and pragmatic: it’s something worth getting excited about. I don’t really care if it turns out to be the next Ruby or not, it’s a language that’s worth knowing and using. (If you haven’t tried it yet, a decent, short introductory book is the recently published Practical Clojure.)

Clojure’s popularity may even bring more attention to Lisp in general (for example, most must-read literature uses Scheme or Common Lisp). Perhaps then, it may indirectly help introduce more traditional Lisp dialects to a new generation of programmers.

No related posts.

For example, IBM’s adapter (through the underlying DBI wrapper) now uses persistent connections, which are especially helpful when dealing with Django – as it lacks connection pooling. (Of course DB2 also has the Connection Concentrator to aid in reducing the usage of server resources and improving scalability.)

Furthermore, the adapter adds support for the DECIMAL datatype, a necessary feature when dealing with money and currencies. Various enhancements and bug fixes were included too; check them out on Google Groups.

As a reminder, DB2 Express-C is an absolutely free of charge version of DB2 and it’s production ready (not a toy version). You can download it from here. Take it for a spin, experiment – chances are you’ll like it. If you need a guide to getting started, be sure to check out this free e-book by my colleagues Raul, Ian, and Rav.

No related posts.

In order to download the screencast, you don’t need a credit card, to provide your address or even your last name. Just head on over to this page and join the newsletter. Upon confirming your subscription, you will immediately receive an email with links to DRM-free, 720p HD files in the formats QuickTime Movie (.mov), AVI and Ogg Theora (.ogv). These videos are in English, prepared by a published serial author on the subject of Python/Django, and narrated by a native English speaker. They also include optional subtitles in the .srt format, as well as the source code which is released under the MIT license, as is customary for ThinkCode.TV to do.

I hope you enjoy the free screencast and stay tuned for the launch in three weeks.

Possibly related posts:

1. Announcing ThinkCode.TV’s new English newsletter
2. Ruby 1.9 screencasts plus an announcement
3. Screencast on how to install DB2 Express-C on Mac OS X

Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google’s webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking infrastructure.

The story so far

This release generated widespread interest among the Python and open source development communities. Rightfully so. There are many reasons to like Tornado. To begin with, it’s fast — and that’s fundamental for a web server. By using nginx as a load balancer and a static file server, and running a few Tornado instances (usually one per core available on the machine) it’s possible to handle thousands upon thousands of concurrent connections on relatively modest hardware; and this isn’t just theory. Tornado has already proven its worth in the field, by allowing FriendFeed to scale graciously.

Tornado is not only a fast web server, it acts as a very lightweight application framework as well. As such, it’s an appealing alternative to well established frameworks to the growing group of developers who’d like to develop “closer to the metal” and avoid the baggage associated with full-fledged web frameworks. The two things combined make Tornado ideal for developing “real time” web services and applications.

The feedback so far hasn’t been all positive though. Criticism of the project has mainly focused on the lack of test coverage and the fact that FriendFeed has opted not to contribute to, and improve on, the existing Twisted Web project (which has similar goals). To make things worse, there were a few nonchalant comments about it as well. Performance issues and lack of ease of use were the reported motivations for starting a new project from scratch.

Dustin Sallings started working on a hybrid solution (henceforth Tornado on Twisted) that would reportedly keep the good parts that Tornado introduced, while using Twisted as its core for networking and HTTP parsing.

At this point I became naturally curious about the speed of these three web servers. Is Tornado really faster than Twisted Web? And what about Tornado on Twisted, would it be faster or slower? Let’s find out.

Benchmark results

I ran a simple Hello World app for all three web servers. All the web servers were run in standalone mode without a load balancer. I stress tested the web servers with httperf using a progressively larger amount of concurrent requests. 100,000 requests were generated for each test. The web servers were run on a desktop machine with an Intel® Core™2 Quad Processor Q6600 (8M Cache, 2.40 GHz, 1066 MHz FSB) processor and 8GB of RAM. The operating system of choice was Ubuntu 9.04 (x86_64).

Without further ado, here are the results:

As you can see Tornado turned out to be faster than the rest of the Python web servers. Handling a peak of almost 3900 req/s with a single front-end and on commodity hardware is nothing to sneer at.

Twisted Web didn’t do too bad either (max. 2703.7 req/s), but the difference in performance is noticeable. Likewise, the performance of Tornado on Twisted was virtually identical to that of Twisted Web.

There you have it. I was curious about the possible outcome and now I know. Remember, this is a report on the numbers I got on my machine, not a research paper. But I hope that you find them interesting nevertheless.

Show me the code

Tornado:

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()

Twisted Web:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
from twisted.web import server, resource

class Simple(resource.Resource):
    isLeaf = True
    def render_GET(self, request):
        return "Hello, world!"

site = server.Site(Simple())
reactor.listenTCP(8888, site)
reactor.run()

Tornado on Twisted:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor

import tornado.options
import tornado.twister
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])

    site = tornado.twister.TornadoSite(application)
    reactor.listenTCP(options.port, site)

    reactor.run()

if __name__ == "__main__":
    main()

UPDATE (September 14, 2009):

• The original version of this post included Unicorn as well. This wasn’t fair however, since it’s not an asynchronous web server.
• EventMachine HTTP Server was added, but I have since decided to remove it as I prefer to let the article be a fair comparison between asynchronous Python web servers.
• I initially used Apache Benchmark (ab). The results were misleading at best. I re-ran the tests with httperf and updated the results above.
• Stock Tornado couldn’t be tested with httperf because their HTTP Server doesn’t implement getClientIP(). I had to manually modify a method to return the remote ip address. This may introduce a very minimal advantage for Tornado, but it should be negligible in this context.
• I modified the examples for Twisted and Tornado on Twisted, to ensure that both took advantage of the epoll-based reactor.

No related posts.

Consider the following series of queries:

SELECT * FROM users WHERE karma BETWEEN 100 AND 499;
SELECT * FROM users WHERE karma BETWEEN 500 AND 999;
SELECT * FROM users WHERE karma BETWEEN 1000 AND 1999;
SELECT * FROM users WHERE karma BETWEEN 2000 AND 4999;
SELECT * FROM users WHERE karma BETWEEN 5000 AND 9999;
SELECT * FROM users WHERE karma BETWEEN 10000 AND 50000;

These each represent the same query and can be transformed into a single parameterized query:

SELECT * FROM users WHERE karma BETWEEN ? AND ?;

Trying to use clever tricks with quotes in order to inject arbitrary SQL code becomes futile. Parameters are considered values, and have no effect on the structure of the query itself.

Parameterized queries are therefore efficient and go a long way towards preventing SQL injection attacks in your applications. They have virtually no downside.

Newbie developers often ignore the existence of this feature and end up irritating seasoned DBAs who have to deal with the consequences of their incompetence. Leon Katsnelson argues that this is such an important matter, that every DBA should forward this Computerworld article to their developers. I tend to agree with how important of an issue that is.

That article provides the following example in Java:

String lastName = req.getParameter("lastName");
String query = "select * from customers where last_name = ?"
PreparedStatement pstmt = connection.prepareStatement(query);
pstmt.setString(1, lastName);
try { ResultSet results = pstmt.execute(); }

Here I’ll show you an example of how to work with parameterized queries from Ruby and Python. I’ll use the Ruby and Python drivers for DB2.

Ruby first:

require 'ibm_db'

conn = IBM_DB.connect("mydb", "db2inst1", "mypassword")

query = "SELECT * FROM users WHERE karma BETWEEN ? AND ?"
pstmt = IBM_DB.prepare(conn, query)

values = [500, 999]
IBM_DB.execute(pstmt, values)

while row = IBM_DB.fetch_array(pstmt)
  puts "#{row[0]}:#{row[1]}"
end

We load the driver (use mswin32/ibm_db on Windows, and ibm_db.bundle on Mac), create a prepared statement, and then bind the two parameter values to it through the execute method. We then fetch the resultset one row at a time and print the value of the first two fields for each record. For fine-tuned control we could have used the IBM_DB::bind_param method.

The Python version is very similar:

import ibm_db

conn = ibm_db.connect("mydb", "db2inst1", "mypassword")

query = "SELECT * FROM users WHERE karma BETWEEN ? AND ?"
pstmt = ibm_db.prepare(conn, query)

values = (500, 999)
ibm_db.execute(pstmt, values)

tuple = ibm_db.fetch_tuple(pstmt)
while tuple:
    print tuple[0] + ":" + tuple[1]
    tuple = ibm_db.fetch_tuple(pstmt)

As you can see, working with parameterized queries is not any harder than dynamically generating SQL queries. Yet the benefits of doing so are huge.

Unfortunately, despite being a very sound choice to base an Object-Relational Mapper (ORM) on, ActiveRecord does not use parameterized queries. Even when it looks like you are passing parameters to a given method, these are actually used to dynamically form an SQL query. Of course you are still free to use parameterized queries in your Rails applications by employing the driver directly. But I really think this is something ActiveRecord should be built upon.

Luckily for Django developers, Django’s ORM uses parameterized queries, thus improving both performance and security with a single design choice. In the Python world you couldn’t get away with ignoring parameterized queries.

For those of you using Rails, all is not lost. DB2 Express-C 9.7 has a killer feature known as the Statement Concentrator, which caches similar queries allowing them to use a shared access plan. It’s not as efficient as using prepared statements in your code, but it’s the best you can do when, as in the case of ActiveRecord, you can’t use parameterized queries directly. Leon’s article explains in greater detail how this feature actually works.

No related posts.

This article is obsolete. Please refer to the following articles for up do date instructions: Ruby/Rails and DB2 | Python/Django and DB2. Thank you!

This is the Python version of a post I made about Ruby a few days ago.

Now that Mac OS X 10.6 is out, it’s time to leave the world of 32 bit computing behind. The pre-installed Python interpreter will run in 64 bit mode by default, so you may need to pay attention when installing some C-based eggs.

Assuming you have DB2 Express-C installed already, the ibm_db Python egg for DB2 can easily be installed by following these simple steps:

$ sudo -s
$ export IBM_DB_LIB=/Users/<username>/sqllib/lib64
$ export IBM_DB_DIR=/Users/<username>/sqllib
$ export ARCHFLAGS="-arch x86_64"
$ easy_install ibm_db

This will install the ibm_db C driver, and the ibm_db_dbi Python module that complies to the DB-API 2.0 specification.

You can verify that the installation was successful my running the following:

$ python
>>> import ibm_db
>>>

Now, for the Django adapter, install Django first (if you haven’t done so already):

$ sudo easy_install django

The Django adapter can then be installed as follows:

$ sudo easy_install ibm_db_django

Finally, if have installed SQLAlchemy and wish to install the DB2 adapter for it, run:

$ sudo easy_install ibm_db_sa

Please let me know if you encounter any issues, I’d be glad to help you.

No related posts.

BlogBabel, the aforementioned site/web app, is a blog indexing and aggregation service that began in 2006. Amongst its features are the ability to detect and show the most popular blog discussions, weekly posts, books, videos, and even popular blog entries based on their location (through geotagging). It also features leaderboards of the most popular blogs.

Its codebase uses Python and Django, and consists of 27,359 physical lines of code (roughly equivalent to 6.46 person-years, according to sloccount). The R&D alone makes this application worthwhile to an interested party.

At this stage, BlogBabel has an Italian interface (located at it.blogbabel.com) and aggregates almost 15,000 Italian blogs and 5 million posts. Changing the interface to make it an international project that’s available in several languages, or switching to English (solely), would not be challenging in the least (they used to run a Spanish version as well, for example, but decided to discontinue it so as to focus on the Italian one).

BlogBabel has been featured in the mainstream Italian media and has had a noticeable influence on the Italian blogosphere. One could argue that it has been the yellow pages of the Italian blogosphere. Because of this, Ludovico Magnocavallo (the site’s creator) received substantial offers to buy BlogBabel in the past, but he turned them down because he wanted to continue building this site. Now however, due to personal circumstances and lack of time/resources, he’s willing to sell this application for what may amount to far less than its true value. And here’s the real bargain, the starting price, without a reserve, is 4,999 Euros. This is of course, a ridiculously low price for the value being offered. But Ludovico believes in letting the market decide.

If I had the funds lying around, I would buy it myself and gear it towards the English speaking world (in conjunction with the pre-existing Italian version). It’s a prepackaged, virtually ready-made startup with a great deal of potential both in its current state and in terms of what it could grow to become.

To recap, the auction includes:

• The domain name blogbabel.com (it.blogbabel.com has a pagerank of 6);
• The full codebase (almost 30,000 lines of code);
• A database containing 3 years worth of data relating to the Italian blogosphere (more than 30 GB, lots of data-mining opportunities);
• 4 hours of work to help you with setting up the site on your own servers.

BlogBabel has been running smoothly for three years, and is currently under-marketed. Optimizing ads, affiliates, and similar sources of revenue wouldn’t be hard at all, especially if one were to aim this site at the English speaking world.

Also, Ludovico has already implemented most of the code that’s necessary to allow users to have accounts (through OpenID), but since these “social features” are not fully implemented yet, they have not been deployed in production. A buyer could decide to disregard them or finish implementing them and roll out a technorati-like service. The winner of this auction could decide to implement support for Twitter, comments on social networks, sentiment analysis, etc, on their own. The possibilities are really limitless when you start with a solid engine and crawler, and already have a great deal of data at your fingertips.

I know Ludovico and he’s a stand-up guy. If you are interested in this great deal, you can bid here. If you have technical questions about this auction, please feel free to contact him directly through eBay.

UPDATE (September 8, 2009): Ludovico received an undisclosed offer for the site and a few years of maintenance work, so the auction for the site alone was suspended.

No related posts.

This article is obsolete. Please refer to the following articles for up do date instructions: Ruby/Rails and DB2 | Python/Django and DB2. Thank you!

I’m glad to announce that the API team has just released version 1.0.2 of the adapter for Django. And on my birthday to boot, what a nice present. This version extends its support to the recently released Django 1.1, as well as incorporating the feedback that was received earlier on. (For installation instructions, take a look at the README file.)

IBM confirms its commitment to support Python and Django, and gives Django well deserved credentials in environments where having IBM’s support counts. Django is becoming an increasingly mature web framework with the potential to do well within the Enterprise world. Having support for DB2 will surely help.

The next step will be working with the Django team to bake DB2 support directly into Django’s releases. The code for the adapter is released under a liberal OSI-compliant license that is compatible with Django’s own BSD, and the API team is more than willing to work on the development and support of the adapter should it become part of Django. We love Django and ponies. Let’s make this happen, guys.

No related posts.