A few months ago I blogged about the fact that IBM was looking for two interns to fill a longterm (paid) internship. The good news is that after an extensive selection process and a hearty dose of governmental bureaucracy (which is not unusual when relocating countries), two students have finally been able to join our DB2 team at the lab.
In case you are curious about who the students are, they’re Marius Butuc from Romania, and Henrique Zambon from Brazil. They’re good guys, feel free to follow them on Twitter (even though Henrique’s stream may be more interesting to you if you speak Portuguese).
The selection process
I received about 100 resumes from students around the globe who wanted to apply for the internship. Of those, I invited the 50 most promising applicants to have an initial phone interview, as I wanted to give a fair chance to everyone (yet at the same time, I didn’t want get the hopes up of those who clearly didn’t have a shot, and thus the remaining applicants were informed that they hadn’t made the initial cut).
Before their first interviews took place, I assigned the 50 preselected candidates a coding project. They could either develop a CSV to HTML converter, a search client using Twitter’s API, or an S3 uploader.
While none of these exercises were rocket science, they were intended to provide us with an general idea of the students’ ability to code. Most candidates went for the CSV to HTML converter. Some chose the Twitter client, and a tiny handful opted for the S3 uploader (one ambitious candidate delivered all three).
After carefully reviewing their code, I held the technical interviews with all of the candidates who had sent in their assignments. During the first interview I asked each person a series of very technical questions, and in some instances we also discussed their assignments. In most cases each interview lasted for an hour.
After the initial round of interviews had wrapped up, I selected the 10 most promising candidates for a second phone call and informed the others that they wouldn’t be continuing on with the interview process.
For the second round of interviews I was joined by my manager, and together we conducted phone calls with each candidate that were based less on the each person’s technical skills. This second interview was meant to test their soft skills, to get a feel for their interests, and to assess their overall attitude.
There were important interviews because they helped us get to know the candidates a bit better. Our aim was to learn about what excited these eager applicants, to see their level of passion for programming, and to observe other skills that were not strictly related to coding. We also discussed the project they’d be working on if they were selected, our team approach (we are more of an Agile startup than the stereotypical image you may have of IBM), and discussed details of the next steps with each of these ten candidates.
At this point Leon (my manager) and I discussed at length which two applicants we wanted to select. All ten students were top notch candidates, so the choice wasn’t an easy one for us. We ended up considering every single bit of information at our disposal about the students, from their their resumes and past experiences to their performance with the technical assignment, what (human and programming) languages they knew, and how they’d faired with both of the interviews.
After a lot of consideration, we ended up selecting Marius and Henrique. (In case you’re wondering, the student who did all three assignment was also from Romania and he was ultimately our third choice, had either Marius or Henrique been unable to accept the internship position.)
What I learned from reviewing so many candidates
- When you receive 100 resumes from very good students from around the world, you quickly come to realize that it’s surprisingly hard to determine someone’s real ability. Just about everyone looked pretty darn good on paper! That’s why I had to extrapolate what made each person unique via specific elements that weren’t based on coursework or grades (more on this later).
- The coding assignment was a huge asset in determining people’s real abilities. Some of the nicest assignments came from those with academically weaker performances. You could clearly see who the hackers and potential future computer science professors were. Without the assignment, the selection process would have been much harder, so I’m glad that it was something we required the applicants to do.
- With just a few exceptions, even good students seemed to struggle with relatively straightforward algorithmic questions, more than any other type of question. Sample questions included: What’s a Red-Black Tree and what would you use it for? Can you explain the P versus NP problem to me? (Note that these were rather open-ended questions, which I would generally follow up with more specific questions to assess the applicants’ theoretical backgrounds — as well as their way of thinking about these kinds of problems.)
- Most students didn’t know much about programming in the real world. In particular, they didn’t seem to be very up-to-date, most having never heard of things like SVN, GIT, MVC, ORM, Agile programming, or NoSQL.
- In general, the qualifications listed on the resumes we saw were greatly exaggerated by mediocre candidates, and somewhat downplayed by the great ones. (It looked like some form of Dunning–Kruger effect was at work.)
- Some candidates, taking advantage of the long distance telephone interview, tried to Google their answers and I could literally hear them typing as I presented them with questions that they weren’t familiar with. Replies such as, “Hmmm…well…”, followed by a 20 second pause and then the all but cut and paste Wikipedia definition. This was very easy to spot and didn’t end up boding well for those candidates who opted for this less than honest route.
- Recruiting is hard. It takes a lot of time and energy; rational decisions are challenging to make because in most cases you really are comparing apple to oranges.
What made a resume stick out to me
- Open Source contributions (a GitHub link would immediately arouse my interest in that candidate).
- A concise but detailed description of practical projects that had been implemented by the student for companies outside of their coursework (whether on the side or through internships).
- Social media involvement in the programming community. Did they have a StackOverflow profile? A Twitter account related to programming? Perhaps a a blog dedicated to the subject? While these points alone certainly don’t mean someone is a good programmer, they are a decent indicator that a person is at least passionate about the subject. Attendance of industry conferences (which some had) was also a strong indicator of interest in the field.
- Knowledge of unusual programming languages. Virtually everyone had Java on their resume, however that alone wasn’t good enough to catch my eye as an interviewer. (Heck, even Python is starting to be pretty common.) When candidates claimed knowledge of Haskell, Scala, Clojure, OCaml, Scheme, Factor, Io, J, etc… that definitely grabbed my attention. Again, this is no guarantee that someone is a good programmer, but chances are that they have an above average interest in our profession or have attended a university where these languages were taught (both positive signs).
While I’m not sure what you’ll get out of this article, I thought it was worth sharing as I found the whole process described above to be a fascinating one. Perhaps if you’re a student in a similar position you’ll find tips on what to do when applying for an internship with a company such as IBM. If you’ve ever found yourself in a similar interviewer’s position, you’ll likely find numerous relatable points in this post.
One final note: if you are curious, here’s Henrique and Marius at our first team party (after a few drinks):
Get more stuff like this
Subscribe to my mailing list to receive similar updates about programming.
Thank you for subscribing. Please check your email to confirm your subscription.
Something went wrong.
Sorry to hear “Some candidates, taking advantage of the long distance telephone interview, tried to Google their answers…”
Maybe you think I am one of them coz I am typing too (I don’t know some tech word in English so I am looking up it in an english-chinese dictionary)
Actually, it was clear that you didn’t Google for the answers. The “cheaters” were the ones reading the definition straight from Wikipedia. Follow up questions revealed that they didn’t really know anything about the subject. 🙂
One of the things I love doing when I’m interviewing someone who’s playing google is to google along with them, and try to find the website they’re reading from. It’s quite hilarious, actually.
It’s most entertaining if you start reading along with them, and then, when asked what you are doing, deny it.
I’ve had this happen to me too, quite funny. No one suddenly pulls out a perfect definition of a programming concept after silence for 60 seconds.
> I don’t know some tech word in English
Sorry to be blunt…then I don’t see what’s the point of your even bothered with trying to get this particular opportunity. (IBM, State-side)
I mean we’re not talking some Liberal Arts-type of position (Philosophy, Western Literature, anyone ?)….I can safely bet that at least 95% of the tech stuff and n languages *YOU* know/learned come from OUTSIDE of your country/culture. so (if I’m interviewer) I really have to question why you would not know ALL of the tech terms/concepts in this international language, i.e. English.
I’m Chinese and I say this with earnest proposition: You guys should really work on your English! (suggest you at least use English tech books/materials)
>> [did google tech term]
> [you have to know it beforehand]
I don’t believe you if you are saying, that chinese did not develop own terms/translations for some technical terms. That mistrust in your word is based on my (anecdotal) experience:
I’m German, we have own words for database-related, starting with “Datenbank” instead of “database”. Imagine my inexistent surprise as I went to France for an internship and found “BDD (base de données)” being the word for “database”. The word that cracked me up was “déclencheur” for “trigger” … a word we Germans just had assimilated into our language.
To wrap it up: Yes, Wenquing could have told that he had to look up a term to be clear on its meaning. But alsoi Yes, an iunternship is about learning new things, and if it is only a new word for a known concept.
so should you
That’s so rude of you to throw crap at Liberal Arts. Fine, being a techie is great. But that never gives you right to behave mean towards other arts or even science for that matter.
It has been seven months since the interview. The process must really have been tough as the two interns were announced just now. Congratulations to Henrique and Marius! 😀
Actually Jose, there are a couple of things to consider. 🙂 My informal announcement arrives a bit late, because Marius and Henrique started working in August. Governmental bureaucracy took a few months to allow them to come over. That said, the selection process itself took a while due to the huge number of interviews we had to do.
btw, the red-black trees question still haunts me :))
Umm… google it? 😉
Kudos to Marius and Henrique!
Nice article. I’d expect a CS course to cover the basics of Software Engineering relevant in the real world, like Agile practices for example, but I’m not sure I would hold the candidate against it. After all, its the only time they can program happily without worrying about estimates and schedules and iterations 😉
I didn’t hold it against candidates, but those who had experience with TDD and were familiar with what’s happening in the industry had an edge all things being equal. Both Marius and Henrique for example, were very knowledgable and passionate about these subjects.
Can’t tell you how many candidates for student assistants/interns we get that were force-fed Java through their undergraduate CS program and have no experience with other languages, source control, working on team projects, or even producing a completed program. I really have to wonder about the quality of modern CS programs.
At what point in the CS curriculum would have have them learn about source control or working on team projects? It seems to me that this would have to come after they’d learned how to produce the source code to be controlled and integrated with that of other developers.
If you figure two years of general liberal arts followed by two years of heavy emphasis on the major, it seems to me you’d be talking about people pretty near to graduation already.
Maybe two years of liberal arts is a terrible way to run a CS degree. Most countries don’t have that disadvantage, they actually start by focusing on the major. So two years of learning algorithms and basics, if you can’t produce source code after that: Fail.
Aside from that, at my university we started using source control in 2nd year, when we did a team project modifying an existing open source program.
Just an opinion, but I’ve never understood the sentiment that googling for information for interview questions is bad, or that an applicant should have memorized a random sampling of 25 out of 1,000 computer science algorithms and theories. Then go on and act surprised that the “top candidates” didn’t know much about real world programming like databases, source control, and server software.
The fact is, real 9 to 5 productive programmers live in a world where Google is constantly used to reference the growing jungle of APIs, and the fancy algorithms and math problems we learned in college only help give us background, they rarely ever directly effect our work.
Of course, I imagine IBM’s labs may be much more academic in nature, but smaller firms adopt the same type of interviewing paradigm. I just don’t agree.
Congratulations to the two Interns who made the cut. It’s an impressive feat to beat out 98 other motivated college applicants!
Thank you for the great overview of the intern hiring process at IBM.
Never memorize something you can look up
Google is important for programmers, and I couldn’t do my job without it. However, for these two positions, we wanted candidates who were familiar with many programming concepts, including some computer science theory. The interview was not mostly about algorithms and nobody was excluded for not knowing such things. Having a strong computer science foundation was considered a plus, and evaluated in the overall scope of the candidate’s profile.
I completely agree. I refuse to interview with companies that hold the pop-quiz style interview. At one lab I worked for we were specifically told not do that. Instead, you should look at their resume and ask them for their involvement in their projects and participation, and what tools/technologies they’ve learned and used.
To ask me “What’s a Red-Black Tree?” when Im ten years out of college and working on government database applications is pointless.
I’ve never understood this attitude. Just because you don’t know what a red-black tree is doesn’t entitle you to feign competence in an area in which you’re actually incompetent.
It’s perfectly fine for an interviewee to say, “Hmm, red-black tree, I remember the term from algorithms class but I don’t remember all the details. Something about ‘rotations’, I think? I could Google it real quick, if it’s going to be important for the next question.”
It’s *never* okay for an interviewee to say, “Oh yes, red-black tree, mmhm, I know all about that. Red-black tree is a type of self-balancing binary search tree, a data structure used in computing science. I have used it to implement associative arrays.” At that point, I don’t care if they actually studied tree algorithms in college or not. As soon as you lie to the interviewer or inflate your resume you deserve all the ridicule you get.
I couldn’t have agreed more, I got curious and looked up what a Red-Black Tree is and now that I’ve seen the wiki page I remember seeing it in school (and after reading the page for a minute I could probably code one for you). But there’s no way I would have remembered on the spot what it was. It’s been too long since I’ve been out of school…yet I can still apply most of that stuff after reading about it for a few minutes
While most would agree it would be quite ridiculous to ask a 10 year veteran of the industry if they could explain what a Red-Black Tree is, I don’t think it’s too much of a stretch to ask current students/recent college grads this question. The article is about hiring interns, not senior software engineers.
I think the important thing to take away here is, if you don’t think the interview question is fair, be honest. Don’t lie.
“The fact is, real 9 to 5 productive programmers live in a world where Google is constantly used to reference the growing jungle of APIs, and the fancy algorithms and math problems we learned in college ”
Exactly, I use Google every day to find information for developing code.
It’s definitely useful to Google for information, but it’s invaluable to know what to search for beforehand. You don’t need to know all of the details of a Red-Black tree, but it’s useful to be familiar with the topic so you know in which direction to look. If you are completely unfamiliar with self-balancing trees, you may never Google that far!
How much time did your group spend in total to select these 2 interns?
So basically, to fill two intern positions, you:
* Reviewed 100 resumes and selected 50
* Reviewed a few dozen coding samples
* Phone interviewed a few dozen applicants
* Selected 10 applicants
* Performed a second phone interview for 10 applicants, along with your manager
* Made the final two selections
That seems like an enormous among of effort overall. How much time would you say you spent in the entire process? How would you justify this expenditure? I can’t see any company I’ve worked at looking favorably at going to such lengths for an intern hire.
Also, it sounds like you didn’t actually meet the candidates in-person until they were actually hired, is that correct?
Correct.
It required time and effort, but we feel it was worth it given that these were 16 month internships.
I get them to do a mbt and iq on line , then the sample code. It takes less time than reading 100 cv only the ones who do something will I read the cv.. I even get the odd reply can I fail a iq test ? I always say it only help me to get a level of what to expect. Some of my lowest iq staff are the best coders.
And I found the best admins and staff from the mbt results.
As someone who has hired interns, programmers, and designers, I’d say that it’s very common to put the same effort into hiring interns as you would anyone else. Eventually, you hope that those interns will turn into super-productive, full-time employees.
It’s just smart business.
We did not look at it as “intern positions”. We were looking to get two top notch people for a really cool project where failure is not an option. What Antonio did was absolutely awesome and worth every minute he spent on it. I should know because I am the one who authorized it.
As Leon explained, these are high profile intern positions. But even if they were not, unless your company has money to throw away, it would be anti-economical to cut time/energy and money devoted to hire interns. Why? A short story from my experience: http://thinkingbox.wordpress.com/2010/09/26/dont-cut-timeenergymoney-invested-in-hiring/
Great article Antonio!
I would also like to mention that many of the students referred to Antonio came to know about these two positions because of the DB2 on Campus program.
More information about the students and the DB2 on Campus program can be found in these two links:
http://blog-db2oncampus.blogspot.com/2010/08/job-internship-opportunities-at-ibm.html
http://www-01.ibm.com/software/data/db2/express/students.html
Cheers,
Raul (Antonio’s colleague at IBM).
Antonio, your approach also works for experienced people as well, and yes, you will spend a lot of time finding good people. I note the significant lack of HR involvment in the selection process, which is generally a Good Thing. I have yet to master the art of the technical phone interview, but I won’t be doing any more real time Google searches. 8>) Thanks for sharing.
I’m currently in the process of searching for a co-op or internship in an unrelated field, engineering. This article gave me valuable insight on what some employers really see when they read your resume. Even though all the programming lingo was beyond me, my resume will most definitely be modified after reading this. Thanks!
wow, inters have a hard life… what do you have to do to get a permie role?
Since we wanted to hire the best students, we had to be very selective. However interns don’t have a hard life in our team. They come to work pretty much whenever they are ready, and leave when they feel like it. They have headset and listen to music as they code. They have high-speed internet access, and nobody will tell them anything if they are checking their personal emails, or visiting a site that is not programming related. When they are tired, they can go play ping-pong, or the company playstation. You get the picture.
What we care about is that they accomplish whatever is assigned to them. As long as they deliver, we don’t really care too much how.
Why just interns all good developers should work like this.. I find the bums on seat mentality so not appealing to keep good dev.. Realistic time lines and good staff.
“Some candidates, taking advantage of the long distance telephone interview, tried to Google their answers ”
I’m curious, why did you consider this a problem? I mean, presumably, when they run into something at work that they don’t know they’ll use Google to figure it out. Unless you don’t allow them internet access during work hours or something?
Now, maybe it would be better if the candidate said something like: “I’m not sure about that one, let me Google it…” well, maybe that would be more honest. If I were the interviewer I’d probably give the candidate points for this.
The issue was that some candidates were reading the answer, something that is both dishonest and not very respectful.
Example:
Q: What’s a binary tree?
A: Hmmm, ohhh, huuuu…
[Typing]
[20 seconds later]
A: “A binary tree is a tree data structure in which each node has at most two child nodes, usually distinguished as “left” and “right”. Nodes with children are parent nodes, and child nodes may contain references to their parents.”
Keep also in mind that some of these students were not really fluent in English. After typing, they would switch to encyclopedic English without a single mistake in it. Only to reverse back to broken English, if I asked further questions that couldn’t be read off Wikipedia.
It was ridiculous to be honest.
Of course, all IBMers have fast internet access (several megabytes per second), browse the web, and look up stuff all the time. They are not even monitored. But reading Wikipedia at an interview is exactly what I’d consider “cheating”.
This was an interesting blog post. It is good when interviewers have a clue, even better when teams have a strong word to say when picking up new members, but that’s not my point.
I never wanted to work in such a big company like IBM, I’m not even sure I qualify from a technical point of view (heck, I only know that a Red-Black tree is a tree with certain properties, which I don’t know, which I would google up, but certainly not during a live phone interview, I agree this is lack of respect and cheating).
But I’ve caught myself thinking that if I would be that guy who passed the interview and got hired, then I would discover that I have several megabytes per second of internet access at work and this is called “fast”, I would feel cheated as well.
Come on, it’s 2010 and it’s IBM, is it really THAT bad? I don’t think that somebody who had a goal to work at IBM and got there would risk his place by abusing internet at work. I just want to say that I’m that kind of geek who would never trade a very fast internet connection at work for a slower one, ping-pong and free cookies.
I’m not sure what you are saying, here. In Canada, having the equivalent of an actual 50-100Mbps connection (both up and down) every time you connect your laptop to the network is definitely fast. For comparison, at home the faster I can get is 4Mbps. Maybe you are arguing that a fast connection is a given for a company like IBM, and I wouldn’t disagree. I only mentioned it because someone asked if employees were allowed to have internet access.
Just out of curiosity but did you not have any American candidates make it to the top 10?
It seems like there would be innumerable candidates in the country qualified for these jobs and with record unemployment, particularly for 20 somethings it seems irresponsible to higher abroad.
We are actually based in Toronto, not in the States. However, we did have candidates from Canada who made the top 10 but were not in the top two.
We didn’t receive many US applications. Each of them though was given a fair chance to get the position.
Ahhh, I assumed with the IBM affiliation you were in the states so I guess that just goes back to the age old adage about assumptions.
Congratulations to your two interns. I am sure they will love IBM and Toronto.
Nice Palmeiras shirt! 😀
So true, We do a simple “make a google map/database app”. I’m amazed at how many people screw it up and seemingly dont take it seriously when its for a job? im mean how would they treat clients then?
It’s a little different when applying for internships. However, as an experienced engineer, if someone asks me to do FREE work for them, even under the pretext of being some sort of interview qualification, that’s a red flag and I’m running away as fast as I can. Even if the request comes from a large company that’s obviously not trying to extract free labor, it says the company doesn’t value my time now, and almost certainly won’t if/when I become an employee.
If “people” don’t take your demands seriously, you might re-examine your expectations through the eyes of your applicants, and see if those expectations are reasonable.
All of the above notwithstanding, I don’t know anybody who only applied for one job. If each application generated a two or three hour project in addition to the time already required to fine-tune the resume, (maybe) create a cover letter, then research the company, group, and hiring manager, the time required balloons beyond anything feasible.
How much does GPA play into your selection process? I’m heading to a career fair tomorrow with a 2.47 and I’m nervous. Twice at my university’s Spring career fair my resume was handed back because I didn’t have a 3.0.
How do I get recruiters to look past the low GPA and notice a strong background of technical and leadership experiences?
(IBM is one of the many companies that will be there. :-P)
I can’t speak for other teams at IBM. I can tell you however that for our team, the GPA score of a candidate wasn’t a determining factor. Best of luck for tomorrow.
As someone who does a lot of interviewing, I have great appreciation for the difficulty of selecting a set of questions that will really get a good read on the quality of a candidate. The process described here was impressive in its depth and balance — thanks, Antonio, for sharing about it.
While it’s easy to go overboard on theory questions — and I prefer to ground them in a real-world problem when possible — I don’t think it’s unreasonable to throw in a few, like they did here, especially if the candidate claims to have some sort of CS background.
I’ve been out of school a long time, too, and I was rusty on this sort of stuff the last time I was job hunting, so…. I hit the books for several weeks to make sure I was ready! Unless you’re just stunningly brilliant, you have to work at standing out in a pool of talented candidates. I’m always amazed when candidates show up for an important day of interviews completely unprepared!
Besides, while it’s always fun to complain that 90% of what you learn in school or from CS books is useless, it’s one of those things where it’s hard to say which 90% is going to be a waste. You never know when a vague memory of algorithm X may give you an aha! moment solving a practical problem. So, for many kinds software work, a candidate with a decent, verifiable background in CS really does have some real-world edge.
I agree.
I have been out of school for well beyond ten years. I have studied in-depth before all interviews.
I study more and more as the years go by – because I know more and more topics and areas as the years go by. Sometimes I re-read material “just because” and it’s amazing what you can learn after deep real-world experience gives you something to tie certain pieces of material to in your mind. Many aha! moments – for sure.
You must do everything reasonably possible that would positively separate yourself from other candidates. If you’re not doing that – then you’re likely not going to be offered the job.
I have been offered every single job that I’ve ever interviewed for. I’m absolutely certain that studying has been a factor in my interview success.
This is what has been freaking me out about the CS field recently. I’ve been on a couple of these ‘technical’ interviews with coding assignments. This is for an internship! An internship! Why is it so damn competitive.
First, it’s a global selection process, second its 100 resumes narrowed down to 50, third, they only take 2 candidates! Even the poor kid that did all three questions didn’t get selected. I hate this type of competition… it’s too much!
You don’t know what a red black tree is (I leaned that in AP CS in high school
Your AP CS class was either extremely superficial or taken by geniuses. Btw, I don’t remember seeing red-black trees in CS GRE examination which should be [quite] a bit higher level than AP CS.
In short, I think you are wrong: not knowing what red-black tree is does not mean much: the data structure is pretty specialized and not that commonly used.
I agree with your point regarding svn/git…
Considering that the internship was for working on a database engine (DB2), the question about Red-Black Tree was appropriate. For students/new grads.
A couple of years back, when I was working in Developer Relations at Apple, I was one of the people who reviewed applications for student attendees to our annual Developer Conference.
I can certainly relate to your description of the enormous number of applicants and the difficulty of choosing among them. Just about every one I saw really deserved to be there.
-jcr
Good article.
Have a look at
http://stevehanov.ca/blog/resume_comic.png
It points in the same direction.
Good article…and I understand about weeding out those reading definitions off of google searches. That is just silly on the canidates part.
But for the rest of the “google guy” issue. IMHO: I’d rather have a guy who can search up a code snippit in 1 minute than a guy who spends a week looking for it in a book or worse, a week looking at his screen.
The “google guy” will get more done.
i am also involved in hiring new staff and interns. it is very tough, because you would like to be fair, and also the risk of losing out on a potential superstar. fortunately i dont have to go through hundreds of applications. for applicants, we have 3 simple rules:
1. must be in pairs from the same university (ideally, we would like to hire in pairs as peer support is something very important for someone who have just started work).
2. cgpa of at least 3.0 (this is for malaysia where the education system is not of the same standard as other countries). most importantly it is an indicator that the person is disciplined enough to complete his assignments and has good time management skills. the hardest employees are those who have poor time management and no respect for datelines).
3. must be an active member of a developer community (this is a must for getting a good developer as it reflects social skills, identifying problems and having a try at solving them. with the internet and open source projects, there really is no excuse for a budding developer not to get involved. even postings in this site would count).
for potential staff, there is an added rule.
4. during the interview, the applicant must bring along a software project that he has done (or even involved in) and talk about his project. it is very important to be able to talk and understand attitudes.
Does IBM regard these interns as the future “stars of the company” or essentially as being on the same ability level as their full time employees?
We expect them to be top students, not top professionals. That’s why we didn’t expect anyone to be able to answer all the technical questions. Some of them were meant to be tougher to see how far a candidate’s knowledge stretched. But again, nobody was penalized or made feel incompetent for not knowing some of the answers. Being students though, we didn’t have a real world programming track record to check out (for the most part) so theory had to be part of the interview. The whole process, despite being lengthy and selective, was very easy going compared to most other companies this large.
For the phone interview, you should make it clear that the applicant should not Google answers and your questions should reflect this.
You are asking a student that is used to giving answers the best they can from resource materials.
There is a big difference asking :
“What is a red-black tree”? and “Do you know what a red-black tree is”?
The first question invites Googling (unless specifically forbade), the second does not.
On a phone interview I never ask a question that can be Googled, there is no point.
I disagree with almost everything you said – because it’s absolutely fabricated.
Everyone knows that an interview is a test – and there’s no such thing as “inviting cheating on a test by asking questions that are capable of being researched”.
Unless a test is specifically referred to as “open book” – then reviewing material is understood to be forbidded.
I agree that it’s important to ask questions that show understanding (eg. “cannot be Googled”)
– However, I am seeing much value in asking questions that can be Googled – in order to see which candidates are willing to cheat.
I take it that the opportunity you provide is way better than the jobs that are typically available?
I only ask because you talked about having lots of excellent candidates from whom to choose; and there are so many blogs from people complaining about their difficulty finding experienced candidates who are even reasonably competent. (Maybe the solution for them is not to require prior expertise in all twelve of the tools, languages and APIs that their team is using.)
I’m surprised that nobody picked up on this fact: 2 more programming positions went to non-US citizens.
There was a time when IBM searched colleges in the USA for candidates for interships back in the early 1980’s.
It’s a sad state of affaires now, and I won’t be buyng IBM for a long, long time now!
We are based in Canada, not in the US. We made the internship opportunity globally available, including of course US and Canadian students. We have a DB2 on Campus program that actively searched local and foreign students. Heck, I even blogged about it in my blog which is mostly read by Americans. However we selected the best candidates, regardless of their country of origin (if a position is open worldwide, discriminating would be highly unethical).
Re above: Why would a company based in Canada be obliged to hire US citizens?
One reason would be that I’m an American…
I really wish I could figure out how to get 100 even slightly decent resumes for paid internships. In my area I can hardly find 5 or 6!
What area?
A lot of IT companies in the US still take for granted the ability to attract and hire the best talent from the world engineering pool. I wonder if this can be sustained. After the economic turmoil and persistent high unemployment in the US, the visas have become harder to get and the salaries have not kept up with the pace of inflation.
Congratulations Henrique and Marius, celebrate for now but don’t lose sight of coming back home or to other countries. Opportunities outside the US are abundant and the american dream is sometimes just that, a dream inherited from previous generations that lived beyond their means.
Hi,
I found the article very enjoyable and very interesting.
I have to admit I had to google Red-Black Tree to move beyond the assumption it was a binary design process. (Yeah, I be that ancient!)
Having done tech interviews (Mainframe primarily), I can say without a doubt that every one is challenging and fun! From the interviewee that was writing down all the questions and answers to a 20 minutee discussion on creating a temp storage queue (and was still wrong).
IBM’ers are not the white shirt gray suit automatons you might envision. Worked on two IBM projects years ago where the only suit in the place was our manager’s. The IBM liason had an intense love of hawaiian shirts and we saw them all!
I envy those two and what they will learn and do and see during their time at IBM.
A very very good article about interviews and processes and people.
Thanks!
I’m curious…What’s the average age of programmers on you team. How old is the oldest. I’m seeing more and more counselors not recommending engineering because the careers are shorter than those of professional hockey players.
Regards.
Keep in mind I’m not a professional programmer…
It’s interesting to see what made a candidate stand out for you. On the surface, the desire for “extracurriculars” and more theoretical knowledge seems strange, but it only makes sense in context.
1) Extracurriculars like open-source development and social media involvement really help indicate passion for a subject. At the end of the day, you want someone who’s really self-motivated as opposed to someone who just wants his paycheck.
As someone who primarily uses programming as a means to an end, I can definitely say I would not code just for fun. I have to have a goal: solve a PDE numerically for example. On the other hand, my roommate (who is a CS major and definitely loves programming) will just code for coding’s sake. He has his own shuffle program for music that he made just for the hell of it.
2) I’m not sure on this, but more theoretical questions may actually be directly applicable for a company like IBM. There may be more of a research emphasis there than at a company that primarily does software development.
Consider those who program at work and again in their free time because of their passion for programming. I wonder whether such employees are going to write their own versions of tools that are already freely available instead of, say, writing documentation or clarifying requirements with the user — because programming is more fun.
I often _study_ programming during my spare time (because I cannot do that at work but need to keep up and improve) — but I don’t _do_ yet more programming after work.
Excellent article! Thank you for the presenting it. You’ve done a great job making me pause to sort through a lot of my own thoughts and opinions about the interview process.
I’m about to start interviewing for my replacement. I’m a one-man operation stuck in the classic “guru” position for a small company. There is no “team”. I report to the VP of Operations. The company has always been satisfied with handing everything to me to “just get it done” so I’m a jack of all trades but master of none – network and server administration, Exchange and database admin, PCI/VISA/MC/AMEX certification, programmer for VB (all versions), TSQL, command line scripts, and a full-blown automation engineer. My “robots” do the work of 5 FTEs.
I wouldn’t survive any interview process with a large company since deep specialization in a particular field is always required. Red-Black tree? I have a lot of them in my production code, but didn’t know the name of it until I Googled it after reading your article. Facebook and Twitter? Sorry, I’m working to keep the company cash flowing instead of talking about it. Attend a conference? Let alone present? That’s taking time away from my family. I’m passionate about my work, but I can’t be public with it.
I know you’re nodding and saying “you shouldn’t have let the company do that to you.” You’re right. I ended up in a great big pile of should. That’s part of the motivation to move on and hire someone to take over for me. The greater reality was the paycheck – not for greed’s sake but for covering the mortgage and feeding the kids. I’m good at what I do (accounting for Dunning-Kruger) but I will never fit a corporate profile of any IT position after being 15 years in small companies.
I admire IBM. Great hardware and software products. Leading innovation. I fondly remember managing AS/400s, writing code on them, and maintaining the databases. They NEVER went down and I miss that in Wintel these days. But, IBM would never look twice at me for a corporate position. I’d be in the bottom 50 for “we need a round peg in this hole and this guy is a dodecahedron.” My options in the job search are limited to other small companies that need gurus. On the other side of the same coin, the person I interview has to have a lot more breadth than depth. The interview I have will be face-to-face, listening to pitch and tone and watching eye movement, and a lot of reading between the lines over war stories like “how would you handle a power outage in the server room?”, “the CEO needs a report to join 5 tables of data by 3PM, or we lose the customer” and “the IPS device just stopped an attack from a Chechnyian IP address, do we need to permanently ban it at the firewall?” Not exactly “can you explain the 7 layers of the OSI?”
And we have to hire by the end of October.
If someone has advice on how to transition to one of those cushy “take a break to play ping-pong” jobs, please let me know!
You couldn’t find any suitable candidates to hire in Canada? So, do you think the issue is that they just didn’t apply?
We had a handful of Canadian candidates, but rather few when compared to those from other countries. I think that the main issue was that the position was for a 16 month internship. A Canadian student may prefer to graduate first, and then find a job, rather than take on such a lengthy internship. In some instances foreign students may have fewer local opportunities, so an internship with such a well known company here in North America may seem more appealing (than it would to a Canadian student). Honestly though, that’s just my guess.
Hey Antonio,
Great article 🙂 You seem to be doing well for yourself, congrats.
My own experience, which isn’t dissimilar to yours, and some of the details you’ve mentioned in your hiring process, and some of the comments here, make me think that we really should have proper and popular Computer Engineering programs. Computer Science should be more academic.
It’s clear that most candidates and most companies … heck, most anybody in the industry, want/need people with more practical experience. I’m routinely stupefied at the number of people who aren’t even aware of revision control. These folk are clearly in it for the degree and the salary that follows, not for the passion of it. With a proper Computer Engineering program, they’d be FAR more useful and happy out of the gate.
Computer Science could then focus on the more hard-core stuff, as I feel it should. (Seems that most of the candidates produced today are neither good at the practical since they lack practice, and don’t care about the hard-core stuff.)
I wonder, has anybody ever interviewed anybody from a college-level program? A 2- or 3-year type thing that focused on the practical? Results? (On the Ops side of thing those candidates are usually worse than useless 🙂
Re: all of this discussion about knowing the terms, googling, and being able to answer the questions posed in the interview: I think that some people are missing the context.
Reynold: Antonio was interviewing students who should have been studying things like algorithms and programming concepts, language semantics, etc. People who are working as a professional developer know what they need to know to get their job done and done well. Is it reasonable to expect an academic to know the academic details? I think so. Is it reasonable to expect, say, a newspaper editor to have great grammar? I think so. Is it reasonable for a professional cook to know how to grow vegetables? It’d be great, but I don’t think that the cook gives a rat’s ass about the mixture of grains that have to be force fed different geese to obtain the best fois gras. Nor do I expect the farmer who knows that to be familiar with 12 different ways of preparing the fois gras. I do expect the farmer, and any health inspectors, to be aware of health regulations but I don’t think that the cook or the general public to care – even if they can google it.
Similarly, I don’t expect, say, someone who specializes in buffer cache management to be familiar with the Java Swing API’s or a fast line-drawing algorithm. If the developer recently took a graphics course then I guess it’s reasonable for them to know the line drawing algorithm.
Maybe that’s something that you should consider in the future, Reynold – the context to see if it is reasonable to expect someone to know that information (or to possess some particular skill).
In the future, maybe I’ll consult you if I need to write a grant proposal (i.e. beg for money). If I need some C coding done, I’ll ask someone else.
Thanks for the message, Bill. When I wrote the comment, I didn’t think that it would come down this way. After reading it again, I do realize how bad it was. I’ve asked Antonio to take down the comment.
I’d like to apologize (to everyone) for the remark I made.
I also appreciate you putting it down in the nicest way possible by saying “If I need some C coding done, I’ll ask someone else.”. I can learn a lot from you.