Meditations on programming, startups, and technology
New Relic

On the Usability of Codecademy

I asked a friend of mine (who isn’t a coder) if she could try out the amazing Codecademy for me.

She is a smart person, but again, she isn’t a programmer. I thought Codecademy would be a great way to introduce her to programming for the web. As a programmer, I thought it was extremely well made and assumed it would work for anyone.

She landed on the homepage and read the instructions.

She wrote her name (“Elizabeth”), then decided however to change it to the shortened form of her name, “Liz”. To do so she highlighted the whole string she had just typed and tried to press the backspace key to delete it. The whole page disappeared and the browser redirected her to the new tab.

Liz: “Why did it do that?”

Me: “You can’t highlight a word like you’d do in Microsoft Word. You’re in a browser, so the backspace button tells the browser to go back to the previous page you visited in that particular tab. You need to delete the string one letter at the time.”

She reopened the page and tried again, this time things were successful.

Next, she had to measure the length of her name. But she forgot the quotes, so she got this:

Liz: “What happened?”

Me: “What do you think happened?”

Liz: “Some kind of error, but what does ‘Liz is not defined’ mean?”

Me: “You accidentally forgot the quotes.”

Liz: “Oh, OK, let me try again.”

She wrote the first quote but pressed enter before immediately realizing she had forgotten the closing one. As such she encountered the following scenario:

Liz: “I messed it, let me try again.”

Liz: “I typed it right this time, why didn’t it work?”

Me: “Since you didn’t close the first string, it thinks that the second line is still part of the first command. Do you know how to get out of this?”

Liz: “No, but let me try.”

No, she didn’t close the string and pressed enter twice or deleted a bunch of characters (this web REPL allows it, kudos to Codecademy). She simply reloaded the page. Thankfully the same exercise (#1) was reloaded. After completing it successfully though, instead of exercise #2, exercise #0 reappeared, asking her to enter her first name again.

After raising an eyebrow at that point, she went along with it. Next she was asked to try some math.

She can’t be faulted for thinking that “for instance” meant she could try something else. Yet, she got an “Oops, try again.” message.

At that point I suspended this tiny hallway test and I asked her what she thought of this system. She found the site to be a bit unintuitive, but above all felt like she didn’t know how to get unstuck when she made an error.

Now, before you all jump in and tell me that my friend is dumb, that she’ll never be a programmer, or that you can’t idiot-proof a design, keep in mind the very point of this post.

Usability is hard because real users use our UIs and software in different and unexpected ways. Codecamedy did a very good job, but even they left plenty of room for improvement. As web developers it’s our duty to give usability and user experience a significant amount of attention. The UI is the software to the end user.


If you enjoyed this post, then make sure you subscribe to my Newsletter and/or Feed.

receive my posts by email

51 Responses to “On the Usability of Codecademy”

  1. Mike says:

    I agree. I have been doing the codecademy lessons for a couple of weeks and they are good, but when you get stuck, it is definitely hard to figure out what is wrong. I am currently on the Fizz Buzz lesson and am having problems.
    Also, I learned to program in high school on QBasic, so I have some idea on how stuff works, but it is has been awhile and I get frustrated fast.
    Is that normal for a new programmer, just sit and stare and try to figure out what went wrong?

  2. Jeffrey says:

    This isn’t just a usability problem; it’s a reflection that people who learn how to code should all be autodidacts and a manifestation of the fantasy that human instructors can and should be completely replaced by automata. But we’ve been trying to do this for forty years now only to find that many smart, motivated students still need the intervention of an instructor (or mentor, coach, etc.) who can answer questions, curate curriculum, and provide structure to learning.

    • Good points. Maybe we shouldn’t all jump on the bandwagon that says college is a waste of money (not saying we should draw too many parallels with Codecademy, of course).

    • Mike McGee says:

      “But we’ve been trying to do this for forty years now only to find that many smart, motivated students still need the intervention of an instructor (or mentor, coach, etc.) who can answer questions, curate curriculum, and provide structure to learning.”

      Couldn’t agree more with this quote, which is exactly why my co-founder and I created Code Academy (http://codeacademy.org), a physical program that teaches beginners how to build web applications.

      We spent about 10 months teaching ourselves HTML, CSS, and Ruby on Rails going through just about every online resource there was, and we still felt that we needed more skills to build the web applications we wanted. Also, the process of learning online (and by yourself) was not fun.

      We knew there had to be a better way to do it, so when we couldn’t find a physical program out there that taught non-programmers how to build web apps, we decided to create one.

      We wanted to create the best environment for a beginner to learn how to code, so we found a great instructor (who is also a professional software developer) an awesome space (where the students could stay 24/7 to learn) and provided awesome mentorship from professional software developers in the field.

      So far our program been a great success! 35 students went through our first program, and now we have 57 students coming to Chicago from all over the world who want to develop and design their ideas into real web applications.

  3. karatedog says:

    This is a problem of interpretation or not knowing the underlying rules.

    For example you write first “tried to press the delete key to delete it” and later you write “You’re in a browser, so the backspace button…”
    Which means you are using a Mac, don’t you?

  4. Ryan S says:

    It is indeed a challenge when creating something brand new. You as the creator have an idea of how things should work and thus you start building. Of course after you have something up and running YOU MUST ask friends like Liz to test. Ask many Liz’s and reiterate and reiterate and reiterate until every Liz gets it! Maybe even use Mechanical Turk to get an even larger tester sample.

    Overall this is what we’ve been doing with http://CodePupil.com since August.

    Of course it will be compared to Codecademy, but it’s complimentary and different.

    • Nick says:

      Nice way to shill your product using a humblebrag within a blog entry about a competitor.

    • Mark says:

      I tried to check out CodePupil, but it insisted on an email address, so I didn’t even get past the home page. :(

      So, you asked for comparison (in a somewhat backhanded way) and one nice thing about codeacademy is that you can try it out without registering in any way.

  5. Jay says:

    You bring up, but gloss over, a very important point. Many people don’t really understand how to use a browser. If someone is going to be using that as the primary vehicle for their UI, they should give an orientation pre-class. Similar to the way a console game walks you through what all the buttons on a controller does.

    • Joey says:

      Well, she still can’t really be blamed. Any text field in a browser allows you to select and delete text. Conceptually what CodeAcademy presents there is akin to a text field (it has the same affordance), yet it doesn’t allow selection because it’s a completely different technology underneath. But you can’t expect a non-expert to know or even realise that. Heck, even I frequently look at the source code of pages just to know how something is implemented.

  6. Thanks for the post. We’re taking all of this and looking at ways to simplify Codecademy so users like Liz will understand. Thanks for bearing with us.

  7. blick black says:

    Excellent post. This is exactly what is wrong with 90% of teaching courses out there. It is easy to focus on what is correct. Unfortunately its the mistakes that really allow people to learn. If you aren’t told the why and what of your mistake, good luck in learning anything fast. Sure if you have all day you can figure it out just like an old school sierra adventure game. This is the biggest difference between good and excellent teachers. Excellent teachers teach by holding the students hand while they are making mistakes and guiding them along to the correct answer. The answer is only part of the process, really the easiest. It’s the journey to the answer that is really the most important part.

    Kudos to you sir.

  8. Sigh says:

    Open Source Daycare strikes again.

    I, the programmer, know how to use this, and therefore everyone else will think like I do.

    Should I take 10 minutes to run untrained people through this and see what kinds of reasonable problems they’d encounter?

    Documentation? Usability? Nah. Ship it.

  9. I had a similar experience teaching 3rd-6th graders how to create e-books. HTML, well formed XML, CSS, fibonacci series, and hex counting were all relatively easy concepts for them. What got them stuck was opening, closing, editing files and moving them to/from a usb drive. It didn’t help that Windows Explorer hides filename extensions by default.

  10. Doug says:

    Interesting post. What’s interesting is that your friend needed a human (you), to successfully use the teaching tool. I’m guessing that removing the online tool altogether would have saved you, and her, time and pain. :)

  11. Marc says:

    I have had the same experience learning with Codecademy. It needs to teach through explaining your errors, but I can imagine it’s tough to scenario plan for every stupid mistake a user can make. I usually just have more questions after completing lessons, and just work out how to complete the lesson, rather than understanding (and remembering/learning) why/how you solve the problem.

  12. Stephanie says:

    Since you can’t anticipate all problems people will run into, maybe a good user forum can help solve this problem. Let more advanced users teach newer users and create a sense of community around it. StackOverflow is nice, but it’s not particularly friendly to beginners. A Codecademy forum can be like a beginner’s SO. Piazza (www.piazza.com) is also an interesting model to look at.

  13. Ronny Karam says:

    An excellent description of a very common UX mistake where designers tend to neglect certain aspects considered as intuitive; a primitive requirement for any computer user.
    It’s actually not made on purpose. When teaching someone how to drive, you’ll never start by showing him how to open the door or how to sit. It doesn’t even cross your mind; to you, those things are natural and obvious.
    And that’s how a designer/programmer feels about the usability of his application.

    Good article. It’s phase 1 in a series of usability tests that should be conducted

  14. David says:

    I actually had the exact same problems as your friend Liz and I am a programmer!

    I was pulling my hair out with the “Fizz Buzz lesson” too. My code was printing out exactly what they wanted, but I think I did it in a different way they expected. I have been using Codecademy’s feedback mechanisms too, but I have no idea if they are reading my feedback.

    I also signed up for “Code Year” after I signed up for Codecademy and I find it quite confusing as to which lessons are associated with code year and which lessons are just Codecademy lessons.

  15. I’m also getting a bit frustrated with Code Academy. As mentioned in this post, they often have ambiguous directions. The FizzBuzz project implied that all the skills and information necessary to complete it had already been covered in the introductory course, but nesting was never mentioned. I don’t mind looking things up and learning from a variety of sources, but I found this annoying. If FizzBuzz had been upfront about it, I wouldn’t have wasted so much time reading and rereading old lessons, wondering what I had missed.

    • Eric says:

      Connie: you shouldn’t have to use a nested structure to program FizzBuzz. Nesting usually refers to putting a structure within the same kind of structure, like a loop within a loop, or a conditional within a conditional. I’m not sure how they are approaching FizzBuzz, but you shouldn’t have to use nesting.

  16. Dan Davidson says:

    I have been doing the Codecademy courses just to brush up on my JavaScript (learnt it mainly through hacking together jQuery scripts, so needed to do it). I already know PHP and some C/Objective-C , so I am very familiar with functions, arrays, conditional statements etc and would call myself a developer.

    I am very interested in the approach they have taken and overall I think it is very well executed. However I have found myself confused at points as to the exact expected output, despite knowing how to achieve the result. For example, today I was doing part 2 of the weekly course and one of the exercises wanted 3 numbers to its power being true (at least from memory this is what they wanted). I made the function, and ran the test 3 times with true being shown. This did not complete the exercise however as it turned out I needed to log it to the console 3 times in the same run. It just wasn’t very clear to me that this was required and when I looked at the q/a page many others had come across the same issue where the exercise could not be completed despite using the correct logic.

    I can imagine this being very off putting for a complete newbie, so let’s hope they can work on making the feedback more instructive.

  17. Dave Herman says:

    Nice post! The folks working on the Racket programming language (http://racket-lang.org) have been doing great research for years on designing pedagogical IDE’s in conjunction with Computer Science curriculum. Their work has really focused a lot of attention on what happens when things go wrong, and empowering students to help themselves get unstuck. It’s a very difficult subject!

    An interesting and relevant recent paper to come out of the group is “Measuring the Effectiveness of Error Messages Designed for Novice Programmers” by Marceau et al:

    http://gmarceau.qc.ca/papers/Marceau-2010-Measuring-Effectiveness.pdf

    Best,
    Dave Herman
    Mozilla Research
    dherman@mozilla.com

    • Amjad Masad says:

      Thanks for the link. I’ve always loved DrScheme’s way of handling errors and was a great help back when I was studying SICP.

      As you may already know we use the browser’s native JS engine which doesn’t give us much control over the runtime environment. Static code analysis may work in simple cases, also logically inferring what’s missing from the type of the error may also work (specially in the case mentioned in the article). That being said we are not planning on being the only authors of content on our site and we’ve already had some community contributed courses through our “Teachers Beta”. And planning on opening up our platform for the public to contribute courses, so it would be great if we had a general better error reporting, debugging, and environment inspection tools.

      We are looking at using Mozilla’s Narcissus interpreter which still needs some work to make it run in all modern standards-compliant browsers and make sure the language semantics are valid. We are also doing some work to make debugging work. Using ES6 generators would be great for pausing execution, specially when coupled with a nice API like TaskJS. I hope they land soon! For now we are converting the interpreter into CPS inorder to get rid of the execution state in the call stack to be able to pause execution at any given point. Right now we have an initial prototype up and running, but my greatest fear would be changing the language semantics and thats why we can’t ship this feature sooner than later. We are doing extensive testing which yielded some errors in our implementation and the original one too, we are and will continue to contribute to the main repo when it makes sense.

      Thanks,
      Amjad Masad
      Codecademy
      amjad@codecademy.com

  18. matthias says:

    Dave has already pointed you to the Racket (formerly PLT Scheme) experience. Our entire project started from the observation that pedagogic IDEs must be designed to be so because existing IDEs don’t deliver. Better still, no existing language is just right. A language must be ‘tamed’ and then ramped up carefully to avoid the ‘hitting the wall’ problem that so many novices encounter after the first few encouraging lessons. See
    http://www.ccs.neu.edu/racket/pubs/#cse2003-fffk
    and
    http://www.ccs.neu.edu/racket/pubs/#jfp01-fcffksf
    for details, plus Guillaume’s study for a measure of effectiveness.

  19. grant hutchison says:

    I teach high school computer science (App Inventor and Python currently) and I gave CodeAcademy a try over the weekend. Overall I thought it was well done. I was actually doing the first few lessons with my seven year old daughter at the keyboard and she found programming easy.

    After a few lessons and the word ‘literal’ was encountered I had to continue without my daughter. I completed the FizzBuzz scenario, but also found it a bit confusing as others have noted.

    It is difficult to determine the proper language to use and examples with purpose (and ease).

    There are some other tools that I have used recently including: singpath (singpath.com) and University of Waterloo CS Circles (http://cemclinux1.math.uwaterloo.ca/~cscircles/wordpress/).

    Antonio – thanks for sharing the experience.

    Codeacademy – Kudos for the site… well done.

  20. [...] building tools for use by developers, especially novices, you need to ask yourself this question again and again, or they’re going to get mighty confused by the error messages. Via Hacker News. Share [...]

  21. Liz Hannaford says:

    Thanks for sharing your experience. It made me smile because my experience is the same as Liz’s! I have NO programming experience so it is frustrating when things don’t work. You explain some of the stuff which is obvious to a programmer (like not being able to highlight and delete!!) but bizarre and annoying to us newbies. I’m getting better at working through my mistakes and, in the long wrong, that’s probably a good way to learn. I’m also blogging about my efforts to learn coding http://hannaford.wordpress.com/

  22. I was indeed thinking CodeAcademy is purely awesome and dead simple. But thats only from a standpoint of somebody who is into HTML, CSS since many, many years, so i might not be the best example. Its interesting to see how the experience for complete beginners is and i think you’ve made a valid point her that could help the CodeAcademy team. Send them an Email to this article. I am sure they’d love the feedback.

  23. Bevan says:

    Interesting – as a reasonably seasoned programmer, it is difficult to evaluate something like this because you just don’t hit the kind of snags that someone who has never written any code would do.

  24. Becca says:

    Thank you for posting this! I had a very similar experience with Code Academy, where I became frustrated by the ambiguity or lack of clarity in some of the examples, as well as sections where I felt like I had to draw upon outside knowledge to complete an exercise instead of drawing upon knowledge gleaned from the previous exercises.

    Although I fall somewhere between a newbie and a programmer (I have only been programming for several months, but possess a basic working knowledge of Python), I got to a point with Code Academy where I felt like I was typing commands without a true understanding of what I was doing. Perhaps this is where the disconnect between an online learning system and a real teacher is the greatest!

  25. Sichun Xu says:

    Great post! I think the root cause of the usability issue is the programming language itself. For code academy, programming language is no longer hiding behind the UI. It is part of the UI. Most programming languages are too rigid and not tolerant to errors, which make them not friendly to beginners. We need a programming language that is fault tolerant, like HTML, and general purpose. I am working on such a language right now. I have put up some thoughts on my blog http://starscript.wordpress.com/. You are welcome to check it out and leave feedbacks. Thanks!

  26. steven says:

    thanks for the post. I am a newbie and i signed up for codeyear because it said i would learn how to code, not learn how to code Javascript based on the assumption that i new other languages. Maybe that was their intention and it got out of hand when they started getting a lot of press and people like michael bloomberg signed up. I got through the first lesson okay and then the second lesson kicks off by telling you that it assumes you have coded in other languages and the knowledge from the first lesson is not enough. What? The hints suck – if you don’t know what to do the hints really don’t help and then you are at a dead end unless you have a friend like you looking over your shoulder. I see above the codecademy has added some Q&A so i will go back and check it out. I thnk the courses are probably pretty good – i just think there are more prerequisites than were advertised…

  27. [...] UX An interesting look at usability, and how real users don’t play along with how the software was ‘designed to be used’. On the Usability of Codecademy. [...]

  28. [...] last week, speaking at Software Quality Days.  A bit to add to the discussion is a description of Codecademy usability by IBM evangelist Antonio Cangiano on his blog Zen and the Art of Programming.  He had a smart but [...]

  29. [...] overjoyed about Codecademy but eventually realised from their usability that it was intended at a reasonably computer literate user. Their problem was that the forgiveness that they employed when teaching the programming novices [...]

  30. Edwin says:

    Thanks for this blog post, I never knew about this Codecademy before. I finished the trial for non-members, and I think this site is pretty good to recommend to some of my acquaintances those are a total newbie in programming.

  31. [...] Programming Zen. Lalu gw baca-baca beberapa postingan di sana & nemuin satu postingan tentang usability test buat website Codecademy. Jadi ceritanya dari postingan tentang usability test itu gw tau tentang website yang namanya [...]

  32. Reminds me of the first time I ever tried to exit from IRB :)
    I had a friend try, try ruby – before it was CodeSchool. He got pretty lost too :(

  33. Mack says:

    I think Codeacademy have many issues but the most obvious one is how hard it is to actually learn something from it. You will be distracted by other things that has nothing to do with coding. Really annoying. If you´re new to programming you will find it very difficult to know what they want you to do in the lessons/projects they have set upp for you. You will get stuck on things for hours MANY times, that´s non programming related. Unfortunately you´ll end up spending A LOT LESS time on learning because of it.

    That being said, i think the overall idea is brilliant and i´m sure in time those issues will be corrected, but for now i would recommend Codeacademy to no one.

  34. Kim says:

    The lay-out and jumping-back are very confusing.

    Although there is a Q&A, and some people try to give helpful responses, I see a lot of people saying, “Me too” to problems that occur and to weird things that happen, like messages about something called a Sandboss [I think that was the word] There are colors which seem to be meaningful but inconsistently applied [e.g., a semi-colon is in black, but we're told it is necessary [but then it doesn't seem to be necessary all the time]]. When “if statements” are introduced, the word “if” is also in black. This means that a normal person reads the word “if” as part of the sentence[s]. Thus, s/he thinks the program is telling her/him that “if . . . , then ” — but there is no “then . . . .” All sorts of vocabulary comes up without ENGLISH explanations. All in all, it is poorly designed for non-geeks.

    As previous posters mentioned, error analysis should be used. It seems the people who put this together did not look at any research or publications on how to teach and learn.

  35. [...] developers have heralded the concept of Codecademy itself. In theory, it sounds like exactly what we need. For developers, however, it’s almost impossible [...]

  36. Braco says:

    I find Codeacademy very difficult to use.I know some things about programming,but I’m still trying to learn.Thanks for sharing!

  37. Furiant says:

    I’m confused as to why everyone thinks we need a different method of learning programming. All the best programmers in the world learned programming without a CodeAcademy-like model. Why suddenly has that method become obsolete? Why does that wheel need to be reinvented? Is it because the human race has evolved in the last 10 years to be incapable of learning that way anymore?

    Get some books; find a quiet space and some free time; concentrate, try hard; find a programmer and offer to buy as many lunches as they want as long as you can spend that lunch asking them questions… I hear beer is good currency with programmers too;

    The above is a known good method of learning. It produces everything from poor to brilliant programmers, and has, reliably, since at least the 1970′s.

    Not everything improves by being dumbed down and made effortless. I think many many things do; but not necessarily programming. And I’m glad the programmers with whom I work all have brains that are not flaccid and fragile, who have greater-than-average attention spans, and who are capable of pulling information together from dozens of unrelated resources, and fitting it all together using their mind alone.

    • Tim says:

      Thanks, that is exactly what I was going to do. It’s good to know it works. I am done with most online learning. Much of it is garbage.

  38. After experiencing similar frustrations when trying CodeCademy on my family members, I had a go at creating my own learn to code site, initially as a fun side project. 18 months later it has become much more than a side project. I have pretting much been working on it full time.

    Check out http://codeavengers.com

    I would love more feedback on how I can improve! Feedback I have been getting the past 12 months is that total beginners, teachers and students prefer my courses over CodeCademy hands down! If anyone cares to disagree, I would love to hear why. Thanks.

  39. Tim says:

    I gave the javascript lesson a good go at it, until I realized (or finally admitted) it is really lacking.

Leave a Reply

I sincerely welcome and appreciate your comments, whether in agreement or dissenting with my article. However, trolling will not be tolerated. Comments are automatically closed 15 days after the publication of each article.

Current day month ye@r *

Copyright © 2005-2012 Antonio Cangiano. All rights reserved.