Exposing The Most Frequent Mistakes In Programming

The Black-box project is a massive data collection initiative by the University of Kent, that sifts through millions of source code compilations looking to identify the most frequent mistakes made by student programmers.

Why is that useful?

Understanding how students learn to program through their common misconceptions and their recurring mistakes is important for many reasons:

  • Produce educational material focused on these issues
  • Render educators more efficient
  • Build IDE’s or programming tools that protect against those errors 
  • Improve the readability and the helpfulness of the errors emitted by compilers
  • Language design – improve the future syntax and design of a language by taking into consideration the syntax barriers students typically encounter


What data is collected?
The data collection is based on BlueJ, a free Java IDE, which is designed specifically in providing usability enhancements to beginners. Blackbox acts as an add-in to BlueJ that collects anonymous information on the ways the IDE is used, the source code of the project the student is working on, as well the errors resulting from its compilation(s).

Although the source code and the compilation errors are the springboard of the research, the researchers also went on to calculate the time it took students to fix a bug by looking forward in time to find the next compilation where the mistake was no longer present. This emerging metric serves as an indication of and how long it typically takes students to learn from their mistakes.

The errors then were classified according a classification resulting from a previous study based again on the Blackbox data, called “Investigating Novice Programming Mistakes : Educator Beliefs vs Student Data”. This in turn used a classification established in an even older study “Identifying and correcting Java programming errors for introductory computer science students” that surveyed educators asking them for their experience on the most frequent mistakes their students made.

They came up with 18 errors grouped into three broad categories; Syntax, Semantic and Type. The errors were subsequently labelled A through R, and were informally categorized as follows: 


 A: Confusing the assignment operator (=) with the
comparison operator (==). For example:if (a = b)
 C: Unbalanced parentheses, curly or square brackets
and quotation marks, or using these different symbols
interchangeably. For example:while (a == 0]
 D: Confusing \short-circuit” evaluators (&& and ||)
with conventional logical operators (& and |).
For example:if ((a == 0) & (b == 0))
 E: Incorrect semi-colon after an if selection structure before the if statement or after the for or while repetition structure before the respective for or while loop.For example:
if (a == b);
return 6;
 F: Wrong separators in for loops (using commas instead of semi-colons).For example:for (int i = 0, i < 6, i++)
 G: Inserting the condition of an if statement within
curly brackets instead of parentheses.For example:if {a == b}
 H: Using keywords as method or variable names.
For example:int new;
 J: Forgetting parentheses after a method call.
For example:myObject.toString;
 K: Incorrect semicolon at the end of a method header.
For example:

public void foo();



L: Getting greater than or equal/less than or equal
wrong, i.e. using => or =< instead of >= and <=.
For example:if (a =< b) …


P: Including the types of parameters when invoking a
method.For example: myObject.foo(int x, String s);

Type errors:

I: Invoking methods with wrong arguments (e.g. wrong
types).For example:list.get(“abc”)


Q: Incompatible types between method return and
type of variable that the value is assigned to.
For example:int x = myObject.toString();

Other semantic errors:

 B: Use of == instead of .equals to compare strings.
For example:if (a == “start”)


 M: Trying to invoke a non-static method as if it was
static.For example:MyClass.toString();
 N: A method that has a non-void return type is called
and its return value ignored/discarded.
For example:myObject.toString();
 O: Control
ow can reach end of non-void method without returning.
For example:

public int foo(int x)
if (x < 0)
return 0;
x += 1;
 R: Class claims to implement an interface, but does
not implement all the required methods.
For example:class Y implements ActionListener { }


In “Investigating Novice Programming Mistakes: Educator Beliefs vs Student Data” the researchers used these 18 errors to contrast the mistakes the educators perceived to be the most frequent with the reality exposed by the errors resulting from source code compilation.

The study came to some surprising findings :

“Our first finding was that educators have only a weak consensus about these frequencies….

Our further result that educators are not very accurate compared to a large data set of students suggests that educators are also not accurate about the frequencies of these mistakes,so any claims that “students always make mistake X” are unlikely to be accurate…..

Our most surprising result was that an educator’s level of experience (as measured by years as and educator, years teaching introductory programming in any language, or years teaching introductory programming in Java)had no effect on how closely the educator’s frequency rankings agreed with those  from the Blackbox data….”

The current research, reported in “37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale
Student Data” aimed to go even further than its predecessor by providing a more detailed investigation into the characteristics of the mistakes, trying to answer the following research questions:

  • What are the most frequent mistakes in a large-scale
    multi-institution data set?
  • What are the most common errors, and common classes
    of errors?
  • Which errors take the shortest or longest time to fix?
  • How do these errors evolve during the academic terms
    and academic year?


To arrive at valuable conclusions, a large data set should exist, and Blackbox had exactly that: participation of 250,000 Java programming novices from institutions all over the world and source code from 37 million compilations.
In more detail, the research used the data collected during the 2013-2014 academic school year, a total of 37,158,094 compilation events, of which 19,476,087 were successful and 17,682,007 were unsuccessful. The analysis of the data was done in Haskell.

After processing that huge amount of data, the list with the most frequent mistakes was compiled (in descending order, thus error C is the most frequent) :


  1. Unbalanced parentheses, curly braces, brackets, and quotation marks, or using these different symbols interchangeably, such as in: while (a == 0].
  2. Invoking methods with wrong arguments or argument types, such as in: list.get("abc").
  3. Control flow can reach end of non-void method without returning, such as in:
    public int foo(int x)
      if (x < 0)
        return 0;
      x += 1;
  4. Confusing the assignment operator (=) with the comparison operator (==), such as in: if (a = b).
  5. Ignoring or discarding the return value of a method with non-void return type, such as in:myObject.toString();.
  6. Use of == instead of .equals to compare strings.
  7. Trying to invoke a non-static method as if it was static, such as in: MyClass.toString();.
  8. Class claims to implement an interface, but does not implement all the required methods, such as in: class Y implements ActionListener { }.
  9. Invoking the types of parameters when invoking a method, such as in: myObject.foo(int x, String s);.
  10. Incorrect semicolon in if statements or for and whileloops, such as in: if (a==b); return 6;.

 But that’s not all. Further insights conclude that:


“Syntax errors show a small peak at the beginning of
the northern hemisphere academic year, but otherwise are
fairly flat.  In the contrast, the slight peaks for the Semantic
and  Type  categories  happen  later.   We  propose  that  this
is due to the order of topics in academic courses:  students
initially struggle with syntax, but as they master this, they
begin to grapple instead with semantic and type errors.
While the Syntax mistakes are almost quick to fix and consistent  over  time, the  Semantic  ones  need  more  efforts and this decreases a little over time. 

The Type curve  is  interesting  as  it  shows  a  sudden  rise  and drop towards the end of the academic year, possibly due to introducing advance types for students.
The Time-to-fix of Syntax mistakes is almost flat, showing that they are quick to fix, but students do not generally get faster at fixing them. 

In contrast, Semantic mistakes show a decrease of thetime needed to fix over the course of the year, Type mistakes
show a strange, unexpected shape. 

The only interpretation we can offer is that with the increase of topics introduced to students, moving from primitive types to generic types and inheritance, students tend to make more mistakes between these ‘more diffcult’ types”



So as a side-product of the research’s findings, can we reach a conclusion as far as typing is concerned? Are dynamic languages easier to work with, since they require no type information at compile time as automatic type coercion is done at runtime? 

In other words, is dynamic typing better since you avoid the Type related mistakes from happening?

It certainly sounds so. Further proof comes from another study,  “An Empirical Investigation into Programming Language Syntax”.

Its findings were that :

“Our results may also have implications for instructors and students in regards to the use of static type systems, because of the observed difficulty novices have with static type annotations.
novices only placed type annotations approximately half the time in our study, across tasks and languages. From a language design perspective, computer code inside of a method can probably remove type annotations under many conditions while still maintaining static typing. However, for method declarations, there may be an unresolvable trade-off between static and dynamically typed languages.”

At first it might appear that dynamic typing helps novices but further down we find that :

“We suspect that some might believe our results imply dynamic typing is better for novices, but we find this explanation implausible…..Thus, claiming that dynamic typing is
better for novices is incorrect without the proper context. “

Instead a system utilizing type inference is the way to go :

“We have derived a type inference system that removes most type annotations inside of methods, keepsannotations in method declarations, and keeps static type checking itself. More study is needed, but compromises like this may afford ease of use for all.”

All these conclusions are a product of data collection done through a single IDE. Imagine an initiative that collected data from the many hundreds of MOOC’s out there. How would that improve the experience of programming and evolve educational material? 

Utilization of such data in the educational industry would lead to big gains almost from day one.


Courtesy I-Programmer