Enablers, Inhibitors and Perceptions of testing in Novice Software Teams – Raphael Pham

To study novice software teams, Raphael used students, so he starts by explaining how software testing is taught at the Leibniz university. After an introductory software engineering course, in which the needs and benefits of testing were explained, all software engineering students have to participate in a software project during their BSc. It lasts from 4 month and has three phases: requirements, design and implementation and is executed by 5 students for a real client.

Last year, Raphael did a Grounded Theory study where 97 students were interviewed in groups of 5, where they were interested in whether the lessons on testing from the course were implemented in real life. The students had to had in tests at the end of the project, but no info was provided on what precisely had to be done. Then, Raphael tried to understand

  • How did they test?
  • What did they test?
  • What made it easier and what made it harder?


Students only test at the end of the projects (each line is one group, each row in one of the three phases)

2014-11-18 11.50.31

So, why does this happen? Students were explicitly taught to test early. So, Raphael of course asked the students why they did not test earlier. One of them answered:
“The things we were testing were changing heavily. Because everything was so dynamic. At the end, there was no time left”
So, it was not the case that they forgot, or did not see the need. They just did not know how to test a changing system, they saw automated tests like technical debt. Students therefore mainly opted for manual labor, even though they knew that this was not really considered testing. “we just executed it, that is not really testing, but we did not have time” As reason for doing it manually, students lists this was faster, easier and did not involve a learning curve for new tools. Some groups did try GUI testing frameworks, but stopped this because there was no good tutorial available.

Even worse: students felt that writing tests was unproductive, and they felt it would be better for their projects to add more features than to write tests.

At the end of the projects, students were disappointed with the effects of the test suite, because (probably) since it was added at the end, the worst bugs were already found by manual execution, and there were no benefits of regression testing.

Preprint: http://t.co/Ka2woGCbCS

My questions:

Since there was no time for my question during Q&A I am posting it here and I hope Raphael will respond 🙂

I am curious as to how the testing was taught. When I was a student, it was explained like something you just should do because if it inherently good. Like eating vegetables or stretching before a workout. It wasn’t until I had developed a huge code base with multiple people working on it, I began to see the value of tests. If I do not want other breaking my code, for example, I add a test. If I want to be sure a new optimization won’t mess up existing code, I add a test. Shouldn’t we try to have students experience how testing can be beneficial rather than just explaining them?

Not that I would know how to do this, by the way.



  1. Raphael Pham

    Hi Felienne!

    Thank you for live-blogging – this is so cool! 🙂

    As far as I know, we offered the students weekly lectures coupled with a tutorials in smaller groups and homework. Tutorials and homework were meant to be practical in nature and should deepen and exercise the topics of the lecture – but I see room for improvement! In our interviews, students complained that they perceived our lectures to be more on the theoretical side and wished for a more practical experience. Some students that did test automatically during their software project course reported to experience *the* “feeling of accomplishment when testing” for the first time.

    So, in this sense, I would advocate a course design that encourages and employs open source development: students are introduced to testing topics and are asked to demonstrate their knowledge on real-world software in tutorials and exercises.

    However, I am not sure how many universities deviate from the “lecture-tutorial-exercises” scheme when it comes to software engineering courses…


    1. felienne (Post author)

      Thanks for the comment!

      It could be that the homework are so small or artificial, that the use if testing does not click yet. And on deviation form the default scheme, not many universities maybe, but we do things differently 🙂 see: http://avandeursen.com/2013/12/30/teaching-software-architecture-with-github/

  2. Pingback: Talk on Software Testing in Novice Software Teams, FSE 2014 | rpham's research.

Comments are closed.