Leif Singer’s PhD Presentation

IMG_20130211_095232

Software engineering has proven benefits, yet adoption is low, there is resistance, because:

  • change is hard
  • inexperienced developers will not see benefits
  • long-term benefits are unclear

Current state: mandating behavior. Leif’s research is different, he wants to persuade developers to use new techniques, instead of forcing them.

Leif’s work is based on innovations theory. This theory consists of 6 different steps, in which different mechanisms exists for persuation.

IMG_20130211_100731

But, there is a gap here, the KAP-gap. Knowledge-Attitude-Practice Where people know about the innovation, yet don’t decide to use it.

If you want people to cross the KAP-gap, you need to study their motivation.

Something else that is important is CSCW (computer-supported cooperative work) CSCW is the field of research that studies social media and groupware.

This increases social transparency, which creates new effects that can be used for improving adoption.

  • behaviour can be imitated
  • there is implicit communication of norms
  • networks diffuse innovation

Empirical studies

Study 1) Influence of social transparency on GitHub: how does GitHub affect testing.
Through semi-structured interviews and a follow up survey, Leif found that a crucial task for project owners is communicating test cultures. As
Open Source is mainly volunteering work, external motivation is hard.

Project owners use: transparency to show newcomers how it is done, in commits, discussions and pull requests. There is also formal communication:
guides, examples, information on continuous integration on project home page: this shows testing matters.

Study 2) The Social Programmer Ecosystem

How do developers use social media. Again, with semi-structured interviews and surveys, Leif studied Coderwall, a websites that create developer profile aggregates, with skills and achievements based on GitHub commits.

Leif found that achievements can motivate programmers to try new things (like a badge for trying Ruby) and like appreciation from peers.
Another interesting thing is that developers actively seek opinions from thought leaders and try out technologies they recommend.

On these results, Leif developerd PAID: a process for actively motivating developers:

  1. Characterize contentWe need to determine whether this is a creative or routine practice. Extrinsic motivators are more suitable from routine tasks, for example. We can also look for opinion leaders, who we can use to influence adoption.
  2. Goals & metrics 

    Choosing metrics is hard, we need to be aware that it cannot be gamed. Is the metric playful and does it impact the real world? For instance, a deduction in salary is not playful and impacts real world, therefor we can expect it will be manipulated.

  3. Choose adoption patternsLeif has developed a catalog of adoption patterns that match the goal and metrics of the project.
  4. Design treatmentIn this step it is important that barriers are low (for instance: no complex install but webapp) and that we concentrate on one issue. Small world networks have the strongest effect on adoption, so this could be enabled to improve chances of success.
  5. Deploy intervention 
  6. Analyse resultsHere, we compare metrics against baseline and we should not steer on metrics alone, but comine this with interviews and questionnaires.

Rinse and repeat.

The catalog of adoption patterns

Leif derived the adoption patterns by starting with a very broad literature review, with papers from psychology, sociology and SE. He studied no less than 634 publications and with grounded theory, he coded them into 24 patterns.


IMG_20130211_103052

All patterns have a name and a problem that they solve in a abstract way and a solution to solve it. Furthermore, they have a rationale
and a discussion. Prerequisites and examples and related patterns from the catalog.

Some examples

  • Normative Behavior: Making normative behavior visible may increase adoption
  • Leader-board: Competition motivates come populations. But may drive others away and losing does not motivate.
  • Challenge (relate to goal setting theory) Clear and challenging (but attainable) goals prompt individuals and increases performance. Learning goals are more effective than performance goals. So try integration testing might me more effective that ‘get 5% more coverage’

Leif tried this in a Software Project Course at the university. In this project, student’s commits were irregular and not always clear.
He performed a quasi-experiment, where test and control group are not random: he used the current batch of students for the tests and previous years as control group.

Looking at PAIP: writing a commit message is a somewhat creative task and there were some early adopters in previous years, so there would likely be some this year.

The goal thus became: both starting the adoption of a new practice and the improvement.

The metrics Leif chose were:

  • # commits per student
  • time between commits
  • #commits per student with message
  • ratio of commits with message
  • length of commit messages

Then, selecting patterns lead to treatments:

  • Normative behavior pattern: news-feed
  • Leader-board: ranking students
  • Points&Levels, Challenge, Progress Feedback: milestones for commits
  • Triggers: Notification for positive events
  • Triggers: Weekly digest

Results

77% improvement in number of commits, 44% increase time between commits (meaning commits were more regular), 75% improvement in length of commits. Also, this year, there were no students with 0 commits and less commits with no message.

Leif concludes with an outlook: society and technology influence each other, so adoption patterns, which are a snap shot of the current situation,
may change over time. However, the core idea: a systematic design of cooperation should be applicable to other field, like electrical engineering.
Also, the impact of these patterns is not restricted to adoption of new technologies, it may also be applicable to employee motivation.