Proposition #2

A method to encourage replication studies is to initiate a replication track, in which results from the previous edition are being challenged.

Unfortunately, research today is mainly aimed at obtaining new results. I find this very strange, challenging new ideas is the heart of research and it is how much of the important scientific knowledge was obtained. From the model of our universe to the model of an atom, all were constructed by scientists constantly questioning each others theories.

Recently replication is in the center of scientific attention, since there have been many cases of fraud. The Stapel case was covered in the Dutch media quite extensively. One would expect that this would change the attitude of funding agencies, however NWO, main research funder in the Netherlands recently stated that they mainly support new and innovative ideas (1).

If funding agencies are not going to change, we researchers should! If a replication study is deemed as important to a researchers resume as a new idea, they will be more popular and eventually will be funded too.

One of the ways to encourage this is to run a specific replication track, in which results of previous papers are put to the test. Just pick one of the previous studies and run it again. All submissions to this track will be judged by:

  • Quality of the replication (was new data collected? were additional tests done?)
  • Assessment of similarities and differences
  • Suggestions for additional tests

Just as PC members are currently invited to review for a certain conference, in a similar manner replicators from the field could be asked to bid on papers to replicate and to perform and describe a replication study.

This is quite easy to implement and could seriously increase the regard of replication studies.

References

1. Spruitjeswetenschap – Daniël Lakens & Sander Koole (Hypothese)

 

1 Comment

  1. dragan gasevic

    This is a nice proposition and very much needed in empirical research in computing. I would dare to say that we hardly have any real research culture in computing. However, I think that the post is not quite correct in stating that no research recognizes importance of replicated studies. That is the very essence of research in medicine, psychology, sociology, education. The same (set of) hypotheses and is tested and refined all over again until enough evidence is found to confirm theories to hold across different conditions, populations, cultures, ethnicities, … I am under impression that we need to discover hot water too often in computing research. And we are so unbelievably ignorant…

    In any case, a great post which will hopefully reach many to awake us from out ignorance…

Comments are closed.