“If conferences and journals start to employ crowd reviewing, this will increase both the speed and quality of reviews”

A bit of background for those not in academia, or even not in computer science, as publishing practices differ over different field. First of all, we CS people publish long papers with new results in conferences and not in journals. The process for a high-profile conference is more or less as follows:

Month 0: You submit a paper
Month 5: You get the notification
Month 8: Camera-ready deadline
Month 9: The actual conference takes place
Month 10: Your paper will be published, that is to say, behind a pay-wall, accessible for those with a subscription

You see how there is almost one year between me submitting and the paper being available online? I know this does not hold for all conferences: for smaller, more focused conferences, this process is typically shorter, but my experiences have mostly been with this type of publishing cycles (ECOOP 2010, ICSE 2011, 2012, ICSM 2013 and ICSE 2013) and it is just too long.

Some more background on the reviewing process: The reviewers (together called the Program Committee or PC) are invited by the organizers of a conference responsible for the paper selects (called Program Chairs) and they bid on the papers they would like to review. Typically, three or four people review your paper after which a discussion follows where a decision is made. Again, this depends on the conference, but usually this discussion takes place in a ‘PC meeting’ and reviewers who have not read your paper may also weight in. Although there have been doubts on the quality of this process, my objections have more to do with the tempo.

Just for fun I did some statistics on these 5 conferences which papers form my dissertation. In total, over 4 years, I have spent no less than 14 months waiting for notifications and even worse, the general public has been waiting for a total of 23 months to be able to read my (already accepted) papers.

We should do better and I propose to do this with crowd reviewing. Why not put your paper online and let people review it? Of course this needs moderation, but the time this could save is huge! But what about the quality, you say? Well, I assert that the success of websites like Quora and StackOverflow shows that it is possible to employ a system of upvoting and reputation to create and validate knowledge. Why not have a similar system for papers? It will save time and quality will be guaranteed by the crowd: bad reviewers lose reputation and their future reviews will count less towards the verdict. With a system similar to LinkedIns new endorsements, even the confidence level of reviewers can be asserted, rather than themselves asserting it as we currently do. So review time will decrease, review quality will increase. And as a bonus, the process will become more transparent.

Program chairs could go over all papers that were sent towards a specific conference, have a look at their crowd reviews and select those best suited for their conference, while rejected papers can be updated and resubmitted.

I know this is a huge paradigm shift and I understand it will not happen for a while. But we should work towards it!

Agree/disagree? Please let me know below or on Twitter.

Recommended further reading:
The future of academic publishing
From Peer Review to Crowd Review