New Peer Review Systems

October 29, 2007

The traditional peer review system is to submit an article to a journal, wait for the editor to get anonymous referee reports and then deliver a decision. We all have horror stories on how inefficient this system is, both in terms of time lost (and Economics seems particularly bad here) and in terms of the arbitrariness of the process. Yet despite all the complaints and the many announcements of its imminent demise in the age of the Internet, this peer review system is still going strong.

What has been done to reform it? Editors have worked hard to reduce decision times, sometimes with success, but there seems to be a lot of habit in the slow response of referees (so editors claim). New models are tried, such as the bepress journals that provide quality ratings and thus avoid authors to submit repeatedly down the journal ladder, the new American Economic Journals that can “automatically” feed on the rejects from the AER, or Economic Inquiry that now asks referees to only provide an up or down vote, thus bypassing the revisions. While these are all important initiatives, they are after all only a variation of the original system.

We can think completely differently. Think of this blog. I rant on a topic, and then others can comment on it and openly declare whether this rant was valuable or not. Why not do this with academic work? An early attempt was done with WoPEc. This was the first RePEc service, similar to IDEAS and EconPapers today, which offered for some time on each paper’s abstract page a discussion section. Participation was minimal and there was very little value added (see an example, I could not find one that actually had comments). This aspect of WoPEc was finally abandoned. A second attempt was organized by SOLE (Society of Labor Economists), that would post every two weeks a new paper to discuss. Again, participation was small, and the project was finally abandoned.

The latest attempt is the Economics E-Journal, which allows registered users to rate and comment on discussion papers. Once the editors find that a paper has generated sufficient interest, it is promoted to the journal, where it can still be discussed. This initiative started this year, so the jury is still out whether it will be successful in the end. So far, it looks very promising.

From time to time, members of the RePEc team are approached and asked whether a discussion section could be added to our services. Given the past experience with WoPEc and the large monitoring costs involved, we are not enthusiastic. Of course if other volunteers are interested in working on this, we may think about it. But first we need to understand whether there is really a demand for this. Maybe RePEc is now too large for this and such initiative should be left to field specific initiatives (SOLE again?).