The traditional peer review system is to submit an article to a journal, wait for the editor to get anonymous referee reports and then deliver a decision. We all have horror stories on how inefficient this system is, both in terms of time lost (and Economics seems particularly bad here) and in terms of the arbitrariness of the process. Yet despite all the complaints and the many announcements of its imminent demise in the age of the Internet, this peer review system is still going strong.
What has been done to reform it? Editors have worked hard to reduce decision times, sometimes with success, but there seems to be a lot of habit in the slow response of referees (so editors claim). New models are tried, such as the bepress journals that provide quality ratings and thus avoid authors to submit repeatedly down the journal ladder, the new American Economic Journals that can “automatically” feed on the rejects from the AER, or Economic Inquiry that now asks referees to only provide an up or down vote, thus bypassing the revisions. While these are all important initiatives, they are after all only a variation of the original system.
We can think completely differently. Think of this blog. I rant on a topic, and then others can comment on it and openly declare whether this rant was valuable or not. Why not do this with academic work? An early attempt was done with WoPEc. This was the first RePEc service, similar to IDEAS and EconPapers today, which offered for some time on each paper’s abstract page a discussion section. Participation was minimal and there was very little value added (see an example, I could not find one that actually had comments). This aspect of WoPEc was finally abandoned. A second attempt was organized by SOLE (Society of Labor Economists), that would post every two weeks a new paper to discuss. Again, participation was small, and the project was finally abandoned.
The latest attempt is the Economics E-Journal, which allows registered users to rate and comment on discussion papers. Once the editors find that a paper has generated sufficient interest, it is promoted to the journal, where it can still be discussed. This initiative started this year, so the jury is still out whether it will be successful in the end. So far, it looks very promising.
From time to time, members of the RePEc team are approached and asked whether a discussion section could be added to our services. Given the past experience with WoPEc and the large monitoring costs involved, we are not enthusiastic. Of course if other volunteers are interested in working on this, we may think about it. But first we need to understand whether there is really a demand for this. Maybe RePEc is now too large for this and such initiative should be left to field specific initiatives (SOLE again?).
The RePEc Blog 
The problem with open discussion — blogging — is that it fills up with trash. The whole point of journals is that they screen, which is a valuable service for readers. Who has time to read a million blogs and leave comments? Moreover, commentary tends to come from people with a low value of time. (If you are feeling insulted, leave a comment…I did)
Moreover, blogs so far haven’t lent themselves to deep thought. There is a lot of value to me if you think about a problem for a long time, sort it out, and then provide the fruits of your thought in a concise, readable way. A sound bite like this one is not an effective way to advance the science. That is why most of the journal experiments, including the economics e-journal, still involve papers and not blog entries. I don’t know if the e-journal will succeed but I sent a paper there to help. I think of their policy as open reviewing, which is also an interesting idea.
Finally, let me correct some mis-impressions about Economic Inquiry. First, “no revisions” is an option, not a requirement. It is a good option if you are sick and tired of referees who ask for a million changes when you are quite happy with what your paper says, or if you are a 5th year assistant professor and want a decision rapidly. There is a drop down box where you select it, or not, when submitting.
Second, there are still reports at Economic Inquiry. The *main* purpose of the review is to evaluate, but we still expect reports. The difference is that, with an acceptance, the author gets to decide what changes to make, if any, based on the reports. That is, if we say yes, the author has the right to publish as is. And under no revisions, we will either say yes or no.
Third, rejected papers are usually published somewhere eventually. There are over 300 economics journals. If you don’t like our policy, by all means, submit to one of the other journals, or just don’t select no revisions, because it is an option.
Fourth, there has been a lot of discussion about my ability to commit. “What about the paper that just needs minor fixes to be publishable?” If it is a good paper I’ll accept it. I have already done so on several papers. If the author doesn’t want to fix minor blemishes, that is the author’s choice and I won’t lose sleep as most published papers have some minor problems. Furthermore, most authors will fix problems even if the editor doesn’t force them to, because they value their reputation. If the problems are sufficiently severe that I have to reject it, then it can go elsewhere. We are getting enough submissions to lose a few.
Finally, this is an experiment. Let’s see what happens with it before we tinker or, as some have asked, scrap it. No revisions are running about 1/3rd of submissions and I have accepted a few and rejected a few.
Preston McAfee
Editor, Economic Inquiry
Experiments on improving peer review are always welcome, but what the
worldwide research community (in all disciplines, economics included)
needs most urgently today is not peer review reform, but Open Access
(OA) to its existing peer-reviewed journal literature. It’s far easier
to reform access than to reform the peer-review system, and it’s also
already obvious exactly what needs to be done and how, for OA — mandate
RePEc-style self-archiving, but for the refereed postprints, not just
the unrefereed preprints — whereas peer-review reforms are still in
the testing stage. It’s not even clear whether once most unrefereed
preprints and all refereed postprints are OA there will still be any
need for radical peer review reform at all; it may simply be a matter
of more efficient online implementation.
So if I were part of the RePEc community, I would be trying to persuade
economists — who already have the preprint self-archiving habit — to
extend their practise to postprints (and to persuade their institutions
and funders to mandate postprint self-archiving in each author’s own
Institutional Repository, from which RePEc can then harvest).
http://users.ecs.soton.ac.uk/harnad/Hypermail/Amsci/0594.html
Click to access peerev.pdf
http://www.eprints.org/signup/fulllist.php
Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/
UNIVERSITIES and RESEARCH FUNDERS:
If you have adopted or plan to adopt an policy of providing Open Access
to your own research article output, please describe your policy at:
http://www.eprints.org/signup/sign.php
http://openaccess.eprints.org/index.php?/archives/71-guid.html
http://openaccess.eprints.org/index.php?/archives/136-guid.html
OPEN-ACCESS-PROVISION POLICY:
BOAI-1 (“Green”): Publish your article in a suitable toll-access journal
http://romeo.eprints.org/
OR
BOAI-2 (“Gold”): Publish your article in an open-access journal if/when
a suitable one exists.
http://www.doaj.org/
AND
in BOTH cases self-archive a supplementary version of your article
in your own institutional repository.
http://www.eprints.org/self-faq/
http://archives.eprints.org/
http://openaccess.eprints.org/
Re Stevan Harnad’s posting:
as a clarification, please note that RePEc does not harvest from author’s own IRs. We provide a service (MPRA, mpra.repec.org) by which authors who lack institutional RePEc archives may post their works to RePEc, only after establishing that they have the legal right to do so. This generally excludes contributions for which the author cannot establish the right to post, e.g., a postprint which is an exact copy of the published article.
We also do not harvest from institutional repositories. Rather, we provide a framework by which institutions can document their publications for inclusion in RePEc services. Thus, we do not have a facility by which an institution can merely say “please harvest from our OAI-compliant IR”. We require that institutions provide metadata on their own server in a standard fielded format (ReDIF). That format can usually be generated mechanically from OAI metadata.
Thanks
Kit Baum
There actually exists in MPRA the possibility to add a comment to an existing paper. Maybe we should advertise this possibility on the welcome page. (At the moment, this option opens on the first submission page.) It would not be much effort.
Ekkehart
I agree with Preston that one must have relatively low value for time to find such time to comment on blogs. I also agree that responding the same day does not allow to “go deep”. That is true for a “standard” blog.
But what about a blog that would pertain to research? Where the topic would not change on a daily basis? This is essentially what SOLE tried: define a topic for two weeks and let people discuss it. Even after the two weeks, discussion was possible.
Some of us are good at creating research, other are sought after as discussants (or both). You get little credit as a discussant, yet it can be very valuable. Why not find a way to reward good discussions? Amazon does this with its reviewers. Could this be done on a research blog?
The one way I see this could work is with niche research blogs that would attract a specialized crowd. Say a blog on immigration and education, or VECMs, or asymmetric information in credit markets. Yes, that specialized. It could bring incredible dynamics into a research community. But I do not see it work on a larger scale. Thus I do not see much of role for RePEc here.
[…] thoughts on various points raised in New Peer Review Systems and the comments that […]
We are happy about the positive reference to our new e-journal Economics.
Since many readers of this blog may not know how exactly it differs from previous initatives, let me give a short explanation of how the Peer Review System of the e-journal works: When a paper has been submitted, the assigned associate editor decides whether to accept the paper for the further peer review process by answering two questions: (i) whether the paper shows promise of making a significant contribution and (ii) whether it meets basic scientific standards. If these questions are answered in the affirmative, the paper is published on the platform of the Economics Discussion Papers where all referee reports, replies of the authors and reader comments are posted. After 8 weeks, based on these input, the associate editor decides whether the paper is accepted or rejected for publication as an article in the e-journal.
Regarding Stevan Harnad´s remark: In the e-journal Economics all readers have open access to reading and downloading papers.
Dennis Snower, Senior Editor, Economics: The Open-Access, Open-Assessment E-Journal
I think that Internet is changing the rules of the Review Systems and that electronic journals such as e-journal will represent the future of the editorial market. We all know how unfair the Review Systems of traditional journals are. I personally had an experience of that kind with the Quarterly Journal of Economics, when I submitted a paper entitled “A contribution to the Positive Theory of Indirect Taxation” and the next day Robert Barro, a QJE’s editor, answerd me that he had rejected the paper on the ground that it didn’t have “enough” economic content for a general audience. So I decided to take a look to the last issues of the QJE. Well, it is interesting to notice that QJE’s editor evaluates “enough” for an economics audience not works on taxation but on the following topics (published between 2004 and 2007):
Obtaining a Driver’s License in India: An Experimental Approach to Studying Corruption*
Bertrand, Marianne – Djankov, Simeon – Hanna, Rema – Mullainathan, Sendhil
The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector*
Jensen, Robert
HIV Breakthroughs and Risky Sexual Behavior*
Sood, Neeraj – Goldman, Dana
Bargaining in the Shadow of the Law: Divorce Laws and Family Distress*
Stevenson, Betsey – Wolfers, Justin
Which Countries Have State Religions?*
Barro, Robert J. – McCleary, Rachel M.
Air Pollution and Infant Health: What Can We Learn From California’s Recent Experience?*
Currie, Janet – Neidell, Matthew
Sexually Transmitted Infections, Sexual Behavior, and The HIV/AIDS Epidemic*
Oster, Emily
The Causes and Consequences of Distinctively Black Names
Roland G. Fryer, Jr., Steven D. Levitt
Legal Origins
Edward L. Glaeser, Andrei Shleifer
Why have Women Become Left-Wing? The Political Gender Gap and the Decline in Marriage
Lena Edlund, Rohini Pande
The Schooling of Southern Blacks: The Roles of Legal Activism and Private Philanthropy, 1910–1960
The Impact of Legalized Abortion on Crime
John J. Donohue III, Steven D. Levitt
Now it is clear that such a system cannot resist. The difference between a journal like the QJE and another like e-journal is that in the former three editors can take arbitrary decisions that nobody can reject, whilst in the second a more democratic voting takes place. In turn, in the former a monopolistic editor decides for all, whilst in the latter the market decides.
I think that 99 per cent of the authors would prefer the second type. I believe that what we have to do is finding the system to promote these second types of journals, increasing their impact factor. Otherwise everyone can understand how the market is closed and biased towards those very few universities which have editors in the journals’ boards.
[…] to encourage peer review bloggers to certify themselves and use an icon on their blog. We discussed earlier whether such blogs would be appropriate in Economics, along with some past experiences. Other […]
[…] Peer Review Systems” By Bill Goffe Several thoughts on various points raised in New Peer Review Systems and the comments that […]