The RePEc Blog

Several thoughts on various points raised in New Peer Review Systems and the comments that followed.

• In a sense, the lag in the review process might be optimal. A publication of most any sort is valuable to the author and one in a leading journal of course has a very substantial return. Journals thus have a good reason to deter papers that aren’t at all appropriate; this was pointed out by Ofer Azar, “The Slowdown in First-Response Times of Economics Journals: Can It Be Beneficial?,” Economic Inquiry, 2007, 45 (1). The constraint here would seem to be editor’s and referees’ time. In economics, the most common cost that journals impose on authors is a lengthy review process. I’d hazard a guess that bepress gets around this by their ranking of papers into different tiers; they don’t have to deter less than stellar papers as they’ll likely get a home there. This is combined with their system where authors who submit there agree to review two papers quickly (a nice example of a virtuous cycle). Another way to speed up the referee process is a system where any reader can submit comments on a paper. But, as Christian points out, this doesn’t seem to attract many comments. It turns out I’ve looked a bit at this and found 5 journals that have tried a reader rating system and none have attracted a sufficient number of comments to make it fly. From here, one option is something that ranks papers after they’ve been out, such as citations. Paul Ginsparg has some thoughts on one approach, as does Hal Varian (now the chief economist at Google). But, these might take years to generate sufficient data to render a judgment on a paper. I think many of us want something that is quicker.

  Another possibility is something like Faculty of 1000 in biology and medicine where a level of review beyond journals takes place. I very much like the summaries in their sample web pages; you don’t see that in economics. One could imagine it working on top of our working paper culture. But, I wonder if some of their success comes from the grant culture in these fields as this is a fee-base service that seems to pay the reviewers? How might one set up something similar in economics with our working paper culture? Journals would likely see it as preempting their role.

• I agree with Preston that perhaps the most interesting part of Economics E-Journal is the open review system (all can read reviews) and the feature that allows authors to publicly respond to referee reports. Both would seem to give referees correct incentives. I would think that journals could implement this quickly and easily at low cost. Also, while deep thinking and working through a paper you are writing is extremely valuable, so is getting feedback and discussing a paper and the ideas in it. I have a first draft of a paper on these topics (see below) but in writing this blog entry I have developed some new insights. I wouldd add that prompt discussion is something that the Internet can aid for those of us without local colleagues in our fields.A very minor point on his post: if you count economics journals by the number in EconLit, there are about 1,240. In short, most any paper should certainly be published!
• I certainly second Christian’s point about blogs and research. All the economics blogs I know of do not discuss serious research that much if at all. Much more common are discussions of current economic events and policy that members of the public find interesting. I don’t know of a one where someone might say, “Say, that paper by Sam Jones in Computational Economics” is interesting because the algorithm he used to calculate…” Also, debates by papers take years given the time to write one and respond. This seems rather silly given today’s technology; after all, journals in the their current form came about when information traveled at the speed of a horse or ship. Perhaps a blog is too quick for complete works, but I understand in law their use is leading to a relative decline in the importance of law journals. One example (first paper found in a Google search) is Guest Blogger: The Start of the Supreme Court’s 2007-08 Employment Discrimination Docket: Federal Express Corporation v. Holowecki. Yes, it discusses current events, but the topic is one that only specialists would seem to care about. You do not seem to see such in economics blogs that I am aware of.

Where does this leave RePEc? Well, I’m not really sure. It is hard to change norms in a field and I am not sure that RePEc could swing it. But, I do not think that a comment system on individual papers would get that much traction. Somehow you want to get the judgment of peers (for promotion, tenure, and annual raises) in a speedy manner. Those two criteria seem to be at odds with each other.
It turns out I have done some thinking on these issues; at the risk of self-promotion they can be found in Next Steps in the Information Infrastructure in Economics. Note that this draft was written for a conference of non-economists, so some parts will strike a very obvious cord to an economist’s ear. I have also have yet to incorporate some very useful comments that Christian kindly wrote. In a nice mix of blog and papers, further comments on the paper are greatly appreciated.