Several thoughts on various points raised in New Peer Review Systems and the comments that followed.
- In a sense, the lag in the review process might be optimal. A publication of most any sort is valuable to the author and one in a leading journal of course has a very substantial return. Journals thus have a good reason to deter papers that aren’t at all appropriate; this was pointed out by Ofer Azar, “The Slowdown in First-Response Times of Economics Journals: Can It Be Beneficial?,” Economic Inquiry, 2007, 45 (1). The constraint here would seem to be editor’s and referees’ time. In economics, the most common cost that journals impose on authors is a lengthy review process. I’d hazard a guess that bepress gets around this by their ranking of papers into different tiers; they don’t have to deter less than stellar papers as they’ll likely get a home there. This is combined with their system where authors who submit there agree to review two papers quickly (a nice example of a virtuous cycle). Another way to speed up the referee process is a system where any reader can submit comments on a paper. But, as Christian points out, this doesn’t seem to attract many comments. It turns out I’ve looked a bit at this and found 5 journals that have tried a reader rating system and none have attracted a sufficient number of comments to make it fly. From here, one option is something that ranks papers after they’ve been out, such as citations. Paul Ginsparg has some thoughts on one approach, as does Hal Varian (now the chief economist at Google). But, these might take years to generate sufficient data to render a judgment on a paper. I think many of us want something that is quicker.
Another possibility is something like Faculty of 1000 in biology and medicine where a level of review beyond journals takes place. I very much like the summaries in their sample web pages; you don’t see that in economics. One could imagine it working on top of our working paper culture. But, I wonder if some of their success comes from the grant culture in these fields as this is a fee-base service that seems to pay the reviewers? How might one set up something similar in economics with our working paper culture? Journals would likely see it as preempting their role.
- I agree with Preston that perhaps the most interesting part of Economics E-Journal is the open review system (all can read reviews) and the feature that allows authors to publicly respond to referee reports. Both would seem to give referees correct incentives. I would think that journals could implement this quickly and easily at low cost. Also, while deep thinking and working through a paper you are writing is extremely valuable, so is getting feedback and discussing a paper and the ideas in it. I have a first draft of a paper on these topics (see below) but in writing this blog entry I have developed some new insights. I wouldd add that prompt discussion is something that the Internet can aid for those of us without local colleagues in our fields.A very minor point on his post: if you count economics journals by the number in EconLit, there are about 1,240. In short, most any paper should certainly be published!
- I certainly second Christian’s point about blogs and research. All the economics blogs I know of do not discuss serious research that much if at all. Much more common are discussions of current economic events and policy that members of the public find interesting. I don’t know of a one where someone might say, “Say, that paper by Sam Jones in Computational Economics” is interesting because the algorithm he used to calculate…” Also, debates by papers take years given the time to write one and respond. This seems rather silly given today’s technology; after all, journals in the their current form came about when information traveled at the speed of a horse or ship. Perhaps a blog is too quick for complete works, but I understand in law their use is leading to a relative decline in the importance of law journals. One example (first paper found in a Google search) is Guest Blogger: The Start of the Supreme Court’s 2007-08 Employment Discrimination Docket: Federal Express Corporation v. Holowecki. Yes, it discusses current events, but the topic is one that only specialists would seem to care about. You do not seem to see such in economics blogs that I am aware of.
Where does this leave RePEc? Well, I’m not really sure. It is hard to change norms in a field and I am not sure that RePEc could swing it. But, I do not think that a comment system on individual papers would get that much traction. Somehow you want to get the judgment of peers (for promotion, tenure, and annual raises) in a speedy manner. Those two criteria seem to be at odds with each other.
It turns out I have done some thinking on these issues; at the risk of self-promotion they can be found in Next Steps in the Information Infrastructure in Economics. Note that this draft was written for a conference of non-economists, so some parts will strike a very obvious cord to an economist’s ear. I have also have yet to incorporate some very useful comments that Christian kindly wrote. In a nice mix of blog and papers, further comments on the paper are greatly appreciated.
The RePEc Blog 
Bill,
Economics journals are slow for two reasons. First, economists tend to write longer and more complicated papers than the typical natural scientist. One-point-per-paper short contributions are much easier to assess. My experience (own papers, Energy Economics) suggests that short papers get through peer-review faster. Second, and perhaps more important, there is a culture of long delays in economics. Many economics referees wait several months before they start working on their review, and then rush things anyway. If I review for an environmental sciences journal, I have the editor on my back within four weeks. This has never happened to me for an economics journal. If culture is indeed the problem, institutional or technical change will not accelerate the review process.
Citations are the only true test for the quality of a paper. Good papers stand the test of time, and only time can tell whether your paper is really good. That said, you do not need to wait till you’ve passed the 100 citations mark to tell whether a paper has potential. Good papers accumulate citations much faster than bad papers.
IDEAS/RePEc can make a contribution here.
“Top items” now has total citations over the lifetime of a paper.
One can also think of a ranking on the basis of citations in the last 12 months only, or filter the current top items for papers that are less than 1, 2, 3, 4, 5 years old. That would quickly reveal the papers with the potential to make it into the pantheon of seminal contributions.
Richard
I agree that, within the established business model, citations are the best assessor of the contribution of a paper. But it is indeed a lengthy process until this metric can actually be revealing anything: the paper first needs to get accepted, then printed, then cited in new papers, which in turn have to be published and printed, and then only do citation services pick them up and count them. Given publication delays, this is completely unreasonable.
This is where RePEc has something to contribute. Indeed, the citation analysis we do is focused on working papers, because this is what we can access the most readily. Thus, you do not need to wait for the citing paper to be published. Furthermore, it takes about a month for a citation to be analyzed, which may be shorter than other citation services. If you consider the recently released discounted citation counts (citations in more recent papers count more), then this will tell you what is being cited frequently very recently. As the link indicates, these are not necessarily recent papers, though. But we can indeed sort this list to lindicate only the very recent papers (say, last three years).
I certainly agree with Richard’s point that technology won’t change social norms. Still, I am heartened that bepress has been able to reduce turnaround times by holding their referees to a quicker response with something of a compact with those who submit there. “Traditional” journals have done this too; for example the QJE reduced its “first response time” dramatically in the 1990s. It apparently used this to encourage the best submissions.
More broadly, social norms do change, and they’ve changed in this very area. Glen Ellison has two very interesting papers on this point: The Slowdown of the Economics Publishing Process and
Evolving Standards for Academic Publishing: A q-r Theory. Both of these were in the July, 2002 Journal of Political Economy (a top-ranked journal for the non-economists out there). The slowdown paper finds that the time from submission to final acceptance has dramatically increased since 1970 or so. Little of the increase is due to more complex papers or “first response time.” Instead, it seems to come from more extensive revisions.
He also finds that papers have gotten quite a bit longer in economics as well as in other fields in the last few decades. Perhaps the most surprising finding is that back then “revise and resubmit” was rare. Economic Inquiry seems to have recognized this and they let authors submit their papers to an up or down review.
My take-home message here is that not only do norms change, but with some effort by journal editors, they could change again. Or, like Economic Inquiry, they can address the long lags to publication in other innovative ways.
It is really exciting that RePEc offers near-real time citation counts of working papers; this is indeed a place where it can make a huge contribution. This is useful not just for “Top Items,” but for counting the few cites that many of us likely to get.
To help RePEc more accurately count cites, one could imagine permanent RePEc “handles” for papers (handles identify RePEc resources). In turn, we’d put these handles in our bibliographies for RePEc software to read for more accurate citation counts. I could be wrong, but I seem to recall that about 1/3 of papers in RePEc don’t have their citations extracted because of the technical difficulties of doing so. After all, papers are written for people, not computers (this is subset of a larger point made by proponents of the “Semantic Web” made about web pages and why searches can be so maddening). I understand that making RePEc handles permanent is non-trivial and getting economists to add another item to bibliographies is quite a hill to climb, but it may well be worth it if we really want to stress one of RePEc’s strengths.
It turns out that this and more is discussed in my paper that I mentioned above: Next Steps in the Information Infrastructure in Economics. More self-promotion I guess…
Bill,
thanks for your thoughts and the links. Just on one point where I happen to strongly disagree. You write:
“In a sense, the lag in the review process might be optimal. A publication of most any sort is valuable to the author and one in a leading journal of course has a very substantial return.”
This kind of rationing does not make sense to me. Long turnover will discourage outstanding as well as mediocre authors alike, so fewer submissions will occur across the quality spectrum. An easier and more efficient way for a journal to reduce submissions would be to return every other submission without even looking at it.
Cheers
Ekkehart
Ekkehart –
I pretty much agree that the delays have gotten “too long.” But, if there was no cost to submitting to elite journals, I suspect they would see substantially more submissions that might be costly to review. This is the basic point that Azar made.
But, along the lines of the argument you made, in his July 2007 paper,
Is Peer Review in Decline? Glenn Ellison finds evidence that some elite authors are indeed by-passing peer reviewed outlets. He argues that this is due to the factors mentioned above: long times to publication and arduous revise and resubmit cycles. Certainly bepress, Economic Inquiry, and others are innovating around these issues. Still, mediocre authors seem to put up with the current system as they have fewer alternatives than elite ones. It will be interesting to see if other journals follow the lead of bepress and Economic Inquiry.
Bill
On “other journals”, I can only speak for Elsevier journals. There are three positive developments. (1) The switch from paper to electronic submissions has substantially cut the processing time — papers being mailed around the world, papers sitting in offices waiting for the editor to return from a conferences, and papers getting lost. (2) Acceptance to pre-publication on the web is 14 days or less. (3) Galley-proofed to print is being capped at 6 months.
Mind you, all this is driven by Elsevier’s medical journals. Economics journals are free-riders on these technological advances.
The crucial delay is the time it takes to review. As an editor, one has limited control — although some editors use that, and others do not.
[…] is not a traditional blog, as it mainly serves as a portal to other blogs. As was discussed (1, 2) a while ago on the RePEc blog, there is currently little in the Economics profession in terms of […]
[…] discussion on this blog (1, 2) covered the issue of peer review of new research, in particular now that the Internet has become […]
[…] This is not a traditional blog, as it mainly serves as a portal to other blogs. As was discussed (1, 2) a while ago on the RePEc blog, there is currently little in the Economics profession in terms […]