More on peer review and blogging

November 15, 2007

A community of research bloggers tired of being confused with “news, politics, family, bagpipes, and so on” blogs has started its own blog at BPR3 (Bloggers for Peer-Reviewed Research Reporting). The idea is to encourage peer review bloggers to certify themselves and use an icon on their blog. We discussed earlier whether such blogs would be appropriate in Economics, along with some past experiences. Other fields seem to have some active research blogs. Do we have any blog in Economics that would qualify?

Inside HigherEd has two articles on academic blogging, one discussing how painful it is, the other how great it is. Both authors are graduate students, but they can still offer interesting perspectives to more seasoned researchers tempted by academic blogging. A few excerpts:

Over the past three years, I’ve learned what it’s like to write in a way most academics never have: namely, for an audience. If this seems like a simple point, that’s because it is. Nor is it one of those profoundly simple points, either: it’s straight simple. When a blogger sits down to slave on her dissertation, article, or book, she doesn’t turn her back on the public sphere. Because in the end, the public sphere is us.

I’m talking about the communities we currently have, only five years in the future, when we’re scattered around the country, unable to communicate face-to-face, but still connected, still intellectually intimate, because we’ll still regularly be engaged with each other’s thoughts. But I’m not only talking about us. There’s no reason our community needs to consist solely of people we knew in grad school. Why not write for people who don’t already how you think about everything? Why not force yourself to articulate your points in such a way that strangers could come to know your thought as intimately as your friends from grad school do?

More than formatting issues, however, I think that everyone needs to realize that having a productive conversation in an online format is very hard work, which is why it happens so rarely. Many bloggers can point out online conversations in which they were pushed to think in a new direction or got genuinely valuable feedback on a question, but as with all human endeavors, there is a high percentage of dross to go along with the occasional gold. Policing comments is a difficult job, and efforts to keep conversations on-topic or ensure that contributors have some substantial knowledge to share will often cause resentment in light of the “democratic” leanings of online communities. All this is on top of the obvious problems with online interaction as opposed to in-person conversations.

As more and more academic resources become available online, hopefully academic blogs will begin to fill a role analogous to the political blogs that link to and comment on particular news stories — that is, bringing new scholarly research to the attention of an interdisciplinary audience. I hope that events like this will help to push more journals toward open-access electronic formats. Failing that, however, academic blogs seem to me to be best-suited as a social outlet for academics who would otherwise feel isolated, creating camaraderie and supplementing the social aspects of disciplinary conferences.

There is also a discussion on this topic on the blog of the Association of College & Research Libraries.

The RePEc blog does not consider itself to be part of the research blog community, indeed our focus is not research but the dissemination of research. Hence, we are interested in understanding the feasibility and the interest in research blogs in Economics. And if any research blogs appear (or already exist) in Economics, RePEc would be more than happy to feature them on this blog and possibly elsewhere.

3 Comments | Dissemination of research in Economics | Permalink
Posted by Christian Zimmermann

Ranking Institutions Within Fields

November 9, 2007

In previous posts, we discussed how to categorize authors by field and then how to rank them within fields. These discussions are still open and I can still be convinced to change the procedure. Today, I would like to make a proposal regarding the ranking of institutions within fields.

We have several options regarding how to count people from an institution for a specific field. In the examples, I assume that author A has 10, or 50%, of his works in macroeconomics, B 3 or 30%, C 10 or 20%, D 1 or 10% and A 0%. I also assume that the 25% and 5 rule applies, as discussed in the post on categorizing authors. Thus, under these rules, authors A, B, and C are considered macroeconomists. The options are:

  1. Count fully all authors considered within the field: A+B+C.
  2. Count all authors considered within the field proportionally to their involvement in the field: 0.5A+0.3B+0.2C.
  3. Count all authors, irrespective whether they qualify as specialists, proportionally to their involvement in the field: 0.5A+0.3B+0.2C+0.1D.

My preference is for option 3. The reasons are the following. The first option fails to properly differentiate between strong specialists and marginal ones. This may also have been a concern when ranking authors, but the issue there was the high volatility of weights at the author level. At the institutional level, this is less of a concern as several authors are aggregated. Note also that only the top 20% institutions will be listed anyway, thus I expect all of them to have several authors within the field. Thus I prefer option 3 over option 1. Then I prefer option 3 over option 2 because it allows to count for authors that may not be specialists but still may contribute to enriching the field. Think for example when a prospective graduate student compares programs. While she cares about the specialists of her field of interest, she may also care about those faculty on the fringes of the field.

Of course I am open to suggestions and can still be swayed to to change my opinion. I plan on implementing this for next month.

1 Comment | RePEc rankings | Permalink
Posted by Christian Zimmermann

Further Thoughts on “New Peer Review Systems”

November 5, 2007

Several thoughts on various points raised in New Peer Review Systems and the comments that followed.

  • In a sense, the lag in the review process might be optimal. A publication of most any sort is valuable to the author and one in a leading journal of course has a very substantial return. Journals thus have a good reason to deter papers that aren’t at all appropriate; this was pointed out by Ofer Azar, “The Slowdown in First-Response Times of Economics Journals: Can It Be Beneficial?,” Economic Inquiry, 2007, 45 (1). The constraint here would seem to be editor’s and referees’ time. In economics, the most common cost that journals impose on authors is a lengthy review process. I’d hazard a guess that bepress gets around this by their ranking of papers into different tiers; they don’t have to deter less than stellar papers as they’ll likely get a home there. This is combined with their system where authors who submit there agree to review two papers quickly (a nice example of a virtuous cycle). Another way to speed up the referee process is a system where any reader can submit comments on a paper. But, as Christian points out, this doesn’t seem to attract many comments. It turns out I’ve looked a bit at this and found 5 journals that have tried a reader rating system and none have attracted a sufficient number of comments to make it fly. From here, one option is something that ranks papers after they’ve been out, such as citations. Paul Ginsparg has some thoughts on one approach, as does Hal Varian (now the chief economist at Google). But, these might take years to generate sufficient data to render a judgment on a paper. I think many of us want something that is quicker.

    Another possibility is something like Faculty of 1000 in biology and medicine where a level of review beyond journals takes place. I very much like the summaries in their sample web pages; you don’t see that in economics. One could imagine it working on top of our working paper culture. But, I wonder if some of their success comes from the grant culture in these fields as this is a fee-base service that seems to pay the reviewers? How might one set up something similar in economics with our working paper culture? Journals would likely see it as preempting their role.

  • I agree with Preston that perhaps the most interesting part of Economics E-Journal is the open review system (all can read reviews) and the feature that allows authors to publicly respond to referee reports. Both would seem to give referees correct incentives. I would think that journals could implement this quickly and easily at low cost. Also, while deep thinking and working through a paper you are writing is extremely valuable, so is getting feedback and discussing a paper and the ideas in it. I have a first draft of a paper on these topics (see below) but in writing this blog entry I have developed some new insights. I wouldd add that prompt discussion is something that the Internet can aid for those of us without local colleagues in our fields.A very minor point on his post: if you count economics journals by the number in EconLit, there are about 1,240. In short, most any paper should certainly be published!
  • I certainly second Christian’s point about blogs and research. All the economics blogs I know of do not discuss serious research that much if at all. Much more common are discussions of current economic events and policy that members of the public find interesting. I don’t know of a one where someone might say, “Say, that paper by Sam Jones in Computational Economics” is interesting because the algorithm he used to calculate…” Also, debates by papers take years given the time to write one and respond. This seems rather silly given today’s technology; after all, journals in the their current form came about when information traveled at the speed of a horse or ship. Perhaps a blog is too quick for complete works, but I understand in law their use is leading to a relative decline in the importance of law journals. One example (first paper found in a Google search) is Guest Blogger: The Start of the Supreme Court’s 2007-08 Employment Discrimination Docket: Federal Express Corporation v. Holowecki. Yes, it discusses current events, but the topic is one that only specialists would seem to care about. You do not seem to see such in economics blogs that I am aware of.

Where does this leave RePEc? Well, I’m not really sure. It is hard to change norms in a field and I am not sure that RePEc could swing it. But, I do not think that a comment system on individual papers would get that much traction. Somehow you want to get the judgment of peers (for promotion, tenure, and annual raises) in a speedy manner. Those two criteria seem to be at odds with each other.
It turns out I have done some thinking on these issues; at the risk of self-promotion they can be found in Next Steps in the Information Infrastructure in Economics. Note that this draft was written for a conference of non-economists, so some parts will strike a very obvious cord to an economist’s ear. I have also have yet to incorporate some very useful comments that Christian kindly wrote. In a nice mix of blog and papers, further comments on the paper are greatly appreciated.

9 Comments | Dissemination of research in Economics, Economics Profession | Permalink
Posted by Bill Goffe

Ranking Authors Within Fields

November 4, 2007

As discussed earlier on this blog, we are trying to categorize authors by fields. The current proposal is to consider someone in a field if either 5 or 25% of her papers announced in NEP where announced in the relevant field report. These parameters are still open for debate, and the reader is welcome to weigh in. Once this is done, the next step is use this methodology to rank authors within fields. There are several ways in which this can be done.

  1. Take the unweighted list of authors in each field. This means the following: Take the list of authors as categorized by the parameters mentioned above. Just rank them within the list assuming the same weight to each.
  2. Do the same, but with weights. Those weights would correspond to the share of the field within each author’s work. So an author who has 4 of 12 papers announced in NEP announced in the NEP-MAC report would have one third of his scores count towards the ranking among macroeconomists.
  3. Do the same, but not restrict to those have passed the threshold to qualify for a field (currently 5 papers or 25% of papers announced in NEP).

My preference would go for the first option. I do not think that the field weights are that precise to allow using them for authors. Also, the ranking of author in a field may drop with the other options for the sole reason that he has published a working paper in another field.

I intend to have the first field rankings available in a month, so voice your opinion before then.

4 Comments | RePEc rankings | Permalink
Posted by Christian Zimmermann

Seeking Volunteers

November 3, 2007

The RePEc project is entirely supported by volunteer effort. Indeed, no one is paid in any way for working on RePEc, in fact RePEc does not even have a budget. A core team is heavily involved, on spare time, and many others participate in smaller, but nonetheless important ways. We are always interested in renewing the blood in the team, so if you wish to help out and provide useful services to the profession, join the team in whatever capacity you can provide. Several of the tasks require familiarity with programming (in particular perl) and with Linux systems, but not all tasks. And like for everyone else on the team, this can be learned on the spot.

Watch this space as we announce calls for volunteers for specific projects. More generally, we are interested in anybody who could help provide some redundancy. Indeed, many projects rely currently on one person, and if this person were not able to help anymore, we would have to scramble. Thus, if you are particularly interested in one or the other aspect of RePEc, why not get involved?

We also want to start a few projects and are looking for people to help out there. Again, watch this space for announcements. Or make your availability known either in the comment section or through repec@repec.org.

Leave a Comment » | RePEc volunteers | Tagged: | Permalink
Posted by Christian Zimmermann

RePEc in October 2007

November 2, 2007

Every month, a short summary of what happened with RePEc is sent to the RePEc-announce mailing list. I will also put that message, slightly adapted, on this blog.

Two new items this month: first we now have a blog, at https://blog.repec.org/ where we discuss issues relevant to RePEc and the Economics community in general. Second, we now attempt to classify authors by fields. See results at http://ideas.repec.org/i/e.html.

Coverage has tremendously increased this month fuelled by 15 archives and the old archives: over 34,000 new items are now listed. Traffic was also very high, setting records on several servers. Total traffic was 642,109 file downloads and 2,280,187 abstract views.

Quite obviously, we passed a few thresholds under these circumstances:

120,000,000 cumulated abstract views
400,000 items with download
300,000 articles listed
250,000 articles with download
90,000 distinct papers announced through NEP
75,000 papers with citations
17,500 NEP reports
1,750 chapters listed

Leave a Comment » | Workings of RePEc | Permalink
Posted by Christian Zimmermann

Easing the life of referees

October 30, 2007

One of the more boring tasks a referee faces is to check the algebra of a submission that looks promising. This is often trivial to do but may be quite time consuming, and it requires concentration. Given programs like Mathematica, the task can be simplified by asking authors to give their calculations in machine readable code. This would render it easier to do such boring checks and would, at the same time, help authors to avoid mistakes.

Mathematica is, however, too expensive. Is there any freeware that could be used for such purposes? If so, RePEc could recommend using it as a standard. This would ease the refereeing process. Referees could concentrate on contents rather than being immersed in ultimately trivial calculations. Such a program must of course be able to do symbolic calculations, like forming derivatives or evaluating symbolic equation systems.

Ekkehart

8 Comments | Dissemination of research in Economics | Tagged: | Permalink
Posted by Ekkehart Schlicht

New Peer Review Systems

October 29, 2007

The traditional peer review system is to submit an article to a journal, wait for the editor to get anonymous referee reports and then deliver a decision. We all have horror stories on how inefficient this system is, both in terms of time lost (and Economics seems particularly bad here) and in terms of the arbitrariness of the process. Yet despite all the complaints and the many announcements of its imminent demise in the age of the Internet, this peer review system is still going strong.

What has been done to reform it? Editors have worked hard to reduce decision times, sometimes with success, but there seems to be a lot of habit in the slow response of referees (so editors claim). New models are tried, such as the bepress journals that provide quality ratings and thus avoid authors to submit repeatedly down the journal ladder, the new American Economic Journals that can “automatically” feed on the rejects from the AER, or Economic Inquiry that now asks referees to only provide an up or down vote, thus bypassing the revisions. While these are all important initiatives, they are after all only a variation of the original system.

We can think completely differently. Think of this blog. I rant on a topic, and then others can comment on it and openly declare whether this rant was valuable or not. Why not do this with academic work? An early attempt was done with WoPEc. This was the first RePEc service, similar to IDEAS and EconPapers today, which offered for some time on each paper’s abstract page a discussion section. Participation was minimal and there was very little value added (see an example, I could not find one that actually had comments). This aspect of WoPEc was finally abandoned. A second attempt was organized by SOLE (Society of Labor Economists), that would post every two weeks a new paper to discuss. Again, participation was small, and the project was finally abandoned.

The latest attempt is the Economics E-Journal, which allows registered users to rate and comment on discussion papers. Once the editors find that a paper has generated sufficient interest, it is promoted to the journal, where it can still be discussed. This initiative started this year, so the jury is still out whether it will be successful in the end. So far, it looks very promising.

From time to time, members of the RePEc team are approached and asked whether a discussion section could be added to our services. Given the past experience with WoPEc and the large monitoring costs involved, we are not enthusiastic. Of course if other volunteers are interested in working on this, we may think about it. But first we need to understand whether there is really a demand for this. Maybe RePEc is now too large for this and such initiative should be left to field specific initiatives (SOLE again?).

10 Comments | Dissemination of research in Economics, Economics Profession | Tagged: , , , | Permalink
Posted by Christian Zimmermann

Baseball World Series

October 28, 2007

Many of our US based readers follow the baseball World Series between the Boston Red Sox and the Colorado Rockies. Major League teams typically have affiliated teams in lower leagues where prospective players try and dream to qualify for the big team. The Colorado Rockies have such a team in Asheville (North Carolina), and its roster includes Matthew Repec at third base…

Leave a Comment » | Going off on a tangent | Permalink
Posted by Christian Zimmermann

Categorizing Authors

October 27, 2007

We are trying to find a way to categorize authors registered with RePEc into fields. There are two obvious ways to do so that we did not like. We went for a third.

Self-categorization at registration

This would allow authors, when they generate or update their profile at the RePEc Author Service, to declare in which field(s) they work in. We see two problems with that: 1) This is not implemented in the current service; 2) Self-categorization is not necessarily accurate, as authors may not make consistent choices.

Using JEL codes of works

Authors have works in their profiles that can help in categorizing them. One way to do so is to use the JEL codes. Given their number (over 900), you obviously do not want to use the full set of codes. But this is not the real problem. A major issue is that relatively few papers and articles are JEL-coded in RePEc (as of today, 109’085 of 543’566, or one fifth). Given the wealth of data, the small proportion is not that problematic. However, items are very inconsistently coded in the sense that some publishers do not use them at all, other put a large number of codes for each item, some put just the top level codes (in some cases the same codes to all papers in a series), some go with very fine codes. As authors tends to publish more with some publishers than others (think of working paper series), all sorts of biases can creep up. Also, these codes are typically self-declared, which can also be problematic.

Using NEP data

Our suggestion is to use data collected with NEP. This project catalogs new working papers by field, the results being announced through emails (subscribe for the report in your field if you have not done so yet). The cataloging is done by human editors help by a nifty expert system. Thus we do not have the problem of self-declaration. Currently, there are 79 active NEP reports, and they have dealt with over 90’000 papers which have been categorized about 260’000 times. Indeed, the same paper can appear in multiple reports. We think that the categorization of works is more consistently performed by NEP editors than publishers. Also, there is no self-categorizing problem. Finally, NEP reports correspond more closely to fields as they are used everyday: they may encompass several or only part of the top JEL codes. (By the way, if you think a field is not represented, volunteer to edit one. It is less work than you think)

Recent working papers of registered authors are disseminated through NEP, thus we can use this data to categorize authors. The subjective factor now how to define whether an author is a specialist in her field. Indeed, one may work in different fields, so there should certainly not be an expectation that all papers fit in the same field. And the NEP editor may also have missed some. In the current implementation, the following rule is applied: an author is considered a specialist in a particular field if, amongst all papers announced through NEP, at least 25% were announced in the relevant NEP report. She is also a specialist if at least 5 papers were announced in that list.

25%

Why 25%? Having a majority of the papers in a field would too high a hurdle for those who work in several fields. One should also factor in that some papers may have been missed by NEP editors.

5

Why 5? Say that one needs, in many cases, about that many papers to obtain tenure. You obtain tenure when you are considered to be a valuable researcher in a field.

Use of this data

How does the categorization pan out with these specifications? See the author list. To see how the fields of an author have been determined, go to the very bottom of her profile. Ultimately, we may use this data to rank authors within fields, and do so as well for institutions. We will discuss this later.

Our question to you

What do you think of the choice of 25% and 5? Please discuss this in the comment section, we truly value your input.

5 Comments | RePEc features | Tagged: , | Permalink
Posted by Christian Zimmermann

« Previous Entries
Next Entries »