Easing the life of referees

October 30, 2007

One of the more boring tasks a referee faces is to check the algebra of a submission that looks promising. This is often trivial to do but may be quite time consuming, and it requires concentration. Given programs like Mathematica, the task can be simplified by asking authors to give their calculations in machine readable code. This would render it easier to do such boring checks and would, at the same time, help authors to avoid mistakes.

Mathematica is, however, too expensive. Is there any freeware that could be used for such purposes? If so, RePEc could recommend using it as a standard. This would ease the refereeing process. Referees could concentrate on contents rather than being immersed in ultimately trivial calculations. Such a program must of course be able to do symbolic calculations, like forming derivatives or evaluating symbolic equation systems.

Ekkehart


New Peer Review Systems

October 29, 2007

The traditional peer review system is to submit an article to a journal, wait for the editor to get anonymous referee reports and then deliver a decision. We all have horror stories on how inefficient this system is, both in terms of time lost (and Economics seems particularly bad here) and in terms of the arbitrariness of the process. Yet despite all the complaints and the many announcements of its imminent demise in the age of the Internet, this peer review system is still going strong.

What has been done to reform it? Editors have worked hard to reduce decision times, sometimes with success, but there seems to be a lot of habit in the slow response of referees (so editors claim). New models are tried, such as the bepress journals that provide quality ratings and thus avoid authors to submit repeatedly down the journal ladder, the new American Economic Journals that can “automatically” feed on the rejects from the AER, or Economic Inquiry that now asks referees to only provide an up or down vote, thus bypassing the revisions. While these are all important initiatives, they are after all only a variation of the original system.

We can think completely differently. Think of this blog. I rant on a topic, and then others can comment on it and openly declare whether this rant was valuable or not. Why not do this with academic work? An early attempt was done with WoPEc. This was the first RePEc service, similar to IDEAS and EconPapers today, which offered for some time on each paper’s abstract page a discussion section. Participation was minimal and there was very little value added (see an example, I could not find one that actually had comments). This aspect of WoPEc was finally abandoned. A second attempt was organized by SOLE (Society of Labor Economists), that would post every two weeks a new paper to discuss. Again, participation was small, and the project was finally abandoned.

The latest attempt is the Economics E-Journal, which allows registered users to rate and comment on discussion papers. Once the editors find that a paper has generated sufficient interest, it is promoted to the journal, where it can still be discussed. This initiative started this year, so the jury is still out whether it will be successful in the end. So far, it looks very promising.

From time to time, members of the RePEc team are approached and asked whether a discussion section could be added to our services. Given the past experience with WoPEc and the large monitoring costs involved, we are not enthusiastic. Of course if other volunteers are interested in working on this, we may think about it. But first we need to understand whether there is really a demand for this. Maybe RePEc is now too large for this and such initiative should be left to field specific initiatives (SOLE again?).


Baseball World Series

October 28, 2007

Many of our US based readers follow the baseball World Series between the Boston Red Sox and the Colorado Rockies. Major League teams typically have affiliated teams in lower leagues where prospective players try and dream to qualify for the big team. The Colorado Rockies have such a team in Asheville (North Carolina), and its roster includes Matthew Repec at third base…


Categorizing Authors

October 27, 2007

We are trying to find a way to categorize authors registered with RePEc into fields. There are two obvious ways to do so that we did not like. We went for a third.

Self-categorization at registration

This would allow authors, when they generate or update their profile at the RePEc Author Service, to declare in which field(s) they work in. We see two problems with that: 1) This is not implemented in the current service; 2) Self-categorization is not necessarily accurate, as authors may not make consistent choices.

Using JEL codes of works

Authors have works in their profiles that can help in categorizing them. One way to do so is to use the JEL codes. Given their number (over 900), you obviously do not want to use the full set of codes. But this is not the real problem. A major issue is that relatively few papers and articles are JEL-coded in RePEc (as of today, 109’085 of 543’566, or one fifth). Given the wealth of data, the small proportion is not that problematic. However, items are very inconsistently coded in the sense that some publishers do not use them at all, other put a large number of codes for each item, some put just the top level codes (in some cases the same codes to all papers in a series), some go with very fine codes. As authors tends to publish more with some publishers than others (think of working paper series), all sorts of biases can creep up. Also, these codes are typically self-declared, which can also be problematic.

Using NEP data

Our suggestion is to use data collected with NEP. This project catalogs new working papers by field, the results being announced through emails (subscribe for the report in your field if you have not done so yet). The cataloging is done by human editors help by a nifty expert system. Thus we do not have the problem of self-declaration. Currently, there are 79 active NEP reports, and they have dealt with over 90’000 papers which have been categorized about 260’000 times. Indeed, the same paper can appear in multiple reports. We think that the categorization of works is more consistently performed by NEP editors than publishers. Also, there is no self-categorizing problem. Finally, NEP reports correspond more closely to fields as they are used everyday: they may encompass several or only part of the top JEL codes. (By the way, if you think a field is not represented, volunteer to edit one. It is less work than you think)

Recent working papers of registered authors are disseminated through NEP, thus we can use this data to categorize authors. The subjective factor now how to define whether an author is a specialist in her field. Indeed, one may work in different fields, so there should certainly not be an expectation that all papers fit in the same field. And the NEP editor may also have missed some. In the current implementation, the following rule is applied: an author is considered a specialist in a particular field if, amongst all papers announced through NEP, at least 25% were announced in the relevant NEP report. She is also a specialist if at least 5 papers were announced in that list.

25%

Why 25%? Having a majority of the papers in a field would too high a hurdle for those who work in several fields. One should also factor in that some papers may have been missed by NEP editors.

5

Why 5? Say that one needs, in many cases, about that many papers to obtain tenure. You obtain tenure when you are considered to be a valuable researcher in a field.

Use of this data

How does the categorization pan out with these specifications? See the author list. To see how the fields of an author have been determined, go to the very bottom of her profile. Ultimately, we may use this data to rank authors within fields, and do so as well for institutions. We will discuss this later.

Our question to you

What do you think of the choice of 25% and 5? Please discuss this in the comment section, we truly value your input.


Welcome to the RePEc blog

October 25, 2007

The RePEc team is opening today this blog with several goals in mind.

  1. Give us the opportunity to explain how RePEc works and what we do.
  2. Discuss some of the policy decisions we need to take.
  3. Give you the opportunity to comment and give us feedback.
  4. Expand to a wider audience some of the discussions we have within the RePEc team.
  5. Give you the opportunity to participate in our exciting project in whatever capacity you propose.
  6. Make people aware of some of the developments in the profession or in the Open Archive movement that are relevant to RePEc and its community.
  7. More generally, discuss the dissemination models for research in Economics and related fields.

It is not our intention to have a new post on a daily basis. We do not want this blog to become a burden as we scratch our heads finding new topics to write about. We want this blog to be useful for all parties. So watch this space on a regular basis and help RePEc improve!