Small changes to RePEc ranking statistics

July 1, 2014

With the next RePEc rankings and statistics to be released in a few days, a couple of small changes will be implemented.

First, we have adjusted the way the recursive and discounted recursive impact factors were used for weighting article pages, documents, and citations. A scaling factor was previously applied twice instead of once. The correction does not affect the rankings in any way, but it allows authors to replicate their scores. New scores are 21.4 times higher.

Second, we have started preventing republished articles from counting in the impact factors of the republishing journal. Also, these articles are dropped from the list of top recent works.


New: 10-year rankings on RePEc

October 20, 2013

RePEc has been publishing rankings of various sorts for over a decade. While many of them can still be considered experimental due to limitations in the data, they have had an impact on the evaluations of institutions, economists and journals in quite a few instances. Gradually, these rankings have been expanded to cover more and more aspects of academic life, as well as slicing them by fields, geography, gender and age. These rankings have typically considered all publications listed in RePEc. This can be a disadvantage for younger economists and publication series (although there are criteria that discount citations by age, for example).

We are now introducing a new set of rankings that limit themselves to publications in the last 10 years. For example, to compute an impact factor for a journal, only articles published in the last 10 years are considered. For an economist, anything published over 10 years ago is dismissed (unless the article version falls within 10 years). The ranking page has links to all those new 10-year rankings.

A few caveats: As samples are smaller than for the general rankings, the 10-year rankings will be more volatile, and any measurement error will be larger. For this reason, the 10-year rankings are not computed for fields and geographies. Also, any criterion that is based on recursive factors will still need some time to stabilize as they have to go through several iterations for them to converge, and they will never fully converge, as new data keeps coming in and data will have to be dropped every year. Finally, we cannot count research from publishers who do not supply publication years.


Proposed changes to RePEc rankings up for vote

April 29, 2013

The RePEc rankings are a popular feature of RePEc. As we gather more information about the profession, we can also refine the criteria that are used for those rankings, as well as add more of them. With this post, we seek from our users their opinion about a few potential changes. For each proposed change a poll is attached, which we hope will help in deciding what to do.

Regarding citation counts

Citation counts are the basis for a series of criteria used in ranking authors and departments. Our citation project, CitEc, uses references from open access material, or those offered directly by publishers in their metadata, or from user contributions. But citations may also appear elsewhere and count be counted. One source is Wikipedia, which has a little less than 3000 references pointing to items in RePEc. Another is the RePEc Biblio, a curated bibliography of the most relevant research in an increasing number of topics (1100 references, but it is just starting). And a third one are blog post indexed on EconAcademics.org, the blog aggregator focusing on economic research (8000 blog mentions so far). The question here is whether references listed on Wikipedia, the RePEc Biblio or EconAcademics.org should count as citations for ranking purposes. All these citations are already listed on the relevant IDEAS pages, but they have so far not counted towards any statistic. As usual, self-citations would not count, as much as possible. For this poll, we want to distinguish whether they should count for the ranking of economists and institutions on the one hand, and journals and series impact factors on the other hand, or both.

Regarding breadth of citations

Citation clubs bias how we try to measure the impact of someone’s research. We have already some criteria that try to measure the breadth of one citations, the number of citing authors, and the same weighted by the citing author’s rank. Another way to measure breadth is to measure how widely an author has been cited across fields. For this, we can measure in how many NEP fields an author is cited. To this effect, the analysis is of course limited to working papers that have been disseminated through NEP, which has currently 92 fields. Again, self-citations are excluded, and this new criterion would only apply to author rankings.

Doctoral advisors and programs

The RePEc Genealogy is a crowd-sourced project that gathers informations about who studied under whom, where and when. From this information, one could determine who is the best dissertation advisor and which doctoral programs are the strongest. Some preliminary rankings are already available in this regard, based on the data that has been gathered so far: 1869 advisors and 499 programs at the time of this writing. It is expected that these numbers would significantly increase once a ranking would be incorporated. Instead of the h-index currently computed, it would be calculated in the same way that institutional rankings are determined: by adding up the scores of the relevant student for each criterion, and ranking within each criterion and then aggregating criterion ranks. As one can expect that only a fraction of authors and institutions can be ranked this way, all the others would be ranked right after the last author or institution with a student. It is to be expected that this ranking would matter mostly for the top ranked authors and institutions. Note that a ranking of economists by graduating cohorts is going to be first released in early May.

Exclusion of extremes

For author and institution rankings, the various criteria are aggregated after excluding the best and worst criterion. This was introduced when there were about 25 criteria. Now, there are 33 for authors and 31 for institutions. Depending on the outcomes of the votes above, there may be even more. Thus one may want to exclude even more extreme scores to avoid taking outliers into account. How many extremes should be excluded on each end? The status quo is one.

Questions or concerns?

Feel free to post a comment below!


The Purpose of Journals

February 14, 2013

The editor of the Economics Bulletin, John Conley, has noted that many things go wrong with economic journals. Here is the abstract of his letter:

This letter calls attention a recent trend in economics publishing that seems to have slipped under the radar: large increases in submissions rates across a wide range of economics journals and steeply declining acceptance rates as a consequence. It is argued that this is bad for scholarly communication, bad for economics as a science, and imposes significant and wasteful costs on editors, referees. authors. and especially young people trying to establish themselves in the profession. It is further argued that the new “Big Deal” business model used by commercial publishers is primarily responsible for this situation. Finally it is argued that this presents a compelling reason to take advantage of new technologies to take control of certifying and distributing research away from commercial publishers and return it to scholarly community.

According to Conley,

The purpose of academic journals is to facilitate scholarly communication, filter for errors, and maintain the record of scientific advance.

This is, in my opinion, an idealized conception that does not reflect  the purpose of economic journals anymore. For economic research, the current economic journals are largely redundant. Conley himself notes this:

I seldom actually read journals  any more. I research topics using Google Scholar, RePEc, SSRN, and so on. It is inconvenient to sign up  with publishers to get tables of contents emailed to me or to login to my university’s library web portal to  search a journal issue by issue. I find it adds very little value over a more general search in any event. In  short, certification remains important to help people gain tenure and promotion and to get a sense of the  quality and centrality of individual scholars. However, neither certification by a journal, nor the collection  of similar papers within the bound or even electronic pages of a specific journal has very much meaning to  me when I am trying to understand where the debate in a subfield is at any given moment. As a result, I  was beginning to come to the conclusion that while they are irritating, commercial publishers are “mostly  harmless” to the research enterprise itself as publishing itself is becoming mostly irrelevant.

This coincides with my own observation: researchers don’t need journals. The main purpose of the journals is currently to ease the work of hiring committees. People publish in order to get a job. The wish to communicate new findings appears secondary in most cases.

Journals could serve worthier aims, however: they are needed by students, college teachers, and others who would like to obtain reliable information but can not as easily  separate the wheat from the chaff as active researchers can.

The important point Conley is making is, however, that the current journal system, although largely irrelevant for research, is nevertheless

bad for scholarly communication, bad for economics as a science, and imposes significant and wasteful costs on editors, referees. authors. and especially young people trying to establish themselves in the profession.

I fear, however, that John Conley’s suggestion to increase the number of journals would not improve the situation very much. As long as hiring committees use the reputation of journals, rather than the reputation of individuals,  a useful system of  “communication, filter for errors, and maintain the record of scientific advance” is practically blocked.

What can be done besides increasing the number of journals? Here some further suggestions.

1. Hiring committees can restrict the number of papers to be considered for judging an applicant to, say, three and disregard all other writings. This may help to reduce the number of publications and thereby reduce the need for further journals; it would also tilt the quality-quantity trade-off in favor of quality. (I think this has been a practice in Berkeley.)

2. Hiring committees that feel incompetent to judge the substantive quality of a contribution and have to resort to statistics of some sort may turn to citation counts of individual authors, as obtainable through  Google Scholar, Web of Science, or RePEc). This is a better solution than the the current practice of relying on the prestige of journals and would take account of the fact that  many papers in top journals are not so good, and medium-quality journals publish excellent articles.


Upcoming changes to rankings

March 26, 2012

The RePEc rankings are among the most visited pages of any RePEc service. While most of those rankings are still experimental and in some cases quite volatile (mostly due to incomplete citation data and not everyone being registered), they have proven to be extremely popular in the profession. While it is generally a bad idea to change definitions in the computation of rankings, sometimes there are good reasons to make adjustments or include more information. Two such changes will happen starting with the next release early April.

User-supplied weights for multiple affiliations

Authors with multiple affiliations currently have their scores distributed across affiliations for institutional rankings. The same applies across the geographic regions associated to each affiliation for the author rankings within regions. This is not going to change. But now that authors can set these weights themselves, author-supplied ones will be used if present. If not, the weights will continue to to be determined according to a formula that is supposed to determine the likelihood of a particular affiliation being the main one. In short: if you have multiple affiliations, set the weights yourself and the rankings will now take it into account.

Two new criteria for authors

The two rankings provided by CollEc, which measure how central an author is in Economics by looking at co-authorship networks, are going to be included for authors. As this concept is not well defined for institutions and regions, it will not be applied for those rankings. As the CollEc rankings change daily, a snapshot will be taken every month the day the new general rankings are produced. As not everyone can be ranked (see blog post for explanation), the unlisted will be ranked just below the last ranked author. These two new rankings will provide even more diversification to the ranking criteria.


Important upgrade for the RePEc Author Service

February 28, 2012

The RePEc Author Service just underwent a major upgrade. One important aspect of it is the treatment of multiple affiliations. There are also other changes of a more cosmetic nature that should help users avoid some common mistakes as well as some administrative and management improvements that a typical user would not notice.

Multiple affiliations

The most requested change was to allow authors with multiple affiliations to either select an order of importance for the affiliations or to select weights for each. This has become important for the rankings, as authors are allocated to their respective affiliations. Some weighting scheme had to be put in place, and the one in place so far was guessing the probability that a particular affiliation is the main one. The risk of error is of course large. With the revision, authors now have to chose the proper weights themselves. This now applies to any authors changing affiliations, and any new registrant. Anybody getting on the affiliation page also has to choose weights. These weights will be enforced for the March 2012 rankings released in early April 2012. Note that weights are public information.

Clearer claim choices

This change has actually been in place for a few weeks. When authors were offered choices of research items to claim as theirs, quite a few got confused and did the exact opposite of what they wanted to do: claim works of others and refuse their own. The form is now much clearer, with green and red backgrounds for the choices. Our observation so far has been that the error rate has been dramatically reduced.

Better action alerts

When an author logs in, he/she will immediately see whether research items or citations are waiting to be claimed. Bright red numbers are then present next to the relevant links. This features will gradually roll in as author accounts are refreshed.

Avoid duplicate entries

Upon registration, there is now a check to avoid someone to register again. Indeed, when moving or changing email address, it is much better to update an existing account than create a second, none the least because this preserves links throughout the RePEc system. Of course, it is still possible that a homonym is registering, so the check can be bypassed.

Name variations

Research items are linked to authors using the name variations they supply. During registration, the system makes suggestions that a registrant can amend (for example: Adam Smith; Smith, Adam; A. Smith; Smith A.). Unfortunately, it was noticed that a not insignificant share of users was deleting valid name variations, in particular to keep just one. The system was then unable to link them with appropriate results. It is now impossible for a new registrant to have less than four name variations.

Better treatment of deceased authors

Unfortunately, some authors pass away, and by now the list has become significant. Instead of leaving those accounts orphaned, they are now aggregated into a master account that can manage them. Indeed, a deceased author may still have new works added to RePEc, and have new citations discovered. Such an account continues to provide useful information and should not be deleted. By the way, this is an opportunity for volunteer to get involved in helping RePEc.

Better monitoring

There is now also better monitoring of the activity on the RePEc Author Service to prevent abuse and errors. This also releases more time for the administrator to deal with other tasks.

Problems?

Should anything appear amiss, do not hesitate to contact the administrator listed on the RePEc Author Service website.

PS: Secure HTTP

The site is now also served under secure http (https), to increase the security of transactions.


About author affiliations

April 26, 2011

When authors register at the RePEc Author Service, they are asked to provide their affiliation(s). Here, I want to clarify a few items about how affiliations are handled within RePEc. It is important that authors maintain their affiliations current, so that the proper institutions can get credit for their accomplishments.

What an affiliation is
An institution that pays the authors for his work. This may include current visiting positions, courtesy appointments and emeritus status. This is basically the institution(s) one would put under one’s name in a publication.

What an affiliation is not
Former place of study or work. Societies or associations. Consulting gigs. Banks where you hold an account (we have seen it all).

How to affiliate yourself in the system
There is a database of institutions derived from EDIRC that is used for affiliations. For universities, affiliations are listed at the department, center or institute level. Search in the database first, and only if you do not find your affiliation, suggest a new entry (90% of received suggestions are already in the database). Only affiliations from the database will count towards rankings, suggestions will not. Accepted suggestions will be converted.

About multiple affiliations
One can have multiple affiliations. But be aware that, for ranking purposes, each affiliation is attributed a share of the author’s scores. This means in particular that an author with affiliations in several countries will not count fully in each. We want to let authors decide what the shares should be, but until this is instituted, the temporary solution is a probabilistic calculation of what the main affiliation could be. The author’s email address, personal homepage and the number of affiliates at each institution are inputs in the formula. Authors can see their weights by following the ranking analysis link in their month email from RePEc. Affiliations not listed in EDIRC get a default value in the calculation.

Removing affiliations
To adjust affiliations, authors should log into the RePEc Author Service and click on “affiliations”. We leave authors authority on what they consider their proper affiliations and will not override their choices. The only exception is when some authority from an affiliated institution asks the author to be removed from the list.

Special cases
Deceased authors are considered to be unaffiliated. We welcome notifications and will adjust records in this respect. In particular, some of the authors with whom we have lost contact may have left us. Note that the latter do not count towards their affiliations either, the presumption being that the reason their email address is not valid any more is that they have changed employment.
Authors with write-in affiliation(s) are ranked in a country if it can be guessed from the URL of the affiliation (country domain), and if not from their email address.


RePEc rankings: adjustments and novelties

March 26, 2010

Given the increased interest in the various RePEc rankings, as well as thanks to comments made by users, I have made various adjustments over the last months. Some are barely perceptible, while some give some new features. Here is a short list.


Aggregate ranking for series and journals
Thanks to popular demand, there is now a ranking that aggregates the four impact factors, the h-index, download counts and abstract views. This ranking is available for journals, working paper series and all series.
Linked series
By linked series, I mean the situation where a series ceased to exist and continues under a new name. This happens in particular for journals that switch publishers, or journals that merge. Various statistics are now merged for ranking purposes. Thus, there is now a unique impact factor for the various series or journals. This was more complex to pursue for the h-index, and obviously all affected series gained from that. However, this increased the scope of self-citations that could be removed for the impact factor calculations.
Note to RePEc archive maintainers: you can link your series or journals to others by adding a new field in your series templates, Followup: or Successor:, followed by the handle of the other series.
Expanded listings
Following a poll a few months ago, the portion of the rankings that is public has been expanded. This will be visible with the ranking released early next month. The ranking of authors and institutions within countries or US states passes from the top 20% to the top 25%, within fields from the top 5% to 10%. The big ones, the top 5% authors and institutions still includes the top 5% in much detail, but now also the next 5% in five 1% “bins.”
US Economics departments
While there is already a ranking of Economics departments, one of the most frequent requests is to have one specifically for the United States. There will be one starting with next month’s update. Link.
Lost authors
We sometimes lose track of some authors when their monthly messages bounce back. Typically this is because they have moved or died. In both cases, they should not be counting towards the ranking of the institutions they are affiliated with. For a few months now, they have not. By the way, you can help rectify their status or their address by alerting us. The list of lost authors is here. See also the known deceased authors.
Peer authors
Registered authors receive every month an email with an analysis of their rankings. This now also includes a list of about 20 peers that are similarly ranked.
Errata
There is no reason an erratum or a correction should count as an additional publication for an author. We now try to drop them from ranking considerations and also to link them to the original article.


Poll on internal disclosure of author statistics

January 14, 2010

The statistics compiled by RePEc are used for various rankings, for example for authors. While we still consider these statistics to be experimental, in particular those pertaining to citations, these numbers are increasingly used for evaluation purposes. We value the privacy of authors and only disclose the statistics for the top ranked ones. Authors get each month a link to their statistics, a link with a code valid only for a month. This avoids a link that may have been disclosed once to be visible forever.

We get more a more requests from department heads to obtain the data for members of their department. Our typical response is to have them ask directly the members of their department to forward them their monthly RePEc emails. But of course, we could also provide directly all the relevant information. The purpose of this poll is to see whether participants in RePEc would favor such a disclosure. The conditions would be:


  1. The request must come directly from the head of the relevant department, by email to Christian Zimmermann
  2. The request contains a link allowing to verify that this person is indeed head.
  3. Statistics would be disclosed only for the members of the department who are currently affiliated with the department, as indicated in their RePEc profiles.
  4. Those with invalid email addresses would be excluded from the analysis, under the presumption that their affiliation may not be current.
  5. The department head would be provided with a link containing the analysis, a link that expires with the next monthly update of the rankings.
  6. The department head needs to be a registered author.
  7. A department would receive at most one disclosure a year.
  8. We reserve the right to refuse disclosing statistics.
  9. By department, we mean any unit with a separate entry in EDIRC.

If you have an opinion about thus please vote below and/or offer a comment. Given the nature of the question, we would require significant more than a majority (two thirds) to offer this service. The poll closes on February 21, 2010.

Update (22/2/2010): This poll is now closed. There was little interest, with only 89 votes, which fell 52-37 in favor of the disclosure. With an approval rate of 58%, it fell short of what I consider a substantial majority for an implementation. Thus, there will be no disclosure to department heads.


Polls on ranking disclosures

October 15, 2009

Rankings have become an important part of RePEc and we regularly get request about non-published rankings. Indeed, depending on the ranking in question, only the top 5%, 10% or 20% among authors or institutions are displayed, depending on the geographic or field aggregation. Given the insistence of some requests, I am now considering whether RePEc rankings should be disclosed in a more extensive fashion. Before making any changes, I am seeking the opinion of users.

But first, let me expose the reasons of the limited disclosure so far. Our interest is to have as many institutions and people participate in RePEc, and keep their data there current. Rankings provide the right incentives for this. Thus RePEc participation is our focus, and rankings are an accessory (and we still consider them to be experimental, as the data is still far from complete). We know, however, that at least some people do not like their poor rankings exposed and would thus remove their registration in RePEc if this were exposed. Thus, too extensive ranking disclosure would defeat their purpose. But I have no idea how widespread this would be. The second reason for limited disclosure is that rankings become less reliable as one goes further down the list. Consider, for example, that 28% of all authors have no recorded citation. Third, full disclosure will create a lot of large files and tables. We have about 22000 authors and 4500 institutions to rank…

The following polls are not binding. There results will help to define what users want. Feel free to discuss aspects that go beyond the options of the polls in the comment section (of this post, not of the individual polls). I will then decide what to do. For both author and institution rankings, the options are: 1) keep things as is, 2) disclose all the way to the top half, 3) keep things as is, but provide rankings for the following one in clusters. For example, rank the top 5% as now, then have a list of the top 6-10%, another for the top 10-15%. 4) Provide full rankings. Polls will be open until November 21, 2009.

Update: Polls are now closed. A post soon will discuss results as well as various adjustments to rankings.




Follow

Get every new post delivered to your Inbox.

Join 198 other followers