This is a guest post by Jan H. Höffler
The ReplicationWiki currently offers a database of 4,484 studies from the social sciences for which empirical methods were used. It lists which of the studies have data and code available online. In cases where replications are known they are classified by their type and results.
The topic of replication has become more and more prominent in the scholarly discourse in recent years. Yet, much needs to be done to make the availability of code and data more mainstream. To highlight how much work still lies ahead, even recent publications on the topic of replication in leading journals are not replicable and contain major flaws. For example, the authors of a study calling to make replication the norm that was published in Nature do not make their replication material available, ignoring the rules on data availability of the journal and the sponsor, the Berkeley Initiative for Transparency in the Social Sciences. Or, a study published in Research Policy came to the conclusion that work published in the top 5 economics general interest journals are less likely to attract replications published in leading journals, although the authors’ own data shows exactly the opposite.
So how can we get more replications to improve on the state of economics and discuss cases like the ones listed above? One important way is to include replication in the education of economists as was suggested by Daniel Hamermesh in his 2007 article on replication in the Canadian Journal of Economics. The ReplicationWiki followed this approach by setting up a teaching initiative that was presented, among others, at the Research Transparency Forum of the Berkeley Initiative for Transparency in the Social Sciences (BITSS) and Annual Meetings of the American Economic Association (2014, 2016). Seminars on replication were held at universities in Germany, Canada, China, and Switzerland and at a workshop in San Francisco with the Institute for New Economic Thinking Young Scholars Initiative, BITSS, and the Project Teaching Integrity in Empirical Research.
The advantage of the wiki approach lies especially in the fact that users can contribute to it without publishing a journal article. A working paper series was started for this purpose. Forum and blog posts can also be included as long as they have a verifiable author and make a contribution regarding the replicability of a published empirical study. On the studies’ discussion pages even very short comments can help other users like “To make the code work I had to add … at line …” or “The data has been moved to the following URL: …”.
For instructors, the wiki can help to identify examples for coursework as it allows searching for studies for which data and code are available, for which software was used that is accessible to the students, and for which a method was used that they should learn about. With the help of JEL codes and keywords preferred topics can also be searched for. Depending on the location of the students, it can also be motivating for them to see if research is available based on data from their home country (click here for an example). If it is not, they may be encouraged to compare results based on data from their country or region with the existing published research. For the students it can be an additional motivation if they can easily share their results with the research community via the ReplicationWiki.
The ReplicationWiki was described in more detail in a journal article. In the 2017 American Economic Review Papers and Proceedings an overview was given of economics journals’ data policies as well as of the distribution of the use of different software packages and of the geographical origin of the data used. In that article, some evidence was also presented that indicates that studies for which replication material is made available may attract more citations. This should be seen as a motivation for authors of empirical work who are willing to share their material to point this out by adding this information to the wiki. The ReplicationWiki has recently added a number of additional features. Now there are overviews of the methods, data sources and software used in the studies. In addition to replications the wiki now also provides information about corrections that have been published and whether studies have been retracted. Complex searches are now possible with a more user-friendly interface.
Initially the wiki covered studies mainly published in the Journal of Applied Econometrics, which already started an online data archive in 1995, the Journal of Political Economy, the American Economic Review and the four American Economic Journals. Now it covers studies published in 231 journals, 36 working paper series & blogs and 27 books. It lists 652 replications, 23 corrections and 14 retractions. As the wiki has been cited from a number of neighboring fields as an example to follow, it is becoming a hub for all social sciences. There have already been contributions in particular from political science and sociology.
The ReplicationWiki’s pages have been accessed more than 6.6 million times so far. It has been mentioned numerous times in the media, and more than 260 users from around the world have registered. As a wiki, it lives off the contributions of its users. We hope to encourage more users to contribute to this tool, or simply use it. In particular, one site feature that could become more valuable with higher participation is the ability to vote which studies should be replicated.
In July 2014, a cooperation with RePEc was started via a link exchange. For studies listed in the ReplicationWiki a link appears in the IDEAS section “Related works & more” under “Lists” like in this case, and on the authors’ pages under “Citations/Wikipedia mentions” like here.
Is your work listed? Check in and add it if not!
The RePEc Blog 