Law and Method, October 2016

Introduction Special Issue Stumbling Blocks in Empirical Legal Research

Gijs van Dijck en Sanne Taekema

Empirical research is common in fields such as criminology, victimology, law and psychology, and public administration. It has become increasingly popular in other law-related fields as well, particularly in the United States. Studies have observed that almost half of the 1998 to 2008 US law review articles includes some empirical content¹x Shari Seidman Diamond and Pam Mueller, ‘Empirical Legal Scholarship in Law Reviews’ 6 Annual Review of Law and Social Science 581, 587 (5.7% being original research). and that, compared to other types of research such as doctrinal research and law and economics research, the number of empirical publications in legal journals in the United States has steadily and substantially increased.²x Michael Heise, ‘Analysis of Empirical Legal Scholarship Production, 1990–2009’ U Ill L Rev 1739, 1745 Richard H. McAdams and Thomas S. Ulen, ‘Introduction to the Symposium on Empirical and Experimental Methods in Law’ University of Illinois Law Review Michael Heise, ‘The Past, Present, And Future of Empirical Legal Scholarship: Judicial Decision Making and the New Empiricism’ University of Illinois Law Review 819. This particularly applies to studies that use quantitative methods.³x Robert C. Ellickson, ‘Trends in Legal Scholarship: a Statistical Study’ 81 Journal of Legal Studies 517, 528. See also Tracey E. George, ‘An Empirical Study of Empirical Legal Scholarship: The Top Law Schools’ 81 Indiana Law Journal 141, 147 (making an estimation for the 2000–2006 period, based on the results of the Ellickson study).
Given the rise of empirical legal scholarship, it comes as no surprise that articles have started to appear on how empirical and legal research is or should be conducted. In 2002, for example, a critical review article by Epstein and King discussed and evaluated current empirical legal research.⁴x Lee Epstein and Gary King, ‘The Rules of Inference’ 69 University of Chicago Law Review 1. They observed that law reviews lack a tradition of publishing articles exclusively devoted to the way in which empirical research is conducted or should be conducted. In their article, they explored whether empirical legal research follows the methodological rules (‘rules of inference’) followed in the natural and social sciences.
Epstein and King (2002) examined 231 American law journal articles that were published in the 1990 to 2000 period that contained the term ‘empirical’.⁵x Added with empirical studies that were published in top journals in the 1995–2000 period, the fifty most cited articles, empirical studies that they came across in other ways, and empirical studies from four peer reviewed journals. Basically, all studies they examined contained some kind of methodological flaw. Their conclusion is not encouraging: ‘the current state of empirical legal scholarship is deeply flawed’.⁶x Epstein and King 6, 6. The article evoked criticism. It was pointed out that ‘rules of inference’ in the social sciences are not always clear and sometimes disputed and that the methodological quality of the Epstein and King publication itself was flawed. See, for example, Frank Cross, Michael Heise, and Gregory C. Sisk, ‘Above the Rules: A Response to Epstein and King’ 69 University of Chicago Law Review 135, 135, Jack Goldsmith and Adrian Vermeule, ‘Exchange: Empirical Research and the Goals of Legal Scholarship’ 69 University of Chicago Law Review 153, 153. Zeiler & Hardcastle (2013) draw a similar conclusion as Epstein and King. After reviewing 197 empirical studies that examined the effect of tort reforms, damage caps in particular, on medical malpractice insurance premiums, they claimed that, from a methodological perspective, it is difficult to determine whether studies that find effects are better than or superior to studies that do not observe an effect.⁷x Kathryn Zeiler and Lorian Hardcastle, ‘Do Damages Caps Reduce Medical Malpractice Insurance Premiums?: A Systematic Review of Estimates and the Methods Used to Produce Them’ in Jennifer H. Arlen (ed), Research Handbook on the Economics of Torts (Edward Elgar Publishers 2013) <http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2179074>. According to them, most studies that examine the impact of tort reforms on malpractice premiums contain one or more methodological flaws, and one flaw can already produce inaccurate results.⁸x Ibid.
Publications like those of Epstein & King (2002) and Zeiler & Hardcastle (2013) at the very least suggest that empirical legal researchers can be, and actually are, confronted with serious pitfalls when conducting empirical legal research. The absence of a community of research groups and journals that critically review and further develop empirical legal research methodology also enlarges the need for evaluative studies and assessments of empirical legal research.
This special issue aims to contribute to the advancement of empirical legal research by critically reflecting on the methods that empirical legal researchers may use. The aim is not to criticize research, but to point out some of the obstacles and stumbling blocks that empirical legal researchers face, while offering suggestions on how to overcome these issues. In this special issue, a number of key methodological issues are discussed that relate to qualitative or quantitative empirical research in law.
The contributions cover various topics, research designs, and methods of data collection. Van den Bos and Hulst deal with some of the stumbling blocks, common mistakes, and controversial issues in experiments conducted in the legal domain. Melville discusses the issue of sensitive interviewing and the challenges that researchers face when conducting sensitive interviews. Webley explains how to conduct case studies and, in this respect, addresses frequent stumbling blocks encountered by researchers and discusses common problems that researchers encounter. Dhami and Belton consider the advantages of conducting multilevel analyses while identifying the problems of estimating single-level models. Finally, Van Velthoven explains the research process and highlights key issues that researchers face when setting up empirical legal research.
The articles have been published over the course of the 2016 volume of Law and Method. Some articles are general, whereas other contributions are technical or focus on one specific issue. This way, this special issue is relevant to both novices with no or limited knowledge of empirical legal research and researchers who have experience in conducting empirical legal research.

Noten

1 Shari Seidman Diamond and Pam Mueller, ‘Empirical Legal Scholarship in Law Reviews’ 6 Annual Review of Law and Social Science 581, 587 (5.7% being original research).
2 Michael Heise, ‘Analysis of Empirical Legal Scholarship Production, 1990–2009’ U Ill L Rev 1739, 1745 Richard H. McAdams and Thomas S. Ulen, ‘Introduction to the Symposium on Empirical and Experimental Methods in Law’ University of Illinois Law Review Michael Heise, ‘The Past, Present, And Future of Empirical Legal Scholarship: Judicial Decision Making and the New Empiricism’ University of Illinois Law Review 819.
3 Robert C. Ellickson, ‘Trends in Legal Scholarship: a Statistical Study’ 81 Journal of Legal Studies 517, 528. See also Tracey E. George, ‘An Empirical Study of Empirical Legal Scholarship: The Top Law Schools’ 81 Indiana Law Journal 141, 147 (making an estimation for the 2000–2006 period, based on the results of the Ellickson study).
4 Lee Epstein and Gary King, ‘The Rules of Inference’ 69 University of Chicago Law Review 1.
5 Added with empirical studies that were published in top journals in the 1995–2000 period, the fifty most cited articles, empirical studies that they came across in other ways, and empirical studies from four peer reviewed journals.
6 Epstein and King 6, 6. The article evoked criticism. It was pointed out that ‘rules of inference’ in the social sciences are not always clear and sometimes disputed and that the methodological quality of the Epstein and King publication itself was flawed. See, for example, Frank Cross, Michael Heise, and Gregory C. Sisk, ‘Above the Rules: A Response to Epstein and King’ 69 University of Chicago Law Review 135, 135, Jack Goldsmith and Adrian Vermeule, ‘Exchange: Empirical Research and the Goals of Legal Scholarship’ 69 University of Chicago Law Review 153, 153.
7 Kathryn Zeiler and Lorian Hardcastle, ‘Do Damages Caps Reduce Medical Malpractice Insurance Premiums?: A Systematic Review of Estimates and the Methods Used to Produce Them’ in Jennifer H. Arlen (ed), Research Handbook on the Economics of Torts (Edward Elgar Publishers 2013) <http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2179074>.
8 Ibid.