This article provides an inventory of the concrete moments of choice, problems and challenges that scholars are confronted with when using legislation, case law and literature commentaries within doctrinal legal inquiry. The inventory is achieved by conducting a systematic literature review and an explorative empirical investigation among legal scholars employed at the Tilburg Law School. This study shows that doctrinal legal inquiry is subjected to more and other moments of choice, problems and challenges surrounding the source-usage process than one might expect. The inventory offers (especially young) legal scholars an understanding of the complexity of doctrinal legal research and a checklist for identifying (possibly) problematic aspects of their own source-usage process.

• 1 Introduction and background

  Within doctrinal legal research, scholars generally do not explicate the procedures and protocols they followed to assure the soundness of the usage of sources. (Deane & Hutchinson 2010; Epstein & King 2002, p. 45; Tijssen 2009, p. 124-126). In applications for research funding, doctrinal legal scholars have – until recently – invariably sufficed by stating that their research method(s) comprise ‘studying legislation, case law and literature’ (Vranken 2005, p. 79). This does not mean that legal scholars are not confronted with concrete or practical choices, problems and challenges within their source-usage process (cf. Bloembergen 1988, p. 78; Herweijer 2003, p. 28; Korrel & Kamstra, 1991, p. 9; Vranken 2014, nr. 106). This lack of explicitness obscures the exact scope of the choices that were made, and the problems and challenges that were encountered (Vranken 2014, nr. 87). Literature on ‘legal research methods’ does not provide the necessary clarification; the nature of this body of literature is on the whole ‘highly theoretical’ and often ignores the complexity of concrete methodological problems encountered by scholars in daily research practice (Manderson & Mohr 2002, p. 159-161; Tjong Tjin Tai 2013, p. 201; Van Gestel, Micklitz & Maduro 2012, p. 20; Van Manen 2008, p. 1927). This contribution aims at filling this hiatus. It answers the question ‘ what concrete methodological moments of choice, challenges or problems legal scholars encounter when utilizing sources in doctrinal legal inquiry’. To answer this question, I will first clarify and operationalize the main concepts employed within this study: ‘doctrinal legal inquiry’, ‘source-usage’ and ‘method(ology)’ (paragraph 2). Subsequently, I will describe briefly the research methods and techniques (systematic literature review and explorative empirical investigation) followed, in order to answer the central question of this paper (paragraph 3). Next, I will discuss the results of this study (paragraph 4) after which I will draw my conclusions with regard to the research question (paragraph 5). The reader should be aware that this study is descriptive; it provides a first inventory of moments of choice, problems and challenges that legal scholars (might) encounter in doctrinal legal research. Methods and techniques that may be used to cope with the identified choices, problems and challenges fall outside of the scope of this article.¹xMore structured empirical research on quality criteria, challenges and research methods and techniques applicable to doctrinal legal inquiry is currently being conducted within my dissertation research project.

• 2 Analytical framework

  Before I present the inventory of concrete moments of choice, problems and challenges, I need to clarify the concepts introduced in the introduction. Firstly, the concept ‘doctrinal legal inquiry’ is in need of elucidation. In short, ‘doctrinal inquiry involves deriving answers to legal questions from the study of existing legal provisions, which are reflected in legislation and case law, as well as commentaries provided on these sources within literature, thereby assuming that these legal provisions are internally coherent and consistent’ (Vranken 2009, p. 546-547).²xI recognize that different definitions of ‘doctrinal legal inquiry’ circulate and that several nuances to the presented definition may exist (Vranken 2014). The focus on doctrinal legal inquiry implies that interdisciplinary legal inquiry, inquiry undertaken in the area of ‘legal theory’, ‘jurisprudence’, ’legal history’ and ‘comparative law’ research fall outside of the scope of this study (cf. Bloembergen 1988, p. 63; Bodig 2011, p. 10; Rubin 2001, p. 8677-8678; Van Hoecke 2010, p. 46).

  Secondly, what does the process of ‘source-usage’ exactly refer to in the present context? In an abstract sense the process of source-usage can be split into two – at least analytically – distinguishable phases of the research trajectory: (1) preparing the research, resulting in a justified and embedded research question (‘preparatory phase’) and (2) answering the research question(s) (‘executive phase’) (Fajans & Falk 2011; Giard 2013, p. 184). This study is concerned with the ‘executive phase’. This phase consists of four distinct sub-phases: (a) the phase of finding or locating the sources relevant to the inquiry, (b) the process of making a selection of these sources, (c) the appreciation (quality and utility judgment) and interpretation (distilling the relevant information from the selected sources and displaying it in a correct and useful way) of the selection of relevant sources, and (d) the synthesis: the effort to forge the derived information into an integrated whole (Aveyard 2010; Pendleton 2007, p. 161; Wren & Wren 1990).

  ‘Methodology’ and ‘method’ are used in many different ways (cf. Cryer et al. 2011, p. 14; Kamstra & Kunneman, 1988) but for me, understanding methodology is closely related to what we understand the field of enquiry to be. It guides our thinking and questioning within a particular scientific discipline (Cryer et al. 2011; Smits 2012, p. 111; Watkins & Burton 2013, p. 2). A research method in the context of the present study by contrast refers to ‘what a scholar actually does to enhance his or her knowledge, test his or her thesis or answer his or her research question’. The research method refers to the scientifically sound (’profound’, cf. Van Gestel & Vranken 2011, p. 910-911) way to acquire knowledge by the employment of coherent research techniques related to the collection and procession of data (Curry-Sumner et al. 2010, p. 22; Van Schaaijk 2011, p. 102).³xThe concepts ‘scientifically sound’ or ‘profound’ in relation to source-usage within doctrinal legal inquiry have not been clarified or operationalized within contemporary legal scholarship. Some insights are, however, provided by Tijssen (2009, p. 60-61) and Vranken (2014, nr. 34, 90). The scope of this contribution does not allow me to elaborate on such aspects. From this notion, methodological choices, problems and challenges refer to the choices, problems and challenges a scholar (may) encounter in his or her process of knowledge acquisition (through the use of sources) in a ‘scientifically sound’ or ‘profound’ way.

• 3 Methodological framework

  In order to answer the research question set out in the introductory paragraph, a systematic literature review was performed in which I collected and categorized the rather scarce and fragmented knowledge available. In conducting this review, I broadly followed the method and techniques developed by Aveyard (2010), Fink (2010) and Hart (2001).⁴xDocumentation on the exact techniques applied in order to conduct the systematic literature review can be obtained from the author on request. To validate (triangulate), complement and if necessary correct my findings, I conducted an explorative empirical investigation among legal scholars employed at Tilburg Law School.

  3.1 Justification of the expert-consultation

  Twenty scholars were asked to fill out a written questionnaire. As the intention was to identify the most complete possible overview of methodological challenges that an explorative study would allow me to achieve, I needed a collection (a sample) of doctrinal legal scholars that was as diverse as possible.⁵xWithin the literature this technique is known as ‘strategic sampling’. Therefore, I consulted scholars active in different legal fields and disciplines (private law, constitutional and administrative law, criminal law, and European and international law) (cf. Tijssen 2009, p. 82). Moreover, I approached scholars with different levels of experience – (applying) PhD-students and (associate) professors (cf. Osborne 2012, p. 3). As a result of the explorative character and limited time and means, I could unfortunately not involve a third factor: the research culture of the institute where the legal research takes place (cf. Cownie 2004; Tijssen 2009, p. 82). The questionnaire consisted of an overview of the challenges identified during the systematic literature review. Respondents were asked to react (acknowledge, refute, comment) on the identified challenges and were provided with the opportunity to formulate supplementary challenges. This resulted in the identification, modification and reworking of the categories of challenges that were the result of the literature review (Cownie 2004). The results have been anonymized.

  3.2 Limitations of this contribution

  The empirical part of this study is admittedly rather explorative and limited in scope. The main problem is the sample selection. While there is substantive reason to believe that opinions related to doctrinal legal methodology are widely diverging between different research institutes and cultures, only (a selection of) scholars employed at Tilburg Law School were consulted (cf. Cownie 2004). Nevertheless, the results of the empirical investigation supplement the systematic literature and add to the reliability and validity of my research findings. Although I suspect that my findings may be more broadly applicable, the results of my expert-consultation can of course not be generalized to all researchers at Tilburg Law School, let alone to doctrinal legal researchers of other faculties or research institutes.

• 4 Research results

  In this paragraph I will present the inventory of the concrete dilemma’s, problems and challenges identified. The challenges are categorized as follows: first, I will discuss challenges related to the collection (finding and selecting) of relevant sources (paragraph 4.1). Subsequently, I will address challenges that relate to the interpretation and synthesis of legal sources (paragraph 4.2). I must emphasize that not all identified challenges are applicable to every doctrinal legal research project or recognizable by all legal doctrinalists (cf. Cryer et al. 2011, p. 14; Van Hoecke 2011, p. 14; Korrel & Kamstra 1991, p. 17; Patterson 2011, p. 236; Smits 2012, p. 56; Toma 1999, p. 543-545; Wendt 2008, p. 86). In general, however, the respondents have indicated that they recognized the choices, problems, and challenges forwarded in the literature. They did, however, provided me with modifications, operationalizations and additions. I once more emphasize that possible methods and techniques that may be used to cope with the identified dilemma’s, problems and challenges (solutions) fall outside of the scope of this article.

  4.1 Collecting relevant sources; a challenging research activity

  1. Scholars have acknowledged that the process of finding relevant sources in contemporary doctrinal legal research may prove to be very difficult indeed (Cohen & Olson 2007, p. 13; Van Gestel et al. 2012, p. 1). The growing number of (digitalized) national and international results in an almost infinite sea of information (Davidson 2010, p. 563; Osborne 2012, p. 11; Posner 2008, p. 850; Verbeke, Vanhove & Hoekx 2009, p. 1788).⁶xAlso explicitly stated by a professor of Criminal Law. It is therefore often impossible to oversee all possible relevant sources related to a specific legal theme. A general inquiry into the area of tort law, that is, an inquiry that is not strictly limited in scope, for instance, will yield hundreds of possibly relevant scholarly publications and thousands of possibly relevant judicial decisions. Even if it would be possible to oversee and judge all (possibly) relevant material, it would still be physically impossible to study such an amount of materials within a single research project (Tijssen 2009, p. 186). Moreover, plurality within legal sources is also growing. Should the legal scholar for instance also search for relevant ‘soft law’, ‘notices’, ‘webpages’, ‘guidelines’, official and non-official ‘communication’, et cetera? Obviously, the ever-growing supply of information is a major challenge to the doctrinal legal scholar who strives for ‘soundness’ and ‘profundity’ in the use of legal sources.

  2. At the same time, the increasing amount and plurality of sources that are available to the legal doctrinalists make it more difficult to find those sources that are specifically relevant to a particular problem definition (cf. Cohen & Olson 2007, p. 13). One of the respondents said: ‘How do I find sources that really tell me something about what I am looking for? (...) I have to search through hundreds of exhaustive scholarly publications and judicial opinions in search of that tiny bit of information that could be of actual relevance for answering my problem definition.’⁷xProfessor of Private Law.

  3. The growing number of (possibly) relevant sources as a result of internationalization and Europeanization deserves specific attention (cf. Fisher et al. 2009, p. 239-243). By now, nearly all fields of the law are predominantly influenced by international and European legal instruments (Van Gerven & Lierman 2010; Sieburgh 2011, p. 2, 7).⁸xAccording to a professor of Criminal Law, ‘the legal world has been globalized and research-wise exploded, especially if one has knowledge of diverse languages and if one does not simply rely on national or even English literature’. It has therefore been suggested that source-usage within doctrinal legal inquiry is not ‘sound’ or ‘profound’ if international and European instruments are not included (Van Gestel & Micklitz 2011, p. 20-21; Vranken 2014, nr. 120). Van Gestel et al. (2007, p. 1453) even take it a step further; they argue that a doctrinal legal research project with regard to an area of the law that is influenced by international or European legal instruments, must also consider relevant sources (case law, legislation and literature commentaries) from abroad that provide information on these international and European instruments. Such sources are part of the present ‘state of the art’ that the researcher should identify and summarize in his presentation of the problem or question at hand. In incorporating information from abroad, the legal scholar is confronted with a variety of additional challenges: sources of what countries should be consulted? What foreign databases to consider? How can foreign sources be collected in a reliable way, taking into consideration the problems of language barriers and unfamiliarity with databases and institutional systems applicable to other jurisdictions?⁹xProfessor of Administrative and Constitutional Law. And if one is to consult foreign sources, is it not necessary to become aware of methodological issues addressed in the (also large) body of literature on the methods and methodology of comparative law? Is it even possible to ‘transpose' an argument expressed in a foreign source to the scholar’s own jurisdiction? To make sense of sources from other legal regimes one needs to become familiar with, again, a large amount of new materials, often unfamiliar both in form and substance, and often also referring to a a plurality of sources.

  4. When the scholar has decided where to look for information an additional, perhaps more trivial, challenge emerges. The scholar should somehow make sure that he does not overlook seminal or capital sources such as legislation, case law or literature commentaries (Parise 2010, p. 13).¹⁰xAlso explicitly addressed by a respondent, Ph.D. candidate of Private Law.This may seem obvious, but several scholars pointed out that this threat is real (Van den Bergh 2004, p. 1435; Van Gestel et al. 2007, para. 6.2). These authors have identified publications in which such reference to such seminal sources was missing.

  5. Subsequently, a challenge was identified that relates to the question how far legal doctrinalists must go in collecting data in order to arrive at ‘sound’ or ‘profound’ source-usage.¹¹xPh.D. candidate of Constitutional Law. Do they have to consult scholarly publications and case law that was published ten years ago? And what about publications that appeared fifty years ago? Do they have to turn to the more philosophical accounts of issues with regard to contract law if they want to evaluate the current rules on the termination of contracts? Of course, the exact answer will largely depend on the particularities of the problem definition. Nevertheless, regardless of the precise problem definition formulated, a scholar will often have to make some tough choices in this regard.

  6. In the past decades, digitalization has strongly influenced the process of collecting legal sources in doctrinal legal research. This results in the following additional challenges (6 to 9). Firstly, the process of collecting relevant sources is complicated by the fact that the growing amount of sources is stored in a growing number of different (physical and digital) databases. Skipping a particular database could lead to missing out on decisive information. The choice of databases the scholar consults may therefore strongly influence his or her research outcomes. Choices should be made with due care. Profound source-usage requires that the research results may not be dependent on coincidental or ad hoc decisions with regard to, for instance, the consulted databases (Korrel & Kamstra 1991, p. 18; Posner 1993, p. 70; Salter & Mason 2007, p. 6; Smits 2012, p. 37). Developing a search strategy among both digital and physical databases thus provides a prominent challenge for the legal scholar.

  7. Berring already provided a structured account on the problems and challenges of electronic research in 1997. Apart from the problems relating to access to digital legal sources, Berring (1997, p. 199) pointed out several serious problems with the electronic search possibilities and strategies. He questions whether electronic database users are aware of the pre-emptive decisions being made for them by the system they are using. According to Berring, ‘[t]he danger of the high-end [electronic] products is that each step in the research process that is carried out automatically by the front end system, is a step taken away from the purview of the researcher. Each decision that is built into the system makes the human who is doing the search one level further removed from the process’. Although Berring’s article was published in 1997, I believe that his concerns are still valid today. The recent dissertation of Van Opijnen (2014) argues that anno 2014 computer ‘algorithms’ that determine what and in what order search results appear do not function optimally. Moreover, it is likely that the majority of the legal scholarly community is totally unaware of what a computer does exactly when a user enters a query into an electronic database. To increase the profoundness of the source-usage process, such knowledge might in some cases be required.

  8. Apart from the technical details of the functioning of electronic databases, searching through the databases with keywords leads to additional problems for the legal doctrinalist. Peoples (2005, p. 663-664) argues that ‘[k]eyword searches provide a list of matches in the database but convey nothing about any relationship between the search results and other information in the database’. Databases fail to classify information in a meaningful way. Moreover, according to Peoples, ‘[o]nline results emphasize the factual elements of a case while broad legal concepts are neglected. The speed of searching electronic databases encourages researchers to rely on bits of rapidly retrieved text to support arguments that are often ill conceived and devoid of an enlightened, broad perspective’. Bintliff (1996, p. 346) added, that finding actual relevant sources on legal concepts or rules often proves to be very difficult: a search for information on legal concepts (such as ‘burden of proof’) ‘can be frustrating because of the many different ways these words are used in case law’. In order for legal scholars to arrive at valid, credible and profound research results, they have to carefully think about such issues before starting electronic searches. As a result of the multiplicity of ways in which legal scholars have cited case law, literature and regulation in the past, Van Opijnen (2014) not only convincingly demonstrates that searching for case law, literature and legislation with key words (for instance case numbers, dates, names, et cetera) is more difficult than one might aspect, but also provides us with several examples to illustrate this point.

  9. Partly overlapping with the last two issues, Van Gestel et al. (2012, p. 15) argued that while search engines and electronic databases become increasingly important, finding relevant sources will become more problematic. They state that ‘searching with key words for example will frequently fail to track legally relevant actions or behaviour’ and that ‘search engines understand word patterns, not (…) the essence of what you need to know’. Apart from the danger that highly relevant sources are overlooked because of semantic reasons, search engines ‘often come up with haystacks of information while you are looking for needles’, (cf. Curry-Sumner et al. 2010, p. 27). It may be clear that the challenges mentioned here do not do justice to the full range of problems that relate to digitalization and computerization. Nevertheless, it provides a promising starting point of getting acquainted with the complexity of computerized research.

  4.2 Interpreting and synthesizing legal sources

  1. The choices, problems and challenges related to the interpretation and synthesis of legal sources demand separate attention. For centuries and from a wide variety of perspectives, scholars have thought and written about the process of interpreting legal sources. On the one hand, (famous) legal philosophers and theorists have often covered the topic (such as Scholten and Wiarda in the Netherlands). More recently, the topic has also been approached from a ‘legal methodological’ perspective (for instance Salter & Mason 2007). It would reach too far and result only in repetition to address even the basics of this theoretical debate on legal reasoning and interpretation. Instead I will focus on more concrete or practical moments of choice, problems and challenges that were forwarded by legal scholars within the recent debate on ‘law and methodology’ and the scholars consulted within the empirical investigation performed within this dissertation.

  2. Consensus seems to exist by now that legislative enactments and case law are often deeply ambiguous texts that therefore raise the question of the possibility of objectivity in interpretation (Posner 2009, p. 273): ‘[O]ver legal sources hovers the specter of hopeless indeterminacy and rampant subjectivity.’ Only rarely, just one interpretation of a particular source is plausible (Vranken 2014, nr. 119). This has led Hutchinson and Duncan (2012, p. 110) to question whether it is even possible to plan the process of analysis and interpretation and to describe this process in an intelligible way. In this sense, one could argue that the whole process of interpretation provides a big challenge for the doctrinal legal scholar.¹²xStated by two respondents, professor with a Private Law expertise and professor with a Constitutional and Administrative Law expertise.

  3. Apart from this general challenge, more concrete challenges have been mentioned. Vranken (2014) for instance addressed the following issues: how does a scholar find out whether a particular source is of high quality or not? How critical must a scholar be with regard to the prevailing view presented within the sources consulted? Can a scholar build on the research done by others or should he or she check whether the prior research was done adequately (cf. Committee Performance-Indicators and Ranking 2007, p. 25)? When and why is a scholar allowed to refer primarily to his or her own prior work (‘self-plagiarism’, cf. Hesselink 2009, p. 4; Van Gestel et al. 2007)?

  4. Another, perhaps more trivial, challenge is that when interpreting, the scholar faces the challenge of not attaching a meaning to the consulted sources that the original author of the source did not intend to present in (or with) the source (Smith 2009, p. 222; Smits 2012; Tjong Tjin Tai 2013, p. 206). The fact that legal scholars often operate with a preordained result in mind (as discussed above) intensifies this challenge within doctrinal legal inquiry. Moreover, it can easily happen that a scholar accepts an incorrect interpretation of the law because he or she has consulted just one source (for instance because new case law emerged, or because the particular interpretation forms an exceptional rather than the prevailing view within the legal community) (Tjong Tjin Tai 2013, p. 205-206). In this context, the explorative study of Dawson (1991, p. 468) may be mentioned. While using a non-representative sample of her own students, Dawson found that the first interpretation of a legal source scholars come across is often the interpretation that remains favoured throughout a particular research project. The legal scholar is challenged to guard against making such ‘automatic inferences’, in order to arrive at ‘sound’ or ‘profound’ source-usage.

  5. After having considered eight scientific journal articles in 2007, Van Gestel et al. have highlighted the danger of conscious or unconscious selective behaviour in the (selection and) interpretation of sources in doctrinal legal inquiry. This challenge is widely recognized within the legal community. Many scholars have emphasized that once a particular approach has been chosen, one is naturally inclined to establish an affirmative answer (‘confirmation bias’ or ‘advocacy scholarship’) (Van Hoecke 2010, p. 35-36; Miller 1968, p. 291-294; Schlag 1991, p. 929; Spitzer 2008; Stolker 2004, p. 1415-1416; Vranken 2014, nr. 117; Wendt 2009, p. 785). This can have undesirable effects. Take the example of concealed argumentation (Hirsch Ballin 1988, p. 84; Vranken 2005). This occurs when a specific unclarity in the law is hidden for ethical or political reasons, or, conversely, when opinions are presented while a more thorough analysis is still possible. This particular challenge becomes increasingly acute when the scholar is personally interested in the analysis, or pursues a specific outcome (Barnhizer 2005, p. 21). Unfortunately, empirical research relating to the manifestations of such ‘advocacy scholarship’ within doctrinal legal inquiry is still missing (Van Gestel 2013, p. 65). Another undesirable effect is described by Vranken (2014, nr. 115). He argues that confirmation bias might mean that legal scholars may too easily regard certain case law or literature to be more relevant than others although this is not evident after critical reflection.

  6. In the same research project, Van Gestel et al. (2007) identified another challenge, that of establishing the mutual authority and cogency (weight or importance) of legal sources. The issue of the relative cogency of legal sources is profoundly contested within contemporary legal scholarship (McCrudden 2006, p. 635; Van Hoecke 2011, p. 12). Moreover, as Vranken (2011, p. 115) argues, ‘the concealed weighing of interest that is anchored within legal results [legislation, case law, and literature] and which results in relevant viewpoints is not set in stone. The validity of viewpoints needs to be tested over and over again, to determine whether their relative worth has changed, and to see whether new viewpoints or factors have arisen’. The scholar is therefore challenged to develop a mechanism for the determination of the mutual relation between different legal sources. The growing plurality of sources that are legally (‘soft law’, ‘notices’, ‘guidelines’, ‘unpublished papers’ and ‘web pages’) relevant makes the identification of this challenge even more urgent (Fisher et al. 2009, p. 235-239). It is unclear how exactly these sources should be treated within scholarly doctrinal legal inquiry. Questions relating to their stringency and (hierarchical) relation to more traditional (national) legal sources have emerged (Hutchinson 2013, p. 26; Van Gerven & Lierman 2010; Van Gestel et al. 2012, p. 15).¹³xAlso stated by a respondent, professor with an Administrative and Constitutional Law expertise. Moreover, these new legal instruments appear to change more rapidly than the traditional legal sources.¹⁴xAs remarked by a respondent, Ph.D. with an International and European Law expertise. This challenge is becoming even more relevant as a result of the ongoing digitalization (Berring 1997, p. 199): ‘As more and more publishers enter the fray, producing a dizzying array of electronic products, things will grow more confusing. Where once a lawyer did not have to think about the information that she was going to use, now choices will be popping up like zits on a teenaged face. (…) In other words, cases are everywhere. Price options will be everywhere. The primacy of the old paper sets is fading, and a vortex of conflicting claims and products is spinning into place.’

  7. The fact that conclusions do not always directly flow from consulted sources is another problem or challenge reported by legal doctrinalists (Epstein & King 2002, p. 32, 34-36, 42; Van Gestel et al. 2007). Epstein and King state that within doctrinal legal inquiry it is often attempted to make inferences from a few exemplary or key cases, without providing any information on, first, what ‘exemplary’ or ‘key’ would mean within their research context; second, why the selected cases are ‘exemplary’ or ‘key’ cases, and whether, third, the selected cases adequately or fairly represent any cases other than those selected by the author. Without such an explanation it is of course possible that the inferences reached employing this approach meets academic standards, but possible is not the same as conclusive. Similar issues have been formulated regarding the use of legislation, for example when determining or interpreting the ‘legislative intent’. Doctrinal legal inquiry often involves consulting the legislative history, including the parliamentary debates held with regard to some specific law or statute. Although such an approach for establishing legislative intent is generally acceptable, it is often problematic because the scholar ‘never tells readers how he surveyed the congressional material, how many of the debates he read, and whether the material cited in the article represents “key” events, a “few exemplary” passages, or a systematic sample of some specific kind’ (Feldman 2009, p. 99-101; Levine 2005, p. 2-3).

  8. Particularly in the use of case law in doctrinal legal research, another challenge for the legal doctrinalist looms large. Several scholars noted the inherently random nature of the publication of case law (Oldfather, Bockhorst & Dimmer 2012, p. 1191; Van Manen 2008; Van Opijnen, 2014). Firstly, case law does not reflect the exact scope of societal problems; the existence of case law is dependent on the decision of people to litigate and to follow the complete legal procedure. Secondly, even if a certain dispute resulted in case law, we must realize that the majority of case law is not published. Empirical research showed that in the Netherlands in the period from 2001-2009 only between 10 and 20 percent of Supreme Court decisions in criminal law cases and between 26 and 38 percent of Supreme Court decisions in tax law cases were published yearly (Van Opijnen 2014, p. 214). For decisions of lower courts, this publication percentage does not rise above 1 percent (Malsch, IJpelaar & Nijboer 2007, p. 105; Van Opijnen 2014, p. 222-223). Moreover, the criteria or procedures used for selecting the case law for publication are (highly) inconsistent, or at least unclear (Malsch, IJpelaar & Nijboer 2007; Van Opijnen 2014). The legal doctrinalist has to therefore carefully assess the case law that was consulted and the conclusions based thereon.

  9. Finally, when collecting sources, the scholar is challenged to confine the broader context and contextual discussion of a particular research project so that it does not subsume the entire project (Hutchinson 2013, p. 26).¹⁵xPh.D. researcher of Private Law; professor of Administrative and Constitutional Law. On the other hand, the scholar should include all possibly relevant sources for answering his or her research question. The inconsistency between those two demands results in several possible dilemmas for the scholar. Do we for instance have to consult scholarly publications and case law that have been published ten years ago? And what about publications that appeared fifty years ago? Do we have to turn to the more philosophical accounts on contract law if we want to evaluate the current positive law on the termination of contracts? And so on.

• 5 Conclusions and implications

  Although I do not pretend to have presented a complete list of methodological challenges in doctrinal legal inquiry, I think I found more and other concrete or practical methodological pitfalls in doctrinal legal research than one might expect. The majority of these dilemmas, problems, and challenges are relatively new to the field of doctrinal legal research (Cnossen & Smith 1997, p. 1; Osborne 2012, p. 11); several challenges specifically relate to the changing source-landscape due to – in short – a growing amount and plurality of doctrinal legal sources and extensive digitalization and computerization (Davidson 2010, p. 563).

  In the phase of collecting information, the growing amount and plurality of (available) information is a particular challenge to the present-day legal scholar. On the one hand, this trend does result in the availability of more relevant information, leading to problems and dilemmas with regard to the selection and assessment of the materials. On the other hand, finding particularly relevant en therefore interesting information becomes more difficult. Digitalization and computerized research increasingly complicate this process. Both technical shortcomings and challenges relating to information overload provide problems for the legal scholar conducting doctrinal legal inquiry. Finally, when the scholar interprets and synthesizes sources, several problems must be dealt with. The main problems here relate to researcher bias (that can have many forms and guises), the determination of authority of, or the hierarchy among legal sources and the process of generalization from particular cases or sources or deriving valid conclusions from them.

  From all this follows that doctrinal legal scholarship (1) is in urgent need of more systematic empirical investigation of the methodological challenges, (2) should search for information on how legal doctrinalists generally deal with the methodological challenges identified, and (3) must investigate what all this means for the way in which doctrinal legal inquiry is practiced and how it may evolve in the near future. In the end, it is the academic legal forum that must assess the identified challenges and decide how to deal with them.

References

• Aveyard 2009
  H. Aveyard, Doing a Literature Review in Health and Social Care: A Practical Guide, Berkshire: Open University Press 2009.

• Barnhizer 2005
  D.R. Barnhizer, ‘Truth or Consequences in Legal Scholarship?’, Hofstra Law Review 2005, p. 1203-1239.

• Van den Bergh 2004
  G.C.J.J. van den Bergh, ‘Echte wetenschap?’, NJB 2004/79, afl. 28, p. 1435-1436.

• Berring 1997
  R. Berring, ‘Chaos, Cyberspace and Tradition: Legal Information Transmogrified’, Berkeley Technological Law Journal (12) 1997, p. 189-212.

• Bintliff 1996
  B. Bintliff, ‘From Creativity to Computerese: Thinking Like a Lawyer in the Computer Age’, Law Librarian Journal (3) 1996, p. 338-351.

• Bloembergen 1988
  A.R. Bloembergen, ‘Iets over object en methode van wetenschap en rechtspraak in het privaatrecht’, in: O.W.M. Kamstra & F. Kunneman (eds.), Nederlandse rechtswetenschap: tussen distantie en betrokkenheid: paradigma’s in de twintigste eeuw, Zwolle: W.E.J. Tjeenk Willink, p. 61-80.

• Bodig 2011
  M. Bodig, Doctrinal Knowledge, Legal Doctrines and Legal Doctrinal Scholarship. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1921305.

• Cnossen & Smith 1997
  C. Cnossen & V.M. Smith, ‘Developing Legal Research Methodology to Meet the Challenge of New Technologies’, Journal of Information, Law and Technology (2) 1997, p. 1-5.

• Cohen & Olson 2007
  M.L. Cohen & K.C. Olson, Legal research in a nutshell, St. Paul: Thomson/West 2007.

• Committee Performance-Indicators and Ranking 2007
  Committee Performance-Indicators and Ranking, Naar prestatie-indicatoren voor rechtswetenschappelijk onderzoek, Den Haag: VSNU 2007.

• Cownie 2004
  F. Cownie, Legal Academics. Culture and Identities, Oxford: Hart Publishing 2004.

• Cryer et al. 2011
  R. Cryer, T. Hervey, B. Sokhi-Bulley & A. Bohm, Research Methodologies in EU and International Law. Oxford: Hart Publishing 2011.

• Curry-Sumner et al. 2010
  I. Curry-Sumner, F. Kirsten, T. van der Linden-Smith & J. Tigchelaar, Onderzoeksvaardigheden. Instructie voor juristen, Nijmegen: Ars Aequi Libri 2010.

• Davidson 2010
  S. Davidson, ‘Way Beyond Legal Research: Understanding the Research Habits of Legal Scholars’, Law Librarian Journal (4) 2010, p. 561-579.

• Dawson 1991
  T.B. Dawson, ‘Legal Research in a Social Science Setting: The Problem of Method’, Dalhousie Law Journal (3) 1991, p. 445-472.

• Deane & Hutchinson 2010
  F. Deane & T. Hutchinson, ADT (Australasian Digital Theses Program). Retrieved from http://www.caul.edu.au/caul-programs/australasian-digital-theses.

• Epstein & King 2002
  L. Epstein & G. King, ‘The Rules of Inference’, University of Chicago Law Review (1) 2002, p. 1-134.

• Fajans & Falk 2011
  E. Fajans & M.R. Falk, Scholarly Writing for Law Students: Seminar Papers, Law Review Notes and Law Review Competition Papers (4 ed.), Eagan: West Publishing Company 2011.

• Feldman 2009
  R. Feldman, ‘Law’s Misguided Love Affair with Science’, Minnesota Journal of Law, Science and Technology (1) 2009, p. 95-116.

• Fink 2010
  A. Fink, Conducting Research Literature Review: From the Internet to Paper. London: Sage 2010.

• Fisher et al. 2009
  E. Fisher, B. Lange, E. Scotford & C. Carlarne, ‘Maturity and Methodology: Starting a Debate about Environmental Law Scholarship’, Journal of Enviironmental Law (2) 2009, p. 213-250.

• Van Gerven & Lierman 2010
  W., van Gerven & S. Lierman, Algemeen Deel veertig jaar later. Privaat- en Publiekrecht in een meerlaags kader van regelgeving, rechtsvorming en regeltoepassing. Mechelen: Kluwer 2010.

• Van Gestel 2013
  R.A.J. van Gestel, ‘Ontspoorde experimenten’, Regelmaat 2013, p. 59-69.

• Van Gestel & Micklitz 2011
  R.A.J. van Gestel & H.W. Micklitz, ‘Revitalising Doctrinal Legal Research in Europe: What about Methodology?’, in: U. Neergaard, R. Nielsen & L. Roseberry (eds.), European Legal Method, Kopenhagen: Djøf 2011, p. 25-75.

• Van Gestel, Micklitz & Maduro 2012
  R.A.J. van Gestel, H.W. Micklitz & P. Maduro, Methodology in the New Legal World. EUI Working Papers LAW No. 2012/13. Available at SSRN: http://ssrn.com/abstract=2069872.

• Van Gestel & Vranken 2011
  R.A.J. van Gestel & J.B.M. Vranken, ‘Assessing legal research: Sense and nonsense of peer review versus bibliometrics and the need for a European approach’, German Law Journal (3) 2011, p. 901-929.

• Van Gestel et al. 2007
  R.A.J. van Gestel, J.B.M. Vranken, H. Gribnau & H. Tijssen, ‘Rechtswetenschappelijke artikelen. Naar criteria voor methodologische verantwoording’, NJB 2007/82, afl. 24, p. 1448-1461.

• Giard 2013
  R.W.M. Giard, ‘Reflecteren over publiceren’, in: G. Van Dijck, R. A. J. Van Gestel, I. Giesen & F. Hammerstein (eds.), Cirkels. Een terugblik op een vooruitziende blik, Deventer: Kluwer 2013, p. 181-190.

• Hart 2001
  C. Hart, Doing a Literature Search: A Comprehensive Guide for the Social Sciences, London: Sage Publications 2001.

• Herweijer 2003
  M. Herweijer, ‘Juridisch onderzoek’, in: J.W.L. Broeksteeg & E.F. Stamhuis (eds.), Rechtswetenschappelijk onderzoek. Over object en methode, Den Haag: Boom Juridische uitgevers 2003, p. 23-33.

• Hesselink 2009
  M.W. Hesselink, ‘A European Legal Method? On European Private Law and Scientific Method’, European Law Journal (1) 2009, p. 20-45.

• Hirsch Ballin 1988
  E.H.M. Hirsch Ballin, ‘Object en methode van de wetenschap van het staats- en het bestuursrecht’, in: O.W.M. Kamstra & F.B. M. Kunneman (eds.), Nederlandse rechtswetenschap: tussen distantie en betrokkenheid: paradigma’s in de twintigste eeuw, Zwolle: W.E.J. Tjeenk Willink 1988, p. 81-100.

• Van Hoecke 2010
  M.A.A. van Hoeke, Is de rechtswetenschap een empirische wetenschap?, Den Haag: Boom Juridische uitgevers 2010.

• Van Hoecke 2011
  M.A.A. van Hoecke, Methodologies of Legal Research: Which Kind of Method for What Kind of Discipline?, Oxford: Hart Publishing 2011.

• Hutchinson 2013
  T. Hutchinson, ‘Doctrinal Research: Researching the Jury’, in: D. Watkins & M. Burton (eds.), Research Methods in Law, London/New York: Routledge 2013, p. 7-33.

• Hutchinson & Duncan 2012
  T. Hutchinson & N. Duncan, ‘Defining and Describing What We do: Doctrinal Legal Research’, Deakin Law Review (1) 2012, p. 83-119.

• Kamstra & Kunneman 1988
  O.W.M. Kamstra & F.B.M. Kunneman, Nederlandse rechtswetenschap: tussen distantie en betrokkenheid: paradigma’s in de twintigste eeuw. Zwolle: W.E.J. Tjeenk Willink 1988.

• Korrel & Kamstra 1991
  P.G. Korrel & O.W.M. Kamstra, Sociaal wetenschappelijke & juridische methodologie. Amsterdam: Melkman 1991.

• Levine 2005
  K.L. Levine, ‘The Law is not the Case: Incorporation Empirical Methods into the Culture of Case Analysis’, University of Florida Journal of Law and Public Policy (2) 2005, p. 283-302.

• Malsch, IJpelaar & Nijboer 2007
  M. Malsch, G. IJpelaar & H. Nijboer, ‘Publicatie van strafrechtelijke uitspraken’, in: M. Malsch & N.F. van Manen (eds.), De begrijpelijkheid van rechtspraak, Den Haag: Boom Juridische uitgevers 2007.

• Manderson & Mohr 2002
  D. Manderson & R. Mohr, ‘From Oxymoron to Intersection: An Epidemiology of Legal Research’, Law Text Culture (1) 2002, p. 159-182.

• Van Manen 2008
  N.F. van Manen, ‘Wat de rechtswetenschap (niet) zo bijzonder maakt...’, NJB 2008/83, afl. 31, p. 1927-1930.

• McCrudden 2006
  C. McCrudden, ‘Legal Research and the Social Sciences’, Law Quarterly Review (4) 2006, p. 632-650.

• Miller 1968
  A.S. Miller, ‘The Myth of Objectivity in Legal Research and Writing’, Catholic University Law Review (3) 1968, p. 290-307.

• Oldfather, Brockhorst & Dimmer 2012
  C.M. Oldfather, J.P. Bockhorst & B.P. Dimmer, ‘Triangulation Judicial Responsiveness: Automated Content Analysis, Judicial Opinions, and the Methodology of Legal Scholarship, Florida Law Review (5) 2012, p. 1189-1239.

• Van Opijnen 2014
  M. van Opijnen, Op en in het web. Hoe de toegankelijkheid van rechterlijke uitspraken kan worden verbeterd, Den Haag: Boom Juridische uitgevers 2014.

• Osborne 2012
  C. Osborne, A Methodical Approach to Legal Research: The Legal Research Plan, an Essential Tool for Today’s Law Student and New Attorney, Washington & Lee Public Legal Studies Research Paper Series. Lexington: Washington and Lee University School of Law 2012.

• Parise 2010
  A. Parise, ‘The 13 Steps of Successful Academic Legal Research’, International Journal of Legal Information (1) 2010, p. 1-18.

• Patterson 2011
  D. Patterson, ‘Methodology and Theoretical Disagreement’, in: U. Neergaard, R. Nielsen & L. Roseberry (eds.), European Legal Method. Paradoxes and Revitalisation, Kopenhagen: Djøf Publishing 2011, p. 227-241.

• Pendleton 2007
  M. Pendleton, ‘Non-Empirical Discovery in Legal Scholarship: Choosing, Researching and Writing a Traditional Scholarly Article’, in: M. McConville & W.H. Chui (eds.), Research Methods for Law, Edinburgh: Edinburgh University Press 2007, p. 159-180.

• Peoples 2005
  L.F. Peoples, ‘The Death of the Digest and the Pitfalls of Electronic Research: What Is the Modern Legal Researcher to Do’, Law Librarian Journal (4) 2005, p. 661-680.

• Posner 1993
  R.A. Posner, The problems of Jurisprudence, Harvard: Harvard University Press 1993.

• Posner 2008
  R.A. Posner, ‘The State of Legal Scholarship Today: A Comment on Schlag’, The Georgetown Law Journal (3) 2008, p. 845-856.

• Posner 2009
  R.A. Posner, Law & Literature (3rd ed.). Cambridge/London: Harvard University Press 2009.

• Rubin 2001
  E.L. Rubin, ‘Legal Scholarship’, International Encyclopedia of the Social & Behavioral Sciences, Amsterdam/New York: Elsevier 2001, p. 8677-8684.

• Salter & Mason 2007
  M. Salter & J. Mason, Writing Law Dissertations. Harlow: Pearson Educated 2007.

• Van Schaaijk 2011
  G.A.F.M. van Schaaijk, Praktijkgericht juridisch onderzoek. Den Haag: Boom Juridische uitgevers 2011.

• Schlag 1991
  P. Schlag, ‘Normativity and the Politics of Form’, University of Pennsylvania Law Review (4) 1991, p. 801-932.

• Sieburgh 2011
  C.H. Sieburgh, ‘De Schoonheid van de ontdekkingstocht naar gewoon geldend recht’, Tijdschrift voor Privaatrecht 2011-1, p. 1-11.

• Smith 2009
  C.E. Smith, ‘Het normatieve karakter van de rechtswetenschap: recht als oordeel’, Rechtsfilosofie & Rechtstheorie, 38(3), 202-225.

• Smits 2012
  J. Smits, The Mind and Method of the Legal Academic, Cheltenham: Edward Elgar Publishing 2012.

• Spitzer 2008
  R.J. Spitzer, Saving the Constitution from Lawyers: How Legal Training and Law Reviews Distort Constitutional Meaning, Cambridge: Cambridge University Press 2008.

• Stolker 2004
  C.J.J.M. Stolker, ‘Wat maakt een juridisch tijdschrift wetenschappelijk?’, NJB2004/79, afl. 28, p. 1409-1418.

• Tijssen 2009
  H.E.B. Tijssen, De juridische dissertatie onder de loep: de verantwoording van methodologische keuzes in juridische dissertaties, Amsterdam: Boom Juridische uitgevers 2009.

• Tjong Tjin Tai 2013
  T.F.E. Tjong Tjin Tai, ‘Verantwoording van bronnen’, in: G. van Dijck, R.A.J. van Gestel, I. Giesen & F. Hammerstein (eds.), Cirkels. Een terugblik op een vooruitziende blik, Deventer: Kluwer 2013, p. 201-208.

• Toma 1999
  J.D. Toma, ‘Understanding Why Scholars Choose to Work in Alternative Inquiry Paradigms’, Research in Higher Education (5) 1999, p. 539-569.

• Verbeke, Vanhove & Hoekx 2009
  A. Verbeke, K. Vanhove & M. Hoekx, ‘Yes we care! Bedenkingen over de (methode der) rechtswetenschap’, Rechtskundig Weekblad 2009/72, afl. 42, p. 1785-1789.

• Vranken 2005
  J.B.M. Vranken, Algemeen deel, een vervolg, Deventer: Kluwer 2005.

• Vranken 2011
  J.B.M. Vranken, ‘Methodology of Legal Doctrinal Research: A Comment on Westerman’, in: M.A.A. van Hoecke (ed.), Methodologies of Legal Research, Oxford: Hart Publishing 2011, p. 111-121.

• Vranken 2014
  J.B.M. Vranken, Mr. C. Asser’s Handleiding tot de Beoefening van het Nederlands Burgerlijk Recht: Algemeen deel: Een Synthese, Deventer: Kluwer 2014.

• Watkins & Burton 2013
  D. Watkins & M. Burton, Research Methods in Law, London / New York: Routledge 2013.

• Wendt 2008
  J.A.I. Wendt, De methode der rechtswetenschap vanuit kritisch-rationeel perspectief, Zutphen: Uitgeverij Paris 2008.

• Wendt 2009
  J.A.I. Wendt, ‘De opgekomen methodenvrees in het rechtswetenschappelijk debat in Nederland. Een voorstel’, NJB 2009/84, afl. 13, p. 782-789.

• Wren & Wren 1990
  C.G. Wren & J.R. Wren, ‘Reviving Legal Research: A Reply to Berring and Vanden Heuvel’, Law Library Journal 1990, p. 463-493.

Noten

• 1 More structured empirical research on quality criteria, challenges and research methods and techniques applicable to doctrinal legal inquiry is currently being conducted within my dissertation research project.

• 2 I recognize that different definitions of ‘doctrinal legal inquiry’ circulate and that several nuances to the presented definition may exist (Vranken 2014).

• 3 The concepts ‘scientifically sound’ or ‘profound’ in relation to source-usage within doctrinal legal inquiry have not been clarified or operationalized within contemporary legal scholarship. Some insights are, however, provided by Tijssen (2009, p. 60-61) and Vranken (2014, nr. 34, 90). The scope of this contribution does not allow me to elaborate on such aspects.

• 4 Documentation on the exact techniques applied in order to conduct the systematic literature review can be obtained from the author on request.

• 5 Within the literature this technique is known as ‘strategic sampling’.

• 6 Also explicitly stated by a professor of Criminal Law.

• 7 Professor of Private Law.

• 8 According to a professor of Criminal Law, ‘the legal world has been globalized and research-wise exploded, especially if one has knowledge of diverse languages and if one does not simply rely on national or even English literature’.

• 9 Professor of Administrative and Constitutional Law.

• 10 Also explicitly addressed by a respondent, Ph.D. candidate of Private Law.

• 11 Ph.D. candidate of Constitutional Law.

• 12 Stated by two respondents, professor with a Private Law expertise and professor with a Constitutional and Administrative Law expertise.

• 13 Also stated by a respondent, professor with an Administrative and Constitutional Law expertise.

• 14 As remarked by a respondent, Ph.D. with an International and European Law expertise.

• 15 Ph.D. researcher of Private Law; professor of Administrative and Constitutional Law.