Making the implicit quality standards and performance expectations for traditional legal scholarship explicit

Abstract

Scholars in search of quality standards for traditional legal scholarship could well end up disappointed. By answering the question concerning what standards legal academics use for evaluating such works—through reviewing the international literature on evaluative standards and interviews with forty law professors—this Article aims at filling this gap. This Article recommends that traditional legal scholarship is judged by using the following criteria: (1) the conceptual design—a clearly formulated research question that is both original and significant and the adequacy of the methods proposed to answer that question; (2) the composition of a particular line of reasoning—does the researcher adhere to principles of accountability, accuracy, balance, and credibility?; and (3) the overall characteristics of a work of scholarship—the readability and persuasiveness of the whole and the extent to which the researcher managed to identify and clarify the presuppositions that may have potentially affected her inquiry.

A. Introduction

As legal academics, we constantly engage in the process of evaluating works of legal scholarship. In a variety of contexts and with different purposes, we either explicitly or unconsciously put quality labels on the work we come across and produce ourselves. We consider some works to be outstanding, inspiring us to start new research projects, or making us want to cooperate with or hire the author. We encounter pretty good works which we believe should be published in our journals, or that we decide to build on when we conduct our own research. And we—all too often, according to some—encounter works that we instantly dismiss as poor quality, sometimes even making us seriously doubt the capabilities of those that authorized publication.¹ While it makes perfect sense that we pass such judgments, it does raise the question of how, and especially on what grounds we do so: What standards do legal academics use for evaluating works of legal scholarship?

Given the long history of legal scholarship, one might expect an answer to that question to be quite highly developed.² One is likely to be disappointed, however. Unlike their colleagues operating in other academic disciplines, legal academics rarely engage, individually or collectively, in a more general and thorough analysis of their quality standards.³ Consequently, what counts as evidence of “outstanding” legal scholarship has remained largely inexplicit, mysterious, and ill-defined. For a variety of reasons, explained in paragraph B, I believe this to result in a suboptimal research environment.⁴ This Article is an effort to take a leap forward through, first, a systematic and critical review of the scarce and fragmented contributions that have proposed quality standards—paragraph C—and, second, a presentation of the results of an empirical study that was aimed at verifying, supplementing, and bringing nuance and deepness to the insights present within that literature—paragraph D.⁵

Because very different forms of legal scholarship exist—from the normative to the descriptive, from narrow doctrinal analyses to large sample size statistical investigations, from case notes to elaborate monographs—and it would be presumptuous to suggest that all of that scholarship should or could hold on to one set of views about quality, I was—like others who have addressed the topic before⁶—forced to confine myself to discussing standards for one type of legal scholarship only. As it still lies at the heart of our discipline, is practiced by legal scholars all over the world every day, and has no other academic disciplines to directly rely on,⁷ my focus rests on what is usually called traditional legal scholarship.⁸ In short, traditional legal scholarship is the type of scholarship that focuses on the way in which judges, legislators, or administrators have decided or should decide on legal questions from a legal—thus, not economic, social, moral, etc.—perspective.⁹

Right from the start, I do have to temper expectations. If the debates on quality standards present in other academic disciplines teach us anything, it is that developing a comprehensive, all-encompassing, operationalized set of quality standards is simply impossible.¹⁰ Any proposition of such a set is necessarily open-ended, assailable, non-absolute, and no more than “a snapshot in time of a set of emergent ideas.”¹¹ That, however, has not withheld scholars operating in other disciplines from having a vibrant debate about their quality standards, and it does not keep me from trying to breathe life into a debate about criteria for considering traditional legal scholarship now. What it does mean, though, is that the reader is encouraged to consider the insights presented in this Article critically, to treat the proposed performance expectations with a healthy form of suspicion, and to actively engage in a further debate about the rules of the game we all like to play.

B. The need for explicating standards or performance expectations

Prior to presenting what I found, let me first express why it is that I—and I am not alone here¹²—think our discipline should be involved in a more conscious dialogue about its quality standards, and, as our community has operated for so long without any, why now would be a good time to begin doing so.

The absence of clearly articulated criteria made several scholars conclude earlier that our method of evaluating the work of our colleagues is essentially intuitive¹³ and/or based on criteria of meritocracy.¹⁴ Besides the fact that intuitionism of this sort—scholars acting as oracles—and the reliance on merit seem seriously out of place in academia, it also—and more crucially—creates specific dangers. Most importantly, when relying on intuition or merit rather than agreed-upon criteria when evaluating works of scholarship, any scholar, no matter how conscientious, may more easily fall prey to personal opinion, bias, or even idiosyncrasy that cannot be justified in general terms.¹⁵ That is especially problematic if one realizes that far-reaching consequences can be attached to the quality judgements we render.¹⁶ An articulated, coherent theory of evaluation may help here, as it can both create altitude and a means of motivating and justifying our judgements.¹⁷

Articulated criteria may, however, also be helpful in the sense that they can guide us in seeking to make our own work the best example of what we understand our scholarship to be, or at least contribute to the avoidance of sloppy science. Here, rather than evaluatively, criteria are used “aspirationally” as a means for members of the scholarly community to improve their work.¹⁸ If we have a clear idea about what evidences outstanding research, we may more consciously focus our research process accordingly. While one could hardly deny that all who are in one way or another pursuing legal research could in this way benefit from articulated criteria, the aspirational function of criteria is especially affecting students and scholars that stand at the beginning of their academic careers.¹⁹ To get acquainted with the standards legal scholars are expected to adhere to in another way than finding out by trial and error what supervisors, peer reviewers, or journal editors apparently expect, would certainly be a welcoming development.²⁰

A final reason for having a debate about our standards that I would like to mention, is that the articulation of criteria may have political value for our discipline.²¹ The importance of providing evidence to governments and other funding agencies that their investment in research provides good value for money in terms of quality and impact has increased in nearly all countries.²² Therefore, whether we like it or not, our sponsors will, and already are, measuring the quality of the products in which they invest. To insulate our work from the imposition of inappropriate criteria developed by other academic disciplines when such assessments are made, we cannot escape from developing and/or clarifying our own standards. As doing so would help us defend the validity of our scholarly activities and objectively demonstrate the quality of the work we produce,²³ we may, moreover, eventually even be able to reverse the diminishing position of traditional legal scholarship on the market for research funds.²⁴

C. State of the art

The absence of a vivid scholarly debate on standards for traditional legal scholarship does not alter the fact that, from various angles, the legal academic community has put effort in the development of explicated quality criteria. Repeatedly, quality standards are the subject of discussion in committees appointed to decide on the (re-)accreditation and/or performance of universities and their respective research groups, programs, and institutes.²⁵ Also, quality standards are being discussed among editorial boards of law journals as they grapple with the question of how to determine whether a scholarly work qualifies for publication. Earlier studies of the instruments used for these purposes (assessment protocols²⁶ respectively author guidelines²⁷) revealed that, if quality standards are mentioned all mentioned, three in particular stand out: Originality, thoroughness, and profundity. In nearly all instances, however, it remains unclear what these exactly mean. All of the studies analyzing assessment protocols and/or author guidelines conclude that standards—if at all mentioned—mostly remain “extremely broad,” “extortionately vague,” and “multi-interpretable.” In these assessment protocols, Van Gestel and Vranken, for instance, did not find much more information than that originality refers to contributing “something new to the scholarly debate on a subject,” thoroughness “describes the extent to which authors should try to find underlying explanations or theories for developments in law (asking why questions instead of how questions),” and profundity is associated with visible competence and the methodological or analytical rigor of a particular inquiry.²⁸

Greater insight into the standards that may be used to judge works of traditional legal scholarship comes from scarce, but available, scholarly contributions.²⁹ Around 1990, the desire of several American legal scholars operating in minority disciplines—feminism and critical race theory—to develop objective standards for judging their work resulted in a small debate in the United States on the different quality standards that apply to traditional and non-traditional legal scholarship. From this debate, two works stand out in particular:³⁰ Rubin’s On Beyond Truth: A Theory for Evaluating Legal Scholarship, and Kissam’s The Evaluation of Legal Scholarship. Inspired by the famous philosophers of science, Jürgen Habermas and Hans-Georg Gadamer, Rubin argues that a piece of traditional legal scholarship should meet standards of “normative clarity,” “persuasiveness,” “significance,” and “applicability.” In order to reach normative clarity, Rubin argues that the scholar is required to clearly identify so-called “controlling norms”³¹—the premises running through a work—and to ensure that specific arguments are appropriately related to, or flow from, these controlling norms.³² Persuasiveness refers to both the extent to which an evaluator believes the work should convince the public decisionmakers to whom it addresses³³ and the authenticity³⁴ of the interpretations of legal texts—case law, statutory provisions, or parliamentary documents—present in the work.³⁵ Significance, then, relates to the extent to which the work is part of the ongoing developments in a particular field; the extent to which the contribution is of interest and importance to the community and influences the field by changing the way people think.³⁶ Finally, applicability refers to the extent to which the evaluator believes that the work contains an identifiable thought that can be used by other legal scholars.³⁷

Kissam approaches the question of quality standards from a different angle. Starting from the more general values or motives for conducting academic scholarship, Kissam considers the achievement of an originality and competence standard necessary for conducting “good scholarship.” According to him, the originality standard may be satisfied if a scholar discovers new knowledge or if the scholar manages to disseminate existing knowledge in an original manner so that it becomes more useful to professional audiences.³⁸ Competence, then, implies that a piece of scholarly work “must be factually accurate, written in a comprehensible manner, and be based on appropriate methods.”³⁹ If scholarly work is both original and competent, Kissam continues, it may be indicated as important or even outstanding depending on the extent to which it affects the lives, works, and thoughts of professionals and academic colleagues.⁴⁰

In Europe, mostly as a response to the increasing focus in universities on the quantitative evaluation of scholarly output,⁴¹ several scholars and committees have—mainly in policy documents—called for a re-focus on the substantive quality of legal publications. Verbeke’s proposition, one of the most elaborate publicly accessible among these types of contributions, is exemplary.⁴² He proposes to differentiate more consciously between academic and non-academic research. Therefore, he suggests one cardinal criterion—”scientificness”—and four additional criteria—”analytical,” “original,” “critical,” and “clear.” According to Verbeke, being scientific means that the research should be solid—for example proper referencing, all claims supported by references or argumentation, all relevant sources checked—comprehensive—for example all aspects of research question covered, all perspectives checked, arguments pro and contra considered and analyzed, open for falsification—and transparent—for example clear research question, source selection strategy clarified, explication of underlying legal, philosophical, ethical, sociological frameworks. With the analytical and critical criteria,⁴³ Verbeke seems to mean that the scholar should move beyond mere description of relevant source material—abstraction, working with a theoretical and conceptual framework, critical assessment, among others. Originality refers to some novelty in content, ideas, approach, and critical assessment, but also in structure or methodology. Finally, being clear means that the publication should present its analysis and argumentation in a clear and stringent style, offering a high degree of lucidity and clarity.

The most recent and elaborate contribution on quality standards for traditional legal research I came across, is a Dutch book written by Vranken. Vranken explores the similarities and differences in the rules of the game that apply to judges and legal academics.⁴⁴ Interestingly, he—more so than Rubin, Kissam, and Verbeke—distinguishes between quality standards related to the development and embedment of a research question and the quality of the composition of an answer to that question. With respect to the research question, Vranken identifies “novelty”—in the sense that the question posed should be able to generate new knowledge—as its most important requirement.⁴⁵ According to Vranken, this also implies that a scholar is required to situate her question against the “backdrop of existing knowledge.”⁴⁶ The newness criterion closely mirrors, but is distinguishable from, the requirement that the question must be worth answering from a scientific and societal perspective—the requirement of “relevance.” Besides being original and relevant, Vranken argues that a research question should also be “precise”; It must allow the reader to get to know exactly what the author means by the core concepts used and what aspects are going to be included and excluded from the project.⁴⁷ The standards Vranken considers to apply to the construction of a line of reasoning in answering a particular research question, seem to be mainly inspired by the Dutch and European “code of conduct” for academic research:⁴⁸ The materials on which the answer is based should be “complete” and up-to-date or—if completeness is not possible because there are simply too many relevant sources—”representative” and including justification thereof; statements should be accompanied by “correct references”; the answer must be “reliable”;⁴⁹ the research materials used by the scholar must be able to “carry the conclusions”; and, the answer must be “persuasive” and “easy to read.”⁵⁰

The problem these accounts share, is that however thoughtful and considerate they may be, it remains unclear whether they represent the ideas and views of the author concerned or those of the scholarly community as a whole.⁵¹ As different authors appear to focus on slightly or partly different standards or maintain different interpretations thereof, I believe it is important that we start approaching the search for standards differently; in a way that allows us to capture the conceptions—and the patterns and variations therein—of the legal scholarly community as a whole. Initiatives hereto are already taken. Scholars operating in a relatively new subdiscipline aimed at empirically studying research assessment methods and techniques have started to take legal scholarship as an object of study.⁵² As far as I know, the first results of these projects are yet to be published. At first sight, however, the current initiatives seem to be designed in such a way that, while they do allow for revealing abstract quality standards that the legal community considers applicable to certain forms of traditional legal research, they do not aim to operationalize those standards in any depth.

D. Empirical insights

Preceded by providing them with the opportunity to openly express their views about standards and performance expectations for traditional legal scholarship, I presented the insights from the literature—in semi-structured interviews—to forty Dutch law professors who regularly publish internationally⁵³ for commentary, discussion, and further operationalization—what do these standards mean for daily research practice?⁵⁴ That allowed me to achieve—exploratory, but still—insights in the standards the legal scholarly community as a whole considers important and the way in which these standards are to be interpreted and operationalized. Interestingly, the data that resulted from the interviews showed that the respondents—like Vranken—felt a need to distinguish among standards that relate to the embeddedness of the research problem or question and the conceptual design (paragraph D.I), the composition of an answer to a research problem or question (paragraph D.II), and the publication as a whole (paragraph D.III). Despite the fact that the respondents recognized and emphasized the importance of most of the standards I encountered within the literature, the data did give rise to the formulation of a set of standards that does not seamlessly coincide with one of the sets proposed before. A brief overview of that set is presented in Table 1. Before addressing the individual standards in more detail, let me remark, though, that while they are distinguishable for analytical purposes, the different standards are, of course, interrelated and sometimes even partly overlapping. Where I found such to be the case, I reported it.

Table 1: Overview of Quality Standards

I. Requirements related to the embedment of the research question and the conceptual design

According to the respondents, an academic traditional legal publication must commence with a “clear problem definition,” preferably shaped as a “concretely and precisely formulated research question or hypothesis.” Here, the scholar should “indicate as precisely as possible what it is that is going to be researched.”⁵⁵ It is important that this question is “linguistically correct,” “easy to understand,” and “does not contain internal contradictions or inconsistencies.” Apart from that, the scholar also has to make sure that “possibly ambiguous or multi-interpretable terms used in the question are conceptualized” before, in, or shortly after the question is presented, allowing author and reader to remain on the same page throughout the publication.⁵⁶

Subsequently, it is expected from the scholar that her contribution is both “original” and “significant.” The respondents identified the fact that an inquiry has a possible “consequence” as the key distinguishing feature of a significant contribution. A scholarly publication has to be of some interest and importance to certain societal actors—societal significance or relevance—and the academic community—scientific or theoretical significance or relevance. “Originality” was defined with respect to the “newness” of a contribution and, more specifically, to the question whether that publication in fact “adds something to the existing body of academic knowledge.”⁵⁷ The respondents emphasized the importance of not leaving the estimation of a publication’s originality and significance to the sole discretion of the reader. Rather, the scholar is well advised to accommodate the reader with a “persuasive justification” thereof. Here, the author may, for instance, argue that her contribution is original in the sense that it identifies or addresses a problem that others have not identified or addressed before, uses a new perspective, theory, or method to readdress a problem others have touched upon before, investigates a known problem by using new and/or more data, and/or proposes and elaborates (a) new argument(s) or solution(s) in relation to a particular problem.⁵⁸ The project’s societal significance can be explained by indicating in what way(s) the knowledge generated can be useful to lawyers, courts, legislators, or litigants, or how it may contribute to solving a societal problem.⁵⁹ The scientific significance, finally, can be justified by substantiating how answering the research question contributes to the development of a particular doctrine, provides insight in the bigger questions the legal discipline faces, or inspires others to tackle other legal problems using a similar approach.⁶⁰ The consequence of these first requirements for the scholar is having to achieve originality and significance is that the scholar is expected to conduct thorough preliminary research with respect to the research topic. After all, in order to be able to indicate the originality and significance of a work, the legal scholar cannot escape from thoroughly investigating and reporting whether other scholars have previously addressed the problem she is interested in, and if so, what perspectives, theoretical approaches, methods, and data were used and/or what solutions were proposed. In other words: The research question must be placed within its disciplinary context.⁶¹

If the researcher succeeds in establishing an original and significant research question, the next thing she is expected to do is clarify and justify the steps necessary—the research methods and techniques—to come to an answer to that question. Of course, those steps depend largely on the exact research question posed. The respondents emphasized that it is important that the researcher at least explicates and justifies the non-self-evident choices made here. With respect to research projects that contain an evaluative or normative component—popular within traditional legal scholarship—the respondents argued that a scholar by now—as a minimum—is required to establish a research framework in which the criteria on the basis of which an evaluative or normative judgement is made are explicated and operationalized.⁶² An obvious example would be a research project in which a particular legal doctrine—for example, civil liability of a governmental agency for insufficient supervision—is evaluated on its compliance with international human rights. In such a research project, the scholar will have to explain what demands flow from human rights and how they can be measured before she will be able to investigate whether the doctrine under scrutiny actually complies with human rights.⁶³ A project where the presence of a framework would already be less self-evident is, for instance, a project that considers whether more intensively regulating the pre-pack would be a good idea. In order to reach a substantiated conclusion to that question, the scholar will, however, also have to develop such a framework. It may, for instance, be constructed from the perspective of the protection of employees, guaranteed by national legislation and/or European law, national and/or international principles of liability or insolvency law, something completely different, or a combination thereof.⁶⁴

II. Requirements related to the composition of a particular line of reasoning between question and answer

1. Accountability

According to the respondents, the traditional legal scholar should be transparent about the complete process that was followed in answering a question. This suggests that the first criterion for excellence in legal scholarship is a principle of accountability. The data revealed that being accountable has a dual connotation: It refers both to the use of references and to the explication and justification of important choices made within a particular research project. What is incontestable is that the accountability criterion implies that the reader must be placed in a position in which she can easily retrieve the materials on which the statements in a publication are based. Therefore, a traditional legal publication should “exhibit a precise and correct body of footnotes.”⁶⁵ This implies that: (1) a reference should actually be included where the scholar draws from the work of others—literature, case law, legislative documents, etc.;⁶⁶ (2) the reference should be precise—indicating the exact page(s) or section were the information can be found; and (3) the reference should be correct in the sense that the research materials referred to in fact contain information that supports the interpretation, argument, or statement of the researcher.⁶⁷

Although important, the fact that a publication contains a precise and correct body of footnotes does not necessarily mean that the researcher sufficiently adhered to the principle of accountability. According to the respondents, the legal scholar is also expected to “justify” the most important,⁶⁸ non-self-evident choices—choices that the majority of readers cannot infer from the references used and that do not flow from common sense or generally shared knowledge—made in composing an answer to a research question.⁶⁹ Justification means: (1) explication that a choice was made; (2) explanation why that choice was made; and (3) clarification of the possible implications the choice may have for the research results.⁷⁰ The last aligns with the thought expressed by nearly all respondents that a researcher cannot and does not have to opt for the theoretically most optimal choice in all instances—consider only time constraints—as long as she honestly reports what the suboptimal choice could mean for the trustworthiness of the findings.⁷¹ Or, as Epstein and King framed it:

The sign of good research is not that every problem be fixed in every project, since that is impossible and trying to do so would mean we would not accomplish much of anything, but rather, that the problems be noted and the uncertainty in substantive conclusions be responsibly and honestly reported.⁷²

An example might be insightful here. Suppose a European constitutional law researcher is interested in a particular legal problem concerning the doctrine of trias politica. Of course, the researcher refers to Montesquieu and Rousseau as the historically most important scholars that have written on the topic, to Conte because he identified the problem the researcher is interested in, and subsequently to two German law professors that have recently addressed the problem from a more modern perspective. Critical questions may probably rise instinctively: Would it not be possible that the researcher reaches different conclusions if she would have consulted a broader scope of research materials? Are there other materials that contradict or bring further nuance to the views represented in the sources used by the researcher; and if so, would it not have been necessary to include these materials in the inquiry? If I understood the respondents correctly, the researcher would have better satisfied the criterion of accountability if she—in addition to precise and correct referencing—had explained why she only used the works of the two German law professors. For instance, by indicating that a systematic search through the international literature revealed that no one else has addressed the problem in the past decade(s) or that those law professors provided a reliable overview of other relevant earlier works, she may assert that a separate study of these materials was regarded unnecessary.

Except for showing that merely referring in a precise and correct way may thus be insufficient to operate in adherence with the principle of accountability, the example shows the close connection between the principle of accountability and the standards I will address hereafter. The way in which the researcher operates in the example above also raises questions that relate to other quality criteria—can we speak of a “balanced” use of research materials? Is it possible to come to “credible” statements by consulting only these two modern scholars? The example finally also clarifies that not all choices made by the researcher require similar elaboration. The choice to use the publications of the two German law professors probably raises more critical questions than the choice to build on the works of Montesquieu, Rousseau, and Conte. Apparently, certain choices are more logical or self-evident than others, and for that reason require less justification.

2. Accuracy

The second criterion suggested by the respondents is that a work of traditional legal scholarship should be “accurate.” In its broadest sense, this criterion means that “what follows after the problem definition must actually contribute to answering it.”⁷³ According to the respondents “there must exist a clear fit between the problem definition and conclusions,” substantiated by a “clear line of reasoning” in between. In an outstanding traditional legal publication, “the diverse components of the research project are connected in a seamless way and the conclusions tie the whole thing together.”⁷⁴ “Needless meandering” is to be prevented as much as possible. This does, however, not alter the fact that a publication can be of enormous complexity and may touch upon many different themes and subthemes. If such is the case, however, the discussion of these (sub)themes must both still in some way contribute to answering the research question and be presented in a logical sequence and at a convenient point in the publication.

For the use of research materials, being accurate means that a scholar should be both “exact” and “meticulous.” What that exactly conveys can be clarified by providing an example. Suppose that a researcher aims to answer the following (sub)question: “Under what conditions can a government agency be held liable for damages that were caused by inadequate supervisory activities of that agency?” Let’s assume that there exists an abundance of research materials—policy documents, case law, and commentaries within the literature—that “only” need to be organized and systematized. Suppose the researcher studies dozens of cases that deal with the liability of non-state, private actors for inadequate supervision. The fact that the researcher studies materials that are all about the same topic, means he operates “meticulously.” Here, however, at least at first sight, the researcher does not live up to the principle of “exactness,” as the research materials do not deal with liability of government agencies. The reader might rightly wonder whether the question about liability of governmental agencies can actually be answered by the consultation of materials on liability of private actors. To compensate for this potential accuracy problem, the researcher must persuasively justify the choice to build on these particular cases related to private actors.

A complicating factor here is that the traditional legal scholar often pursues research aimed at positioning a new societal or legal development within the legal system, predicting how a legal (sub)system may evolve, and/or improving a legal (sub)system, instead of clarifying a legal doctrine by merely organizing and systematizing existing research material.⁷⁵ In such cases, there often exist little materials of which the relevance for answering the question posed is beyond doubt. The researcher has to rely heavily on the associations she or others before her were able to draw; associations with legal principles, international law, legal doctrines in other fields of the law—internal comparative law—or other jurisdictions—external comparative law—insights from other academic disciplines, or a combination thereof.⁷⁶ These associations subsequently determine what research materials are consulted. The consequences flowing from the principle of accuracy would then be that the researcher is expected to: (1) be exact and meticulous in the elaboration of the association itself; and (2) explain why the association itself does not erode the accuracy of the inquiry.⁷⁷

3. Balance

Besides the principles of accountability and accuracy, the respondents also considered a principle of “balance” to be applicable to traditional legal scholarship. To be able to reach balance, it is indispensable that “the legal scholar is as complete as possible and necessary in her consultation of research materials that are relevant to the investigation.” If inquiry is pursued on a relatively limited field and based on a sharp, focused, and specific research question, and/or is predominantly oriented on typically national sources, achieving such absolute completeness in the consulted materials is actually practically “possible and therefore also necessary.”⁷⁸ If the research question is, however, broader—which is, for instance, soon the case if it addresses a topic with an international or European dimension—the respondents recognized the practical impossibility, and therefore also unnecessity, of achieving absolute completeness.⁷⁹ If such is the case, that “does not mean that the legal scholar should not strive to be as complete as she can possibly be,” but rather that “making deliberate selections becomes inescapable.” Through selection, the researcher must then make sure that she “consults sources that offer a representative picture of the important different interpretations, arguments, and solutions that have been proposed within the conglomerate of relevant source materials.”⁸⁰ The reader won’t always be able to estimate whether the researcher succeeded in this respect. Therefore, the researcher is well advised to anticipate by explaining why and how she thinks she managed to achieve such a representative image, especially at the points where the completeness of the inquiry might be challenged.⁸¹

Being as complete as possible is, in itself, still insufficient to achieve a balanced inquiry. The researcher should also make sure that different insights represented in the research materials obtain an “honest” position within the publication. In that sense, completeness and honesty are complementary. “Attaching defendable cogency” to the different insights found in the research materials is no exact science. That does not, however, imply that the traditional legal scholar enjoys unrestricted freedom in this respect. At least, adhering to the principle of honesty means that relevant interpretations and arguments found may not—consciously nor unconsciously—be omitted or undervalued without any form of justification thereof.⁸² In addition, the researcher should consider the extent to which a particular insight is supported by the scholarly community—is it mainstream or a conflicting position?—as well as the quality—does the work live up to academic standards?—and the authority—Supreme Court or lower court?—of the source that presented it. While support, quality, and authority are important, those are not the only decisive indicators for positioning a particular insight. Of course, the relevance for the research question also matters. According to the respondents, it is not problematic if the researcher prioritizes certain arguments above others for that reason, as long as she makes sure that important alternative arguments are presented to the reader as well.⁸³

4. Credibility

Even if the researcher adheres to the principles of accountability, accuracy, and balance, the answer to a research question might still be unconvincing. That is because none of the standards discussed thus far tell us something about the truth-value or—as truth is a somewhat contested concept—the “credibility” of the claims the researcher makes within her publication. Each claim, whether descriptive, interpretative, generalist, evaluative, or theoretical, should be credible.⁸⁴ The respondents emphasized that a descriptive inference—in a strict sense, paraphrasing or summarizing a particular source—is only credible if it is correct.⁸⁵ Paraphrases may not contain any errors, summaries may not omit important elements nor wrench the original text from its context, and certainly out of the question is that a concealed interpretation—or even an opinion—is presented as a descriptive inference.⁸⁶

If the researcher moves beyond the descriptive level, what it takes to be credible becomes much less straightforward. Scholars operating in the field of legal theory have dedicated numerous pages on what constitutes a credible interpretation of a legal source. While it is impossible to reproduce the insights represented in that body of literature here, let me at least highlight two important notions. First, legal theorists debate whether it is even possible to come to a more or less “clean” interpretation or whether every interpretation is in fact a concealed normative statement of the interpreter.⁸⁷ Second, legal theorists are stuck in a continuous discussion about the aim that should be pursued by interpretation: Is it about determining what the author(s) of a particular source meant—sometimes referred to as the “originalist approach”—or is it about determining what the source might mean in its contemporary context—sometimes referred to as the “contextualist approach”?⁸⁸ Fortunately, the respondents adhered to a more pragmatic approach. They spoke of the need to achieve a “defensible” rather than a “correct” interpretation. If a consensus exists among the vast majority of the legal scholarly community about the “right” interpretation of a particular source, the researcher is expected to subscribe to that uncontroversial interpretation or persuasively justify why she wishes to deviate. If such a consensus does not exist, for instance, because of the simple fact that the source has not been interpreted before, the researcher seems to enjoy more, though definitely not unrestricted, freedom. According to the respondents, the researchers may, for instance, never reach beyond the intention of the author(s) of the interpreted material⁸⁹ and must ensure that a reader that deeply compares the interpretation with the interpreted may not get the idea that there exists insufficient congruence between the two.⁹⁰ Researchers should also preferably clarify the technique(s) used for reaching the interpretation and are advised to honestly report on the possible insecurities that were encountered during the interpretation process.⁹¹ Finally, if there exists no consensus as to the right interpretation of a source that has been interpreted before, the researcher will—apart from the things mentioned before—be additionally required to pay attention to the way in which her particular interpretation relates to, coincides with, or deviates from the existing interpretations.

For statements or claims based on a cluster of research materials rather than individual materials—is the claim about the scope of the duty of care of a house-buyer that was made on the basis of three judicial decisions valid in the sense that the claim holds if other judicial decisions of the Supreme Court are consulted?—the credibility criterion implies the following. Again, if there exists a generally accepted interpretation of a body of research materials that is supported by a large part of the academic community, the researcher should either subscribe to that interpretation or persuasively justify why deviation is preferred. Often, however, that scenario will not be applicable. The researcher then has different options to choose from. She may decide to generalize on the basis of studying all research materials on a particular theme or on the basis of a sample thereof. While the respondents highly preferred the former, they also recognized that the researcher sometimes has no other choice than to opt for the latter. In that case, the credibility of the generalization may be evaluated by a combination of: (1) considering whether the claim made holds for materials that fall outside the selection; and (2) the persuasiveness of the justification of the process that has led to the claim made.⁹²

The credibility of evaluative and normative claims—is this particular judicial opinion or regulatory provision “right” or “wrong” or, in a legal context, “legitimate” or “non-legitimate”?—or classifications—is this judicial opinion an illustration of position A or position B?—depends on the consistent employment of an explicated framework that indicates the criteria on which the claim or classification is based—for instance, does it align with prior case law or norms introduced by the European Court of Human Rights? The credibility of evaluative claims is thus not concerned with the rightness of the evaluation criteria formulated—that would be a question of whether the right criteria are formulated on the basis of sufficiently valid theoretical foundations—but rather with the question whether they were applied consistently.⁹³

Finally, traditional legal scholars also bring theoretical constructs to a study: The scholar argues that a particular court took a more “conservative” approach, that the introduction of a certain new legal provision is an “act of resistance” towards developments on the commercial market, or that a particular court decision is an illustration of the court making a judgment in order to promote the principle of “legal certainty.”⁹⁴ Here, the labels “conservative,” “act of resistance,” and “legal certainty” are introduced by the researcher. Depending on the importance of such labels for the research question, they either must be explained shortly after the presentation thereof⁹⁵ or close to the place where the construct is introduced. As far as the credibility of the use of such theoretical constructs is concerned, there are at least two things that are important. The first refers to the credibility of the theoretical concepts attached to the description or interpretation of a particular phenomenon itself. For example, is the changed attitude of the Supreme Court indeed a sign of the adoption of a conservative approach—and what is the scholar’s understanding of conservative?—or is it rather something else?⁹⁶ In other words: Is conservative—or perhaps more importantly the operationalization of that concept in the publication of the scholar—the right theoretical concept to address what has happened within the judicial opinion? The second aspect refers to the credibility of the theoretical connections made within a particular publication.⁹⁷ If we want to claim that the fact that the Supreme Court “shifted its perspective”—an interpretation—results in the Supreme Court taking a conservative approach—a theoretical construct—because of the fact that, for instance, the European Court of Human Rights has altered its attitude on that particular aspect of the law, we need to make sure that the conservative approach is really rooted in this fact and not predominantly caused by something else.⁹⁸

III. Readability, persuasiveness, and the identification of presuppositions potentially affecting the inquiry

Apart from the quality standards discussed above, the respondents also mentioned standards that relate to the publication as a whole. The respondents indicated that the quality of a publication also depends on whether it is “linguistically correct” and “easy to read.” That means that “no grammar or spelling mistakes must be made,” that “all sentences and paragraphs in the publication must be fully focused and well formulated,” and that “the writing must be pursued with such clarity that the publication is easy to understand and a pleasure to read.”⁹⁹ The readability of a publication also increases if the author manages to be as brief and concise as she can possibly be to get her message across. The only information that should be brought into the actual publication is the information that is really necessary for understanding the proposed answer to the research question.

A traditional legal publication should also be persuasive. According to the respondents, persuasion can be achieved by obtaining excellent writing and structuring, but more often they associated persuasion with argumentation. According to the respondents, “the argument(s) of the author should be focused, logical, rigorous and sustained” in order to be convincing.¹⁰⁰ The respondents have emphasized that a publication may possess a strong point of view, as long as that does not go hand-in-hand with subjective inquiry. Persuasiveness is enhanced if the scholar can substantiate her point of view with both arguments that speak for the particular point of view taken and arguments that speak against it. The better the scholar succeeds in countering the contra-arguments, the more persuasive the publication becomes. In this sense, thus, the persuasiveness of a publication is strongly determined by the extent to which other quality standards—especially balance and credibility—are met.

Finally, the respondents stated that the difference between an excellent work of traditional legal scholarship and a work of less quality may also rest in the extent to which the researcher actively identifies, explicates, and estimates the implications of the interest, presuppositions, and assumptions on her behalf that may have consciously or unconsciously¹⁰¹ influenced any aspect—question, theory, or particular statements—of the inquiry.¹⁰² As Epstein and King stated before, every research project may be influenced by hundreds of factors.¹⁰³ Of course, this standard is not about mapping all of these factors. That is not even possible, as many of those factors relate to the subconscious. It does, however, mean that the researcher should put sufficient effort in trying to identify those presuppositions or assumptions. As a minimum, then, the researcher is expected to explicate these presuppositions and assumptions. Where possible, the researcher is also well advised to factor in contradictory presuppositions and assumptions.

E. Conclusions

That brings my search of what evidences academic excellence to a temporary ending. In this contribution, I have tried to answer the question: “What standards do legal academics use in evaluating works of traditional legal scholarship?” On the basis of a systematic review of the literature on quality criteria for traditional legal scholarship and an empirical investigation that allowed me to reach deeper insights, I established nine criteria divided over three categories that a researcher—aspirationally—or reader—evaluatively—may employ. I found that a traditional legal publication must, (1) contain a concretely and precisely formulated research question or hypothesis that allows for (2) an original and (3) significant contribution to both society and academia and that (4) is substantiated with adequate methods and research techniques for answering it. Subsequently, in composing the answer to a traditional legal research question, the researcher must (5) use precise and correct references and justify important choices made, (6) be accurate in the sense that the arguments and materials used must actually contribute to answering the research question posed, (7) operate in a balanced way—a complete or representative use of research materials and a fair distribution of weight to the different arguments and insights present within these materials—and (8) achieve a high level of credibility in her descriptions, interpretations, evaluations and use of theoretical constructs. Finally, a traditional publication must also be (9) readable and (10) persuasive, and (11) may not—at least not unaccounted for—be influenced by the presuppositions or assumptions of the author herself.

While this enumeration constitutes the brief answer to the question posed in this Article, a few remarks have to be made. First, the retrieved standards are still formulated in a somewhat abstract way. As was also remarked by some of my respondents, I believe that more insights in the exact manifestation of each of the standards in research and evaluation practice can only be achieved by providing examples of how they play out in particular research scenarios. Also, the enumeration may give the impression of a “checklist.” Such has however never been the aim of this project. Therefore, I hope the reader does not absorb the information provided in such a way. I do not believe it to be a good idea to hold on to the different standards for dear life in future research projects. Rather, it is important to understand the more general, common principles behind those standards and use this understanding to find a balance between being systematic and pragmatic in order to efficiently produce high-quality academic research.

At the start of this contribution I expressed the hope that our discipline engages in a more vivid debate about its quality standards. While this Article provides a first step in this direction, I call upon others to come up with alternatives, modifications, or reinforcements. I address this call to legal scholars—who can contribute by starting to publish more often on the quality of legal research—but also to faculty boards, graduate schools, publishing houses, and journal editors. As the latter actors are specifically involved in the business of evaluating works of traditional legal scholarship, they could certainly do more than they do now to clarify how they do so.