Contested evidence: a Dutch reimbursement decision taken to court

Abstract

This paper examines a remarkable lawsuit in health care rationing. The Patients Association for Interstitial Cystitis sued the Dutch National Health Care Institute for alleged misconduct against Interstitial Cystitis patients, as the Institute decided that bladder instillations with chondroitin sulphate or hyaluronic acid are no longer covered by the basic health insurance. The patients’ organisation challenged the Institute for basing its standpoint on scientific evidence; overruling clinical expertise and patients’ experiences. While scientific advice is often solicited in public health issues, simultaneously, the authority of scientific advice is increasingly being questioned in the public domain. Also, the judiciary is frequently called upon to adjudicate in rationing decisions. Based on an ethnographic study of the National Health Care Institute, drawing on insights from the field of Science and Technology Studies, we analyse this lawsuit as a negotiation of what knowledge counts in reimbursement decisions.

1 Introduction

The Netherlands was recently host to a remarkable lawsuit in health care. The Patients’ Association for Interstitial Cystitis (ICP) sued the health care rationing body of the Netherlands, the National Health Care Institute (hereafter ‘the Institute’), for alleged misconduct against interstitial cystitis patients. In 2013, the Institute issued its authoritative standpoint that bladder instillations with chondroitin sulphate or hyaluronic acid [see Box 1] should no longer be covered by the basic health insurance. Despite the fact that patients claimed to benefit from the treatment, the Institute concluded that there was insufficient scientific evidence supporting its effectiveness. The patients’ organisation legally challenged the Institute for basing its standpoint on scientific evidence, overruling clinical expertise and patient experiences.

Box 1 Bladder instillations

Bladder instillations are a medical treatment used for various bladder conditions, e.g. bladder cancer, interstitial cystitis, painful bladder syndrome. Bladder instillations are a common procedure for urologists and patients can even do the procedure independently in a home setting. During treatment a fine tube (catheter) is inserted into the bladder. The bladder is then filled with a solution, including a pharmacological cocktail of e.g. chondroitin sulphate or hyaluronic acid. The solution remains inside for a period of time (30 minutes to two hours) and is then drained.

In court, the Institute’s use of the principles of evidence-based medicine (EBM) was disputed. The singularity of this case lies in the fact that such a ‘battle of the evidences’ is the exact opposite of what the founders of EBM had in mind. According to David Sackett, the aim of EBM is to ‘integrate the best external evidence with individual clinical expertise and patients’ choice’ (Sackett et al., 1996: 72; emphasis added). The Institute’s evidence-based decision had quite the reverse effect; it led to a momentous clash between experimental and experiential forms of knowledge.

Patients, urologists and the Institute ended up in the courtroom of Amsterdam negotiating the appropriateness of different forms of knowledge for this decision. Drawing on insights of the field of Science and Technology Studies (STS), we studied the legal proceedings as a collective deliberation of what type of knowledge counts in reimbursement decisions. In court, what meaning is given to the concepts ‘evidence’, ‘expertise’ and ‘experience’? What evidential weight is attributed to these respective forms of knowledge? Why were the Institute’s principles of EBM contested in this lawsuit?

This lawsuit is not an isolated case. Over the past decades, the judiciary has increasingly been called upon to adjudicate in rationing decisions (Syrett, 2013). Increased litigation has been attributed to a waning deference to medical judgement, more explicit rationing strategies, a more rights-conscious citizenry and the rise of patient support organisations (Syrett, 2013: 1). Health care rationing institutes find themselves on shifting ground. They have to come to terms with the declining authority of science, a growing recognition and involvement of patients in policy decisions (Wait and Nolte, 2006; Moreira, 2011) and increased judiciary interference (Syrett, 2013; Wang, 2013; Morales, 2015). These are delicate issues for knowledge institutions whose authority, legitimacy and very raison d’être lay, originally, in their acknowledged expertise as a basis for policy decisions. With this analysis, we aim to gain insight into the way in which court cases help rationing bodies like the National Health Care Institute to reflect on their epistemic conventions and – ultimately – to work towards more socially robust rationing decisions.

2 Theorising knowledge disputes

Health care rationing decisions frequently trigger heated discussions in the media and politics. It is commonly assumed that scientific evidence provides the factual, medical–technical and non-political base for such thorny, moral decisions. Yet, STS scholars have shown that there is no given distinction between fact and value (e.g. Latour, 1993) or between objective knowledge and subjective ideas (e.g. Daston and Galison, 1992). Rather, such distinctions are ‘man-made’. What we currently accept as valid knowledge claim results from a collective human endeavour that is contingent to our political and socio-historical setting.

In contemporary knowledge societies, scientific knowledge has become an integral element of politics and governance. At the same time, the authority of scientific advice is being challenged more frequently in the public domain (Bijker et al., 2009; Horstman, 2014). As knowledge has come to play a more central part in health care governance, the question of whose knowledge counts for such decisions seems to have become politically and morally charged (cf. Moreira, 2012). Some scholars have found that the processes of scientification of public policy often go together with processes of politicisation of scientific knowledge (Hoppe, 1999; Weingart, 1999). We regard the lawsuit on bladder instillations as one of those politicisations of scientific knowledge.

3 Methods

This paper is the product of ongoing ethnographic research into ‘how knowing is done’ (de Laet, 2012: 429) being carried out at the National Health Care Institute, a central governing body in Dutch health care. It is within this research context that bladder instillations emerged as a prominent case, exemplary of the Institute’s ‘real life’ struggles. Knorr-Cetina introduced the notion of ‘epistemic culture’ that refers to the historically, politically and socially grounded practices of creating and warranting knowledge (Knorr-Cetina, 1999, 2007). Every place that produces knowledge claims, be it a laboratory or a knowledge institute, is host to a set of ‘practices, arrangements and mechanisms’ which – in that area of professional expertise – make up ‘how we know what we know’ (Knorr-Cetina, 2007: 363). The Institute too is governed by such knowledge routines.

The first author collected empirical data through direct observation, in-depth conversations and document analysis between October 2013 and October 2015. The Institute’s management provided her with an in-house desk (attended one to three days a week), and access to archives. During two years of fieldwork, the first author attended public and closed meetings of internal working groups (weekly), the Institute’s Executive Board (two-weekly), as well as expert meetings of advising committees (Package Advisory Committee, Health Care Quality Committee, Scientific Advisory Committee, Health Care Professions Committee), staff fora and informal lunches. She recorded the ‘practices of knowing’ at the Institute in detailed field notes: what types of studies or expertise are selected in decision-making processes? What preferences exist with regard to methods or disciplines? What are the shared beliefs regarding what a credible knowledge claim looks like?

A few days into the fieldwork, on 18 October 2013, a staff member with many years’ experience referred to the issue of bladder instillations as ‘a hot potato’ (notes#18102013-24). Following this lead, the first author familiarised herself with the case in five in-depth conversations with the staff members involved. With the lawsuit pending, these conversations were replete with references to documents: research papers, journal articles, policy documents and procedural manuals in support of the Institute’s standpoint. At least as many references were made to the letters of complaint, studies, expert opinions and testimonials brought forward by the plaintiff, the ICP. Necessarily, we delved into the Institute’s archives to retrieve all documents on the case and searched PubMed for the journal articles and medical guidelines cited. The following response to a request for an interview illustrates the informational weight these documents had for staff members: what do you want? ‘just a little chat about it? [or] the documents?’ (notes#28102013). Legal records, policy and research papers, internal emails, memos, letters and testimonials were crucial informants in understanding the specificities of the case. Documents did indeed prove to be the prominent ‘artefacts’ of modern knowledge practices (Riles, 2006). In this paper, we cite many such documents, as they possess the conciseness necessary to depict this knowledge dispute satisfactorily.

We closely followed the lawsuit from the writ of summons in summary proceedings on 4 December 2013, until the judicial verdict in proceedings on merits on 19 November 2014. The staff members involved (both policy advisors and legal staff) kept us informed and we carried on tracking the paper trail and attended board meetings and meetings of advisory committees addressing the issue. Two years of ethnographic fieldwork permitted not only a close study of the case, but also of its impact on the Institute: the worries, doubts and reflexivity it brought with it. Being present enabled us to observe the Institute’s continuing struggle to reconcile the unambiguous formal rationality of EBM with the ‘messiness’ of case-based practical rationality (Jenkings and Barber, 2004: 1765; Gkeredakis et al., 2011; Russell and Greenhalgh, 2012, 2014) and the inevitable lack of evidence they often have to grapple with in making rationing decisions.

All authors engaged in an iterative process of (joint) close reading of field notes and legal documents. Sensitised by the literature on EBM and its commentaries (e.g. Sackett et al., 1996; Petticrew and Roberts, 2003; Booth, 2010; Greenhalgh et al., 2014; Parkhurst and Abeysinghe, 2014; Greenhalgh et al., 2015), we progressively focussed on three different forms of knowledge: experimental evidence, clinical expertise and patient experience. We culled from the data any written or verbal statement on the truthfulness, usefulness and significance of these forms of knowledge. In our analysis of the legal proceedings, we focussed on the deliberations of weighing up these respective forms of knowledge against one another. As a ‘member-check’, we discussed our analysis in staff fora, a general board meeting, and with a research team at the Institute.

Working from an STS perspective, we abstained from making any claims about the truthfulness, usefulness or political significance of particular forms of knowledge. Similarly, we make no claims about the appropriateness of the principles of EBM for rationing decisions. It is only by refraining from taking a stand of our own that we can conduct a systematic research into the way other parties give meaning to the notions of ‘evidence’, ‘expertise’ or ‘experience’. By studying how others weigh these respective forms of knowledge and evaluate the appropriateness of the principles of EBM, we aim to learn what is currently required in substantiating and legitimating rationing decisions.

4 Towards evidence-based health care rationing (1965–2006)

The Netherlands has standard basic health insurance for all citizens provided by private health insurance companies. The 2006 Health Insurance Act (Zorgverzekeringswet, Zvw) obliges everybody who resides – or pays payroll tax – in the Netherlands to take out basic health care insurance from a private insurance company. These private insurance companies are expected to negotiate the prices, services, and quality of care on behalf of their insured clients (Van de Ven and Schut, 2008: 779). The market is heavily regulated (Leiber et al., 2015). Income-related subsidies make basic health care insurance affordable for all citizens. A risk adjustment scheme compensates insurance companies for clients with predictably high medical expenses (Van de Ven and Schut, 2009). Benefits covered from basic insurance policy are broadly indicated by law (Zvw art. 10). The Minister of Health formally requests advice from the Institute to specify the medical–technical content of care that can be reimbursed from the basic health insurance. The Institute is asked to take a standpoint in cases lacking clarity about the reimbursement of specific treatments or where there is discord in the field. These standpoints are authoritative in the field. It is in this capacity that the Institute published a negative standpoint in 2013, stating that bladder instillations with chondroitin sulphate or hyaluronic acid were not eligible for reimbursement from the basic health insurance (CVZ [Dutch Health Care Insurance Board] ¹ , 2013a).

In reaching this reimbursement decision, the Institute applied the principles of EBM. The way reimbursement decisions are made in the Netherlands developed over time. As long ago as 1965, the Dutch Sickness Funds Act (Ziekenfondswet, Zfw) determined that, for a medical treatment to be reimbursed from public funds, it must be considered ‘customary’ by the medical professional community:

care provided by medical specialists, as referred to in article 8, para. 1, under a, of the Sickness Fund Act, includes medical, surgical and obstetric care that is determined, in scope, by what is common practice in the professional community (Benefits in Kind (Sickness Fund) Decree, art.8, emphasis added)

From the late 1970s, this so-called ‘common practice criterion’ was much debated within the context of Dutch health care legislation (Van Donk and Tusschenbroek, 2005; De Groot, 2006). Over a period of four decades, the 1965-criterion was reinterpreted and reformed by jurisprudence of the Central Appeals Tribunal (CRvB) and the European Court of Justice (ECJ). First, in the 1979 Acupuncture arrest, the CRvB ruled that ‘common practice’ should be interpreted as the ‘attitude within the profession of medical science and medical practice’ (21 December 1979). Later, the ECJ further developed the interpretation of ‘common practice’ in its 2001 ruling in the Smits-Peerbooms arrest. The court decided that ‘common practice’ could not be interpreted as ‘current perceptions among Dutch professionals’, but as care which ‘has been deemed sufficiently tried and tested in the international world of medical science’ (ECJ 2001, C-157/99). When the Health Insurance Act was implemented in 2006, the much criticised criterion of ‘common practice’ was replaced by a criterion that covered the overtones of the CRvB and ECJ jurisprudence:

The content and range of the forms of care or health services are in part determined by established medical science and medical practice and, where there is no such criterion, by what is regarded in the relevant professional field as responsible and adequate care and services (Health Insurance Decree (Besluit Zorgverzekeringen, Bzv art. 2.1, emphasis added))

This new criterion was an answer to national and European litigation, and to a general call for EBM in the health care sector itself. Legal staff at the Institute had interpreted the ECJ’s 2001 ruling as follows: ‘the court of appeal seems to be referring here to what is actually known as “evidence-based medicine”’ (internal memo 2006).

Starting in the 1970s, EBM gained strong traction in clinical practice and policy rooms, amid growing recognition of ‘the vagaries of clinical experience’ (Armstrong, 2002: 1772) and the inadequacies of knowledge based on personal experience alone. EBM emphasised the use of external, experimental evidence in medical decision making. The randomised-controlled trial (RCT) was seen as the ‘gold standard’ for determining the efficacy of a treatment: evaluating the impact of medical interventions on groups rather than on individuals and through controlled, experimental rather than observational design. The 2006 criterion shows that, in reimbursement policy too, there was a move away from confidence in knowledge based on doctor–patient interactions and their ‘common treatment practices’ (art.8, Sickness Fund Act) and a move towards the new evidence-based paradigm.

Since the introduction of the 2006 Health Insurance Act, the Institute has used the principles of EBM ‘in order to determine whether care complies with the established medical science and medical practice criterion’ (CVZ, 2007). The EBM method currently functions as a tool for validating and weighing medical knowledge from both medical practice and science. It does this, by placing different forms of medical knowledge in a hierarchy ² :

A1) systematic review of at least two A2 level studies carried out independently of one another;

A2) good quality and sufficiently large randomised double-blind, comparative clinical research (RCT);

B) comparative research, but without all the characteristics of A2;

C) non-comparative research;

D) ‘experts’ opinions (CVZ, 2007: 8–9)

The status of evidence within this hierarchy is based on the level of rigorousness in study design to reduce susceptibility to bias. The Institute selects medical-scientific information (scientific articles, medical guidelines, etc.) and ranks it according to its evidential value (level of evidence A1 to D). The ‘cardinal principle of Evidence-based Medicine is that strong evidence supersedes weaker evidence’ (CVZ, 2007: 8-9). The RCT and the systematic review form the apex of this hierarchical order. For positive reimbursement advice, the Institute requires at least one systematic review (A1-level) or two RCTs (A2-level) that show unambiguously positive results about the therapeutic effect of the treatment. As systematic reviews and RCTs are not always available or feasible, the Institute may accept lower levels of evidence in decision making. This may be the case if (double-) blind research is impossible, if the disorder is very rare, or if starting an RCT is no longer viable (because the treatment is a long-standing practice or endorsed by strong international consensus).

The EBM method offers the Institute a general procedure for weighing different forms of evidence and negotiating compatibilities between them. According to the Institute, the EBM method facilitates the ‘meticulous, explicit and judicious use of the current best evidence’ for reimbursement decisions (CVZ, 2007). From an STS perspective, we could say that the EBM hierarchy provides the Institute with a ‘logic of knowledge consumption’ (Knorr-Cetina, 2007: 368). The Institute implemented EBM as -to use the phrase coined by Knorr-Cetina- ‘epistemic machinery’ (1999): a working method, technique or tool of knowing. This does not mean that the formal rationality of EBM is always this clear-cut when used in the practice of decision making. As a staff member commented:

The difficulty [is] that often no good RCTs are available. You often have to be satisfied with ‘lower evidence’. There is a ‘check-list of appropriate evidence’ […] it lists all the reasons for diverging from higher evidence. But every case is still difficult. Every case is different and presents its own problems (notes#31102013)

In the case of bladder instillations, the staff members involved actually thought ‘last year we were starting on a clear-cut case, but it turned out to be fairly complicated’ (notes#31102013).

5 Contested evidence

In 2011, an increase in the prices of three specific brands of bladder instillations (Uracyst, Cystistat, Gepan) attracted the attention of insurance companies. Confronted with increased costs, insurance companies started asking questions about the added value of bladder instillations with chondroitin sulphate or hyaluronic acid (0.9% NaCL). The Institute was asked for a standpoint on the issue. The Institute’s decision-making process started with a literature search. They searched for experimental evidence from double-blind and placebo-controlled RCTs. This is why:

As interstitial cystitis is a disorder that is characterised by a fluctuating course and there is no standard treatment, randomised studies are needed in which bladder instillation with washes containing chondroitin sulphate and/or hyaluronic acid is compared with placebo washes. As the symptoms and outcome parameters are subjective, double-blind studies are preferred (CVZ, 2013a: 10).

Double-blind and placebo-controlled RCTs keep both patient and doctor in the dark about the treatment received. In EBM, it is generally felt that such an experimental design curtails a possibly biased cognition and facilitates a more ‘objective’ measurement of the treatment effect.

The Institute consulted two databases, Cochrane Library and MEDLINE, specialised in biomedical literature, RCTs and systematic reviews. Two double-blind RCTs (A2-level evidence) were selected and a prospective comparative study (B-level evidence). The selected RCTs (Nickel et al., 2012, 2010) showed improvements after treatment with bladder instillations with chondroitin sulphate or hyaluronic acid, but the difference with the placebo group was not statistically significant. The prospective comparative study (Shao et al., 2010) showed statistically significant improvements (in micturition, Visual Analogue Scale (VAS) pain score and bladder capacity), but these were not statistically tested between the groups and the study was of low quality (CVZ, 2013a: 11). The Institute also consulted international guidelines and reimbursement standpoints. In four international guidelines and an (outdated) standpoint of the Canadian Agency for Drugs and Technologies in Health, treatments with bladder instillations were either not mentioned, or had a low level of recommendation due to inconclusive evidence (CVZ, 2013a: 13). From these findings, the Institute concluded that there was insufficient evidence of the effect of this treatment to arrive at a positive reimbursement standpoint.

Before publication, the Institute invited the Dutch Urological Association (NVU) to comment on the findings. The NVU did not contest that ‘the outcomes […] show that at the moment there is insufficient reliable research that provides evidence of the effect of the said washes in the indication fields mentioned’ (excerpt from email). The NVU did, however, express some doubt about the conclusiveness of the experimental evidence:

By the way, this does NOT prove that the washes are NOT effective, which can be confirmed, based on the experience of urologists who frequently use the said washes (CVZ, 2013a: 21).

The Institute, however, required positive proof of efficacy in order to include a treatment in the basic package: ‘care must above all be effective. This demands evidence showing that care does what it is meant to do’ (National Health Care Institute, 2014). Note that this requirement conflates situations where there is definitive evidence of no treatment effect, with situations where there is no conclusive evidence of a treatment effect. The only way towards a positive reimbursement decision would be conclusive proof of the treatment’s effectiveness. That is why the Institute – despite the NVU’s reservations – had to conclude that bladder instillations with chondroitin sulphate or hyaluronic acid did not fulfil the ‘established medical science and medical practice’ criterion as ‘the existing evidence is insufficient for attributing a possible effect of using intravesical treatment with chondroitin sulphate and/or hyaluronic acid to treat interstitial cystitis’ (CVZ, 2013a: 19). One staff member noted: ‘if new positive studies are published, [the Institute] can revise their standpoint’ (email#16122013). But for the time being the matter had been settled. In July 2013 the Institute published its standpoint that bladder instillations with chondroitin sulphate and/or hyaluronic acid would no longer be covered by the basic health insurance.

Soon after this, the Dutch Urological Association (NVU) and the Patients’ Association for Interstitial Cystitis (ICP) took up battle positions against the Institute. First, in a letter to the Institute, the NVU held an earnest plea for continued reimbursement of the instillations. The NVU mentioned the tremendous diversity in the manifestation of interstitial cystitis (IC), the groups of patients, the pathophysiology and the diagnostic findings. Different treatment plans are available for IC (e.g. bladder instillations, antibiotics, painkillers, neuroleptics, neuromodulation, botuine-toxine-A injections and cystectomy with urine deviation), and yet ‘the various different complaints relating to interstitial cystitis are extremely persistent and difficult to treat’ (letter from the NVU). The NVU argued that:

In practice of state-of-the-art clinics that treat patients with IC, this means it is not possible to produce such hard scientific evidence to show which treatments work and which do not. Large series are impossible due to the diversity of the groups of patients and the way in which IC presents, which means that hard evidence will never be supplied. These patients need customised treatment and in the opinion of the experts these bladder fluids are indispensable (letter from the NVU).

Can this complex disorder be adequately captured by an experimental research design? Can the results of the two RCTs (with sample populations of respectively 65 and 98 people) be projected onto the entire population of IC patients? The Institute acknowledged the difficulty of the issue. One of the scientific advisors to the Institute would later put it as follows: ‘It is increasingly difficult to be able to say in general whether something does or doesn’t work… It depends increasingly on how well you define the target group’ (notes#10022014). Together, the NVU and the Institute decided that ‘the best option would be to attempt to identify, based on the existing studies, a subgroup that would benefit from the instillations’ (email#14012014). On the basis of such a subgroup analysis, the Institute may decide to reimburse the washes for specific groups of IC patients, for example patients that show Hunner’s lesions. However, after reviewing the issue, the Institute’s staff concluded that ‘based on the literature […], there is a vague idea that a subgroup exists, but identifying the subgroup is still not possible’ (email#14012014). In the end, it was left to the NVU to come up with a guideline and specified treatment protocol for (different groups of) IC patients. The Institute suggested that, if such a guideline contained an ‘extremely accurate description of the indication and the expertise required of the urologist’, the Institute might decide that ‘no RCTs will be necessary for such a limited group, as we can regard a lower level of evidence as acceptable’ (email#140102014). To this day, however, such a guideline has not been made.

The ICP, too, wrote a letter to the Institute in which it lodged a notice of objection. ‘You were wrong’, the ICP wrote to the Institute, ‘not to take into account in the report – or insufficiently – the opinion of the professional group, which is that this is a case of responsible and adequate care’ (letter from the ICP). This issue was deliberated in court (we will address this issue later). The letter also included several testimonials in which patients claimed that bladder instillations with chondroitin sulphate or hyaluronic acid were less invasive and less painful compared to regular washes without this pharmaceutical cocktail. One of the testimonials read:

The effects of GEPAN can clearly be observed. After bladder instillation washes, the frequency of urination is considerably reduced and nocturnal frequency is also reduced. Furthermore, and naturally because of this, one can also speak of a greatly reduced urge and pain (exhibit 17).

Though the Institute never actually denied that individual IC patients might benefit from a treatment with bladder instillations, they did not take patient testimonials into account. A staff member explained:

The fact that individual patients claim to benefit from the treatment is not sufficient to conclude that the treatment complies with ‘established medical science and medical practice’. [The Institute] does not issue statements about and on behalf of individual patients (email#16122013).

The Institute needed RCTs. Apart from the testimonials, the ICP presented new publications from 2013 showing that patients benefitted from washes with chondroitin sulphate and/or hyaluronic acid. The Institute did decide to review those publications. In an updated standpoint, published at the end of November 2013, they wrote:

Due to a letter of objection from the Patients’ Association for Interstitual Cystitis, a supplementary literature search was carried out for studies dated from November 2012 up to and incl. 27 November 2013. No new randomised studies were found (CVZ, 2013b: 11).

The Institute did find a meta-analysis of a non-comparative study by Nickel et al. (2008) and the two earlier mentioned RCTs by Nickel et al. (2010, 2012) in which: the ‘pooling of these 3 studies revealed that bladder instillation with chondroitin sulphate for interstitial cystitis is effective’ (2013b: 9). However, the Institute concluded that ‘the way this meta-analysis was carried out was methodologically incorrect, because the studies differ from the study set-up, which resulted in a distortion of the results’ (CVZ, 2013b: 11). So the updated standpoint conveyed the same message as before: that the existing evidence was insufficient for attributing a possible effect to bladder instillations with chondroitin sulphate and/or hyaluronic acid and the washes would no longer be covered by the basic health insurance. Staff at the Institute recalled that:

the studies and patient case histories supplied by the ICP did not form a reason for [the Institute] to alter its current standpoint. We confirmed this on 26 November 2013 in a letter to the ICP. […] On 4 December the ICP summoned [the Institute] to appear in summary proceedings (internal memo#07012014).

The legal proceedings started with summary proceedings (ECLI:NL:RBAMS:2014:66) and were followed by proceedings on the merits (ECLI:NL:RBAMS:2014:7727).

6 Legal proceedings

The lawyer of the ICP stated before the court:

I do not contradict that the efficacy of a treatment is a prerequisite for being eligible for inclusion in the insured benefits package. I only contradict that this efficacy has to be proved according to the EBM method […], I feel that the element of ‘practice’ is insufficiently accounted for in the EBM method […] What everyday practice thinks about something has too little evidential value in the EBM method (hearing#30042014)

According to the lawyer of the ICP, clinical practice is given too little weight in the EBM method and the Institute’s standpoint – wrongfully – relied predominantly on RCT-level evidence. The ICP presented a declaration signed by 84 urologists (of the 432 urologists registered with the Royal Dutch Medical Association (KNMG) at that time), claiming that consensus exists amongst urologists that the bladder instillations are effective for some patients. She referred to an article by the founders of EBM (stressing that external evidence alone is never enough, nor is solely clinical expertise, both are essential for medical decision making (Sackett et al., 1996)) to claim that knowledge from ‘everyday medical practice should be placed on the same level as other forms of evidence’ (hearing#30042014).

In response to the urologists’ declaration, the Institute argued that in this case 2 RCTs were available; ‘at such time that such scientific literature is available, the opinions of experts are in fact no longer relevant (strong evidence replaces weaker evidence)’ (email#17122013). The chairman of the Institute’s Scientific Advice Committee explained further: ‘The EBM method uses not only RCTs […]; this form of research is required in particular for those treatments of which we have reasonable doubt about their efficacy’ (hearing#30042014). In the case of bladder instillations, there was such a considerable doubt. A staff member explained:

Some of the patients have a functional syndrome… and these are subjective outcome measures that you examine, i.e., improvement or reduction in symptoms. We were therefore explicit in saying ‘we want randomised studies and not case series because … the placebo group also had a large response’ (notes#03032014).

Weighing the evidential value of clinical expertise against that of RCTs, the judge was of the opinion that the Institute provided grounds showing that, in this case, there was indeed ‘a need of double-blind, controlled research in order to objectively substantiate the experience of doctors’ (ECLI:NL:RBAMS:2014:7727). Hence, in the judicial verdict on 19 November 2014, the court found it was:

justified that, despite the importance of practice experience of doctors – which is acknowledged by the Institute – this is of less consequence than the results of double-blind, controlled studies (ECLI:NL:RBAMS:2014:7727, emphasis added).

The two RCTs by Nickel et al., however, did not remain undisputed in court. In the original standpoint, the Institute stated that: ‘both studies included patients numbers of, respectively 65 and 98 patients, and were under-powered. The duration of follow-up was short’ (2013a: 15). In the course of the lawsuit, the Institute invited an independent research institute (ME-TA) to review the literature using a new EBM method: Grading of Recommendations Assessment, Development and Evaluation (GRADE). GRADE is presented as an innovative EBM method in which the hierarchy of evidence is maintained, yet evidence can be upgraded or downgraded according to its (methodological) quality. The method increases the flexibility of the traditional hierarchy. The GRADE method is regarded as a transparent, explicit and systematic method for weighing evidence not only according to the rigorousness of the study design, but also the quality of that evidence (e.g. Guyatt et al., 2011). ME-TA’s application of the GRADE method to the literature led to the downgrading of RCTs due to quality considerations (underpowered, inadequate duration of follow-up). However, none of the other forms of evidence were upgraded. Working from the cardinal principle that strong evidence supersedes weaker evidence, the RCTs still superseded other forms of knowledge available. The GRADE method did not alter the Institute’s conclusions. The judicial verdict read:

The conclusion of this [GRADE] study is also that the existing evidence is insufficient to be able to attribute a possible effect to the treatment with these bladder instillation products […] Therefore, insofar as it could already be said that the Institute did not make use of the most advanced method for assessing care against [established medical science and medical practice] to arrive at its [initial] standpoint on 23 July 2013, this may remain without consequences, as the newer method did not lead to any different result (ECLI:NL:RBAMS:2014:7727).

After all, the evidential value of RCTs – despite serious quality considerations – was considered to be higher than the evidential value of clinical expertise.

In what follows, we focus on deliberations weighing patient experiences against RCTs. In court, the ICP reproached the Institute for failing to consult patients’ organisations in its standpoint. The ICP stated that it ‘was never involved in the Institute’s research. We were only informed after the standpoint had already been determined’ (hearing#30042014). According to the ICP, this was a breach with the ‘due care principle’ in policymaking. In its defence, the Institute claimed that it is standard procedure to consult professional associations of medical specialists (in this case the NVU), but not necessarily patients’ organisations. The Institute takes into account the ‘specific circumstances and interests of the group of patients’ only when there are questions about the feasibility of scientific research or if ‘it is necessary to be satisfied with a lower level of ‘evidence’. This is […] not the case here’ (statement of defence). The Institute explained that in this case ‘two RCTs had been carried out, proving that it is possible to carry out a double-blind RCT involving this disorder’ (statement of defence). As RCTs were available, they took precedence over patient preferences and testimonials for this reimbursement decision. The Institute claimed:

specifically because this involves a heterogeneous group of patients, it is necessary to carry out research among subgroup of patients in order to demonstrate that the treatment is effective for specific groups of patients. The Institute has always maintained that, when such research becomes available, a new assessment can take place that may result in a different opinion (statement of defence).

During the legal proceedings, the ICP presented journal articles and abstracts of promising studies. It made no difference, however, as none of these studies matched the evidential power that was attributed to the two RCTs used in the Institute’s standpoint. All in all, the judge ruled that:

In the opinion of the court, it is irrelevant whether failing to consult the patients’ association is incompatible with the due care principle, as it has been established that the information that the ICP could have supplied – and which has been supplied during these proceedings – did not lead to a different assessment (ECLI:NL:RBAMS:2014:7727, emphasis added).

Furthermore, the ICP noted that ‘no attention whatsoever was given to “soft principles”, such as quality of life, patient satisfaction and the experience of patients and care providers’ (hearing#30042014). According to the Institute, however, individual patient preferences do not play any significant role in collective coverage decisions (CVZ, 2007: 10). A director at the Institute pointed out the reason why RCTs are valued over patient preferences:

the use of non-personal statistical data on a group level makes it possible to weigh up the personal interests of one group of patients against those of another group of patients. Every euro that you spend on bladder instillation liquids is no longer available for spending on care for elderly persons suffering from dementia, to name just one… (notes#25062015)

For health economic purposes, RCTs permit a meta-assessment of health benefits between different treatments. The use of RCTs and systematic reviews enables one to calculate the expected gain from different treatments, thus facilitating a choosing between them. One of the board members explained that the Institute always has to consider the limited public funds available and is tasked to ensure ‘that the general public, who pay the premiums, are paying for something that is worthwhile’ (notes#30062015). Without evidence of the effectiveness of a treatment, the government could be paying for care that is actually ineffective or – in a worst case scenario – is facilitating care that might be doing more harm than good. In the end, the court decided, based on the same line of reasoning, that:

It is the intention of the legislator to confine the basic benefits package to those forms of care that can be trusted to be effective. This is to be tested against ‘established medical science and medical practice’. Whether care satisfies this legal criterion must be determined as objectively as possible. In the opinion of the court, the Institute has given – with its selected method [EBM] – a correct application of this criterion (ECLI:NL:RBAMS:2014:7727)

7 Discussion

In this article, we showed how and why the Institute reached a negative reimbursement decision with regard to bladder instillations with chondroitin sulphate or hyaluronic acid for the treatment of IC. Our analysis of the decision-making process showed the tension between EBM’s formal logic and its practical rationality: the struggle to be accountable to both ‘good’ EBM and local contingencies (Jenkings and Barber, 2004: 1765). The opinions of clinical experts and patients’ experiences with the treatment, though positive, were considered unreliable due to subjective symptoms and outcome measures and a known risk of placebo effect. The RCTs were underpowered, inadequate with regard to duration of follow-up and performed on a problematically heterogeneous population. The trials did detect some positive effect of the treatment, but the effect was not statistically significant. This could be due to the fact that the treatment simply did not work, but the lack of statistical significance could also be due to the fact that it was an underpowered trial. (It is still plausible that the treatment actually worked, but the RCTs were too small to reveal this.) Ultimately, the Institute concluded that there was ‘insufficient evidence’: there was no proof that the washes were effective, nor was there any conclusive evidence that they were not. Actually, reimbursement decisions are frequently associated with uncertainty, as data on effectiveness, safety, etc. are seldom complete (Boon et al., 2015). Nonetheless, an occupational burden rests on the Institute to take a standpoint, despite this lack of certainty. In line with its general policy, the Institute concluded that without conclusive evidence of its effectiveness, the treatment cannot be reimbursed from public funds.

In court, patients and urologists challenged the appropriateness of the EBM method for making this decision. Before the judge, patients, urologists and policy makers negotiated whether either experimental knowledge from RCTs or experiential knowledge from the clinic ought to play a role in this reimbursement decision. How did a method, aimed at integrating experimental evidence, clinical expertise and patient experiences result instead in a lawsuit in which these forms of knowledge were weighed against one another? The lawsuit on bladder instillations highlights a paradox in EBM’s success in both clinical practice and policy rooms. Originally, EBM aimed at an integrative assessment of the best external evidence with individual clinical expertise and patients’ choice for decision making in the clinic (Sackett et al., 1996: 72). In the 1990s EBM also gained strong traction in policy-making circles. Making an integrative assessment for a national, stringent reimbursement decision in a policy room, however, means that such an assessment can no longer take place at individual case-level in the clinic. A negative reimbursement decision, unavoidably, means less discretionary room for professionals and patients, as a treatment is simply no longer (financially) available. This is how an evidence-based policy decision can, paradoxically, inhibit evidence-based decision making in clinical practice. In line with this, several authors have urged caution in the transportation of the EBM hierarchy from clinical medicine to the realm of policy making (Petticrew and Roberts, 2003; Booth, 2010; Parkhurst and Abeysinghe, 2014). Nonetheless, the court ruled that, with the EBM method, the Institute had given a correct application of the criterion of ‘established medical science and medical practice’.

Studies have shown that the judiciary has long been reluctant to interfere in rationing decisions and has only recently evinced greater willingness to do so (Syrett, 2013; Wang, 2013; Morales, 2015). When they do ‘courts are not normally empowered to substitute their view of the correct decision for that of the original decision maker, but only to ensure that the agency has the lawful authority to take the decision in question and that it complies with principles of procedural fairness’ (Syrett, 2013: 7). In line with this, the Dutch court ruled ‘procedural fairness’ in case of the Institute’s negative reimbursement decision. As an inevitable performative effect, the verdict also ratified the principles of EBM, the idea that ‘objectivity’ is obtained through experimental research, and that the value of an RCT exceeds that of other forms of knowledge.

The ICP and NVU challenged the evidential value of RCTs and of EBM’s epistemic propriety, but to no avail. Their objections, however, resonate with the general critique on RCTs that a ‘patient’s unique and complex predicament is poorly served by applying a recommendation derived […] from an average result in a select population sample’ (Greenhalgh, 2002: 396). It seems that this holds especially true for RCTs performed on heterogeneous populations. IC is not a single condition. It is a term used in clinical practice to refer to a ‘spectrum of urological symptoms’ with different underlying causes (often unknown) (Offiah et al., 2013: 1243). Those suffering from it are likely to respond differently to treatment (Nordling et al., 2012; Offiah et al., 2013). For this reason, the ICP and NVU criticised the adequacy of RCTs as a means to gauge the effect of IC treatments. Acknowledging the issue, the Institute insisted throughout the lawsuit that if new subgroup studies were to become available, they could eventually reconsider their standpoint for specific groups of IC patients with proven benefit from this treatment. Yet, with the suspicions that rests on functional disorders and placebo-sensitive treatments, how viable is it that such studies will actually be taken up by the medical community? A scientific advisory member to the Institute remarked: ‘euhm, who would want to carry out an RCT involving something that doesn’t work?’ (notes#30302014). The invitation to conduct multiple trials on sub-populations might be hypothetically rigorous, but it is often practically unrealistic. This type of practical consideration, however, was dismissed in court.

In court, the question of procedural fairness was largely conflated with the issue of doing ‘epistemic’ justice (Fricker, 2007). This term refers to justice in a way that all parties are duly recognised ‘in their capacity as knower’ and have a fair share of our ‘collective interpretative resources’ at hand in order to manifest what they know (Fricker, 2007: 1). Some scholars have warned that the principles of EBM are actually biased against patients (Greenhalgh et al., 2015) and that patients are easily and often ‘epistemically marginalised’ in the bureaucracy of health care systems (Carel and Kidd, 2014: 530). For example, most clinical research has ‘minimal patient input’, which often entails that studies reflect the outcomes that matter to researchers, rather than those that matter to patients (Greenhalgh et al., 2015: 2; Wiering et al., 2016). Furthermore, greatly reduced levels of credibility are assigned to patient testimonies (Carel and Kidd, 2014) and EBM’s hierarchy of evidence in particular tends to ‘devalue patient experiences’ (Greenhalgh et al., 2015: 3). Those who are familiar with the clinical picture of functional disorders will recognise that this holds especially true for patients suffering from medically unexplained physical symptoms like (some cases of) IC, fibromyalgia, chronic fatigue syndrome, etc. This is also a group of patients with the least investigative resources at their disposal to make sense of their experiences (cf. Fricker, 2007). As mentioned earlier, IC and its treatment do not easily attract research attention or funds, in contrast to, for example, breast cancer. This means that IC patients are often left at the mercy of a few dedicated experts (see Shao et al., 2010). These are the ‘epistemic margins’ from which these patients have to argue their case: a highly reduced credibility and few entry points or resources to generate the required experimental evidence. This raises the question of what ‘procedual fairness’ signifies, if those who are subdued to these equal procedures have such unequal epistemic means at their disposal?

Lawsuits add to valuable learning processes (Syrett, 2013) and it seems that most of the learning is done outside the courtroom. Being at the Institute allowed us to witness the ‘tinkering’ done to improve EBM’s principles for the practice of rationing decisions. As Russell et al. (2011: 2) noted, the rationing debate has long moved away from ‘early naive faith’ in the ability of EBM to straightforwardly guide us in complex rationing decisions. Since 2012, the Institute has been experimenting with conditional reimbursements, by nevertheless including treatments that do not fulfil the legal criterion of ‘established medical science and medical practice’ – for a specific period of time – into the insured package. This is subject to the condition that during that period data are collected on the efficacy of the care (CVZ, 2012; Boon et al., 2015). Aware of the shortfalls of EBM’s strict hierarchy, the Institute started implementing GRADE, a method that allows more flexibility within the original hierarchy and makes it possible to include patient panels in decision-making processes. Furthermore, the Institute now consults patient organisations more systematically in connection with rationing decisions. Together with the Institute for Medical Technology Assessment, the Institute drew up an ‘appropriate evidence’ questionnaire: a guide to whether RCTs can reasonably be expected in specific cases (Heymans et al., 2013). The above shows that, although judiciary interference is often experienced as a negative incident, lawsuits encourage institutional reflexivity.

The academic debate on health care rationing entails a long-established discussion about institutional processes, deliberative procedures and technical approaches (e.g. Daniels and Sabin, 1997; Ham and Roberts, 2003; Klein and Maybin, 2012). We took an ethnographic approach that focussed on the working of these processes, procedures and rationality in practice, going back and forth between principles and policy ideals, and the ‘messiness’ of real-life decision making (cf. Jenkings and Barber, 2004; Gkeredakis et al., 2011; Russell and Greenhalgh, 2012, 2014). This type of analysis highlighted a set of practical issues with EBM that were not directly addressed in court. We argue that ‘epistemic fairness’ deserves our explicit attention, just as much as procedural soundness does, if not in court, then at least in academic and policy circles.