The growing prevalence of discourses on transparency is becoming ‘irresistible’ (Marshall et al., Reference Marshall, Shekelle, Davies and Smith2003: 145) and being reinforced policy and practice in many health systems (Tu and Cameron, Reference Tu and Cameron2003; Blomgren, Reference Blomgren2007; Clarke and Oakley, Reference Clarke and Oakley2007; Levay and Waks, Reference Levay and Waks2009; Pelone and Riccardi, Reference Pelone and Riccardi2013; Kraska et al., Reference Kraska, Krummenauer and Geraedts2016). However, there has been limited comparative evidence of the behavioural responses to public reporting by health system stakeholders, and explanation of those responses (Marshall et al., Reference Marshall, Shekelle, Davies and Smith2003) and hardly any conceptual analysis of what these developments mean for the medical profession or health systems. This paper seeks to remedy those gaps in knowledge in relation to public reporting of the outcomes of cardiac surgery in England and California.
The paper is organised into three main sections. First, we examine the emergence and development of public reporting as a form of transparency and accountability. We also present a conceptual framework of institutional change by which these developments may be understood. Second, we present two qualitative case studies to examine the behavioural responses of stakeholders following the inception of public reporting of mortality associated with cardiac surgery. These case studies examine the responses to public reporting at the micro, meso and macro levels in the English and Californian health systems. Third, we interpret these findings in terms of the conceptual framework.
1. Understanding public reporting in context
In the past two decades, transparency and openness have become commonplace features of most health systems. Arguably, the prime example of transparency in health systems is the public reporting of (medical or surgical) performance. The dissemination of process and/or outcome measures to the public via the internet represents a major shift for health system policy-makers, the staff that work in them and the patients treated by them. This paper examines the responses by surgeons, health care executives and policy-makers to public reporting in England and California. (Patient perspectives are not included in this study.)
Underlying the emergence of public reporting in health systems is a series of socio-political and cultural changes (Exworthy et al., Reference Exworthy, Smith, Gabe and Jones2010). First, trust in medical professionals (and the wider health system) has been undermined by a series of ‘scandals’ which have often revolved around the way in which (medical) performance is managed, usually by their peers (Scott and Ward, Reference Scott and Ward2006; Calnan and Rowe, Reference Calnan and Rowe2008; Timmermans and Oh, Reference Timmermans and Oh2010; Dixon-Woods et al., Reference Dixon-Woods, Yeung and Bosk2011). As a result, doctors in many health systems are no longer in sole charge of determining standards of (their) performance, monitoring those standards and taking action to remedy ‘poor’ or reward ‘good’ practice.
Second, advances in information technology have fostered the emergence of public reporting. The ability to accumulate, store and disseminate comparative data to public audiences has enabled medical practice to become auditable and comparable (Power, Reference Power1997; Waring and Currie, Reference Waring and Currie2009; Jeacle and Carter, Reference Jeacle and Carter2011).
Third, the advent of ‘new public management’ (in many health systems) has instituted new forms of accountability and effectiveness, which are often promulgated by clinical hybrid managers (Lynn, Reference Lynn1998; Llewellyn, Reference Llewellyn2001). Neo-liberal policies of ‘consumer’ choice and competition have placed greater emphasis on market-based approaches in health systems (Davis et al., Reference Davis, Collins, Schoen and Morris1995). Both choice and competition have had limited impacts but have challenged professional ownership of medical performance (Greener, Reference Greener2003; Bevan et al., Reference Bevan, Evans and Nuti2019). Cost containment has also been implicit in discourses of new public management and so public reporting has been framed, to some extent, by this. Often, this has been described as a search for better value health care (Gray, Reference Gray2011).
Fourth, the growth of a consumerist culture has ‘encouraged patients to become proactive and informed about the choices they make regarding their health by accessing independent knowledge’ (Gabe et al., Reference Gabe, Exworthy, Jones and Smith2012; also Roberts, Reference Roberts1999).
Collectively, these changes have made transparency a normative doctrine in health systems; stakeholders ‘must’ claim the need for greater transparency as if being against it would be morally corrupt (Hood, Reference Hood2007; Meijer, Reference Meijer2009). This is despite questions about its definition, implementation and consequences. For example, clinicians could manipulate the data and/or their behaviour if incentives become too explicit (Smith, Reference Smith1995). Similarly, the surfeit of (often, esoteric) information might also constrain patients in their health care decision-making (Tsoukas, Reference Tsoukas1997; Strathern, Reference Strathern2000). Therefore, public reporting may not necessarily lead to ‘improvements’ (Pawson, Reference Pawson2002).
The advent of public reporting has been enabled by these developments but, for it to be effective at improving performance, it requires the presence of key factors, namely the widespread availability of information on a regular basis and the clear presentation of performance rankings (Hibbard, Reference Hibbard2008; Bevan et al., Reference Bevan, Evans and Nuti2019). This ‘programme theory’ of public reporting thus relies on four stages: identification, naming, sanction and recipient response (Pawson, Reference Pawson2002; Exworthy et al., Reference Exworthy, Smith, Gabe and Jones2010). However, each has potential unintended consequences which might subvert the overall effectiveness of public reporting: culprit misidentification, dissemination dissimulation, sanction misapplication and unintended outcome.
The origins of contemporary public reporting in health care can be traced back to developments in Pennsylvania and New York states in the 1980s (Chassin, Reference Chassin2002; Hussey et al., Reference Hussey, Luft and McNamara2014). California represents a particularly advanced example of public reporting in the USA (compared to other states), having been in operation for over two decades. The diffusion and transfer of this policy within the USA and to other countries took place during the 1990s and 2000s (Dolowitz et al., Reference Dolowitz, Hulme, Nellis and O'Neill1999). For example, in recent decades, the English health system has become more attuned to US health policy developments including public reporting (Marshall et al., Reference Marshall, Shekelle, Brook and Leatherman2000; Walshe, Reference Walshe2001; Ham, Reference Ham2005; Exworthy. and Freeman, Reference Exworthy, Freeman, Marmor, Freeman and Okma2009; Oliver and Brown, Reference Oliver and Brown2011). There has been a notable focus on comparisons between the UK and the USA (e.g. Ham, Reference Ham2005; Exworthy et al., Reference Exworthy, Bindman, Davies and Washington2006; Exworthy and Freeman, Reference Exworthy, Freeman, Marmor, Freeman and Okma2009; Oliver and Brown, Reference Oliver and Brown2011; Bevan and Fasolo, Reference Bevan, Fasolo and Oliver2013) and specifically, England and California (e.g. Feachem et al., Reference Feachem, Sekhri and White2002). (The English reference here is deliberate since greater divergence has been apparent in the devolved nations of the UK (Greer, Reference Greer2008; Sanderson et al., Reference Sanderson, Allen, Peckham, Hughes, Brown, Kelly, Baldie, Mays, Linyard and Duguid2013).) By contrast, Scotland was an early pioneer of public reporting in the UK (Mannion and Goddard, Reference Mannion and Goddard2003; Marshall et al., Reference Marshall, Shekelle, Davies and Smith2003).
It is significant that cardiac surgery has been at the forefront of public reporting initiatives in the USA and UK. Some see it as a highly prestigious specialty of medicine (Katz, Reference Katz1998; Creed et al., Reference Creed, Searle and Rogers2010) and yet is also open to greater measurability (compared to other specialties) (Harrison, Reference Harrison2002).
Descriptive accounts of public reporting are valuable but they also need to be explained conceptually. The case of cardiac surgery might be comparable but the framing and implementation of public reporting needs to be understood better, both now and in the future for other health systems. Here, we seek to do that by hypothesising whether two contrasting case studies might be converging towards a generic model of public reporting or diverging along separate lines. Convergence would indicate a form institutional isomorphism (Beckert, Reference Beckert2010) whereas divergence would point towards path dependency (Greener, Reference Greener2005); a useful way to interpret this in terms of Scott's (Reference Scott2008) institutional pillars. This approach is consistent with wider institutional theory which posits institutions as the array of interests and ideas, norms and practices as well as formal structures and rules (Alvesson and Spicer, Reference Alvesson and Spicer2019). Scott's notion of institutional pillars is insightful as it argues that institutions comprise regulative, normative and cultural-cognitive elements, which Scott termed ‘pillars’ (Checkland et al., Reference Checkland, Harrison, Snow, McDermott and Coleman2012; Muzio et al., Reference Muzio, Brock and Suddaby2013; Brunton, Reference Brunton2017). The pillars are not mutually exclusive; they interact together so that each shapes and is shaped by the behaviours by individuals and groups, at different institutional levels – at the clinical–patient interface, at organisational levels and at policy/institutional levels. The pillars are thus useful heuristic devices, transcending institutional boundaries. The regulative pillar refers to the espoused rules and sanctions of the institution, as pronounced in policy documents or legislation. The normative pillar comprises the standards and principles of institutional practices; this implies the expected duties and obligations of (say) surgeons in conducting their clinical practice. The cultural-cognitive pillar addresses the deeply seated assumptions and beliefs underlying actions; such ‘symbolic systems’ of the profession might, for example, confer reputational effect on public reporting (Shekelle et al., Reference Shekelle, Lim, Mattke and Damberg2008). This framework can explain institutional patterns of change, especially those which are often portrayed as clashes between dominant and challenging logics, such as managerialism and professionalism (Ocasio et al., Reference Ocasio, Thornton, Lounsbury, Greenwood, Oliver, Lawrence and Meyer2017). These pillars have the advantage of transcending the macro–meso–micro levels. They also help in comparative analyses.
2. Methods
The behavioural and policy responses to public reporting (in the form of mortality rates associated with individual named surgeons) are now examined empirically, drawing on data collected at the micro, meso and macro levels in the context of California and England. (This replicates Adler and Kwon's (Reference Adler and Kwon2013) distinction between individual, organisation and institution, as well as the inter-connected nature of Scott's ‘pillars’ framework.) The micro level refers to surgeons’ views and behaviour (Hannan et al., Reference Hannan, Stone, Biddle and DeBuono1997; Barr et al., Reference Barr, Bernard, Sofaer, Giannotti, Lenfestey and Miranda2008); hospitals which provided cardiac surgery were either large, multi-specialty centres or (smaller) units dedicated to surgery. The meso level to managerial decision-making processes (Dranove et al., Reference Dranove, Kessler, McClellan and Satterthwaite2003); and the macro level to the external regulatory and policy environment (Gabe et al., Reference Gabe, Exworthy, Jones and Smith2012). This multi-level approach helps gauge the degree of enrolment and resistance to public reporting among surgeons, health care executives and health policy-makers, as well as the interactions between different levels and stakeholders in the health system. (Other studies have considered the impact of public reporting on outcomes; e.g. Bridgewater et al., Reference Bridgewater, Grayson, Brooks, Grotte, Fabri, Au, Hooper, Jones and Keogh2007; Behrendt, and Groene, Reference Behrendt and Groene2016.)
Two separate studies were conducted to investigate the responses to public reporting amongst institutional agents, their use of the data and consequences. The micro level of the English study involved one hospital and its local organisational and managerial setting (meso). The macro level comprised of the external regulatory and policy environment which would affect all similar hospitals. Data collection entailed (16) interviews with senior and junior surgeons, data analysts, (hospital and purchaser) managers, regulators and policy-makers (some of whom were surgeons). The interview sample was selected by virtues of their position, role and/or involvement in public reporting; this was identified by the head of surgery department (as a key informant), documentary evidence and observation. (Additionally, the English case study comprised of participant observation of surgeons in (10) morbidity and mortality (M&M) meetings and in daily practice (over 2 days); given the comparative aspect of this paper, these data are not used here.) The USA study undertook a similar approach albeit in one state (California). It is hard to generalise across the diversity of the USA but California has previously proven to be a useful comparator of the UK or England (e.g. McDonald and Roland, Reference McDonald and Roland2009; Hussey et al., Reference Hussey, Luft and McNamara2014). At the micro and meso levels, (10) interviews were held with practising surgeons in two hospitals, data analysts at the state level, regional purchasers and health service researchers. These individuals were identified with the aid of an academic researcher based in California, and documentary evidence. The two hospitals were selected because of the contrasting organisational status; one comprised of a large health plan and the other was an independent organisation. The macro level was different in that federal health policy has a different resonance in each state; therefore, macro level interviews were not undertaken. Instead, corroborative evidence from policy documentation and published research was garnered. Interviews in both case studies involved similar questions (Boxes 1 and 2); interview were semi-structured, allowing supplementary questions to probe answers in more depth. Interview questions were derived from a programme theory of public reporting (Exworthy et al., Reference Exworthy, Smith, Gabe and Jones2010), which sought to identify behavioural responses in terms of how public reporting should and does work in practice as well as the consequences (intended and otherwise) for different stakeholders. These formed the a priori themes for subsequent analysis.
Box 1. Interview schedule for California participants
1. Describe your job in relation to California Coronary Artery Bypass Graft Outcomes Reporting Program (CCORP)
2. What would you describe as the key aspects of CCORP which have been instrumental in its uptake by clinicians and organisations?
3. Would you like to see CCORP changed?
• Expanded, eliminated, refocused, new measures
4. How are your analyses of CCORP data being used beyond public reporting?
• E.g. Pay-for-performance, quality improvement, health plan designation of cardiac centres of excellence.
5. What other strategies are being used to manage surgical performance and improve it?
6. What might CCORP have achieved in the next 2 to 5 years?
7. Have you (or your organisation) encountered resistance by clinicians or hospitals to report data?
8. What have been the unintended consequences from CCORP?
9. What has been the impact of CCORP upon surgeons in training?
Box 2. Interview schedule for England participants
1. How was poor surgical performance monitored before (personally and within the profession, their experience)?
2. What has been your role in implementing public reporting (locally/nationally)?
3. What discussions with colleagues took place locally about public reporting?
4. What were staff concerns about implementing public reporting?
5. Has there been any resistance to public reporting?
6. What has been the impact of public reporting upon relationships with staff, with managers, with external agencies?
7. How has public reporting affected surgical work?
In both studies, patients were not included in the data collection because the focus of enquiry was upon professional and institutional responses to public reporting. Also, the resources available to both studies precluded the inclusion of patients. Even though evidence suggests a limited use of public reporting data by patients (Marshall et al., Reference Marshall, Shekelle, Davies and Smith2003; Fung et al., Reference Fung, Lim, Damberg and Shekelle2008), patient perspectives remain significant because they are liable to change in light of IT advances, previous patient behaviour and health system incentives. They are, however, beyond the scope of this paper.
All interviews (except for one) were audio-recorded (with permission) and transcribed. Interviews lasted 45–75 min. Contemporaneous, observational field notes were written in full. Analysis of interview transcripts and observational field notes was undertaken in a similar way for both studies; this involved researchers reading and coding transcripts initially before their widespread application. As the English case study was conducted before the Californian one (and despite the contextual differences), there were fewer discrepancies in the coding process for the latter. A software programme was not used to analyse transcripts given the relatively small sample size. The ‘framework’ approach guided the data search for commonalities and differences, thus identifying themes within and between different levels (Ritchie and Spencer, Reference Ritchie, Spencer, Bryman and Burgess1994; Gale et al., Reference Gale, Heath, Cameron, Rashid and Redwood2013). We drew descriptive inferences based on a priori and emergent themes before applying them to Scott's conceptual perspective. In what follows, we present our findings in terms of micro, meso and macro levels, each with sub-themes. Interviewee names have been anonymised.
A number of limitations are manifest in this comparative study. First, the complexity of the study (viz. two case studies, three levels of empirical data) militates against a more extensive account within the scope of the journal. Second, a longitudinal study would have helped to gauge the ways in which public reporting has evolved. Some data offer a corroborative perspective but this is inevitably partial. Third, a more extensive data collection could have given a more rounded assessment. More staff (e.g. nurses) and patients could have been interviewed but the focus of both case studies was on the changing nature of medical professionalism. More observational data would also have provided more depth of insight. These factors illustrate the limitations of the study and also highlight areas of future research.
3. Results
3.1 Micro level
In both studies, some surgeons opposed the ways in which surgery is measured and data were attributed to named individual surgeons. Initially, they did so in a mix of passive and active resistance, especially where individual/organisational reputation was at stake; this has become more muted over time. In England, all respondents noted that some surgeons remained resistant to public reporting although others surgeons were more accepting, as illustrated by Derek and Ralph.
‘There are quite a few dissenting voices yeah, quite a few, the laggards for want of a better word, decent people…’ (Derek, surgeon and regulator, England).
‘There's also an aspect to which having the results published has been cathartic and we've done it, we've been there, you know. Any fall out has happened, we've done it over a number of years now and I think people are quite relaxed about it’ (Ralph, surgeon, England).
Public reporting had even prompted resistance in the form of resignations from the surgeons’ professional body.
‘This whole issue has been quite controversial and in some areas, some members have actually resigned from the Society’ (Larry, surgeon and professional leader, England).
Similar resistance and resentment to public reporting were also apparent in California.
‘Surgeons have been resistant to measurement’ (Arthur, researcher, California).
Resistance concerned the unit of analysis in both forms of public reporting. Naming of individual surgeons within specific hospitals, it was argued, gave misleading inferences about team-based care.
‘There's a lot of push-back [by surgeons to public reporting] which we are philosophically sympathetic to because if you really believe in team care and integrated care and co-ordination – all that stuff, picking individual physician as a unit of analysis is kind of silly’ (Daniel, purchaser, California).
However, Californian respondents thought practicing surgeons had become more accepting of public reporting over time. This might be gauged by the falling numbers of appeals made to the state-wide body overseeing the public reporting.
‘So we've been through it [appeals by surgeons] three times and it's [numbers appealing] gone down each time. The first time we probably had about in the twenties. We learned a lot. People were appealing things that it affected their ratings … and the next time it was less and then this last time which was two years ago, we just had a handful’ (Hannah, data analyst, California).
A central concern at the micro level was whether public reporting was voluntary or mandatory. Public reporting schemes have mostly shifted from voluntary to mandatory disclosure. In California, public reporting (California Mortality Reporting Program; CMRP) was initially voluntary at the hospital level (Parker and Damberg, Reference Parker and Damberg2005).
‘About 25 to 30% of hospitals did not participate but this was largely because they were not STS [Society for Thoracic Surgeons] members and did not have to computer software to enable them to report mortality data’ (Jack, data analyst, California).
However, by 2005 report, non-participation was 36% (Parker and Damberg, Reference Parker and Damberg2005: 3). In the early-mid 2000s, coronary artery bypass graft (CABG) reporting in California became compulsory (California CABG Outcomes Reporting Program; CCORP) with the passage of legislation (Senate Bill 898 in 2001) (Lee and Carlisle quoted in Parker and Damberg, Reference Parker and Damberg2005: v). The concern that public reporting would be mandated (and thus introduce greater regulation of surgeons) had been recognised at the outset.
‘I know when I was in New York, in the mid '80s going to early '90s that the whole discussion was even the validity of a database, the participation of a database and one of the main attacks on having this database that it would be eventually used as a policing’ (Henry, surgeon, California).
In England, voluntary participation has been prized by the surgical specialty which secured surgeons’ participation as public reporting moved from an internal educational exercise to external audiences (Exworthy, Reference Exworthy1998; Keogh et al., Reference Keogh, Spiegelhalter, Bailey, Roxburgh, Magee and Hilton2004). According to two English respondents, about one-sixth of cardiac surgeons did not disclose their mortality rates in the 2000s. Indeed, as one English respondent commented:
‘[The regulator] never ever said that publication of individual surgeon's results should be mandatory’ (Barry, surgeon and policy-maker, England).
The fear expressed by most English surgeons was that reporting would become mandatory and linked to more explicit control of their practice, with a loss of clinical autonomy.
‘I think these things [public reporting] are very good when used internally and professionally. However, when these figures go in the so-called public domain, I think this is where I feel a little bit uncomfortable with, because then… the data needs a lot of knowledge to interpret it, to understand it’ (Tim, junior surgeon, England).
By 2013, the re-launch of public reporting in 10 specialties referred to a high(er) degree of participation in England. Regarding this initiative, Bridgewater (Reference Bridgewater2013) cites legal advice, indicating that individual results should not be published without individual surgeons' consent. However, he claims that data about non-participating doctors will be relayed to local executives and purchasers but not to the public. He, nonetheless, reassures that consent rates are ‘more than 96%’; hence, non-participation appeared to have to fallen to 4%.
The use of publicly reported data by patients in their decision-making varied by circumstance and context between England and California. In both case studies, surgeons commonly stated that patients did not use the data.
‘My experience is that very few patients tend to come and want to discuss the results or discuss the stuff on the website’ (Rupert, surgeon, England).
‘Public reporting has not, in my mind in the last 5 years, has not really affected patient behavior and it hasn't really affected cardiology referring. …Over the last 5 years,… one of the things that we worried about… is that when we share this data, patients aren't going to come and see me. That's not borne out. Cardiologists don't see this data. I mean they see it but send the patient. Their human behavior is that they know that the high risk and they continue to send [patients] to you’ (Henry, surgeon, California).
These views corroborate the conclusions of Marshall et al. (Reference Marshall, Shekelle, Davies and Smith2003) and Shekelle et al. (Reference Shekelle, Lim, Mattke and Damberg2008) in reviews of extant evidence, mainly from the USA. (This would also explain the apparent ‘failure’ of naming and shaming schemes in England; Bevan et al., Reference Bevan, Evans and Nuti2019.) However, the reasons diverge. The English policy of ‘patient choice’ (which encouraged patients to select between ‘competing’ hospitals) had a mixed uptake (Greener, Reference Greener2003; Fotaki, Reference Fotaki2006). Moreover, cardiac surgery is a tertiary service and requires referral from a cardiologist, so patient choice is less relevant (and another reason for their exclusion from this study). In California, patients’ decision-making is heavily mediated by the institutional rules of their health insurers. Jerry described one patient from a nearby town who wanted to be treated at his facility.
‘Their insurance won't pay for it. So the people are trapped by their insurances much more so in the State of California than any other state. So a great example is [the town's] hospital that has a very low [volume of patients]…and even if they wanted to leave there, the majority of patients in that area can't leave. They're trapped by their insurance’ (Jerry, surgeon, California).
Although the ‘exit’ strategy was less available (compared to the voice and loyalty strategies) (Hirschman, Reference Hirschman1970) in California and England, there are signs of patients becoming less reliant on traditional sources of information about clinical performance (Tritter, Reference Tritter2009).
‘25% of patients that I did last year came to me independent of their primary care doctors or their cardiologists' recommendation. Well, it was 90% just a decade ago’ (Jerry, surgeon, California).
It had been feared that public reporting might dissuade surgeons from operating on high risk patients, afraid that their reported outcomes would be ‘worse’. However, although recognising that risk-taking was inherent in surgery, English and Californian surgeons presented a narrative which balanced clinical need against low trust in the data.
‘It [treating a high risk patient] would affect my published performance number, so yeah, so it would just make me unhappy. But it would not stop me doing it, and the reason I say it is because of the risk algorithms we have are relatively poor’ (Damon, surgeon, England).
‘Some surgeons, probably younger ones, are keeping track of how they're doing. They are probably maybe less willing to take on a patient if they think it's going to push them into a poor performance category… There is avoidance of certain patients and it's more subtle than the data can turn out’ (Jack, data analyst, California).
This reasoning moderates any reputation effect of public reporting because the surgeon's peers would tacitly acknowledge the limitations of algorithms of case-mix adjustments. This might invoke the notion of equality of competence in a contemporary form (Causer and Exworthy, Reference Causer, Exworthy, Exworthy and Halford1999).
3.2 Meso level
In England, hospital executives reported that public reporting was loosely connected to organisational imperatives such as quality improvement or cost control. Executives seemed initially content to allow surgeons to develop their own professionally-led initiatives.
‘We've [hospital administration] tended to dip in and out of things … So we've done mortality in isolation and we've looked at rates of complaints or something. We've never quite brought the whole thing together to really use it to help improve what we're doing rather than just measure non-compliance [of public reporting]’ (Dave, senior executive, England).
Executives noted the need for greater transparency and that this could be advantageous to their organisation in future. Although such comments may have over-stated the impact of public reporting, it does signal the thinking of executives in how these data could be used.
‘I know that when [former chief executive officer of the case-study hospital] – his view was that we should publish it [data] because in the world of the Foundation Trusts [i.e. more autonomous NHS organisations] and free [patient] choice, extended choice, you know, to be that hospital which is open and honest about its mortality rates, even if the public actually probably don't understand [the data], it gives the sense of this is a hospital with nothing to hide’ (Ralph, surgeon, England).
More recently, public reporting has seemingly stimulated efforts within organisations to improve quality, a process which was probably more advanced in California than England, according to respondents. This was a recognition that the focus of public reporting was shifting towards health care systems and away from individual surgeons.
‘I'm not a fan of individual surgeon [level reporting]; I think it is clinical teams. And the other point I wanted to make… when organisations fail, 99 times out of a 100 it's managerial, it isn't clinical’ (Barry, surgeon and policy-maker, England).
‘I think the fundamental issue … I think almost always the problem is not the person, it's the system that let the person down and … that's what we concentrate on. We don't try to sling blame. We try to concentrate on trying to improve the system … to make it possible for a person, any person to succeed. I always say that you could put anybody in my job right here right now and they would succeed without a question’ (Jerry, surgeon, California).
Bevan et al. (Reference Bevan, Evans and Nuti2019) argue a similar point by suggesting that trust and altruism are suitable for improvements sought at an individual level but ill-suited for systemic improvements.
By contrast, the Californian hospitals in this study were using publicly reported data in aid of quality improvement and service re-organisation. The initial ‘loose coupling’ between hospital executives and surgeons was now becoming more integrated (Kitchener et al., Reference Kitchener, Caronna and Shortell2005). This coupling challenged the viability of surgical programmes with a low volume of surgery, for example.
‘One mortality at a small programme [hospital] can make you bad and they think it's going to be a punitive measure rather than just trying to elevate their entire programme’ (Jerry, surgeon, California).
Jerry cited examples where ‘those [surgical] programmes were really forced to close’ as a result of public reporting. He claimed that ‘most of the programmes they need to be closed are closed already’ meaning that the focus was shifting towards quality improvement, rather than removing low volume centres. Surgeons recognised the influence of organisational systems upon their surgical practice and public reporting. One chief surgeon explained this regarding his and another hospital within the local area where he worked.
‘We're in a very similar healthcare model, which is a capitated, integrated management system, not only from physicians but with hospital… We're comparing similar patients …, we have similar electronic records… It makes more sense to compare data and our outcomes in that way as opposed to us comparing to [another local, independent hospital]. How can you compare and then come to any reasonable conclusions because their population's different, their machines are different, their technology's different and so their products will be different. Their outcomes will be different’ (Henry, surgeon, California).
However, other Californian hospitals did not work this way. Another surgeon explained the consequences of the different hospital system in which he worked:
‘25% of my patients have no insurance and now for the [Californian] state that's true, for heart surgery. [Hospital X] has zero, so it's a completely different model… No homeless person ever goes to [hospital X]. They have been trained to go to [public hospital Y]. So it was 18% in 2010 for [public hospital Y] of cardiac surgery which was not reimbursed at all… If you don't have insurance, the chance that you're going to need, you know, for the first 30 days – the care is rehab facilities, visiting nurses, all those don't exist for 18% of the patients at [public hospital Y]. [Public hospital Y] has to eat it’ (Jerry, surgeon, California).
Note that each surgeon highlighted different aspects. Henry noted the benefits of an integrated system while Jerry pointed to its impacts of exclusionary approaches.
Linking reimbursement claims data to public reporting had generated controversy in California. According to one respondent, one such initiative in 2008 had created a ‘huge firestorm of opposition and conflict within the state’ and had led to ‘law suits and litigation and political stuff that was very debilitating for everybody’ (Daniel, purchaser, California). In the USA, such financial incentives have arguably been more extensively developed than in England (Rosenthal and Dudley, Reference Rosenthal and Dudley2007; Farrar et al., Reference Farrar, Yi, Sutton, Chalkley, Sussex and Scott2009; McDonald and Roland, Reference McDonald and Roland2009) which have generated some resistance amongst surgeons.
‘If there's… money attached to it, or referrals or some other financial impact, then there's going to be a lot more contention about how my named data appears on the ranking’ (Daniel, purchaser, California).
There had been similar pressures (and some progress) to link publicly reported data to reimbursement at a system or organisational level. Note, for example, Dave's comment (see above) about ‘doing mortality in isolation’ or the English initiative linking quality improvement to additional financial reimbursement, called CQUINs (commissioning for quality and innovation) (Murray, Reference Murray2017). However, in the past few years, emphasis has been on financial constraints associated with austerity (Charlesworth et al., Reference Charlesworth, Roberts, Lafond, Exworthy, Mannion and Powell2016).
In both case studies, hybrid ‘surgeon-executives’ helped to broker the interests of surgeons and the wider institutional context.
3.3 Macro level
At this systemic level, we examine the role of the profession, the impact of the media and the salience of mortality as the publicly-reported measure.
First, the surgical profession was instrumental in promoting the introduction of publicly reported surgical data in England but less so in California. In England, some of the former leaders of the profession have become senior figures in the health ministry (Department of Health) and the health care regulator (Care Quality Commission). Their professional interest in and development of surgical audit in the 1990s transferred to a wider audience initially in cardiac surgery in the 2000s and subsequently to other specialties (from 2013). These English surgical leaders sought to accommodate external pressures (such as patient choice and competition) with the specialty's members’ interests. However, this accommodation had strained the cohesion of the specialty. The leaders (who were supportive of public reporting) tried to maintain collegiality with surgeons (some of whom were resistant).
‘I don't draw a distinction between myself and my colleagues. I am still a surgeon I am not a… bureaucrat’ (Barry, surgeon and policy-maker, England).
Another English senior surgeon recognised the need to foster public reporting while stressing the continued need for professional ownership of it.
‘So we have to have some sort of measurement and far better that we do it and do it professionally and well, than have it imposed on us … I suppose some people would feel it's being imposed on us but …I would reject that. I think we are still leading the way with it and we get other people to help us with it… I know there's a bit of resentment, certainly amongst our members, that we're scrutinising ourselves so closely and indeed, being scrutinised from outside’ (emphasis added) (Larry, surgeon and professional leader, England).
Decisions about the form and scope of public reporting had involved the surgical specialty which enabled it to retain significant control.
‘I do very much agree with a point that performance needs to be monitored, because without monitoring of performance… then you don't really have an accurate idea as to where you're going, whether you're going through a difficult period, a bad patch’ (Ikram, junior surgeon, England)
However, the surgical profession in California was much less involved in the inception of public reporting. Unlike England, it was not the Californian surgical profession which engaged initially but rather the California Medical Association and the cardiologists.
‘Now at that point they [the Californian state] had to engage the professional communities. Interestingly they did not go to the Society of Thoracic Surgeons; they went to the … California Chapter of the American College of Cardiology who in some ways are their competitors, right, and they went to the California Medical Association’ (Jack, data analyst, California).
Respondents noted that Californian cardiac surgeons were less cohesive as a profession and did not engage with the state-led initiative into public reporting. The surgeons did subsequently become more involved. Respondents in California suggested that the state's public reporting has been prompted by experience elsewhere in the USA rather than a surgeon-led initiative. One recounted how a purchaser in the state instigated public reporting.
‘They [purchasers] said back in the- the 90s- 95 or 96 so, this sort of thing ‘New York's been doing it successfully. We're a huge state, there's a lot of people here getting bypass surgery in California. We should be-you know the west coast should be able to do what the east has done’, and so they put some money, you know, into that argument’ (Jack, data analyst, California).
Indeed, the larger states (such as New York or California) have had the resources and capability to undertake programmes of public reporting unlike smaller states.
Second, the role of the media, often using freedom of information, has been instrumental in introducing public reporting. In New York State, for example, Newsday magazine published hospital data while in England, the Guardian newspaper adopted a similar approach (Hannan et al., Reference Hannan, Kilburn, Racz, Shields and Chassin1994; Keogh et al., Reference Keogh, Spiegelhalter, Bailey, Roxburgh, Magee and Hilton2004). Media-led public reporting seemed to accelerate developments; indeed, one respondent explained that surgeon-led initiatives were over-taken by the Guardian's strategy:
‘We were boxed into it [public reporting]…we were into trying to get good measurements of results for individual units; what then happened was we were subject to a freedom of information act request from the Guardian’ (Barry, surgeon, England).
However, in California, the impetus for public reporting was driven less by any particular media concern (as in New York). Initial interest there was sparked by a purchaser group (Pacific Business Group on Health; PBGH) who, in 1995, collaborated with the State-wide office for health planning. The Californian media were initially interested but since then, however, the urgency of public reporting for them has waned over the years. One respondent ‘measured’ the decline in interest by column inches in the newspaper:
‘We … issued this report in April and we had – how many press calls? Even with a brand new measure in here [in the report] which we kind of think, ‘Oh we'll add another- another outcome measure’ – keep it fresh, and we have less press calls than we ever had’ (Hannah, data analyst, California).
This decline was corroborated by a surgeon:
‘My last interview was when the last report came out … I gave an interview to [local newspaper], you know – page 10, small paragraph. You know the newspapers are done with this topic. So it's not getting any publicity per se. It should be news but it's not. The people are burnt out from it, they hear the data all the time, they're burnt out from it’ (Jerry, surgeon, California).
The ‘light’ of transparency, shone by the media, may not necessarily generate continuing improvements in health care quality (cf. Tsoukas, Reference Tsoukas1997). Even over a decade ago, Chassin (Reference Chassin2002) claimed that ‘press coverage, so copious at the outset of the programme, is waning. A Lexis-Nexis search in April 2002 revealed no newspaper reports on the most recent public release of CSRS [cardiac surgery reporting system] data in January 2001’ (p. 49) (cf. Huesch et al., Reference Huesch, Currid-Halkett and Doctor2014).
Third, common to both case studies was the measure of mortality rates for named individual surgeons working within specific hospitals. In England, the (30 day post-operative) mortality rate for CABG (the main surgical procedure) was 1.6% in the 40 UK surgical centres (2008–2009) (http://heartsurgery.cqc.org.uk/Survival.aspx). In California, the ‘observed mortality rate’ for ‘isolated CABGs’ (involving no other procedure) by hospital was 2.24% in 120 hospitals (undertaking CABG surgery) (2007–2008) (http://www.oshpd.ca.gov/HID/Products/Clinical_Data/CABG/07-08Breakdown.html). Such rates have been falling before and since public reporting. However, it was the use of the rate and its symbolic value which were significant here.
‘Mortality rates are really for the bad apples’ (Craig, surgeon, England).
‘The surgeon has a responsibility to the programme to say, 'Well, you know, if we chuck up all these high mortalities and let you go, then we may be fined or shut down or penalised’ (Henry, surgeon, California).
The 30 day period of measurement was problematic since, in some hospitals, patients were discharged without adequate means of care at home.
‘Why we're using operative mortality as the only indicator of publicly reported to me is a complete mystery’ (Daniel, purchaser, California).
In California, the mandated use of the ‘isolated CABG’ mortality rate has, some respondents claimed, caused difficulties. Reductions in mortality rates and a decline in the number of CABG procedures meant this was a less relevant measure.
‘One thing that has kind of maybe pushed this a little more into the spotlight is just recently the in-hospital mortality rate for PCI [percutaneous coronary intervention, or angioplasty] has inched above the in-hospital mortality rate for isolated CABG’ (Hannah, data analyst, California).
‘With deep sternal wound infection, it's measured over a 30-day period so somebody may be discharged 5 days after surgery but the surgeon is still responsible for their patient for the 25 days post discharge and during that period of time they, you know, the patient's ability to manage their wound effectively is compromised’ (Eric, quality improvement facilitator, USA).
‘We're down the 1–2% range [of mortality]. It doesn't discriminate performance of the variable that people are concerned about when they undertake the treatment’ (Daniel, purchaser, California).
English surgeons echoed concerns with mortality as a performance measure. However, despite these concerns, they too continued to use it.
‘I think it's [mortality rate] not a panacea that some of its advocates would suggest… Mortality's often inappropriate because it's just not very discriminatory you know, when things are low… but in a lot of cases, yes, you'll have a 1 or 2% mortality rate… Actually what does it tell you if somebody's got a 10% mortality rate. That's all it just tells you’ (Ralph, surgeon, England).
4. Discussion
By comparing the behavioural responses of stakeholders at micro, meso and macro levels in case studies of public reporting in California and England, it has been possible to discern the extent to which these two health systems have been converging. Patterns of convergence were evident (around measurement and the use of data) but equally, strong differences persisted (especially regarding professional and institutional interests). In each case, the interaction between ideas and interests of institutions and individuals generated the observed outcomes. Beyond description, these findings require explanation; Scott's (Reference Scott2008) pillars framework is thus instructive here.
The regulative pillar contrasts the legislative approach of California (which mandated state reporting of surgical mortality) with the (initially) professionally-led strategy in England. These approaches somewhat constrained health systems from moving over time towards more relevant measures of surgical performance. Californian public reporting had also been initiated by purchasers’ concerns for the comparative value of health care spending between providers whereas purchasers (commissioners) in England had taken a much more detached role, preferring instead to allow health care providers (i.e. hospitals) to manage the process. In the English case study, the medical profession was largely in control of how performance was measured and the data were used.
Sanctions for ‘poor’ performance were apparent mainly in California with purchasers removing low volume, poor quality providers. Fewer explicit sanctions were evident in England. However, the incentives for ‘outstanding’ performance were much less distinct, as public reporting often was primarily concerned with ‘moving performance from awful to adequate’ (Bevan et al., Reference Bevan, Evans and Nuti2019). Regulators and policy-makers in both cases were seeking to address similar challenges of health system performance and were increasingly using public reporting to do so. Public reporting had become a normative doctrine in both case studies. Although other measures (such as financial control) were used to address these challenges (especially in England), public reporting offered a seemingly robust and easily comparable performance measure (namely, mortality). However, in both cases, data quality undermined the ability to manage surgical performance, at least initially. Moreover, the salience of public reporting has lessened as mortality rates have improved over time (thereby reducing variations between the ‘best’ and ‘worst’ surgeons, and emphasising further need to consider other measures such as patient experience). As a result, public and media interest has waned. California's earlier experience of public reporting offers a dynamic picture of public reporting over time and thus offers a perspective upon the trajectory of public reporting in England. Some convergence had taken place but distinct differences remained.
Regarding the normative pillar, the institutional imperative to public reporting appeared strong in both cases (especially as surgeons’ participation became mandatory in both cases). In California, the institutional pressure came from the State and the purchaser group, with surgeons playing a minor role in framing the scope and remit of public reporting. Yet, once introduced, surgeons were involved in public reporting as a means towards improving the quality of services. In England, surgeons’ leaders recognised the need to foster ownership of public reporting; surgical leaders in the specialty and in policy-making were crucial mediators between surgeons and institutional interests. These leaders facilitated the introduction of public reporting. Although such approaches might indicate an awareness among the medical profession of the potential for greater external control, they do also denote a re-professionalisation of medicine which sought to reset the position and status of surgeons vis-à-vis other stakeholders. Such re-professionalisation was marked by the emphasis on accountability (redolent of managerialism) and new forms of engagement with patients (qua consumers) (Elston, Reference Elston, Gabe and Calnan2009). Regarding the first of these, health care executives were connecting public reporting with institutional imperatives (such as pay-for-performance) but the use of such data in quality improvement was variable. This was more prevalent in California than England. Second, the use of these data by patients was hardly apparent in either case study. Patients rarely used the data and the public voice featured little. Media reports took increasingly scant notice of the data. This may be exacerbated by the esoteric measure of surgical performance (i.e. adjustment of mortality rates according to high risk patients) and whose dissemination is often opaque (Hildon et al., Reference Hildon, Allwood and Black2012). Suddaby and Viale (Reference Suddaby and Viale2011) argued that although the profession has sometimes been instrumental in devising and disseminating measures of ‘their’ members’ performance, often for reasons of public interest, such measures tended to be understood largely only by the professionals themselves. Moreover, the partial use of publicly reported data may help explain behavioural responses which underlined the ‘reputation pathway’ as being likely to generate most improvement in mortality rates (Hibbard et al., Reference Hibbard, Stockard and Tusler2003), as opposed to a selection (by patients) pathway.
As for the cultural-cognitive pillar, the ‘transparency doctrine’ appeared relentless in both cases. Initial resistance to public reporting by surgeons had been evident in both cases but this was declining. Evidence presented here showed hardly any overt resistance to the notion of transparency by surgeons. This reflected a shift in which participation had initially been voluntary but had become mandatory. In some ways, this might increase the incentive for gaming (e.g. avoiding high risk patients) if there were more explicit penalties for ‘poor’ performance (Smith, Reference Smith1995). Moreover, most surgeons and their professional leaders in both cases sought to incorporate public reporting into their norms of professionalism (e.g. as a tool of quality improvement) and thereby deflect possible external threats to their autonomy. Although public reporting did enrol most surgeons, participation was sometimes begrudging; minimal resistance was thus apparent (Westaby, Reference Westaby2014). If done begrudgingly, this suggests that public reporting did not fully match surgeons’ own professional logics (i.e. autonomy and self-interest). As public reporting in England had been pioneered by the surgeon's own professional body, this dissonance was perhaps less marked in England compared with California. In any case, it appeared to be leading towards a greater re-stratification within the profession between the knowledge elite (generating the evidence and devising case-mix adjustments), the administrative elite (applying the doctrine of public reporting in terms of incentive regimes) and rank-and-file surgeons (who were the object of and subject to public reporting) (Miller, Reference Miller1992; Freidson, Reference Freidson1994).
Overall, although public reporting was implemented in different ways at different time points, there were some signs of apparent convergence in the process and outcomes of public reporting in both cases. Surgeons in both cases were increasingly aligned with the institutional aims and practice of public reporting. It had become not only a ‘system logic’ (Tuohy, Reference Tuohy1999) but possibly also an ‘electronic panopticon’, providing continuous surveillance of surgeons’ performance (Bain and Taylor, Reference Bain and Taylor2000; Exworthy, Reference Exworthy2015); this strategy might be effective than overt cost control. Yet, there were also signs of divergence or perhaps more strictly path dependence (Greener, Reference Greener2002) which reinforced distinctions, both geographically between cases and temporally with the status quo ante. This was perhaps most evidence in the regulatory pillar. For example, the longer period of implementation in California, together with stronger organisational incentives, had perhaps led to an approach which integrated public reporting more closely with organisational system and processes. The evidence from the English case study suggested this integration was still nascent. This might reflect the sense in which external agents (in the English case) were more hesitant in challenging surgeon's (poor) performance and/or the strength of professional control in the nature and content of public reporting.
In summary, the ‘pillars framework’ is a useful heuristic device to demonstrate how the behavioural and policy responses to public reporting by health system stakeholders are shaped by the nature of public reporting and the context into which it is implemented. The responses also shaped dynamically by subsequent iterations of public reporting, most evidently in California. These interactions between cardiac surgeons, organisational interests and health system imperatives show the far-reaching effects that public reporting has had and continues to have. As a result, on-going divergence persists despite pressure for greater convergence.
5. Conclusion
Public reporting is likely to remain an important aspect of all health systems but will be shaped by a health system's own imperative and approach to transparency. Despite strong imperatives towards convergence, the context of each health system matters. Analyses need therefore to be sensitive to the context-specific factors of each system in framing the interaction between their regulatory, normative and cultural factors. These factors will both shape and are shaped by actors’ decisions and behaviours. This interaction will also be framed by the extent of policy learning between health systems. Specifically, the American approach to public reporting has been instrumental to approaches in England. Although international comparative studies (such as this) help illuminate the impact of such intersections (Dolowitz et al., Reference Dolowitz, Hulme, Nellis and O'Neill1999), policy learning will also reflect the ways in which actors ‘piece together’ various aspects of information and experience into their system-specific approach (Freeman, Reference Freeman2007).
Although greater transparency is a laudable objective for any health system, it is not clear that public reporting will necessarily be as effective as its proponents intended. Alone, it will not act as sufficient stimulus to restore trust in and reputations of the medical profession, to enable to patient decision-making, or to enable quality improvement. Indeed, the growing need for cost savings in all health systems may lessen the salience and impact of public reporting. Yet, public reporting is re-shaping the surgical profession itself and the relations between surgeons and their organisations, and their patients.
Acknowledgements
The ‘English’ study was funded by the ESRC (RES-153-25-0100); we are grateful to them for their funding. The ‘California’ study was made possible with help from staff from the Institute of Health Policy Studies, University of California San Francisco (UCSF). We thank all the research participants in this study. An earlier version of the paper was presented at the American Sociological Association conference. We are grateful to the Editors and reviewers of Health Economics, Policy and Law for their continued support of this paper and the constructive comments.
Open access
