The Residual Lesion ScoreReference Nathan, Trachtenberg and Van Rompay1 is an initiative predicated on the concept that specific features of the surgical repair of complex CHD have sufficiently disproportionate impact on surgical outcomes that success in correcting these critical features should be prioritised. The primary hypothesis for this study is that defining the post-operative findings with the highest predictive capacity for outcomes would enable better intra-operative decision making in terms of allocation of time and effort, need for intra-operative revision, and potential need for re-operation. Initially labelled as the technical performance score,Reference Larrazabal, Del Nido and Jenkins2,Reference Nathan and Karamichalis3 the name was revised to recognise that the success of the surgical intervention is critically dependent on pre-operative anatomy and patient-specific factors in addition to surgical technique. This prospective multicentre study tests the applicability and validity of the Residual Lesion Score for five common congenital cardiac operations: arterial switch operation for dextro-transposition of the great arteries, Norwood procedure for single ventricle heart disease and repair of tetralogy of Fallot with pulmonary stenosis, complete atrioventricular septal defect, and coarctation or interruption of the aorta with ventricular septal defect.
A modified Rand Delphi processReference Sackman4–Reference Nathan, Karamichalis and Liu7 was used to specify the post-operative echocardiographic and clinical measurements considered potentially predictive of outcome for inclusion in the scoring system.
The Residual Lesion Score is determined predominantly by echocardiographic measurements, so the study was designed to optimise standardisation of these measurements by utilising an echocardiographic core lab with expertise in paediatric CHD. A major goal for the core lab was to assure that the scoring system would not be biased by inadequate measurements. To help understand the reliability and reproducibility of the measurements making up the score, the design of the study included some extra layers of evaluation including having all measurements repeated at the study sites by a paediatric cardiologist with expertise in imaging and collecting data regarding the quality of each image being used for measurement. The hope was that this information would help us understand the causes of missing data and/or low reproducibility and this would then inform future iterations of the Residual Lesion Score. The main results of the study have been published along with an overview of the design of the echo core lab.Reference Nathan, Trachtenberg and Van Rompay1,Reference Nathan, Levine and van Rompay6
The goal of this paper is to present our experience with the extra steps that were included to assure that the data involved in the scoring system were of acceptable quality. This should inform future efforts in that investigators can build on our experience and avoid some of our challenges. In addition, we hope to highlight some of the challenges in conducting this type of research in infant populations in general, and specifically in those with CHD. This paper outlines further details of the systems for imaging review while highlighting some important lessons learned about.
Materials and methods
Residual Lesion Score development
The Residual Lesion Score was based on lessons learned from the previous workReference Larrazabal, Del Nido and Jenkins2,Reference Nathan and Karamichalis3,Reference Nathan, Pigula and Liu8–Reference Douglas, DeCara and Devereux15 with the technical performance score. This scoring system divides each operation into subcomponents and aims to classify the results of each subcomponent separately to understand how the outcome for the individual subcomponents affects the overall outcome of the operation. The measurements included in the Residual Lesion Score were chosen by an expert panel, utilising the Rand Delphi method of reaching consensus. Figure 1 summarizes the steps involved in the Rand Delphi method. A general description of this process has been previously published Reference Nathan, Trachtenberg and Van Rompay1,Reference Nathan, Levine and van Rompay6
Figure 1. Development of the Residual Lesion Score modules using the modified Rand Delphi methodology. ASO = arterial switch operation; CAVSD = complete atrioventricular septal defect; DC = discharge; dTGA = dextro-transposition of the great arteries; IO = intra-operative; RLS = Residual Lesion Score; TOF/PS = tetralogy of Fallot; VSD = ventricular septal defect.
Data collection form development
The development of data collection forms was undertaken once the recommendations from the Rand Delphi process were available. The goals for this part of the process included the following:
-
1. Assuring that the Rand Delphi process had not missed key variables. After review of the Rand Delphi recommendations, a few variables were added that were not part of the score but were likely to impact data interpretation, such as measures of systolic ventricular function.
-
2. Adding fields for grading data quality as well as fields that allowed reviewers to provide reasons for missing data. The core lab was interested in understanding if the measures chosen by the Rand Delphi process could be made with enough confidence to be included in future iterations of the scoring system. We hypothesised that variables that evaluators felt were more difficult to measure were likely to demonstrate higher inter-observer variability and may not correlate strongly with the assessment of surgical results by other methods.
-
3. Standardizing data entry to minimise error. Customized data capture forms were designed to evaluate the 5 operations at 3 time-points: pre-operative, intra-operative, and pre-discharge (post-operative); 15 forms in total.Reference Nathan, Trachtenberg and Van Rompay1,Reference Nathan, Levine and van Rompay6
Core lab organisation and measurement of intra- and inter-reader variability
Sites reviewed data from all five of the operations being evaluated. The core lab did not review the data from the arterial switch operation in an effort to streamline financial and human resources. The rationale was presented in prior publications.Reference Nathan, Levine and van Rompay6
Core lab reviewers were blinded to all clinical data with the following exceptions: the patient’s cardiac diagnosis, the operation that was performed, the time point for each echocardiogram (pre-operative, intra-operative, or discharge), height, and weight. The core lab consisted of two investigators. Investigator 1 did the initial reading for three of the lesions and then re-read 10% of randomly selected studies for each of those three lesions. Investigator 2 did the initial reading for one lesion and then re-read 10% of randomly selected studies for that lesion. Once this was completed, investigator 1 read 10% of randomly selected studies initially read by investigator 2 and investigator 2 read 10% of randomly selected studies initially read by investigator 1.
Core lab training of sites
A Technical Reference Manual was created with detailed instructions for both image acquisition and for the review and measurement of echocardiographic variables. The echocardiographic view, phase of the cardiac cycle, and anatomic location for each measurement were standardised, but sites were permitted to use their local protocols for image acquisition. This meant that sites were not equivalent in their use of intra-operative imaging, sedation for transthoracic echo, and timing of post-operative evaluation. If adequate views were not available for the measurements that were integral for assigning the Residual Lesion Score, then sites were directed to leave the data field blank and provide the reason that the measurement could not be made.
In an effort to increase measurement consistency and assure expert evaluation of quality, each site was asked to identify one faculty echocardiographer to oversee study performance. This role included data review and entry as well as ensuring that involved personnel were well versed in the study requirements.
After study completion, sites were surveyed about study implementation, barriers to obtaining the required imaging, and the experience of the faculty echocardiographer (Appendix 1). The results revealed wide practice variation with respect to personnel and data entry as well as a consensus that the role of the faculty echocardiographer was a larger time commitment than expected, creating conflict with other clinical roles.
Results
Personnel involved in data acquisition
There were 17 sites total and 12 responded to a questionnaire about local practices. Table 1 highlights some of the data. At all sites, echocardiograms were performed by personnel familiar with the study protocols. Four centres focused training on a subset of sonographers who performed the majority of studies, while the remaining sites trained all personnel in the lab. Fellows were allowed to perform pre-operative studies on enrolled patients at five sites.
Table 1. Site variation in personnel involved with image acquisition/interpretation
For intra-operative exams, two sites restricted the number of physicians who performed these exams, thus requiring some alteration in the attending schedule to accommodate scheduling for the operating room. The remainder allowed all staff to perform the intra-operative echocardiogram. Ten sites allowed fellows and four sites allowed sonographers to participate in intra-operative imaging. Among seven sites that used epicardial imaging, the surgeon performed imaging at six sites and the cardiologist at one site.
Three sites limited performance of the discharge echocardiogram to a subset of sonographers, which occasionally required an alteration in sonographer schedules. Seven sites allowed fellows to perform discharge echocardiograms.
Site data entry and quality review
The survey revealed that there was considerable variability among sites with respect to the process for measuring variables and recording them in the data collection forms. At eight sites, the faculty echocardiographer filled out the data forms. At sites where the faculty echocardiographer did not fill out all forms, this individual still reviewed the data to check for quality and completeness of data entry.
Eight sites had a formal process for quality review of study echocardiograms. Of those, the faculty echocardiographer was in charge of quality review at five.
The variability that we found between sites when we reviewed the questionnaire data gave us some information about targets for improving variation in the future. We knew when we designed the protocol that we could decrease variation by mandating that only a select group of imagers at each site be allowed to participate, thus improving image consistency. Many protocols in the current era do just this, requiring on-line sonographer training and submission of practice exams. These strategies work well when the timing of exams is predictable. For a study like this one, there was not an opportunity for this type of stringent personnel requirement as exams were done when clinically necessary, including evenings and weekends.
Discussion
The main results of the Residual Lesion Score study, carried out by the Pediatric Heart Network, have been previously published.Reference Nathan, Levine and van Rompay6 In brief, the initial analysis revealed that the components of the Residual Lesion Score were predictive of outcome for some operations more than others and that the measured residual lesions accounted for a smaller percentage of the variation in outcome than was expected. Further analysis should increase our understanding of exactly when residual lesions had the most impact and which infant populations were at the highest risk for a suboptimal outcome when residual lesions were present. Because post-operative imaging data were central to the scoring system, the echocardiographic core lab was a major component of the study design.
Centralized echocardiographic core labs are often used in multicentre research to assess for outcomes and/or complications of a therapeutic intervention. A core lab improves the reliability and quality of the datasets in multicentre research and guidelines for the development and responsible conduct of echocardiographic core labs have been published.Reference Parthiban, Levine and Nathan16 These guidelines stress the importance of consistency, integrity, and accountability in data evaluation and reporting. Core labs have also become key drivers of advances in technology for the secure export, de-identification, and storage of imaging data. For this study, where the primary predictor was heavily reliant on imaging in infants with complex heart disease, a centralised core lab was created with the above goals and the added priority of assuring that the data evaluators had special expertise in CHD and in the interpretation of multiple echocardiographic modalities: transthoracic, transesophageal, and epicardial. Because of the complexity of the data, fully trained paediatric cardiologists were enlisted for all aspects of data evaluation. This represented a unique requirement in the design, but we felt that this resource investment was key to assuring high quality data that would optimise the ability to detect potentially subtle differences in surgical results.
Technical challenges
Over 3000 imaging studies were part of the dataset. The collection, storage, and evaluation of the imaging for this study presented some unique challenges due to the length of the echocardiograms, the complexity of the protocols, the complexity of the heart disease, and the young age of the patients.
The first challenge involved image transfer. Echocardiograms done for assessment of CHD in infants are commonly quite lengthy as they typically include comprehensive imaging focused on anatomic variation. Many studies submitted to the core lab were over 200 clips long, some with individual clips that were over 500 frames long. The size of the imaging files created problems for de-identification and transfer, necessitating significant modification to the original storage plan and delaying core lab access to the primary data. We may have avoided these delays if we had limited data transfer to a smaller subset of the imaging done at the sites. Though this was considered, it would have increased the burden at the sites as they would have had to review the imaging and edit the studies prior to sending the dicom data. Another strategy that some have used is to ask that images for research be added onto the end of the clinical exam so that the views needed for the research project can be easily separated from the clinical study prior to transfer. This can work well but it assumes that the patient will tolerate a longer exam and in the infant population this can be quite variable. For our study, we chose to modify our storage platform to allow for larger files as this was least likely to impact the sites and the patients. The second technical barrier involved development of the data collection forms. In addition to fields for recording diverse anatomic details, quantitative measurements of two dimensional and Doppler data and qualitative assessment of variables such as valve regurgitation, we wanted to incorporate fields for grading image quality and citing reasons for missing data. There were no existing data entry systems that could accommodate these needs, so the programming had to be written from scratch. The multi-faceted functionality involved a fairly complex skip logic algorithm and took longer to develop and test than expected, contributing to delays. The benefit to spending time getting this right is that the modifications needed for this complicated data collection are now available for use by others and have already been modified for use by other research teams. However, this came at quite a cost with respect to time and resources and highlighted that commercial products developed for use in adult studies where imaging is largely focused on valve and ventricular function are not easily modified for the evaluation of CHD.
Human resource challenges
Inherent in the design of this core lab was our goal of having consistent, high quality evaluation of the data along with expert assessment of data quality and barriers to collecting quality data. To that end, measurement and data entry were done by a small group of paediatric cardiologists specialising in imaging, a high value resource. To accomplish the goals of the study, the entirety of these lengthy studies needed to be reviewed. At the core lab, the time needed for review was amplified by the fact that images were not acquired in a standardised order, were sometimes done on unfamiliar equipment, and were not always displayed using standardised orientation. This was a particular challenge for imaging acquired in the operating room where there are often fewer anatomic landmarks visible on the stored image. In the future, we would advocate for a standardised image acquisition order, particularly in the operating room. In most studies involving infants, it is not reasonable to limit the length of an anatomic evaluation but if the views are standardised, it should make review more efficient.
At the sites, the human resource challenge involved the role of the faculty echocardiographer. The person in this role was responsible for image review, data extraction, filling data collection forms, training personnel, and often writing local echo reports. Having a paediatric cardiologist do all of these tasks increased confidence in the consistency and quality of the dataset, but became a significant resource burden when combined with competing clinical demands. In retrospect, the tasks probably should have been assigned more broadly. The faculty echocardiographer could have been assigned those tasks that needed a high level of expert judgement, while delegating more routine measurements and data entry to other team members. Other potentially helpful strategies might have been to include more than one faculty echocardiographer at the larger study sites and more than two cardiologists at the core lab.
Challenges in infant echocardiography
Finally, our experience highlights the importance of considering feasibility and logistics carefully when it comes to performing echocardiography in infants. Performing echocardiograms in infants with heart disease has a few unique challenges, including the need for very detailed and thorough anatomic surveys prior to and after surgical intervention, the inability to enlist patient cooperation with study performance, and the small size of some of the anatomic structures being evaluated which mandates high resolution images. For transthoracic echocardiograms, detailed imaging can be challenging in awake infants who may be tachycardic and mobile, particularly those with acoustic windows that are restricted by chest wounds and bandages. Though qualitative data may be achievable, acquiring images that are adequate for precise measurement of small structures may not always be possible. It became clear early in the study that some of the desired variables could not be reliably measured, resulting in a high percentage of missing data points, necessitating protocol modification. Prior work has shown that there can be unacceptable variation in echocardiographic measures in infants after heart surgery.Reference Parthiban, Levine and Nathan16 The importance of considering patient compliance and measurement feasibility when studying infants seems obvious, yet is not always included in the research design. It is common practice for core lab protocols to include comprehensive lists of required imaging views that are not necessarily tailored to infants; a one size fits all approach. Our experience reinforces the importance of considering a different strategy for infant populations. Most importantly, we would advocate for sponsors to write protocols that include a prioritised list of views and required measurements so that the most important data for the study are collected before moving on to less critical, supportive data. Other strategies include improving imaging conditions, such as the use of a safe sedation strategy for image acquisition, planning to image during other sedated procedures, ensuring that imaging is done before other noxious procedures such as blood draws, and making sure that data collection windows are broad enough to allow for a second try at the echocardiogram if the first attempt is unsuccessful. Finally, for complex research studies like this one, it may make sense to consider short pilot studies aimed at assessing the quality and feasibility of data acquisition before finalizing the imaging section of the research protocol.
Conclusion
The Residual Lesion Score study utilised both a core lab and site review of echocardiograms for five operations in 1,149 infants with CHD. The study was completed as planned, but barriers to completing echocardiographic data assessment included underestimates of the technical and human resources required as well as the feasibility of performing detailed echocardiograms in this young population. Some of the technical barriers and feasibility issues are unique to studies involving infants with CHD: lack of existing data entry programmes that can accommodate the details of abnormal cardiac anatomy, the need for long clips and lengthy echocardiograms to evaluate anatomic details, the reliance on high resolution images, and the inability to enlist infant cooperation with study acquisition. The human resource issue was largely a consequence of the specific design of this core lab. Though the use of a small group of imaging experts for all data extraction likely improved the dataset quality, it is not a sustainable model in most of today’s academic centres due to competing demands on clinician’s time. Difficulty in compensating experts for participation in clinical research at academic centres is not a new concern. Study design groups and funding sources need to consider ways in which the important resources of expertise and time can be valued and compensated in clinical research.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1047951123003037
Acknowledgements
We would like to thank the study coordinators, surgeons, echocardiographers, echocardiogram uploaders, and registry coordinators from the 17 sites, and data coordinating centre staff for their invaluable contributions to this study. (Appendix 2: Residual Lesion Score (RLS) Study Investigators).
Financial support
This study was supported by grants (U24HL135691, U10HL068270, HL109818, HL109778, HL109816, HL109743, HL109741, HL109673, HL068270, HL109781, HL135665, HL135680) from the National Heart, Lung, and Blood Institute, National Institutes of Health. Meena Nathan was supported by a K23 grant-NHLBI/NIH HL119600.
Competing interests
None.
Ethical standard
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national guidelines on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008, and have been approved by the institutional review boards at each of the author’s affiliated institutions and the Data Coordinating Center.
