Quality analysis of publicly available information about hypoplastic left heart syndrome

Abstract

Purpose:

Publicly available health information is increasingly important for patients and their families. While the average US citizen reads at an 8^th-grade level, electronic educational materials for patients and families are often advanced. We assessed the quality and readability of publicly available resources regarding hypoplastic left heart syndrome (HLHS).

Methods:

We queried four search engines for “hypoplastic left heart syndrome”, “HLHS”, and “hypoplastic left ventricle”. The top 30 websites from searches on Google, Yahoo!, Bing, and Dogpile were combined into a single list. Duplicates, commercial websites, physician-oriented resources, disability websites, and broken links were removed. Websites were graded for accountability, content, interactivity, and structure using a two-reviewer system. Nonparametric analysis of variance was performed.

Results:

Fifty-two websites were analysed. Inter-rater agreement was high (Kappa = 0.874). Website types included 35 hospital/healthcare organisation (67.3%), 12 open access (23.1%), 4 governmental agency (7.7%), and 1 professional medical society (1.9%). Median total score was 19 of 39 (interquartile range = 15.8–25.3): accountability 5.5 of 17 (interquartile range = 2.0–9.3), content 8 of 12 (interquartile range = 6.4–10.0), interactivity 2 of 6 (interquartile range = 2.0–3.0), and structure 3 of 4 (interquartile range = 2.8–4.0). Accountability was low with 32.7% (n = 17) of sites disclosing authorship and 26.9% (n = 14) citing sources. Forty-two percent (n = 22) of websites were available in Spanish. Total score varied by website type (p = 0.03), with open access sites scoring highest (median = 26.5; interquartile range = 20.5–28.6) and hospital/healthcare organisation websites scoring lowest (median = 17.5; interquartile range = 13.5–21.5). Score differences were driven by differences in accountability (p = 0.001) – content scores were similar between groups (p = 0.25). Overall readability was low, with median Flesch–Kincaid Grade Level of 11^th grade (interquartile range = 10^th–12^th grade).

Conclusions:

Our evaluation of popular websites about HLHS identifies multiple opportunities for improvement, including increasing accountability by disclosing authorship and citing sources, enhancing readability by providing material that is understandable to readers with the full spectrum of educational background, and providing information in languages besides English, all of which would enhance health equity.

Hypoplastic left heart syndrome (HLHS) accounts for roughly 2–3% of congenital heart defects (CHDs),¹ yet it is responsible for 23% of cardiac deaths that occur within the first week of life.² Long-term survival for patients with HLHS has continued to improve – contemporary surgical management is associated with over 60–70% 10-year survival.³ Management of these patients is complex with significant parental involvement required for optimal outcomes, particularly during the interstage periods between Norwood (Stage 1), bidirectional Glenn (Stage 2), and Fontan (Stage 3) operations.^4,5 As such, parental education and understanding are essential for the long-term management and follow-up of patients with HLHS. In the modern era, most patients and/or their families access electronic resources to develop an understanding of their health problems.^6,7 Importantly however, the average US citizen reads at an 8^th-grade reading level.⁸ (In the United States of America, students in 8^th grade are usually 13 or 14 years old, and 8^th grade in the United States of America is equivalent to Year 9 in the United Kingdom, where there is a similar average reading level of 11–14 years old.^8,9 ) Poor health literacy has been linked to increased mortality in other contexts, including acute heart failure, mental illness, and diabetes.^10–12 Thus, providing quality online resources about HLHS at a reading level appropriate for the average US/UK citizen is important to improve the usefulness of these materials and ultimately improve the outcomes achieved for these patients. Furthermore, providing quality online resources about HLHS at a reading level appropriate for the average US/UK citizen will enhance overall health equity.

Others have evaluated the quality and readability of publicly available electronic health resources in a variety of different settings.^13–16 These studies consistently find that available online health resources are written above the average US citizen reading level and that website quality is often poor, a combination that challenges improved health equity. In one of these analyses evaluating online resources for patients with peripheral artery disease,¹³ poor readability was reflected with a median Flesch–Kincaid Grade Level Score of 10.7 (interquartile range = 10–12) and poor quality was reflected with a median total quality score of 19 out of 47 (interquartile range = 15–30), which accounted for accountability, content, interactivity, and structure. While these analyses highlight areas for improvement in other pathologies, the literature lacks analyses of the quality of online resources available to patients with HLHS and their families.

The purpose of this study is to assess the quality and readability of publicly available electronic patient resources about HLHS.

Methods

We queried four search engines for the search terms “hypoplastic left heart syndrome”, “HLHS”, and “hypoplastic left ventricle”. These search terms were chosen in order to avoid irrelevant results that more broad terms would return. The search engines queried were

• Google,

• Yahoo!,

• Bing, and

• Dogpile.

Google, Yahoo!, and Bing are native search engines, while Dogpile is a “metasearch” engine that compiles results from multiple other native search engines. These queries were completed in December 2022. Queries were completed on a Chrome browser in incognito mode to avoid the impact of browser cookies and search history on results returned by the queries performed.

The first 30 website results for each search term from searches on Google, Yahoo!, Bing, and Dogpile were combined into a single list. Exclusion criteria included:

• Duplicates,

• Commercial websites,

• Physician-oriented resources such as academic research literature,

• Disability websites that describe how to access governmental benefits, and

• Broken links.

Of note, disability websites are not designed to help patients and their families understand their conditions.

Websites were grouped into four types:

• Hospital/healthcare organisation,

• Governmental agency,

• Professional medical society, and

• Open access.

We defined websites as hospital/healthcare organisation if associated with an institution providing direct patient care; governmental agency if associated with a government-operated organisation; professional medical society if associated with a professional society of any speciality of medicine; or open access if unable to be categorised as any of the preceding types.

The assessment tool developed by Ingledew to assess quality of websites in regard to patient–physician collaboration was modified to assess websites about HLHS (Appendix 1). ¹⁷ Using a two-reviewer system (Fig 1), websites were graded for:

Figure 1. Analysis of online HLHS resources by website type. Total (p = 0.03) and accountability (p = 0.001) scores varied significantly by website type, with open access websites scoring the highest and hospital/healthcare organisation websites scoring the lowest. Abbreviations: HLHS, hypoplastic left heart syndrome.

• Accountability,

• Content,

• Interactivity, and

• Structure.

Reviewers underwent training to become familiar with the evaluation systems. Scoring discrepancies among the two reviewers were resolved by consensus. To assess readability, standardised readability scores were calculated by two reviewers via an open access website that offers tools to assess readability on four scales (Fig 2):

Figure 2. Readability distribution of online patient resources. Distribution of websites by readability as assessed by the Gunning Fog, Flesch–Kincaid, Coleman-Liau, and Linsear Write indices. Using all four indices, the majority of online patient resources for hypoplastic left heart syndrome were written above the 8^th-grade reading level recommended by the Food and Drug Administration.

• Gunning-Fog,

• Flesch–Kincaid,

• Coleman-Lau, and

• Linsear Write.

Nonparametric analysis of variance was performed to assess differences in websites, types of websites, and readability across websites. Inter-rater agreement was assessed by calculating a Cohen’s Kappa coefficient between reviewers.

Results

Websites

After exclusion criteria were applied, 52 websites remained and were assessed by two reviewers. Inter-rater agreement between reviewers was high (Kappa = 0.87). Website types included:

• 35 hospital/healthcare organisation (67.3%),

• 12 open access (23.1%),

• 4 governmental agency (7.7%), and

• 1 professional medical society (1.9%).

Six websites (11.5%) were based out of the United Kingdom, one (1.9%) out of Australia, one (1.9%) out of India, and the remaining 44 (84.6%) websites were based out of the United States of America. Lists of the top five websites in content, readability, and overall score are presented in Table 1.

Table 1. Top websites in content, readability, and overall

Quality

The quality of online resources for patients with HLHS was variable, with a total score range of 23.5 (range = 8–31.5). The median total score for the 52 websites was 48.7% (19 out of 39 [interquartile range = 15.8–25.3]): accountability was 32.4% (5.5 out of 17 [interquartile range = 2.0–9.3]), content was 66.7% (8 out of 12 [interquartile range = 6.4–10.0]), interactivity was 33.3% (2 out of 6 [interquartile range = 2.0–3.0]), and structure was 75% (3 out of 4 [interquartile range = 2.8–4.0]). Accountability was low with 32.6% of sites disclosing authorship and 26.9% citing sources. Less than half (42.3%) of websites were available in Spanish. Total score varied by website type (p = 0.03), with open access sites scoring the highest (67.9%, median = 26.5; interquartile range = 20.5–28.6) and hospital/healthcare organisation websites scoring the lowest (44.9%, median = 17.5; interquartile range = 13.5–21.5). Score differences were primarily driven by differences in accountability (p = 0.001), with open access websites scoring 67.6%, 11.5 out of 17 (interquartile range = 6.5–13.0) and hospital/healthcare organisation websites scoring 11.8%, 2 out of 17 (interquartile range = 2–7). The timing of all three procedures in staged palliation (Norwood, bidirectional Glenn, and Fontan) was discussed by 76.9% (40 of 52) of websites. One website (1.9%) discussed timing of one or two of the procedures, while 11 (21.2%) did not mention timing at all. All other categories, including content (p = 0.25), interactivity (p = 0.15), and structure (p = 0.33) were similar between groups.

Content

Content was assessed using a custom, HLHS-specific 12-point scale. HLHS was defined in 100% (52 of 52) of websites. However, less than half (40.4%, 21 of 52) discussed the incidence and/or prevalence of HLHS. Presenting symptoms of HLHS (e.g., blue baby, low oxygen saturation, dyspnoea, and cyanosis during feeding) were discussed by 75.0% (39 of 52) of websites. Discussion of diagnostic tests, which required inclusion of echocardiography, was present in 78.9% (41 of 52) of websites. The Norwood, Glenn, and Fontan procedures were discussed by 73.1%, 75.0%, and 75.0% of websites, respectively. Other surgical therapies, including ventricular assist device implantation and heart transplantation, were discussed by 75.0% (39 of 52) of websites. Only 23.1% (12 of 52) discussed surgical complications in more than two sentences, while 13.5% (7 of 52) discussed this topic in one to two sentences. Finally, complications of HLHS (e.g., growth delay, death) were mentioned by 71.2% (37 of 52) of websites.

Readability

Overall readability of the 52 websites was low, with median Flesch–Kincaid Grade Level of 11^th grade (interquartile range = 10^th–12^th grade), which did not vary significantly by website type (p = 0.29). Similar readability levels were found using other calculators, including:

• The Coleman-Liau index (median = 11^th grade; interquartile range = 10^th–12^th grade; p = 0.83),

• The Gunning Fog index (median = 13.6; interquartile range = 12.7–14.7; p = 0.99), and

• The Linsear Write formula (median = 9^th grade; interquartile range = 7^th–11^th grade; p = 0.56).

Using Flesch–Kincaid Grade Level, only 15.4% (8 of 52) of websites were written at or below an 8^th-grade reading level, with 17.3% (9 of 52) of websites written at a college or higher reading level.

Discussion

Patients and their families have increasingly taken charge of how they process information about their health; therefore, there is increased need for physicians to help guide them to quality resources. ^18,19 The majority of patients now research their conditions before meeting with their physician but remain unlikely to discuss the findings of this research during visits with physicians. ²⁰ Poor patient literacy is linked to poorer health outcomes and health inequity in many populations. ²¹ Many websites involving health data on the internet provide misleading or false information. ^22,23

The average person in the United States of America reads at a 7^th- or 8^th-grade level. ^8,24 This fact is one of many reasons why organisations like the Centers for Disease Control and National Institutes of Health of the United States of America advise that websites be written at an 8^th-grade level in plain language. ^25,26 Readability is an important consideration when healthcare organisations are creating online materials that may be consumed by patients. ⁸ Our data, however, show that the majority of websites about HLHS are written at an 11^th-grade reading level or higher, and only 15.4% of websites are written at an 8^th-grade level or lower. This finding represents a gap between information available and the average reader, and this gap should be addressed by organisations that create health information websites.

Overall, our analysis revealed that hospital/healthcare organisations had the lowest scoring websites, while open access websites had the highest scoring websites. However, overall scores were low, with the average score being 19 out of 39 (48.7%). Quality of websites was especially impacted by accountability, showing a need to increase source citation and authorship disclosure.

While websites frequently discussed the definition of HLHS, diagnostic steps, and surgical procedures, few discussed complications of surgery. Only 23.1% of websites discussed surgical complications in more than two sentences and another 13.5% of websites mentioned surgical complications in one to two sentences, demonstrating a need to increase the number of websites offering insight into surgical complications for patients and their families, to help them in making informed and balanced decisions. As an essential component of informed consent, the discussion of the risks of surgery likely occurs during the patient–surgeon interaction. ²⁷ However, since patients frequently search for information online, incorporating discussion of surgical complications into publicly available online resources represents an opportunity for improving patient education and enhancing health equity by allowing fully informed consent to occur. ²⁰ When incorporating discussion of surgical complications, others have found that citing the potential likelihood of complications using point estimates (e.g., 3% risk) or risk ranges (e.g., 1–5%) is associated with improved and consistent patient understanding when compared to using verbal descriptors (e.g., “uncommon”). ²⁸ In that survey-based study of 296 American adults, the probability of complications associated with surgical or antibiotic treatment of acute appendicitis was presented using these three techniques (verbal descriptors, point estimates, risk ranges). ²⁸ There was significantly higher variability in patient perception of risk when presented with verbal descriptors. Thus, when incorporating increased discussion of surgical complications associated with staged palliation for HLHS into publicly available online resources, using point estimates or risk ranges may be associated with improved patient understanding of these risks, allowing for patients/parents to make informed decisions with realistic expectations.

Spanish-speaking patients, a fast-growing demographic in the United States of America, often have lower health literacy than native English speakers. ^29,30 Despite this, less than half (42.3%) of websites about HLHS are available in Spanish. Addressing this gap in accessibility is especially important considering that additional difficulties faced by these patients are often exacerbated by language barriers. ³¹ Encouragingly, Villa Camacho and colleagues found that online patient education materials written in Spanish for breast cancer were written at a lower grade reading level than equivalent English-language versions (5.49 ± 0.50 versus 7.77 ± 1.95, p < 0.01) and were more likely to meet American Medical Association recommendations for patient educational resources (82.9% versus 40.4%, p < 0.01). ³² Thus, while availability of Spanish-language online resources for HLHS is poor, readability of available resources may be appropriate, although this remains unknown for HLHS. Increasing the availability of online HLHS resources translated to Spanish is necessary to improve patient understanding in this subgroup of patients and their families.

Ultimately, the low overall score of most websites and high levels of variability highlight the need to improve websites broadly, especially those designed for patients and their families. This need for improvement is especially true for the domain of accountability, which was overall low and was the main driver of differences in scores among website types. Key methods to improve accountability would be highlighting sources of information and displaying authorship on all patient-facing and family-facing websites.

This study has important limitations. Websites are designed with goals that may not be patient-centred and/or family-centred, and readability and content may vary based on the intended audience. Due to their nature, websites are constantly changing, and this study was only a snapshot of one day in the history of each website. Search engine results are impacted by user search history and other factors like location, which could change some of the top results found by patients. While we excluded websites that were not patient-centred, they may still be encountered by patients and their families in their search results.

Conclusions

Our evaluation of popular websites about HLHS identifies several opportunities for improvement, including increasing accountability by disclosing authorship and citing sources, enhancing readability by providing material that is understandable to readers with the full spectrum of educational background, and providing information in languages besides English.