How should we measure research impact in paediatric cardiology?

This edition of Cardiology in the Young includes two interesting analyses examining research publications in our field. In the first, Sew ¹ et al use citation frequency to evaluate the most cited manuscripts and authors in paediatric cardiology. Their analysis provides a powerful review spanning from the 1939 landmark publication by Robert Gross and John Hubbard ² on surgical ligation of a patent arterial duct to the 2014 work by Ariane Marelli ³ and colleagues on the lifetime prevalence of CHD in Quebec, Canada. The second manuscript by Loomba and colleagues ⁴ uses impact factor ⁵ and the h-index ^6,7 to evaluate citation potential for research published in the paediatric cardiology-focused journals Cardiology in the Young, Pediatric Cardiology, and Annals of Pediatric Cardiology, compared to two adult-centric cardiology journals, Circulation, and the Journal of the American College of Cardiology. During the 2014 calendar year, 45 paediatric cardiology manuscripts were published in Circulation and JACC while 783 were published in the3 paediatric cardiology journals. Articles published in Circulation and JACC were cited more frequently; however, the h-index was higher for paediatric cardiology journals, and perhaps most noteworthy, amongst the 50 most cited paediatric cardiology manuscripts, 28 were published in paediatric cardiology journals compared to 22 in Circulation or JACC. The authors make a compelling case that publication in paediatric cardiology journals in no way diminishes the potential for citation, and that authors should use other factors, most importantly potential readership, when deciding where to submit research findings.

These papers beg the question – how should the impact of research be measured? The analyses cited above rely on several metrics – citation frequency, impact factor, and the h-index. ⁷ What do these measures represent and are they reliable estimates of research quality or impact? At the most basic level, the number of peer-reviewed published manuscripts and the number of times manuscripts are cited seem like reasonable measures of the impact of a researcher. However, even at this level, as Sew and colleagues suggest, publication biases in favour of work from certain countries or institutions, more common conditions, positive findings, agreed-upon practice, readership interest, etc., can strongly influence the ability to publish. Even amongst researchers with similar number of publications, different researchers may have had different levels of impact, where a few individuals made critical contributions, and others were also included as co-authors but could never have conceived of or completed the research independently. Research and researchers also differ in how much original content to incorporate in manuscripts, with some studies yielding multiple divided or duplicative publications.

Citation indices were designed to quantify the impact of research beyond number of publications, but are also imperfect measures. The journal “impact factor”, represents the average number of citations for manuscripts published by a journal over the preceding 2 years, ⁵ but this measure will vary based on how large a specialty is. For example, adult cardiology is a much larger field than paediatric cardiology, so adult-centric manuscripts are inherently likely to be cited more frequently. Circulation, which publishes many more adult-centric manuscripts than paediatric, has a 2020 impact factor of 23.6 meaning that the average manuscript published in Circulation in 2018 and 2019 received 23.6 citations. By comparison, there is no paediatric cardiology journal with an impact factor above 2.0. Yet, as Loomba and colleagues demonstrate, 28 out of our field’s 50 most cited manuscripts were published in these “lower impact” journals. In general, it is difficult to compare the number of citations as a measure of impact in fields of differing sizes. There can also be cultural differences amongst specialties, such as the common practice of naming operations after individuals, and then always citing first reports, both less common in medical specialties. Similar to the impact factor, the h-index is used to compare journals but is also applicable at the author level. The h-index is defined as the maximum value of h such that the given author/journal has published h papers that have each been cited at least h times. ⁶ It was designed to overcome biases from extreme citations (i.e., impacting the average number of citations), but is itself skewed by the overall volume of publications. As Loomba et al demonstrate, the h-index was actually higher for the paediatric cardiology specialty journals. This finding is slightly misleading as only 45 manuscripts were published in Circulation or JACC, meaning the maximum attainable h-index was 45, while there were 783 manuscripts published in paediatric cardiology journals. Despite nuances and limitations, Loomba and colleagues provide strong evidence that publishing in a lower impact specialty journal does not necessarily limit citation potential.

So how should we measure the impact of a researcher in paediatric cardiology? Is it the number of manuscripts published? The number of manuscripts in high-impact journals? The number of times work has been cited? A combination of quality and quantity such as the h-index? What about other criteria like successful mentorship, groundbreaking discovery, or substantial contributions informing clinical care? The honest answer is that all of these factors are important and no one index or measure can reliably quantify the impact. It is important, though, that aspiring researchers in our field are inspired and encouraged to do impactful research, and avoid wasting precious time or resources on studies with minimal impact. There is value in appreciating the most important contributors to our field, and it is an honour to those listed in Sew et al’s top 100 most cited manuscripts, ¹ documenting people and studies that have undoubtedly moved the needle in our field in very meaningful ways. In general, though, there are many valuable contributions that won’t be cited hundreds of times in the medical literature. Novel case reports offer valuable insight, negative studies are just as important as groundbreaking discoveries, and pilot studies or experiential reports provide the foundation for future breakthroughs. Similarly, the contributions of secondary authors should not be undervalued, as many people in different roles are needed for a study to be successful, and only successful studies can have an impact. In addition, research is not the only way to have an impact on the field, as master clinicians who rarely publish a manuscript define clinical excellence, treat many patients, train many individuals, and inspire others to demonstrate the value of their work by research.

These two articles highlighting published manuscripts and citations by Sew et al ¹ and Loomba and colleagues ⁴ should encourage an important dialogue for research mentors and junior colleagues about what constitutes impactful research, what type of research is worth the effort, and how we can best convey our discoveries to our community. These are even more important questions than a discussion of imperfect, or even misleading measures of impact. Ultimately the end game is the best possible outcome for patients, and all contributions that support this goal should be encouraged, including meaningful, interpretable research findings, with wide dissemination.