Hostname: page-component-857557d7f7-v2cwp Total loading time: 0 Render date: 2025-11-20T19:53:55.192Z Has data issue: false hasContentIssue false
  • English
  • Français

Reevaluating Heterogeneity in Evidence Synthesis

Published online by Cambridge University Press:  23 September 2025

Alkistis Elliott-Graves Open the ORCID record for Alkistis Elliott-Graves [Opens in a new window]
Alkistis Elliott-Graves*
Affiliation:
Department of Philosophy, Bielefeld University , Bielefeld, Germany
*
Email: a.elliott-graves@uni-bielefeld.de
Get access Rights & Permissions [Opens in a new window]

Abstract

Evidence synthesis has recently seen an enormous increase across many different scientific disciplines. Despite its popularity, it has also been the subject of significant criticism. One of the main critiques of evidence synthesis is the existence and treatment of heterogeneity between primary studies. The aim of this article is to reexamine heterogeneity in evidence synthesis, including perspectives from evolutionary biology, ecology, and conservation. I argue that while some of the critiques of heterogeneity remain valid, there are contexts where it is much less problematic. Furthermore, I propose that heterogeneity can be useful because analyzing it can provide valuable information.

Information

Type
Article
Information
Philosophy of Science , First View , pp. 1 - 20
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Beckage, Brian, Gross, Louis J., and Kauffman, Stuart. 2011. “The Limits to Prediction in Ecological Systems.” Ecosphere 2 (11):1–12. https://doi.org/10.1890/ES11-00211.1.Google Scholar
Berchialla, Paola, Chiffi, Daniele, Valente, Giovanni, and Voutilainen, Ari. 2020. “The Power of Meta-Analysis: A Challenge for Evidence-Based Medicine.” European Journal for Philosophy of Science 11 (1):7. https://doi.org/10.1007/s13194-020-00321-w.Google Scholar
Berlin, Jesse A., and Golub, Robert M.. 2014. “Meta-Analysis as Evidence: Building a Better Pyramid.” JAMA 312 (6):603–6. https://doi.org/10.1001/jama.2014.8167.Google Scholar
Boutron, Isabelle, Page, Matthew J., Higgins, Julian P. T., Altman, Douglas G., Lundh, Andreas, and Hróbjartsson, Asbjørn. 2019. “Considering Bias and Conflicts of Interest among the Included Studies.” In Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed., edited by Julian, P. T. Higgins, Thomas, James, Chandler, Jacqueline, et al., chap. 7. Chichester, UK: John Wiley & Sons. https://doi.org/10.1002/9781119536604.ch7.Google Scholar
Bruner, Justin P., and Holman, Bennett. 2019. “Self-Correction in Science: Meta-Analysis, Bias and Social Structure.” Studies in History and Philosophy of Science Part A 78 (December):93–97. https://doi.org/10.1016/j.shpsa.2019.02.001.Google Scholar
Cadotte, Marc W., Mehrkens, Lea R., and Menge, Duncan N. L.. 2012. “Gauging the Impact of Meta-Analysis on Ecology.” Evolutionary Ecology 26 (5):1153–67. https://doi.org/10.1007/s10682-012-9585-z.Google Scholar
Carpenter, Christopher J. 2020. “Meta-Analyzing Apples and Oranges: How to Make Applesauce Instead of Fruit Salad.” Human Communication Research 46 (2–3):322–33. https://doi.org/10.1093/hcr/hqz018.Google Scholar
Catford, Jane A., Wilson, John R. U., Pyšek, Petr, Hulme, Philip E., and Duncan, Richard P.. 2022. “Addressing Context Dependence in Ecology.” Trends in Ecology & Evolution 37 (2):158–70. https://doi.org/10.1016/j.tree.2021.09.007.Google Scholar
Chen, Haoyu, and Jhanji, Vishal. 2012. “Survey of Systematic Reviews and Meta-Analyses Published in Ophthalmology.” British Journal of Ophthalmology 96 (March):896–99. https://doi.org/10.1136/bjophthalmol-2012-301589.Google Scholar
Colautti, Robert I., Ricciardi, Anthony, Grigorovich, Igor, and MacIsaac, Hugh J.. 2004. “Is Invasion Success Explained by the Enemy Release Hypothesis?” Oikos 7 (8):721–33.Google Scholar
Cook, D.I., Gebski, V.J. and Keech, A.C. (2004), Subgroup analysis in clinical trials. Medical Journal of Australia, 180:289–291. https://doi.org/10.5694/j.1326-5377.2004.tb05928.x Google Scholar
Cook, David I., Gebski, Val J., and Keech, Anthony C.. 2004. “Subgroup Analysis in Clinical Trials.” Medical Journal of Australia 180 (6):289–91. https://doi.org/10.5694/j.1326-5377.2004.tb05928.x.Google Scholar
Cuijpers, Pim, Griffin, Jason W., and Furukawa, Toshi A.. 2021. “The Lack of Statistical Power of Subgroup Analyses in Meta-Analyses: A Cautionary Note.” Epidemiology and Psychiatric Sciences 30 (December):e78. https://doi.org/10.1017/S2045796021000664.Google Scholar
Deeks, Jonathan J., Higgins, Julian P. T., and Douglas, G. Altman, on behalf of the Cochrane Statistical Methods Group. 2019. “Analysing Data and Undertaking Meta-Analyses.” In Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed., edited by Julian, P. T. Higgins, Thomas, James, Chandler, Jacqueline, et al., chap. 10. Chichester, UK: John Wiley & Sons. https://doi.org/10.1002/9781119536604.ch10.Google Scholar
Dettori, Joseph R., Norvell, Daniel C., and Chapman, Jens R.. 2022. “Fixed-Effect vs Random-Effects Models for Meta-Analysis: 3 Points to Consider.” Global Spine Journal 12 (7):1624–26. https://doi.org/10.1177/21925682221110527.Google Scholar
Doak, Daniel F., Estes, James A., Halpern, Benjamin S., et al. 2008. “Understanding and Predicting Ecological Dynamics: Are Major Surprises Inevitable?” Ecology 89 (4):952–61. https://doi.org/10.1890/07-0965.1.Google Scholar
Egger, Matthias, Ebrahim, Shah, and Davey Smith, George. 2002. “Where Now for Meta-Analysis?” International Journal of Epidemiology 31 (1):1–5. https://doi.org/10.1093/ije/31.1.1.Google Scholar
Elliott-Graves, A. (2023). Ecological Complexity. Cambridge: Cambridge University Press.Google Scholar
Fieseler, Clare. 2021. “The Case against the Concept of Biodiversity.” Vox. https://www.vox.com/22584103/biodiversity-species-conservation-debate.Google Scholar
Fletcher, Samuel C. 2022. “Replication Is for Meta-Analysis.” Philosophy of Science 89 (5):960–69. https://doi.org/10.1017/psa.2022.38.Google Scholar
Fontelo, Paul, and Liu, Fang. 2018. “A Review of Recent Publication Trends from Top Publishing Countries.” Systematic Reviews 7 (1):147. https://doi.org/10.1186/s13643-018-0819-1.Google Scholar
Foo, Yong Zhi, O’Dea, Rose E., Koricheva, Julia, Nakagawa, Shinichi, and Lagisz, Malgorzata. 2021. “A Practical Guide to Question Formation, Systematic Searching and Study Screening for Literature Reviews in Ecology and Evolution.” Methods in Ecology and Evolution 12 (9):1705–20. https://doi.org/10.1111/2041-210X.13654.Google Scholar
Gurevitch, Jessica, Koricheva, Julia, Nakagawa, Shinichi, and Stewart, Gavin. 2018. “Meta-Analysis and the Science of Research Synthesis.” Nature 555 (7695):175–82. https://doi.org/10.1038/nature25753.Google Scholar
Hardwicke, Tom E., Serghiou, Stylianos, Janiaud, Perrine, et al. 2020. “Calibrating the Scientific Ecosystem through Meta-Research.” Annual Review of Statistics and Its Application 7 (1):11–37. https://doi.org/10.1146/annurev-statistics-031219-041104.Google Scholar
Heger, Tina, and Jeschke, Jonathan. 2014. “The Enemy Release Hypothesis as a Hierarchy of Hypotheses.” Oikos 123 (6):741–50. https://doi.org/10.1111/j.1600-0706.2013.01263.x.Google Scholar
Higgins, Julian P. T. 2008. “Commentary: Heterogeneity in Meta-Analysis Should Be Expected and Appropriately Quantified.” International Journal of Epidemiology 37 (5):1158–60. https://doi.org/10.1093/ije/dyn204.Google Scholar
Higgins, Julian P. T., Savović, Jelena, Page, Matthew J., Elbers, Roy G., and Sterne, Jonathan A. C.. 2019. “Assessing Risk of Bias in a Randomized Trial.” In Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed., edited by Julian, P. T. Higgins, Thomas, James, Chandler, Jacqueline, et al., chap. 8. Chichester, UK: John Wiley & Sons. https://doi.org/10.1002/9781119536604.ch8.Google Scholar
Higgins, Julian P. T., Thomas, James, Chandler, Jacqueline, et al., eds. 2019. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Chichester, UK: John Wiley & Sons. https://doi.org/10.1002/9781119536604.fmatter.Google Scholar
Higgins, Julian P. T., and Thompson, S. G.. 2002. “Quantifying Heterogeneity in a Meta-Analysis.” Statistics in Medicine 21 (11):1539–58. https://doi.org/10.1002/sim.1186.Google Scholar
Holman, Bennett. 2019. “In Defense of Meta-Analysis.” Synthese 196 (8):3189–211. https://doi.org/10.1007/s11229-018-1690-2.Google Scholar
Houlahan, Jeff, McKinney, Shawn, Anderson, Michael, and McGill, Brian. 2017. “The Priority of Prediction in Ecological Understanding.” Oikos 126 (1):1–7. https://doi.org/10.1111/oik.03726.Google Scholar
Ioannidis, John P. A. 2005. “Why Most Published Research Findings Are False.” PLOS Medicine 2 (8):e124. https://doi.org/10.1371/journal.pmed.0020124.Google Scholar
Ioannidis, John P. A. 2016. “The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-Analyses.” Milbank Quarterly 94 (3):485–514. https://doi.org/10.1111/1468-0009.12210.Google Scholar
Ioannidis, John P. A. 2018. “Meta-Research: Why Research on Research Matters.” PLOS Biology 16 (3):e2005468. https://doi.org/10.1371/journal.pbio.2005468.Google Scholar
Ioannidis, John P. A., Patsopoulos, Nikolaos A., and Evangelou, Evangelos. 2007. “Uncertainty in Heterogeneity Estimates in Meta-Analyses.” BMJ 335 (7626):914–16. https://doi.org/10.1136/bmj.39343.408449.80.Google Scholar
Jukola, Saana. 2017. “On Ideals of Objectivity, Judgments, and Bias in Medical Research—a Comment on Stegenga.” Studies in History and Philosophy of Biological and Biomedical Sciences 62 (April):35–41. https://doi.org/10.1016/j.shpsc.2017.02.001.Google Scholar
Kaunisto, Sirpa, Ferguson, Laura V., and Sinclair, Brent J.. 2016. “Can We Predict the Effects of Multiple Stressors on Insects in a Changing Climate?” Current Opinion in Insect Science 17 (October):55–61. https://doi.org/10.1016/j.cois.2016.07.001.Google Scholar
Konno, Ko, Gibbons, James, Lewis, Ruth, and Pullin, Andrew S.. 2024. “Potential Types of Bias When Estimating Causal Effects in Environmental Research and How to Interpret Them.” Environmental Evidence 13 (1):1. https://doi.org/10.1186/s13750-024-00324-7.Google Scholar
Koricheva, Julia, and Gurevitch, Jessica. 2013. “Place of Meta-Analysis among Other Methods of Research Synthesis.” In Handbook of Meta-Analysis in Ecology and Evolution, edited by Koricheva, Julia, Gurevitch, Jessica, and Mengersen, Kerrie, 3–13. Princeton, NJ: Princeton University Press.Google Scholar
Koricheva, Julia, and Gurevitch, Jessica. 2014. “Uses and Misuses of Meta-Analysis in Plant Ecology.” Journal of Ecology 102 (4):828–44. https://doi.org/10.1111/1365-2745.12224.Google Scholar
Koricheva, Julia, Gurevitch, Jessica, and Mengersen, Kerrie, eds. 2013. Handbook of Meta-Analysis in Ecology and Evolution. Princeton, NJ: Princeton University Press.Google Scholar
Kovaka, Karen. 2022. “Meta-Analysis and Conservation Science.” Philosophy of Science 89 (5):980–90. https://doi.org/10.1017/psa.2022.68.Google Scholar
LaCaze, Adam, and Osimani, Barbara, eds. 2020. Uncertainty in Pharmacology: Epistemology, Methods, and Decisions. Boston Studies in the Philosophy and History of Science. Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-030-29179-2.Google Scholar
Lawton, J. H. 1999. “Are There General Laws in Ecology?” Oikos 84 (2):177–92. https://doi.org/10.2307/3546712.Google Scholar
Maron, John L., and Vilà, Montserrat. 2001. “When Do Herbivores Affect Plant Invasion? Evidence for the Natural Enemies and Biotic Resistance Hypotheses.” Oikos 95 (3):361–73. https://doi.org/10.1034/j.1600-0706.2001.950301.x.Google Scholar
Marsden, Emma, Morgan-Short, Kara, Thompson, Sophie, and Abugaber, David. 2018. “Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field.” Language Learning 68 (2):321–91. https://doi.org/10.1111/lang.12286.Google Scholar
Maziarz, Mariusz. 2022. “Is Meta-Analysis of RCTs Assessing the Efficacy of Interventions a Reliable Source of Evidence for Therapeutic Decisions?” Studies in History and Philosophy of Science 91 (February):159–67. https://doi.org/10.1016/j.shpsa.2021.11.007.Google Scholar
Mitchell, Sandra D. 2000. “Dimensions of Scientific Law.” Philosophy of Science 67 (2):242–65. https://doi.org/10.2307/188723?refreqid=search-gateway:148949eeb4f88afdc4c4ac3f69c73275.Google Scholar
Mitchell, Sandra D. 2002. “Ceteris Paribus—an Inadequate Representation for Biological Contingency.” Erkenntnis 57 (3):329–50.Google Scholar
Mouquet, Nicolas, Lagadeuc, Yvan, Devictor, Vincent, et al. 2015. “Predictive Ecology in a Changing World.” Journal of Applied Ecology 52 (5):1293–310. https://doi.org/10.1111/1365-2664.12482.Google Scholar
Mueller-Langer, Frank, Fecher, Benedikt, Harhoff, Dietmar, and Wagner, Gert G.. 2019. “Replication Studies in Economics—How Many and Which Papers Are Chosen for Replication, and Why?” Research Policy 48 (1):62–83.Google Scholar
Nakagawa, Shinichi, and Cuthill, Innes C.. 2007. “Effect Size, Confidence Interval and Statistical Significance: A Practical Guide for Biologists.” Biological Reviews 82 (4):591–605. https://doi.org/10.1111/j.1469-185X.2007.00027.x.Google Scholar
Nakagawa, Shinichi, Koricheva, Julia, Macleod, Malcolm, and Viechtbauer, Wolfgang. 2020. “Introducing Our Series: Research Synthesis and Meta-Research in Biology.” BMC Biology 18 (1):20. https://doi.org/10.1186/s12915-020-0755-0.Google Scholar
Nakagawa, Shinichi, Lagisz, Malgorzata, Jennions, Michael D., et al. 2022. “Methods for Testing Publication Bias in Ecological and Evolutionary Meta-Analyses.” Methods in Ecology and Evolution 13 (1):4–21. https://doi.org/10.1111/2041-210X.13724.Google Scholar
Nakagawa, Shinichi, Noble, Daniel W. A., Senior, Alistair M., and Lagisz, Malgorzata. 2017. “Meta-Evaluation of Meta-Analysis: Ten Appraisal Questions for Biologists.” BMC Biology 15 (1):18. https://doi.org/10.1186/s12915-017-0357-7.Google Scholar
Nakagawa, Shinichi, and Poulin, Robert. 2012. “Meta-Analytic Insights into Evolutionary Ecology: An Introduction and Synthesis.” Evolutionary Ecology 26 (5):1085–99. https://doi.org/10.1007/s10682-012-9593-z.Google Scholar
Nakagawa, Shinichi, and Santos, Eduardo S. A.. 2012. “Methodological Issues and Advances in Biological Meta-Analysis.” Evolutionary Ecology 26 (5):1253–74. https://doi.org/10.1007/s10682-012-9555-5.Google Scholar
O’Connor, Mary I., Gonzalez, Andrew, Byrnes, Jarrett E. K., et al. 2017. “A General Biodiversity–Function Relationship Is Mediated by Trophic Level.” Oikos 126 (1):18–31. https://doi.org/10.1111/oik.03652.Google Scholar
Osimani, Barbara. 2020. “Epistemic Gains and Epistemic Games: Reliability and Higher Order Evidence in Medicine and Pharmacology.” In Uncertainty in Pharmacology, vol. 338, edited by Adam LaCaze and Barbara Osimani, 345–72. Boston Studies in the Philosophy and History of Science. Cham, Switzerland: Springer. https://link.springer.com/chapter/10.1007/978-3-030-29179-2_15.Google Scholar
Parker, John D. et al. (2006) Opposing Effects of Native and Exotic Herbivores on Plant Invasions.Science311,1459-1461. DOI:10.1126/science.1121407Google Scholar
Pilotto, Francesca, Kühn, Ingolf, Adrian, Rita, et al. 2020. “Meta-Analysis of Multidecadal Biodiversity Trends in Europe.” Nature Communications 11 (1):1. https://doi.org/10.1038/s41467-020-17171-y.Google Scholar
Pretis, De, Francesco, Jürgen Landes, and Osimani, Barbara. 2019. “E-Synthesis: A Bayesian Framework for Causal Assessment in Pharmacosurveillance.” Frontiers in Pharmacology 10 (December):1317. https://doi.org/10.3389/fphar.2019.01317.Google Scholar
Pyron, R. Alexander. 2017. “We Don’t Need to Save Endangered Species. Extinction Is Part of Evolution.” Washington Post. https://www.washingtonpost.com/outlook/we-dont-need-to-save-endangered-species-extinction-is-part-of-evolution/2017/11/21/57fc5658-cdb4-11e7-a1a3-0d1e45a6de3d_story.html.Google Scholar
Raerinne, Jani. 2014. “Evolutionary Contingency, Stability, and Biological Laws.” Journal for General Philosophy of Science 46 (1):45–62. https://doi.org/10.1007/s10838-014-9271-7.Google Scholar
Romero, Felipe. 2016. “Can the Behavioral Sciences Self-Correct? A Social Epistemic Study.” Studies in History and Philosophy of Science Part A 60 (December):55–69. https://doi.org/10.1016/j.shpsa.2016.10.002.Google Scholar
Sánchez-Tójar, Alfredo, Moran, Nicholas P., O’Dea, Rose E., Reinhold, Klaus, and Nakagawa, Shinichi. 2020. “Illustrating the Importance of Meta-Analysing Variances alongside Means in Ecology and Evolution.” Journal of Evolutionary Biology 33 (9):1216–23. https://doi.org/10.1111/jeb.13661.Google Scholar
Sánchez-Tójar, Alfredo, Nakagawa, Shinichi, Sánchez-Fortún, Moisès, et al. 2018. “Meta-Analysis Challenges a Textbook Example of Status Signalling and Demonstrates Publication Bias.” eLife 7 (November):e37385. https://doi.org/10.7554/eLife.37385.Google Scholar
Schielzeth, Holger, and Nakagawa, Shinichi. 2022. “Conditional Repeatability and the Variance Explained by Reaction Norm Variation in Random Slope Models.” Methods in Ecology and Evolution 13 (6):1214–23. https://doi.org/10.1111/2041-210X.13856.Google Scholar
Senior, Alistair M., Grueber, Catherine E., Kamiya, Tsukushi, et al. 2016. “Heterogeneity in Ecological and Evolutionary Meta-Analyses: Its Magnitude and Implications.” Ecology 97 (12):3293–99. https://doi.org/10.1002/ecy.1591.Google Scholar
Siddaway, Andy P., Wood, Alex M., and Hedges, Larry V.. 2019. “How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses.” Annual Review of Psychology 70: 747–70. https://doi.org/10.1146/annurev-psych-010418-102803.Google Scholar
Spake, Rebecca, Bowler, Diana E., Callaghan, Corey T., et al. 2023. “Understanding ‘It Depends’ in Ecology: A Guide to Hypothesising, Visualising and Interpreting Statistical Interactions.” Biological Reviews 98 (4):983–1002. https://doi.org/10.1111/brv.12939.Google Scholar
Spake, Rebecca, O’Dea, Rose E., Nakagawa, Shinichi, et al. 2022. “Improving Quantitative Synthesis to Achieve Generality in Ecology.” Nature Ecology & Evolution 6 (12):1818–28. https://doi.org/10.1038/s41559-022-01891-z.Google Scholar
Stegenga, Jacob. 2011. “Is Meta-Analysis the Platinum Standard of Evidence?” Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 42 (4):497–507. https://doi.org/10.1016/j.shpsc.2011.07.003.Google Scholar
Sterne, Jonathan A. C., Hernán, Miguel A., McAleenan, Alexandra, Reeves, Barnaby C., and Higgins, Julian P. T.. 2019. “Assessing Risk of Bias in a Non-Randomized Study.” In Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed., edited by Julian, P. T. Higgins, Thomas, James, Chandler, Jacqueline, et al., chap. 25. Chichester, UK: John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119536604.ch25.Google Scholar
Taylor, Amy, and Munafò, Marcus. 2016. “Triangulating Meta-Analyses: The Example of the Serotonin Transporter Gene, Stressful Life Events and Major Depression.” BMC Psychology 4 (1):23. https://doi.org/10.1186/s40359-016-0129-0.Google Scholar
Turchin, Peter. 2001. “Does Population Ecology Have General Laws?” Oikos 94 (1):17–26.Google Scholar
Watkins, Hannah V., Yan, Helen F., Dunic, Jillian C., and Côté, Isabelle M.. 2021. “Research Biases Create Overrepresented ‘Poster Children’ of Marine Invasion Ecology.” Conservation Letters 14 (3):e12802. https://doi.org/10.1111/conl.12802.Google Scholar
Whittaker, Robert J. 2010. “Meta-Analyses and Mega-Mistakes: Calling Time on Meta-Analysis of the Species Richness–Productivity Relationship.” Ecology 91 (9):2522–33. https://doi.org/10.1890/08-0968.1.Google Scholar
Worrall, John. 2002. “What Evidence in Evidence-Based Medicine?” Philosophy of Science 69 (S3):S316–30. https://doi.org/10.1086/341855 Google Scholar