Comparison of Heterogeneity Measures in Meta-Analysis
DOI:
https://doi.org/10.6000/1929-6029.2025.14.30Keywords:
Meta Analysis,, I^2 heterogeneity measure, R_b heterogeneity measure, H heterogeneity measure, Tau2 heterogeneity measure, simulationAbstract
Background: Heterogeneity assessment is critical in meta-analysis, as it determines the appropriateness of combining studies and affects result reliability. Cochran’s Q is the traditional test, nevertheless, it has low statistical power, so many researchers resort to using heterogeneity measures to quantify the heterogeneity.
Aim: This article aims to compare the performance of the most commonly used heterogeneity measures through simulation.
Materials and Methods: We compared the performance of four heterogeneity measures (, , , H) across various homogeneous and heterogeneous patient-event probabilities [], various sample sizes (n) and number of studies (k), using RMSE (Root mean squared error) and BIAS values in simulation scenarios. Additionally, Cochran’s Q Type-I error rate and power were evaluated using the same simulation scenarios.
Results: and H outperformed other measures in large samples, while , and were preferable for small studies.
Conclusion: Researchers can use the simulation results from this study to select an appropriate heterogeneity measure for their meta-analysis work. This approach is expected to prevent time loss due to unnecessary subgroup analyses in situations where heterogeneity appears to be present but is actually absent.
References
O'rourke K. An historical perspective on meta-analysis: Dealing quantitatively with varying study results. Journal of the Royal Society of Medicine 2007; 100(12): 579-82. DOI: https://doi.org/10.1177/0141076807100012020
Haidich AB. Meta-analysis in medical research. Hippokratia 2010; 14(Suppl 1): 29-37.
Mikolajewicz N, Komarova SV. Meta-Analytic Methodology for Basic Research: A Practical Guide. Frontiers in Physiology 2019; Volume 10 - 2019. DOI: https://doi.org/10.3389/fphys.2019.00203
Smith T. Proposing Alternative Methods for Testing Heterogeneity of Studies in a Meta-analysis: Illinois State University; 2021.
Higgins JPT. Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified. International Journal of Epidemiology 2008; 37(5): 1158-60. DOI: https://doi.org/10.1093/ije/dyn204
Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods 2006; 11(2): 193-206. DOI: https://doi.org/10.1037/1082-989X.11.2.193
Jackson D. Assessing the Implications of Publication Bias for Two Popular Estimates of Between-Study Variance in Meta-Analysis. Biometrics 2007; 63(1): 187-93. DOI: https://doi.org/10.1111/j.1541-0420.2006.00663.x
Thorlund K, Imberger G, Johnston BC, Walsh M, Awad T, Thabane L, et al. Evolution of Heterogeneity (I2) Estimates and Their 95% Confidence Intervals in Large Meta-Analyses. PLOS ONE 2012; 7(7): e39471. DOI: https://doi.org/10.1371/journal.pone.0039471
Patsopoulos NA, Evangelou E, Ioannidis JP. Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. International Journal of Epidemiology 2008; 37(5): 1148-57. DOI: https://doi.org/10.1093/ije/dyn065
von Hippel PT. The heterogeneity statistic I2 can be biased in small meta-analyses. BMC Medical Research Methodology 2015; 15(1): 35. DOI: https://doi.org/10.1186/s12874-015-0024-z
Baujat B, Mahé C, Pignon J-P, Hill C. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Statistics in Medicine 2002; 21(18): 2641-52. DOI: https://doi.org/10.1002/sim.1221
Mittlböck M, Heinzl H. A simulation study comparing properties of heterogeneity measures in meta-analyses. Statistics in Medicine 2006; 25(24): 4321-33. DOI: https://doi.org/10.1002/sim.2692
Borenstein M LVH, Julian P. T. Higgins, Hannah R. Rothstein. Introduction to Meta‐Analysis: John Wiley & Sons, Ltd; 2009. 450 p. DOI: https://doi.org/10.1002/9780470743386
Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 2002; 21(11): 1539-58. DOI: https://doi.org/10.1002/sim.1186
Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I2 in assessing heterogeneity may mislead. BMC Medical Research Methodology 2008; 8(1): 79. DOI: https://doi.org/10.1186/1471-2288-8-79
Crippa A, Khudyakov P, Wang M, Orsini N, Spiegelman D. A new measure of between-studies heterogeneity in meta-analysis. Statistics in Medicine 2016; 35(21): 3661-75.
Crippa A, editor A new measure of between-studies heterogeneity in meta-analysis. XXVIIIth International Biometric Conference; 2016; Victoria. DOI: https://doi.org/10.1002/sim.6980
Van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Statistics in Medicine. 1993; 12(24): 2273-84. DOI: https://doi.org/10.1002/sim.4780122405
Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003; 327(7414): 557-60. DOI: https://doi.org/10.1136/bmj.327.7414.557
Peterson K. Six Modifications Of The Aligned Rank Transform Test For Interaction. The Journal of Modern Applied Statistical Methods 2002; 1(1). DOI: https://doi.org/10.22237/jmasm/1020255240
Hsiung T-H, Olejnik S. Type I Error Rates and Statistical Power for the James Second-Order Test and the Univariate F Test in Two-Way Fixed-Effects ANOVA Models under Heteroscedasticity and/or Nonnormality. The Journal of Experimental Education. 1996; 65(1): 57-71. DOI: https://doi.org/10.1080/00220973.1996.9943463
Paul J, Barari M. Meta-analysis and traditional systematic literature reviews—What, why, when, where, and how? Psychology & Marketing 2022; 39(6): 1099-115. DOI: https://doi.org/10.1002/mar.21657
Stroup DF, Thacker SB. Meta analysis. In: Britannica TEoE, editor. Encyclopedia Britannica 2024.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Policy for Journals/Articles with Open Access
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
Policy for Journals / Manuscript with Paid Access
Authors who publish with this journal agree to the following terms:
- Publisher retain copyright .
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work .
