Reasearch Awards nomination

Email updates

Keep up to date with the latest news and content from Critical Care and BioMed Central.

Open Access Highly Accessed Research

Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study

Zhongheng Zhang*, Xiao Xu and Hongying Ni

Author Affiliations

Department of Critical Care Medicine, Jinhua Municipal Central Hospital, 351 Mingyue Street, Jinhua City, Zhejiang 321004, PR China

For all author emails, please log on.

Critical Care 2013, 17:R2  doi:10.1186/cc11919

Published: 9 January 2013

Abstract

Introduction

Small-study effects refer to the fact that trials with limited sample sizes are more likely to report larger beneficial effects than large trials. However, this has never been investigated in critical care medicine. Thus, the present study aimed to examine the presence and extent of small-study effects in critical care medicine.

Methods

Critical care meta-analyses involving randomized controlled trials and reported mortality as an outcome measure were considered eligible for the study. Component trials were classified as large (≥100 patients per arm) and small (<100 patients per arm) according to their sample sizes. Ratio of odds ratio (ROR) was calculated for each meta-analysis and then RORs were combined using a meta-analytic approach. ROR<1 indicated larger beneficial effect in small trials. Small and large trials were compared in methodological qualities including sequence generating, blinding, allocation concealment, intention to treat and sample size calculation.

Results

A total of 27 critical care meta-analyses involving 317 trials were included. Of them, five meta-analyses showed statistically significant RORs <1, and other meta-analyses did not reach a statistical significance. Overall, the pooled ROR was 0.60 (95% CI: 0.53 to 0.68); the heterogeneity was moderate with an I2 of 50.3% (chi-squared = 52.30; P = 0.002). Large trials showed significantly better reporting quality than small trials in terms of sequence generating, allocation concealment, blinding, intention to treat, sample size calculation and incomplete follow-up data.

Conclusions

Small trials are more likely to report larger beneficial effects than large trials in critical care medicine, which could be partly explained by the lower methodological quality in small trials. Caution should be practiced in the interpretation of meta-analyses involving small trials.