Systematic reviews and often concomitant meta-analyses are designed to summarize the existing and sometimes overwhelming amount of published literature on a specific topic. Ideally, a systematic review and meta-analysis is reproducible, rigorous and strengthens the evidence on the specific topic, but there are potential pitfalls in designing such studies.

O’Byrne et al. (BJOG 2021;) performed a systematic review and meta-analysis regarding the pregnancy outcomes of pregnant women with chronic inflammatory disease exposed to biologics. In the past years some other summarized data have been published on this relevant topic. The authors of the current study tried to distinguish their study from these previous systematic reviews and meta-analyses by using broad inclusion criteria and by expanding the control group population. For reference, a study from Tsao et al. (Rheumatology2020;59:1808–1817) included 24 individual studies and approximately 5600 pregnancies in women with inflammatory systemic diseases exposed to biologics. They compared the outcomes with a disease-matched control group. Komaki et al. (Journal of Autoimmunity 2017;76:38-52) studied 5600 pregnancies of women with immune mediated diseases and the use of anti-tumor necrosis factor-α agents. They included 13 studies and both a disease-matched control group and a general population control group. O’Byrne et al. in contrast, also included case series and case reports in addition, resulting in a study population of more than 11000 exposed cases.

Then, the methods of data analysis were fitted to the wide design of the study. The most applicable method of measuring an effect of an exposure on maternal and fetal outcomes is with relative effect measures, also known as odds ratios. In the current study the authors used proportions in their meta-analyses. Proportions represent the absolute risk of an outcome in the exposed and non-exposed groups and are not perfectly suited to draw conclusions about the association of an exposure and an outcome. But the wide study design, in particular the inclusion of case series and case reports, left the authors with no choice but to use proportions.

Further, this broad inclusion has two more major drawbacks. The meta-analysis showed in most analyses a high heterogeneity. This can be attributed to the inclusion of different study designs and comparisons of different diseases and medications, as the authors mentioned. In addition, there is a serious risk of bias, firstly due to the high selectivity applied when choosing the population of case series and case reports, and secondly due to the use of proportions, making it impossible to adjust for confounders.

Well-designed systematic reviews and meta-analyses can generate a clear overview and provide valuable information for clinicians and patients alike. However, although including a large number of cases is recommended to strengthen the evidence of a systematic review/meta-analysis, this should not be done at any cost - it can sometimes weaken the quality of the study. The most appropriate method should be applied and individualized to the specific study.