Multiple systematic reviews on the same question
Once a clinical question has been formulated, it is useful to search the literature to check if there are any existing systematic reviews that have already attempted to answer the same or a similar question.
If multiple systematic reviews are identified and these reach similar conclusions, this increases confidence in those conclusions. But it is possible that the systematic reviews have the same flaws, so it is important to appraise the systematic reviews and to take care not to overestimate the strength of evidence just because more than one systematic review draws the same conclusion.
A more challenging situation arises when the reviews reach conflicting conclusions to the same clinical question. This could be a difference in order of magnitude of effect size (although this could mean one conclusion is clinically relevant while the other is not), whether there is agreement of an effect, or even a difference in the direction of the effect. This can occur because of different search methods, inclusion criteria, or analysis methods.
The following questions may be helpful:
- Are all the reviews valid? Do they contain any methodological flaws?
- Is the difference between the reviews clinically important — i.e., would they cause you to make a different clinical decision? (e.g., is the direction of the effect similar, even if the size of the effect differs?)
- Are the reviews asking the same question? They might actually be looking at different aspects of a similar question: for example, they may be looking at different subgroups within a population, different outcomes, or different related interventions (e.g., different drugs in a drug class, or different psychological treatment protocols)
- Do the reviews identify the same or different primary studies? This will depend on the reviews’ search methods, their inclusion criteria, and also when they carried out the search. Before drawing any conclusions, check the primary data the reviews analyze. If the reviews identified different studies, think about why this was
- In general, if both reviews are valid, use appropriate selection criteria, and ask the same question; the review with the most comprehensive search and based on the most data is least likely to be biased.
Epistemonikos, a free-to-use database of over 400,000 systematic reviews, helps identify potentially relevant systematic reviews and to compare the primary studies included.
In general if two (or more) valid systematic reviews:
- Retrieve the same studies (or one reports the same studies as all the others plus a few more)
- Report the same outcomes in the same population, and
- Reach the same conclusions,
you might choose to use only the most recent review (by search date), and supersede all older reviews.
The exceptions are:
- If there are very few data available to answer the clinical question, you might choose to consider all the systematic reviews found.
- If the reviews identify the same RCTs, but an older review has superior methods to a more recent review (e.g., it performs an appropriate and clinically useful meta-analysis or provides additional data on the RCTs while the later review does not), it might be appropriate to consider all reviews, taking into account the higher quality methods of the older review.
- If the reviews come to different conclusions and the difference is not explained by the addition of material subsequent to the older review, or by superior methods in the later review, it might be helpful to consider both reviews and also attempt to think about possible explanations for the difference in conclusions.
- If the studies retrieved by the reviews are the same, but the reviews performed different analyses (of outcomes or populations), it might be useful to consider both reviews and the overlap between them.
- If the reviews contain some but not all of the same studies (a common scenario), it might be worth considering several reviews. But an older review can be superseded if it contains studies excluded by a later review on appropriate methodological grounds. However, if there doesn’t seem to be a good reason for the inclusion of different studies in reviews on the same question, or if each review gives different information, it might be appropriate to use multiple reviews. Understanding the relationships between these reviews is helpful, to make clear that you are not double-counting primary studies (and thus overestimating the strength of the evidence).
Related BMJ Blogs
How much health evidence is there in the world?
How systematic reviews can reduce waste in research
Content created by BMJ Knowledge Centre