Once the clinical question for a systematic review has been formulated the next step is to write the protocol. This allows decisions to be made in advance regarding the evidence that will be included, and how it will be reported. This approach reduces the potential for bias associated with decision-making when the findings of the systematic review are known. For example, prior knowledge of the evidence could influence inclusion criteria or how results are analysed.

The SR toolbox is an online catalogue providing summaries and links to the available guidance and software for each stage of the systematic review process including protocol development.

Search strategies

Set parameters for searches at the protocol stage so that they will identify as many studies as possible that meet the eligibility criteria (see below). Consider filters such as date, language, type of study and publication status so that relevant studies are identified without introducing bias in these areas.

More on search strategies.

Eligibility criteria

The PICOT system is used as a framework for establishing the details of the inclusion/exclusion criteria.[1]

  • Population included
  • Intervention assessed
  • Comparison tested
  • Outcome involved
  • Time frame measured

Study types that would be considered for inclusion also need to be prespecified e.g., whether the systematic review will be limited to randomised controlled trials, excluding crossover studies etc. This may affect the search filters used as well as decisions made at the screening stage.

Quality assurance

Predefine how risk of bias will be assessed including any tools to be used. Detail the number of individuals involved and how arbitration will be carried out if required.

A large number of checklists are available, with different ones for each type of trial. The SR toolbox provides an extensive list of these, some examples include:

  • Revised Cochrane risk-of-bias tool for randomised trials (RoB 2)
  • JADAD scale
  • Newcastle-Ottawa Scale (NOS) or ROBINS-I for nonrandomised studies
  • QUADAS-2 for diagnostic accuracy studies
  • Critical Appraisal Skills Programme (CASP) checklists

More on appraising the evidence

Evidence synthesis 

The statistical methods to be used to answer the review question are planned in advance and specified in the protocol. It is important this is based on the question and set out in the protocol to avoid any bias or spurious conclusions from post-hoc analyses.

For a proposed meta-analysis include information on when results can be combined or not (e.g., class effects of interventions, active versus inactive comparators, shorter and longer timeframes etc.). Other areas for consideration include which effect measure to use (e.g. relative risk, odds ratio, hazard ratio, risk difference etc) and how to assess heterogeneity. Predefine the method for meta-analysis. For dichotomous outcomes this can be a fixed-effect (Mantel-Haenszel, Peto odds ratio or inverse variance) or random-effects (DerSimonian and Laird inverse variance) model. For continuous variables consider if mean difference or standardised mean difference will be used, and whether the analysis will include change scores (change-from-baseline measurements) or just post-intervention values.

More on evidence synthesis

Subgroup analyses

These must be defined in the protocol. They should be limited in number with a clear rationale for each. Post-hoc subgroup analysis (i.e., conducted without prespecification) risks spurious conclusions..

Importance of outcomes and certainty of evidence

Predetermine which outcomes are critical and which are important (up to about 7 outcomes if required); outcomes which are not important (including usually surrogate or interim outcomes) should not be included in the systematic review. This prevents post-hoc presentation of the outcomes with the largest effect sizes as being most important, or the inclusion of an outcome just because studies often include it (rather than it being required for clinical decision making).

The protocol should include how the certainty of the body of evidence will be assessed (e.g., GRADE) as this will affect the interpretation of results.

Publishing the protocol

All systematic review protocols should be prospectively registered on PROSPERO, some journals also publish protocols. This reduces duplication of effort and research waste. Changes to the methods outlined in the protocol should only be made if unavoidable and must be transparently declared.

Example of a protocol template

Review question Insert clinical question in PICO format
Objectives What are the aims of the review e.g. effectiveness, cost effectiveness, diagnostic accuracy. For a straightforward systematic review of interventions this will be very similar to the review question, however for a qualitative review or network meta-analysis this section may include important additional information
Eligibility criteria Include detailed PICOT information. 

Outcomes should be divided into main and additional (primary and secondary).

Information on types of study to be included (and which studies if any would be considered if no studies of first preference type are found).

Other criteria e.g., minimum sample size

Search List databases to be searched. 

Include information on date or language restrictions and study filters, if these are to be used, with reasons.

Review strategy Include information on how risk of bias and certainty of evidence will be assessed.

Detail methods for evidence synthesis including any tools that will be used. 

List any prespecified subgroup analyses with reasons.

Plan if meta-analysis is not appropriate (e.g., report ranges of results).

PROSPERO registration number xxxxx

References

1. Brown P, Brunnhuber K, Chalkidou K et al. How to formulate research recommendations. BMJ. 2006;333:804–6.https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1602035/