Writing a protocol
Once the clinical question for a systematic review has been formulated, the next step is to write the protocol. This allows decisions to be made in advance regarding the evidence that will be included, and how it will be reported. This approach reduces the potential for bias associated with decision making when the systematic review's findings are known. For example, prior knowledge of the evidence could influence inclusion criteria or how results are analyzed.
The SR toolbox is an online catalog providing summaries and links to the available guidance and software for each stage of the systematic review process, including protocol development.
Search strategies
Set the parameters for searches at the protocol stage so that they will identify as many studies as possible that meet the eligibility criteria (see below). Consider filters such as date, language, type of study, and publication status to identify relevant studies without introducing bias in these areas.
Eligibility criteria
The PICOT system is used as a framework for establishing the details of the inclusion/exclusion criteria:[1]
- Population included
- Intervention assessed
- Comparison tested
- Outcome involved
- Time frame measured.
Study types that would be considered for inclusion also need to be prespecified e.g., whether the systematic review will be limited to randomized controlled trials, excluding crossover studies, etc. This may affect the search filters used as well as decisions made at the screening stage.
Quality assurance
Predefine how the risk of bias will be assessed, including any tools to be used. Detail the number of individuals involved and how arbitration will be carried out if required.
Many checklists are available, with different ones for each type of trial. The SR toolbox provides an extensive list of these. Some examples include:
- Revized Cochrane risk-of-bias tool for randomized trials (RoB 2)
- JADAD scale
- Newcastle-Ottawa Scale (NOS) or ROBINS-I for nonrandomized studies
- QUADAS-2 for diagnostic accuracy studies
- Critical Appraisal Skills Programme (CASP) checklists.
More on appraising the evidence
Evidence synthesis
The statistical methods to be used to answer the review question are planned in advance and specified in the protocol. It is important this is based on the question and set out in the protocol to avoid any bias or spurious conclusions from post-hoc analyses.
For a proposed meta-analysis include information on when results can be combined or not (e.g., class effects of interventions, active versus inactive comparators, shorter and longer timeframes, etc). Other areas for consideration include which effect measure to use (e.g., relative risk, odds ratio, hazard ratio, risk difference, etc) and how to assess heterogeneity. Predefine the method for meta-analysis. For dichotomous outcomes, this can be a fixed-effect (Mantel-Haenszel, Peto odds ratio, or inverse variance) or random-effects (DerSimonian and Laird inverse variance) model. For continuous variables, consider if the mean difference or standardized mean difference will be used, and whether the analysis will include change scores (change-from-baseline measurements) or just post-intervention values.
Subgroup analyses
These must be defined in the protocol. They should be limited in number with a clear rationale for each. Post-hoc subgroup analysis (i.e., conducted without pre-specification) risks spurious conclusions.
Importance of outcomes and certainty of evidence
Predetermine which outcomes are critical and which are important (up to about seven outcomes if required); outcomes which are not important (including usually surrogate or interim outcomes) should not be included in the systematic review. This prevents post-hoc presentation of the outcomes with the largest effect sizes as being most important, or the inclusion of an outcome just because studies often include it (rather than it being required for clinical decision making).
The protocol should include how the certainty of the body of evidence will be assessed (e.g., GRADE) as this will affect the interpretation of results.
Publishing the protocol
All systematic review protocols should be prospectively registered on PROSPERO; some journals also publish protocols. This reduces duplication of effort and research waste. Changes to the methods outlined in the protocol should only be made if unavoidable and must be transparently declared.
Example of a protocol template
| Review question | Insert clinical question in PICO format |
| Objectives | What are the aims of the review? E.g., effectiveness, cost-effectiveness, diagnostic accuracy. For a straightforward systematic review of interventions, this will be very similar to the review question. However, for a qualitative review or network meta-analysis this section may include important additional information |
| Eligibility criteria | Include detailed PICOT information
Outcomes should be divided into main and additional (primary and secondary) Information on types of study to be included (and which studies, if any, would be considered if no studies of first preference type are found) Other criteria e.g., minimum sample size |
| Search | List databases to be searched
Include information on date or language restrictions and study filters, if these are to be used, with reasons |
| Review strategy | Include information on how risk of bias and certainty of evidence will be assessed
Detail methods for evidence synthesis including any tools that will be used List any prespecified subgroup analyses with reasons Plan if meta-analysis is not appropriate (e.g., report ranges of results) |
| PROSPERO registration number | xxxxx |
References
1. Brown P, Brunnhuber K, Chalkidou K et al. How to formulate research recommendations. BMJ. 2006;333:804–6.https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1602035/