Our first posts in this blog provided an introduction to network meta-analysis and its key assumptions. In this third post we look at how to plan a network meta-analysis.
Most network meta-analyses will consist of six key stages – developing the protocol, performing the systematic review, conducting the feasibility assessment, preparing the statistical analysis plan, running the analyses and reporting the results. In this post, we will give a brief overview of each of these stages, and suggest strategies for ensuring that your network meta-analysis project goes smoothly.
At the start of the project, you should consider the audience for your systematic review and network meta-analysis. If you are aiming to develop a manuscript for a peer-reviewed journal, then it is important to make sure that your project is compatible with publication guidelines such as the PRISMA extension statement for network meta-analysis . Likewise, if you are aiming to develop a submission for a specific health technology assessment (HTA) body, then you will need to consider their guidelines on systematic review and network meta-analysis.
At the outset, it is often possible to come up with a reasonable estimate for how much time will be required to develop the protocol, perform the systematic review and conduct the feasibility assessment. However, it can be more difficult to estimate the time required to prepare the statistical analysis plan, network meta-analyses and report. Until the feasibility assessment is complete, the number of networks that are feasible and the nature of these networks is often unknown. This information is important for estimating the time required to prepare the statistical analysis plan and run the network meta-analyses. Hence, systematic review and network meta-analysis projects are often planned in two stages – the protocol, systematic review and feasibility assessment stage is planned up front, and the statistical analysis plan, network meta-analysis and reporting stage is planned once the feasibility assessment is complete.
Developing the protocol
The first step is to a develop a protocol for the systematic review and network meta-analysis. The protocol should define the research question and specify the methods for the systematic review. The research question will often be defined using the PICO format . This format is used to specify the Population, Interventions, Comparators and Outcomes of interest. The methods for the systematic review should define the search strategy and the study selection and data extraction processes.
At the protocol stage, it is usually not possible to fully specify the methods for the statistical analyses. Unlike in clinical trials, where the investigators get to define exactly what data will be collected from each patient; in systematic reviews, the data available from each study is variable. In our protocols, we usually provide a brief overview of the methods we plan to use, but leave the specific details for the statistical analysis plan.
A systematic review and network meta-analysis project usually involves a team of systematic reviewers, statisticians and clinicians. The protocol will usually be drafted by a systematic reviewer, but it is important that it is reviewed by the whole team. For example, the clinicians can confirm whether the research question covers all of the clinically important outcomes and the statisticians can confirm whether the data extraction will capture all of the information required for the analysis.
Once the protocol is finalised, you should consider registering it on a database such as PROSPERO. Registration is now expected by some journals.
Performing the Systematic Review
The systematic review includes the literature search, study selection and data extraction. Unless the review is very small, this stage will usually be conducted by a team of reviewers. The key outputs from this stage usually include:
- a PRISMA diagram , to illustrate the study selection process,
- a summary of the eligible studies and
- a data extraction spreadsheet.
Conducting the Feasibility Assessment
Once the systematic review is complete, a feasibility assessment is required in order to determine which of the studies can be appropriately combined in a network meta-analysis. Feasibility assessments involve a careful review of study design and patient characteristics, in order to evaluate whether the studies are sufficiently similar. The Pharmaceutical Benefits Advisory Committee (PBAC) guidelines suggest a list of characteristics for review (see Appendix 4). Our previous blog on the assumptions of network meta-analysis discusses what is meant by ‘similar’ in the context of network meta-analyses.
If some differences between studies are identified, then it is often useful to discuss these with a clinician, in order to determine if they are likely to impact the network meta-analysis. For example, if some studies only evaluate older patients, a clinician could advise whether the treatments are likely to impact older patients differently. If there are important differences between studies, then options include: excluding studies, conducting subgroup analyses or adjusting for the differences between studies using meta-regression.
Once the studies that are appropriate to combine in a network meta-analysis have been identified, network diagrams for each outcome can be produced. For standard network meta-analyses, connected networks are required (our first HTA blog provided an introduction to connected and disconnected networks). We’ll be looking at what can be done with disconnected networks in a future blog.
If you’d like to know more about feasibility assessments for network meta-analysis, then the York Health Economics Consortium have produced a short video tutorial on this topic.
Preparing the Statistical Analysis Plan
Although there is no requirement to produce a statistical analysis plan for a network meta-analysis, we almost always recommend one. The statistical analysis plan describes the available data and sets out the plan for conducting and reporting the statistical analyses. It helps the statistical analysis phase of the project to run more efficiently because it provides a clear plan for what analyses will be conducted and what output will be produced.
Running the Analyses
This stage incorporates processing the data, conducting the network meta-analyses and outputting the results in a useable format. The time it takes to run the analyses will depend on the complexity of the network meta-analysis models, the number of outcomes of interest and the number of studies.
Processing the data involves reading the data extraction spreadsheet into the statistical software and calculating any missing values required for the network meta-analysis (for example, for continuous outcomes, estimating missing standard errors from confidence intervals, or other data). This stage is most efficient when the systematic review team and statistical team work together to design a data extraction spreadsheet that is convenient for both extracting and processing the data.
Network meta-analyses can be conducted using a range of statistical software. For Bayesian models, we usually use R in conjunction with specific Bayesian software such as WinBUGS, OpenBUGS or JAGS. For frequentist network meta-analysis models, Stata and SAS are popular choices.
The output from this stage will depend on the specific project but will usually include tables and forest plots. For Bayesian models, it is also straightforward to produce the probabilities that each treatment is the best and other summaries of the treatment ranks, such as SUCRA plots .
Reporting the Results
A single systematic review and network meta-analysis project may be reported in a variety of ways. Usually there will be a complete systematic review and network meta-analysis report (possibly hundreds of pages long!) that fully documents all aspects of the project. The aim of this report is to ensure that the work is reproducible in the future. Depending on the goal of the project, the systematic review and network meta-analysis may also be incorporated into submissions to HTA bodies, or published as a manuscript in a peer-reviewed journal.
For publications, it is really important to consider the timing of the systematic review searches. Journals will often reject publications if they think the search is out-of-date. Hence, if publication is a goal, then it is important to start preparing for the manuscript whilst the systematic review and network meta-analysis is ongoing. If there are any delays to the project, it may be best to incorporate an update search once the feasibility assessment and statistical analysis plan have been drafted.
We hope you’ve found this post informative. If there are any particular topics you’d like to see covered in the future then please let us know. If you are looking for an in depth intro to network meta-analysis YHEC and Quantics run joint courses – currently available in-house, on request.
 Hutton B et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Annals of internal medicine. 2015; 162 (11): 777-84. Available from: http://www.prisma-statement.org/documents/PRISMA%20NMA%20Annals%202015.pdf
 Sackett DL, Richardson WS, Rosenberg W, Haynes RB. How to practice and teach evidence-based medicine. New York: Churchill Livingstone. 1997.
 Liberati A et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine, 6(7). Available from: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000100
 Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. Journal of clinical epidemiology. 2011; 64(2):163-71. Available from: http://www.sciencedirect.com/science/article/pii/S0895435610001691