![[SIGN thistle header]](../../../images/page-header-template.jpg)
NOTES on the USE of Methodology Checklist 1: Systematic Reviews and Meta-analyses
Section 1 identifies the study, the reviewer, the guideline for which the paper is being considered as evidence, and the key question(s) it is expected to address. The reviewer is asked to consider a series of aspects of study design and to make a judgement as to how well the current study meets each criterion. Each relates to an aspect of methodology that research has shown to be likely to influence the conclusions of a study.
For each question in this section you should use one of the following to indicate how well it has been addressed in the study:
1.1 The study addresses an appropriate and clearly focused question.
Unless a clear and well defined question is specified in the report of the review, it will be difficult to assess how well it has met its objectives or how relevant it is to the question you are trying to answer on the basis of the conclusions.
1.2 A description of the methodology used is included.
One of the key distinctions between a systematic review and a general review is the systematic methodology used. A systematic review should include a detailed description of the methods used to identify and evaluate individual studies. If this description is not present, it is not possible to make a thorough evaluation of the quality of the review, and it should be rejected as a source of Level 1 evidence. (Though it may be useable as Level 4 evidence, if no better evidence can be found.)
1.3 The literature search is sufficiently rigorous to identify all the relevant studies.
A systematic review based on a limited literature search – e.g. one limited to Medline only – is likely to be heavily biased. A well conducted review should as a minimum look at Embase and Medline, and from the late 1990s onward, the Cochrane Library. Any indication that hand searching of key journals, or follow up of reference lists of included studies were carried out in addition to electronic database searches can be taken as evidence of a well conducted review.
1.4 Study quality is assessed and taken into account.
A well conducted systematic review should have used clear criteria to assess whether individual studies had been well conducted before deciding whether to include or exclude them. If there is no indication of such an assessment, the individual papers included in the review must be obtained and their methodology evaluated.
1.5 There are enough similarities between the studies selected to make combining them reasonable.
Studies covered by a systematic review should be selected using clear inclusion criteria. These criteria should include, either implicitly or explicitly, the question of whether the selected studies can legitimately be compared. It should be clearly ascertained, for example, that the populations covered by the studies are comparable; that the methods used in the investigations are the same; that the outcome measures are comparable; and the variability in effect sizes between studies is not greater than would be expected by chance alone.
Section 2 relates to the overall assessment of the paper. Question 2.1 asks you to rate the methodological quality of the study, based on your responses in Section 1 and using the following coding system:
| ++ | All or most of the criteria have been fulfilled. Where they have not been fulfilled the conclusions of the study or review are thought very unlikely to alter. |
| + | Some of the criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions. |
| - | Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter. |
The code allocated here, coupled with the study type, will decide the level of evidence that this study provides.
Question 2.2 asks you to indicate whether a review with poor or relatively poor methodology is likely to overstate or understate any effect identified.
Section 3 asks you to identify the types of study covered by the review, and to provide a brief summary of the conclusions of the review as well as your own view of its strengths and weaknesses, and how it will help to answer the key question.