Notes on the use of Methodology Checklist 2: Randomised Controlled Trials

Section 1 identifies the study, the reviewer, the guideline for which the paper is being considered as evidence, and the key question(s) it is expected to address. The reviewer is asked to consider a series of aspects of RCT design and to make a judgement as to how well the current study meets this criterion. Each relates to an aspect of methodology that research has shown makes a significant difference to the conclusions of a study.

For each question in this section you should use one of the following to indicate how well it has been addressed in the study:

1.1 The study addresses an appropriate and clearly focused question

Unless a clear and well defined question is specified, it will be difficult to assess how well the study has met its objectives or how relevant it is to the question you are trying to answer on the basis of its conclusions.

1.2 The assignment of subjects to treatment groups randomised

Random allocation of patients to receive one or other of the treatments under investigation, or to receive either treatment or placebo, is fundamental to this type of study. If there is no indication of randomisation, the study should be rejected. If the description of randomisation is poor, the study should be given a lower quality rating. Processes such as alternate allocation, allocation by date of birth, or day of the week attending a clinic are not true randomisation processes and it is easy for a researcher to work out which patients received which treatment. These studies should therefore be classed as Controlled Clinical Trials rather than RCTs.

1.3 An adequate concealment method is used

Allocation concealment refers to the process used to ensure that researchers are unaware which group patients are being allocated to at the time they enter the study. Research has shown that where allocation concealment is inadequate, investigators can overestimate the effect of interventions by up to 40%. Centralised allocation, computerised allocation systems, or the use of coded identical containers would all be regarded as adequate methods of concealment, and may be taken as indicators of a well conducted study. If the method of concealment used is regarded as poor, or relatively easy to subvert, the study must be given a lower quality rating, and can be rejected if the concealment method is seen as inadequate.

1.4 Subjects and investigators are kept ‘blind’ to treatment allocation

Blinding refers to the process whereby people are kept unaware of which treatment an individual patient has been receiving when they are assessing the outcome for that patient. It can be carried out up to three levels. Single blinding is where patients are unaware of which treatment they are receiving. In double blind studies neither the doctor nor the patient knows which treatment is being given. In very rare cases studies may be triple blinded, where neither patients, doctors, nor those conducting the analysis are aware of which patients received which treatment. The higher the level of blinding, the lower the risk of bias in the study.

1.5 The treatment and control groups were similar at the start of the trial

Patients selected for inclusion in a trial must be as similar as possible. The study should report any significant differences in the composition of the study groups in relation to gender mix, age, stage of disease (if appropriate), social background, ethnic origin, or comorbid conditions. These factors may be covered by inclusion and exclusion criteria, rather than being reported directly. Failure to address this question, or the use of inappropriate groups, should lead to the study being downgraded.

1.6 The only difference between the groups is the treatment under investigation

If some patients received additional treatment, even if of a minor nature or consisting of advice and counselling rather than a physical intervention, this treatment is a potential confounding factor that may invalidate the results. If groups were not treated equally, the study should be rejected unless no other evidence is available. If the study is used as evidence it should be treated with caution.

1.7 All relevant outcomes measured in a standard, valid and reliable way

The primary outcome measures used should be clearly stated in the study. If the outcome measures are not stated, or the study bases its main conclusions on secondary outcomes, the study should be rejected. Where outcome measures require any degree of subjectivity, some evidence should be provided that the measures used are reliable and have been validated prior to their use in the study.

1.8 What percentage of the individuals or clusters recruited into each treatment arm of the study dropped out before the study was completed?

The number of patients that drop out of a study should give concern if the number is very high. Conventionally, a 20% drop out rate is regarded as acceptable, but this may vary. Some regard should be paid to why patients dropped out, as well as how many. It should be noted that the drop out rate may be expected to be higher in studies conducted over a long period of time. A higher drop out rate will normally lead to downgrading, rather than rejection of a study.

1.9 All the subjects are analysed in the groups to which they were randomly allocate (intention to treat analysis)

In practice, it is rarely the case that all patients allocated to the intervention group receive the intervention throughout the trial, or that all those in the comparison group do not. Patients may refuse treatment, or contra-indications arise that lead them to be switched to the other group. If the comparability of groups through randomisation is to be maintained, however, patient outcomes must be analysed according to the group to which they were originally allocated irrespective of the treatment they actually received. (This is known as intention to treat analysis.) If it is clear that analysis was not on an intention to treat basis, the study may be rejected. If there is little other evidence available, the study may be included but should be evaluated as if it were a non-randomised cohort study.

1.10 Where the study is carried out at more than one site, results are comparable for all sites

In multi-site studies, confidence in the results should be increased if it can be shown that similar results were obtained at the different participating centres.

Section 2 relates to the overall assessment of the paper. It starts by rating the methodological quality of the study, based on your responses in Section 1 and using the following coding system:

++ All or most of the criteria have been fulfilled. Where they have not been fulfilled the conclusions of the study or review are thought very unlikely to alter.
+ Some of the criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.
- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The code allocated here, coupled with the study type, will decide the level of evidence that this study provides.
The aim of the other questions in this section is to summarise your view of the quality of this study and its applicability to the patient group targeted by the guideline you are working on.

Section 3 asks you to summarise key points about the study that will be added to an evidence table at the next stage of the process. It is important that you complete this section as fully as possible, and include actual data from the study wherever relevant.

[SIGN 50 Annex C] [Checklist]

Scottish Intercollegiate Guidelines Network
SIGN 50: A guideline developer's handbook <Methodology <Home