Section 6: Systematic literature review

Guidelines based on a consensus of expert opinion or on unsystematic literature surveys have been criticised as not reflecting current medical knowledge and being liable to bias.19,20 SIGN guidelines are therefore based on a systematic review of the evidence. Systematic review is defined as “an efficient scientific technique to identify and summarise evidence on the effectiveness of interventions and to allow the generalisability and consistency of research findings to be assessed and data inconsistencies to be explored”.21

The SIGN approach leads to guidelines that are essentially the direct product of the systematic review. There is no separate report of the review or its conclusions, though all the stages of the review process are thoroughly documented (see below). Because the reviews are largely undertaken by members of SIGN guideline development groups working part time on the project, and within a limited timescale, their coverage of the literature may be more limited than those carried out by dedicated systematic review groups such as the Cochrane Collaboration. Nevertheless, the essential elements of systematic review are met:

The benefits of the SIGN approach derive from the close involvement of guideline developers with the synthesis of the evidence base, allowing them to apply their considered judgement when deriving recommendations (see Chapter 7), and from encouraging a sense of ownership of the guideline amongst all those involved in the process.

6.1 ADDRESSING PATIENT ISSUES IN THE LITERATURE SEARCH

Incorporating the patient’s perspective from the beginning of the development process is essential if it is to influence the coverage of the final guideline. One of the measures used to achieve this is to conduct a specific search on patient issues in advance of the first meeting of the guideline development group.

This search is designed to cover both quantitative and qualitative evidence, and is not limited to specific study designs. It is carried out over the same range of databases and sources as the main literature review, but will normally include both nursing and psychological literature even where these are not seen as particularly relevant to the later searches of the medical literature.

The use of this literature search is discussed in more detail in Chapter 4.2

6.2 USING EXISTING GUIDELINES

The guidelines identified in the scoping search carried out for the original guideline proposal (see Chapter 3.4) will be presented to an early meeting of the guideline development group to allow it to consider what has been done already.

In some cases good quality, directly relevant guidelines will have been produced on some of the issues that fall within the remit of the new guideline. In these circumstances reference will be made to the existing guidelines rather than repeating work that has already been completed. All guidelines must be evaluated using the AGREE instrument and be shown to have followed an acceptable methodology before they can be considered for use in this way.

In other cases existing guidelines may not be directly relevant to NHSScotland, or may be found to have methodological weaknesses. If these guidelines are based on a well conducted systematic review, the guideline group may be able to use the evidence base from those guidelines as a starting point for its own review.

As more good quality guidelines are being produced by other agencies, SIGN is considering use of the ADAPTE instrument 22,23 to adapt guidelines produced elsewhere for use inNHSScotland. A trial of this process started in April 2007, looking at the guideline on obesity produced by the National Institute for Health and Clinical Excellence.

6.3 DEFINING KEY QUESTIONS

The training in critical appraisal and guideline development offered to members of SIGN guideline development groups encourages them to break down the guideline remit into a series of structured key questions using the PICO format:

Patients or population to which the question applies

Intervention (or diagnostic test, exposure, risk factor, etc.) being considered in relation to these patients

Comparison(s) to be made between those receiving the intervention and another group who do not receive the intervention

Outcome(s) to be used to establish the size of any effect caused by the intervention.

The Patients or population to be covered by the literature searches is largely defined by the presence of the particular condition that the guideline will cover. It should be made clear at this stage, however, which age groups are to be covered. For searching the main medical databases these can be split into:

Consideration should also be given as to whether any particular ethnic or social groups have particular needs in relation to the topic under review. If it is thought that any group needs particular consideration in relation to a key question (people of African origin who have sickle cell disease, for example, may need a different approach to antibiotic treatment) the needs of these groups should be specifically addressed in the key questions and subsequent literature searches.

It is worth emphasising here that questions should be addressed even if it is not thought there will be any good evidence. If there is in fact no good evidence, then highlighting it as an area for research is a useful outcome in itself.

Exclusion of any group from the population covered by the guideline should be identified when setting the key questions, and reasons given for their exclusion.

The Interventions (which in this context includes diagnostic tests, risk factors, risk exposure) must be specified clearly and precisely. The only exception is in drug therapy where drug classes should be used in preference to specific agents unless there is a clear reason for focusing on a named agent.

The decision on Comparisons is mostly between placebo / no treatment, or comparison with alternative therapies. It should be borne in mind that where there is an existing treatment comparisons with placebo or no treatment are not ethically acceptable.

It is important to specify Outcomes in advance, and to think of these in terms of what outcomes will influence the views of guideline group members as to how effective a particular intervention is. For some questions there will be a wide range of outcomes used in the literature, and if useful comparisons are to be made across studies it must be made clear which of these outcomes are important.

As far as possible outcomes should be objective and directly related to patient outcomes (eg length of time to next cardiovascular incident or survival time, rather than just reductions in blood pressure). It is also important to include outcomes that are important to patients, rather than focusing entirely oin clinical outcomes.

These questions then form the basis of the literature search, which is undertaken by a SIGN Information Officer.

Definition of a set of clear and focused clinical questions is fundamental to the successful completion of a guideline development project. It is also important to be realistic about the number of questions that can be addressed in a single guideline if the final product is not to be too large to be useable. A large number of key questions also implies a very high workload for the developers, and care must be taken to ensure this is kept within manageable limits. Where the number of questions reaches 40 or more, serious consideration must be given as to whether the scope of the guideline needs to be redefined.

Deciding the key questions is entirely the responsibility of the guideline development group who must apply its knowledge and experience to ensuring the questions address the key issues in the area to be covered by the guideline. The Information Officer working with the group will provide guidance on the formatting of the questions, and ensure they are in a format likely to produce useable results. They will also ensure that the key questions address most, if not all, the issues identified through the patient consultation exercise (see Chapter 4.2).

6.4 IDENTIFYING AND SELECTING THE EVIDENCE

The literature search must focus on the best available evidence to address each key question, and should ensure maximum coverage of studies at the top of the hierarchy of study types (see Annex B). SIGN uses a set of standard search filters that identify:

These search filters are available from the SIGN website.

The systematic literature review procedure is illustrated in Figure 10.

Figure 10: Systematic literature review

[Figure 10: Systematic literature review]

In order to minimise bias and to ensure adequate coverage of the relevant literature, the literature search must cover a range of sources. As a minimum, SIGN requires searches to cover the Cochrane Library, Embase, Medline, NHS Economic Evaluations Database (NEED) and the Internet. It is expected that in most cases the search will also cover additional sources specific to the topic under review.

The period that the search should cover will depend on the nature of the clinical topic under consideration, and will be discussed with the guideline development group. For a rapidly developing field a 5 or 10-year limit to the search may be appropriate, whereas in other areas a much longer time frame might be necessary.

All the main search strategies are subject to an independent review by an Information Scientist based elsewhere in NHS Quality Improvement Scotland.

SIGN does not undertake hand searching of key journals as part of the literature review. It is accepted that this means some relevant trials may be missed, and introduces the possibility of a degree of bias in the process. However, given time and resource constraints, it is not feasible for this to form part of the process.

A listing of the Medline search strategies used for the guideline, plus notes of any significant variation on other databases, is published on the SIGN website at the time of the National Meeting associated with the guideline. This strategy will remain on the website as part of the supporting material for the guideline when it is published.

Before any papers are acquired for evaluation, sifting of the search output is carried out to eliminate irrelevant material. A preliminary sift of each search result is carried out by staff at the SIGN Executive, normally by the individual that carried out the search. Papers that are clearly not relevant to the key questions are eliminated. Abstracts of remaining papers are then examined and any that are clearly not appropriate study designs, or that fail to meet specific methodological criteria, will also be eliminated at this stage.

A final sift is carried out by one or two individuals from the guideline development group, who will reject other papers that do not meet specific clinical or other exclusion criteria that have been agreed by the development group. Only when all stages of search result sifting have been completed will the remaining papers be acquired for evaluation.

All sifting is carried out according to an agreed protocol setting out the criteria used to select papers for inclusion or elimination from the process.

In practice, a single search does not cover all the questions being addressed within a guideline. Different questions may be best answered by different databases, or may rely on different levels of evidence. Information Officers take an iterative approach to the task, carrying out a search for high level evidence in the first instance. After the results of this search have been evaluated, the questions may be redefined and subsequent searches focused on the most appropriate sources and study types. This iterative process is illustrated in Figure 10.

6.5 EVALUATING THE EVIDENCE

Once papers have been selected as potential sources of evidence, the methodology used in each study is assessed to ensure its validity. The result of this assessment will affect the level of evidence allocated to the paper, which will in turn influence the grade of recommendation that it supports (see Chapter 7).

The methodological assessment is based on a number of key questions that focus on those aspects of the study design that research has shown to have a significant influence on the validity of the results reported and conclusions drawn. These key questions differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process. SIGN has based its assessments on the MERGE (Method for Evaluating Research and Guideline Evidence) checklists developed by the New South Wales Department of Health,24 which have been subjected to wide consultation and evaluation. These checklists were subjected to detailed evaluation and adaptation to meet SIGN’s requirements for a balance between methodological rigour and practicality of use. Copies of these checklists and accompanying notes on their use are included in Annex C.

The assessment process inevitably involves a degree of subjective judgement. The extent to which a study meets a particular criterion – eg an acceptable level of loss to follow up – and, more importantly, the likely impact of this on the reported results from the study will depend on the clinical context. To minimise any potential bias resulting from this, each study must be evaluated independently by at least two individuals. Any differences in assessment should then be discussed by the full group. Where differences cannot be resolved, an independent reviewer will arbitrate to reach an agreed quality assessment.

[Contents][Back][Top] [Next]

Scottish Intercollegiate Guidelines Network
SIGN 50: A guideline developer's handbook <Methodology <Home