7.1 SYNTHESISING THE EVIDENCE
Guideline recommendations are graded to differentiate between those based on strong evidence and those based on weak evidence. This judgement is made on the basis of an (objective) assessment of the design and quality of each study (as discussed in Chapter 6) and a (perhaps more subjective) judgement on the consistency, clinical relevance and external validity of the whole body of evidence. The aim is to produce a recommendation that is evidence based, but which is relevant to the way in which health care is delivered in Scotland and is therefore implementable.
It is important to emphasise that the grading does not relate to the importance of the recommendation (see also Chapter 7.2.3), but to the strength of the supporting evidence and, in particular, to the predictive power of the study designs from which these data were obtained. Thus, the grading assigned to a recommendation indicates to users the likelihood that, if that recommendation is implemented, the predicted outcome will be achieved.
The process for synthesising the evidence base to form graded guideline recommendations is illustrated in Figure 11.
Evidence tables are compiled by SIGN Executive staff based on the quality assessments of individual studies provided by guideline development group members. The tables summarise all the validated studies identified from the systematic literature review relating to each key question. They are presented in a standard format to make it easier to compare results across studies, and will present separately the evidence for each outcome measure used in the published studies. These evidence tables form an essential part of the guideline development record and ensure that the basis of the guideline development group’s recommendations is transparent. An example evidence table is shown in Annex D.
7.2 CONSIDERED jUDGEMENT
It is rare for the evidence to show clearly and unambiguously what course of action should be recommended for any given question. Consequently, it is not always clear to those who were not involved in the decision making process how guideline developers were able to arrive at their recommendations, given the evidence they had to base them on. In order to address this problem, SIGN has introduced the concept of considered judgement.
Under the heading of considered judgement, guideline development groups summarise their view of the total body of evidence covered by each evidence table. This summary view is split into three parts.
7.2.1 JUDGING THE LEVEL OF EVIDENCE
In the first chapter, the guideline group comments on:
At this point the guideline group is asked to note the overall levels of evidence addressing this specific key question.
7.2.2 JUDGING THE IMPACT OF THE EVIDENCE
For the next step, the guideline group is asked to consider other factors that may influence its eventual grading of a recommendation. These factors are:
The group are finally asked to summarise its view on all of these issues, both the quality of the evidence and its potential impact, before making a graded recommendation. This summary should be succinct, and taken together with its views of the level of evidence represent the first draft of the text that will appear in the guideline immediately before a graded recommendation.
Finally, the group is asked to consider the importance of the recommendation(s) it has just made. Importance is not necessarily related to strength of evidence, but should reflect the extent to which the group believes the recommendation will impact on the health status or quality of life of the patients concerned.
Where the group has indicated that a recommendation is a key recommendation, it is asked to provide a justification for why this recommendation should be highlighted in the final guideline. All key recommendations will be identified as such in the published guideline, and will appear in the Quick Reference Guide.
Guideline development groups are provided with a pro forma in which to record the main points from their considered judgement. An example of this form and the associated notes for users is included in Annex C.
SIGN formerly used the levels of evidence developed by the US Agency for Health Care Policy and Research (AHCPR, now the US Agency for Health Research and Quality, AHRQ).25 As a number of limitations were becoming apparent in that system, a review was carried out and new levels of evidence and associated grades of recommendation were developed. Following extensive consultation and international peer review, the new grading system was introduced in Autumn 2000.26, 27 The current grading system is shown in Annex B.
The assignment of a level of evidence should involve all those on a particular guideline development group or subgroup involved with reviewing the evidence in relation to each specific question. The allocation of the associated grade of recommendation should involve participation of all members of the guideline development group. Where the guideline development group is unable to agree a unanimous recommendation, the difference of opinion should be formally recorded and the reasons for dissent noted.
On occasion, guideline development groups find that there is an important practical point that they wish to emphasise but for which there is not, nor is there likely to be, any research evidence. This will typically be where some aspect of treatment is regarded as such sound clinical practice that nobody is likely to question it (it could be regarded as “clinical common sense”). These are shown in the guideline as Good Practice Points (GPP), and are marked .
It must be emphasised that these are not an alternative to evidence based recommendations. Indeed, the existence of any evidence relating to a key question, however low quality it might be, excludes the possibility of using a good practice point to make a recommendation relating to that question. Examples of how GPPs might be used include:
The revised grading system is intended to place greater weight on the quality of the evidence supporting each recommendation, and to emphasise that the body of evidence should be considered as a whole, and not rely on a single study to support each recommendation. It is also intended to allow more weight to be given to recommendations supported by good quality observational studies where RCTs are not available for practical or ethical reasons. Through the considered judgement process guideline developers are also able to downgrade a recommendation where they think there are important inconsistencies in the evidence base, evidence is not generalisable, not directly applicable to the target population, or for other reasons is perceived as being weaker than a simple evaluation of the methodology would suggest.
7.4 RESOURCE IMPLICATIONS
(This chapter is undergoing a separate detailed review and will be added shortly.)
7.5 CURRENT AREAS FOR DEVELOPMENT
The SIGN Methodology Development Group was established to consider new developments in guideline methodology, and to attempt to answer specific questions on methodological issues. It is currently looking at the following questions:
Qualitative studies as evidence: Qualitative methods are increasingly being used to inform practice in some aspects of medical care. At present, there is no mechanism for incorporating such studies in the evidence base. Some progress has been made on methods of identifying qualitative studies, and in evaluating their methodological quality. The use of qualitative evidence to identify issues of concern to patients, and to help identify key questions to be addressed in the guideline is becoming an established part of SIGN methodology. A pilot exercise looking at the formal inclusion of qualitative evidence in developing a SIGN guideline has been carried out28 and will form the basis of future developments in this area.
Revision of the grading system: The grading system described in Chapter 7.3 is an improvement on the previous system, but still has weaknesses that need to be addressed.29 SIGN has been participating in the international GRADE project aimed at developing a methodologically sound system that can be applied across countries and cultures.30,31
Whether and to what extent the GRADE approach should be adopted by SIGN is under discussion, but whatever is decided there remains a problem in dealing with different types of evidence. GRADE addresses evidence of effectiveness where it is possible to clearly quantify benefits and harms. In other questions addressed by guidelines evidence is more likely to be presented in narrative form. As the grading system develops, means of dealing with both types of evidence in a rigorous manner will be required. Whatever changes are made are likely to be evolutionary rather than revolutionary in nature.