Indicator Evidence

The indicator evidence is a central criterion which must be evaluated independently of the objective of the assessment. The indicator evidence is also of special importance for the public discussion.

The criterion indicator evidence should not be mistaken for the criterion indicator can be influenced by providers. In regard to indicator evidence for outcome indicators, scientific evidence has to be given that the measured outcome can generally be influenced by providers (interrelation between outcome and process/structure measures). With regard to the criterion indicator expression can be influenced by providers, one can assess whether the evaluated provider can actually influence the health care outcome under the given health care structures.

The criterion indicator evidence has been newly developed in the context of QUALIFY. The determination of indicator evidence is based on the commonly used strength of evidence, but it goes beyond that. Additionally, the agreement of experts in health care processes is being integrated into the assessment: the stronger the consensus, the higher the indicator evidence. A similar procedure is known from experience of developing guidelines, in which the strengths of evidence by consensus are transferred to recommendation grades.

In principle, this methodological proceeding is outlined in Evidence based Medicine (EbM):

EbM is the integration of best research evidence with clinical expertise and patient values. When these three elements are integrated, clinicians and patients form a diagnostic and therapeutic alliance which optimizes clinical outcomes and quality of life (http://www.cebm.utoronto.ca/intro/whatis.htm).

The assessment of the indicator evidence in determining indications for a procedure can be difficult. The assessment of a positive risk-benefit relationship is complex and must be done individually for each quality indicator.

In the assessment of outcome indicators, high complication rates will frequently be associated with poor quality of care. Nevertheless, it must be proven in the context of an evidence-based assessment that the occurrence of complications was not unavoidably, but may be influenced by the medical treatment team (interrelation between outcome and process/structure measures). To achieve a high strength of evidence, the frequent “intuitive evidence” from clinical experience must be substantiated with data. This is done for example with studies that show that single processes optimize the clinical outcome reflected by the quality indicator. In such a way, identified processes must have relevant effects on the indicator. For instance, prophylactic measures, which can only be applied reasonably to some of the patients represented in the quality indicator, are to be assessed in respect to the effect on the overall outcome.

An example is the indicator “postoperative wound infections”: the evidence basis of this indicator is determined by proving the avoidance of wound infections through appropriate perioperative management. The latter could, for example, be antibiotic prophylaxis, if its effectiveness for the patients described in the quality indicator has been proven in studies.

For outcome indicators, in addition to proof of an interrelationship between outcome and process/structure measures, the definition of the appropriate time for the outcome measurement is required. This time point affects the criterion availability of data. For example, if wound infections are only systematically captured in the hospital setting, the data for the quality indicator “wound infections” are only available if the hospitalization of the operated is longer than the time it would take for a wound infection to occur.

BQS has evaluated quality indicators with “low indicator evidence” as methodologically unsatisfactory and therefore rejected them. However, depending on the objective of the assessment a higher indicator evidence can be demanded.

Definition
Indicator evidence examines whether there is evidence for the quality indicator whose optimal expression indicates better clinical care. Therefore, scientific evidence from literature and guidelines is taken into consideration as well as the practical clinical experience in systematic consensus procedures. The definition depends upon whether one is dealing with a structure, process or outcome indicator.

With structure indicators:

Scientific evidence that in the presence of the measured structure an improved outcome occurs (interrelationship between structure and outcome).

With process indicators:

Scientific evidence that in the presence of the measured processes an improved outcome occurs (interrelationship between process and outcome).

Indicators for indication setting (as a special case of process indicators):

Fulfilling the measured indication criteria leads to a positive risk-benefit relationship (interrelation between indication setting and outcome).

With outcome indicators:

1. Scientific evidence that the measured outcome can be influenced by the provider (interrelationship between outcome and process/structure measures)
2. Definition of the appropriate time point for measuring the outcome: scientific evidence that at the measured time point of the outcomes, substantial statements are possible.

Core Statement

With structure indicators: the presence of the measured structure leads to an improved outcome.

With indicators for indication setting: fulfilling the measured indication criteria leads to a positive risk-benefit-relationship.

With process indicators: the presence of the measured processes leads to an improved outcome.

With outcome indicators: the measured outcome can be influenced by the provider.

Information Base for the Assessment

The information base for the assessment of indicator evidence are national and international methodologically high-quality guidelines, national expert standards (nursing care), and clinical expertise. If there are no guidelines/national expert standards published that addresses the care aspect reflected in the quality indicator, studies from the literature (primary literature) for the assessment of the indicator evidence is taken into account (see assessment process).

Assessment Process

The assessment process of the criterion indicator evidence is basically different from the assessment of the other criteria in QUALIFY: It is determined by an algorithm which takes into consideration the scientific evidence from guidelines and the primary literature. If studies of higher evidence are not feasible for ethical or practical reasons, consensus grades of experts will also be included in the assessment. This expert consensus is preferably obtained from guidelines. If it is not available, the core statements have to be evaluated additionally through the assessment committee in a two stage Delphi procedure (Appendix 3). In the area of nursing care national expert standards are considered equal to the medical guidelines in the assessment process.

The assessment takes place in two steps: first, the level of evidence is determined, and then the level of evidence is assigned a grade for the indicator evidence.

Step 1: Determination of the Level of Evidence (Evidence Grade)

In this step, a level of evidence derived from scientific evidence is assigned to every quality indicator. The search for guidelines and studies follows the algorithm in Figure 1. An important characteristic of the assessment of the level of evidence is that it was principally, for reasons of efficiency, based on the knowledge from guidelines and national expert standards. In the ideal case, it is possible to find a statement which exactly reflects the contents of the quality indicator and which has an explicitly assigned level of evidence. This level of evidence is then transferred to the quality indicator. The strength of evidence is classified according to AWMF and AZQ (2001, Table 2). If other classifications of the strength of evidence were used in guidelines or national nursing expert standards, a transfer into those of the AWMF and AZQ is done.

*AWMF: Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften (Association of the Scientific Medical Societies) in Germany, http://www.awmf.org)

Table 2: Classification: Level of evidence of therapeutic studies in levels of evidence according to AZQ (AWMF and AZQ, 2001)

If methodologically high quality national or international guidelines or national nursing expert standards are absent or insufficient, or contradictory information regarding the quality indicator of concern is found, or references to the level of evidence are missing, a new literature search and assessment is done. In this case, the question for a search should be formulated according to the commonly used evidence-based medicine (EbM) PICO scheme. (http://www.cebm.net/focus_quest.asp). In order to find the relevant scientific publications necessary for the assessment of the quality indicator, the following should be made concrete:

• P – which Patient group is relevant for the assessment of the quality indicator
• I – which Intervention should be considered
• C – which intervention can serve as a Comparison group (Comparison intervention)
• O –  which concrete Outcomes should be considered

The exact procedure of the search (databases and used search terms in the databases) and their results are depicted transparently. The strength of evidence is derived from the design of studies supporting the quality indicator. (AWMF and ÄZQ 2001, Table 2).

Step 2: Allocation of Indicator Evidence

The level of evidence determined in the first step is now transferred into an indicator evidence according to the algorithm in Figure 2:

Figure 2: Indicator evidence from evidence level and strength of consensus (K1, K4)

With quality indicators of evidence level IIa and IIb, the indicator evidence rating “high” will automatically be assigned. The assessment of indicator evidence is thereby completed.

For the evidence levels III and IV, first the question comes up as to why studies of higher evidence levels are not available. Are study designs of higher quality (analogous levels of evidence I or II) suitable for the underlying research question and are they feasible? The fact that a lower level of evidence exists must be accepted until clarifying study results are available. This results in the following concrete assessment:

• With quality indicators of evidence level III whose underlying statements make a study design with higher levels of evidence practically possible or feasible, the indicator evidence “medium” is automatically given. The assessment of the indicator evidence is thereby completed.
• With quality indicators of the evidence level IV, whose underlying statements make study designs with higher levels of evidence practically possible or feasible; the indicator evidence “low” is automatically given.
• With quality indicators of evidence level III and IV, where study designs with higher levels of evidence for the underlying research question are not possible for example for ethical reasons (e.g. intraoperative specimen X-ray after preoperative wire marking in breast tumors), the determination of evidence has to be supplemented with consensus expert opinions: In this case, this is the best available evidence. As a result, the evidence derived from the literature can be strengthened as well as weakened. Even in cases of assessment by consensus, one will resort to a possibly already existing broad consensus within the medical and nursing communities, documented in S-2 or S-3 guidelines in high quality international guidelines (e.g. assessment by DELBI, a German guideline assessment instrument) or in national nursing expert standards.

Only if in guidelines or nursing expert standards regarding the underlying statement of the respective quality indicator, a consensus has not yet been obtained, the degree of consensus is newly determined according to a two staged modified Delphi procedure. The result of the Delphi rounds is a certain strength of consensus (K1 to K4) for the indicator under assessment.

Therefore, the respective indicator evidence results from the combination of the external level of evidence (see step 1) with the strength of consensus. The determination takes place according to the algorithm described in Figure 2.

Assessment Stages
Very high indicator evidence
High indicator evidence
Medium indicator evidence
Low indicator evidence