Ability of Statistical Differentiation

A method for the calculation of the necessary minimum sample size (observations per hospital and measurement period) for the ability to discriminate was developed. This was done because operational procedures were not known in which the possibility to detect differences is in the foreground, but without the necessity of observing differences.

Regarding the method of determining the minimum sample size, the ability to discriminate will be measured and not discrimination itself. Whether in the hospitals different outcomes actually occur is of subordinate interest at this point. What matters more is the possibility to detect a qualitative difference if it is present.

The statistical discriminatory power of a quality indicator is dependent on the setting, because the number of cases from the individual hospitals and the proportion of them which meet the minimum sample size is the information base for the assessment.

An assessment of the change sensitivity was not done in this context.

Definition
The ability of statistical differentiation detects differences (variability) between hospitals (discriminatory power) or in the results of the same hospital over time (change sensitivity). The basic population underlying the quality indicator may not drop below a certain number in the hospitals (minimum sample size) in order to be able to detect statistical differences.

If the case number of a hospital falls below this minimal number (minimum sample size) a statistically significant differentiation either into “good quality” or into “poor quality” is not possible.

The quality indicators’ ability to discriminate is considered as adequate if a certain number or proportion of hospitals in the application setting reaches the minimum sample size which is necessary to determine a statistically significant difference between “good quality” and “poor quality”.

Case 1: If lower outcome rates mean “better quality” (e.g. with complication rates) and if

• pg is the limit, up to which an outcome is still considered a “good quality“ and
• pp is the limit, from which on an outcome is considered a “poor quality“,

the minimum sample size n then is calculated under the consideration of the significance level a as follows:

n = min (n ? IN: (pgn + (1-pp)n) <= a)

Case 2: If however higher outcome rates mean “better quality” (e.g. with proceedings according to guidelines) and if

• pg is the limit from which on an outcome is considered a “good quality“ and
• ps is the limit up to which an outcome is considered a “poor quality“,

the minimum sample size is calculated as follows:

n = min (n ? IN: (1-pg)n + ppn) <= a)

However, for change sensitivity, the approach of minimum sample size calculation should be different. It has to be calculated by common methods with the aim of statistically detecting relevant (= minimum) differences within a hospitals’ results (between consecutive measurement intervals).

Core Statement
The following statement is assessed: “The ability to discriminate meets the requirements of the purpose of the analysis”.

Information Base for the Assessment
The first step is the calculation of the minimum sample size, which has to be performed in a hospital in order to be able to determine a significant deviation from the defined “good quality” and the defined “poor quality”. In the application of the QUALIFY instrument for BQS procedure, the attainable “good quality” and observable “poor quality” were determined using the 5% and 95% percentiles of hospital results related to the quality indicator under consideration. As a result, the minimum sample size should allow to detect a statistical difference in the observed results of the 5% of hospitals with the best and the 5% of hospitals with the poorest results. For the assessment of the ability to discriminate, the proportion of hospitals will be determined which reach this minimal case number of procedures. This information is then made available as a basis for the assessment.

Assessment Process
In the first step, the information base will be made available and understandable to all evaluators. For a sufficient ability to discriminate, the proportion of hospitals must not be too small for which a differentiation from the 5% best hospitals or the 5% worst hospitals is not possible because of the case number. The following assessment suggestions are submitted for the statement “the ability to discriminate meets the requirements of the purpose of the assessment”:

• “applies” if the proportion of hospitals able to discriminate is >= 75%.
• “rather applies” if the proportion of hospitals able to discriminate is >= 50 to < 75%.
• “rather does not apply” if the proportion of hospitals able to discriminate is >= 25 to < 50%.
• “does not apply” if the proportion of hospitals able to discriminate is < 25%.

In order to consider a differentiated view of the relevance of the information base (depending on the quality indicator), the assessment committee conducts a final assessment of the statement.

Assessment Stages
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention