|
|
ReliabilityAn important prerequisite for reliability is clear and unambiguous definitions. The two criteria clarity of definitions and reliability thus show interactions. Definition Two procedures are usually distinguished:
Core Statement Information Base for the Assessment For the methodological assessment of quality indicators, two measurements per measured object are not always available. As a result, the reliability of the quality indicators may not be determined with the known methods. Therefore, an additional procedure was developed for QUALIFY which allows an estimation of the reliability on the basis of available data without multiple measurements. The determination of the reliability of the quality indicators with this method is done through an analysis of the variability of the outcomes of individual hospitals between consecutive time intervals. The indicator is evaluated as “reliably measurable” when its expression in consecutive quarters does not show statistically significant differences (please note: the reverse conclusion does not apply because a change in the expression can also be due to a change in the quality characteristic measured). Specifically, the quality indicator rates per quarter are calculated separately for each hospital over two years, if possible. Then, the differences between the rates in subsequent quarters are tested for significance for each hospital. Afterwards, 75% confidence intervals are calculated and tested for overlapping. This corresponds in each case to an alpha error level of 25% which is selected in this height because in this case it is about confirming the null hypothesis (read here: there are no significant differences between the samples under comparison). If for one quality indicator, for example data from 1,000 hospitals from eight quarters are available, then based on this method a total of 7,000 results of tests of significance can be calculated. The basic idea of this approach is that the fewer quarter comparisons that turn out to be significant, the more likely that this can be seen as an indication of the reliability of the quality indicator under consideration. Assessment Process If in the data comparison of the results of the quality indicators, no difference or only a few differences are seen between the data of BQS documentation and the data from a second collection (in source data validation), this will be presumed to be a reliable capture of the quality indicators. However, the higher the proportion of differences is, the lower the reliability has to be assessed. BQS suggests the following gradation: Quality indicators measure reliably when the differences that can be detected are not more than 5% of comparisons. If the proportion is above 5% and up to 10% then the quality indicator measures “rather reliably”. If the proportion is above 10% and up to 20% it is considered “rather not reliable”. If the proportion of differences exceeds 20% then it is a quality indicator that is “not reliable”. In the context of the substitute procedure the assessment process takes place as follows: On the basis of the results of the quarter comparisons, the following assessment stages occur: If the proportion of significant quarter comparisons is above 10% then the quality indicator is “not reliable”. If the proportion is between 5% and 10% the quality indicator can be considered as “rather reliable”. A quality indicator on the contrary will be considered “reliable” if 5% or less of all quarter comparisons is significant. For each individual quality indicator this reliability assessment was submitted to the evaluators as a suggestion for use. In addition, for this assessment procedure the data fields and their definitions which are necessary for the measurement of the quality indicators were listed. It is pointed out explicitly that with this proceeding, the expertise of the experts should be especially considered.
Comments |