Appendix 3: Two-Stage Modified Delphi Procedure
The two-stage Delphi procedure generally includes two Delphi rounds. Prior to the first Delphi round, the evaluators will be provided with the findings of the original literature search including a summary of the results of the literature search.
The two-stage modified Delphi procedure relies on one questionnaire. The questionnaire contains the adjusted core statement for each of the quality indicators to be assessed.
Example for a process indicator: performing an intraoperative X-ray of a specimen from breast tumors results in a decrease of the recurrence rate.
Example for an outcome indicator: wound infections after hysterectomy can be reduced by the provider by using certain procedures (e.g. antibiotic prophylaxis).
The questionnaire allows the following answers:
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention
In addition, the possibility exists to enter a written comment (open-ended statement) and suggest additional literature.
- First Delphi round:
Data collection: The questionnaire is mailed to all evaluators by regular mail or email. A deadline is set for returning the completed questionnaires. If necessary, the evaluators are reminded to submit their vote.
Analysis: From the first Delphi questionnaire, the results including the means are shown. Furthermore, all comments from and additional literature suggested by the evaluators are compiled without mention of the author. This information is mailed without comments to all evaluators as a basis for the second and final Delphi round.
Example of a response with seven evaluators: 1, 1, 1, 2, 2, 3, one abstention. Four questionnaires have an additional comment.
Presentation of the results after the first round: 3 times 1 = “does not apply”, 2 times 2 = “rather does not apply”, 1 times 3 = “rather applies”, 0 times 4 = “applies”, one time “abstention”.
Mean: (1+1+1+2+2+3) / 6 = 1,67 (The abstention is not included in the mean).
Listing of the four comments.
- Second Delphi round:
Data collection: The data collection is done in the same way as in the first Delphi round.
Analysis: From the assessments of the core statement in the second Delphi round, the strength of consensus is (K1 to K4) is determined. Therefore, the respective indicator evidence results from the combined level of evidence and strength of consensus. The determination occurs according to the algorithm in Figure 2.
| Strength of Consensus 1 (K1) |
Consensus in the Delphi procedure, all participants |
| Strength of Consensus 2 (K2) |
Consensus in the Delphi procedure, some of the participants |
| Strength of Consensus 3 (K3) |
Informal Consensus or agreement by the majority in the Delphi procedure |
| Strength of Consensus 4 (K4) |
No agreement |
Whether a consensus, a majority agreement or no agreement is present is determined according to the following formula:
Initially the number of assessments with an agreement to the core statement (all assessments with 3 or 4) is determined. Furthermore, the valid votes are counted (assessment 1 to 4, abstentions are exempt).
Consensus is assumed if
- >= ¾ of all valid assessments are 3 = “rather applies” or 4 = “applies”
(rounding down: Example ¾ of 10=7)
and at the same time
- >= 50% of participants give a valid assessment (assessments 1 to 4)
The consensus is informal when it was achieved in a simple vote, e.g. by raising hands and not in a Delphi procedure.
Majority agreement is assumed if
- > 50% and < ¾ of all assessments are 3 = “rather applies” or 4 = “applies”
and at the same time
- >= 50% of participants give a valid assessment (assessments 1 to 4)
No agreement: <= 50% assessments are 3 = “rather applies” or 4 = “applies” or
> 50% abstentions
All participants: doctor, nurse, patient representative
Comment:
The Delphi procedure can be modified in terms of a broadband Delphi procedure. In contrast to the standard Delphi procedure, the question is first discussed openly in the group. Also, the results of the first Delphi round are discussed before the second assessment. However, as in the standard Delphi procedure, the assessment is to be conducted strictly anonymously (anonymity among the evaluators).
Possible advantages of the broadband Delphi procedure are a less-time consuming process in the second Delphi round, a higher response rate than in the email procedure and an acceleration and strengthening of consensus building through the information exchange.
A possible disadvantage of the broadband procedure is opinion building through group dynamics / pressure.
