Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection

Research output: Contribution to journalResearch articleContributedpeer-review


  • Dorien De Meyer - , Ghent University (Author)
  • Jan Kottner - , Ghent University, Charité – Universitätsmedizin Berlin (Author)
  • Hilde Beele - , Ghent University (Author)
  • Jochen Schmitt - , Center for Evidence-Based Healthcare (Author)
  • Toni Lange - , Center for Evidence-Based Healthcare (Author)
  • Ann Van Hecke - , Ghent University (Author)
  • Sofie Verhaeghe - , Ghent University, VIVES University of Applied Sciences (Author)
  • Dimitri Beeckman - , Ghent University, Royal College of Surgeons in Ireland, Örebro University (Author)


Objective: The objective of this study was to compare two different rating scales within one Delphi study for defining consensus in core outcome set development and to explore the influence of consensus criteria on the outcome selection. Study Design and Setting: Randomized controlled parallel group trial with 1:1 allocation within the first Delphi round of the Core Outcome Set in the Incontinence-Associated Dermatitis project. Outcomes were rated on a three-point or nine-point Likert scale. Decisions about which outcomes to retain were determined by commonly used consensus criteria (i.e., [combinations of] proportions with restricted ranges, central tendency within a specific range, and decrease in variance). Results: Fifty-seven participants (group 1 = 28, group 2 = 29) rated 58 outcomes. The use of the nine-point scale resulted in almost twice as many outcomes being rated as “critical” compared with the three-point scale (24 vs. 13). Stricter criteria and combining criteria led to less outcomes being identified as “critical”. Conclusion: The format of rating scales in Delphi studies for core outcome set development and the definition of the consensus criteria influence outcome selection. The use of the nine-point scale might be recommended to inform the consensus process for a subsequent rating or face-to-face meeting. The three-point scale might be preferred when determining final consensus.


Original languageEnglish
Pages (from-to)23-31
Number of pages9
JournalJournal of clinical epidemiology
Publication statusPublished - Jul 2019

External IDs

PubMed 30922885
ORCID /0000-0002-8671-7496/work/152545154


ASJC Scopus subject areas


  • Consensus, Core outcome set, Criteria, Delphi-procedure, Dermatology, Incontinence-associated dermatitis