Gamified Expert Annotation Systems: Meta-Requirements and Tentative Design

Research output: Contribution to book/Conference proceedings/Anthology/ReportConference contributionContributedpeer-review

Contributors

  • Simon Warsinsky - , Karlsruhe Institute of Technology (Author)
  • Manuel Schmidt-Kraepelin - , Karlsruhe Institute of Technology (Author)
  • Scott Thiebes - , Karlsruhe Institute of Technology (Author)
  • Martin Wagner - , University Hospital Heidelberg (Author)
  • Ali Sunyaev - , Karlsruhe Institute of Technology (Author)

Abstract

Poorly annotated data is a common problem for data-intensive applications like supervised machine learning. In domains like healthcare, annotation tasks require specific domain knowledge and are thus often done manually by experts, which is error-prone, time-intensive, and tedious. In this study, we investigate gamification as a means to foster annotation quality through annotators’ increased motivation and engagement. To this end, we conducted a literature review of 70 studies as well as a series of 16 workshops with a team of six experts in medical image annotation. We derive a set of seven meta-requirements (MRs) that represent the desired instrumental and experiential outcomes of gamified expert annotation systems (e.g., high-quality annotations, a sense of challenge) as well as a tentative design that can address the derived MRs. Our results help to understand the inner workings of gamification in the context of expert annotation and lay important groundwork for designing gamified expert annotation systems that can successfully motivate annotators and increase annotation quality.

Details

Original languageEnglish
Title of host publicationThe Transdisciplinary Reach of Design Science Research - 17th International Conference on Design Science Research in Information Systems and Technology, DESRIST 2022, Proceedings
EditorsAndreas Drechsler, Aurona Gerber, Alan Hevner
PublisherSpringer Science and Business Media B.V.
Pages154-166
Number of pages13
ISBN (print)9783031065156
Publication statusPublished - 2022
Peer-reviewedYes
Externally publishedYes

Publication series

SeriesLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13229 LNCS
ISSN0302-9743

Conference

Title17th International Conference on Design Science Research in Information Systems and Technology
SubtitleThe Transdisciplinary Reach of Design Science Research
Abbreviated titleDESRIST 2022
Conference number17
Duration1 - 3 June 2022
Website
Degree of recognitionInternational event
LocationUniversity of South Florida
CitySt. Petersburg
CountryUnited States of America

Keywords

Keywords

  • Annotation, Annotation quality, Expert annotation, Gamification