Gamified Expert Annotation Systems: Meta-Requirements and Tentative Design
Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review
Contributors
Abstract
Poorly annotated data is a common problem for data-intensive applications like supervised machine learning. In domains like healthcare, annotation tasks require specific domain knowledge and are thus often done manually by experts, which is error-prone, time-intensive, and tedious. In this study, we investigate gamification as a means to foster annotation quality through annotators’ increased motivation and engagement. To this end, we conducted a literature review of 70 studies as well as a series of 16 workshops with a team of six experts in medical image annotation. We derive a set of seven meta-requirements (MRs) that represent the desired instrumental and experiential outcomes of gamified expert annotation systems (e.g., high-quality annotations, a sense of challenge) as well as a tentative design that can address the derived MRs. Our results help to understand the inner workings of gamification in the context of expert annotation and lay important groundwork for designing gamified expert annotation systems that can successfully motivate annotators and increase annotation quality.
Details
| Original language | English |
|---|---|
| Title of host publication | The Transdisciplinary Reach of Design Science Research - 17th International Conference on Design Science Research in Information Systems and Technology, DESRIST 2022, Proceedings |
| Editors | Andreas Drechsler, Aurona Gerber, Alan Hevner |
| Publisher | Springer Science and Business Media B.V. |
| Pages | 154-166 |
| Number of pages | 13 |
| ISBN (print) | 9783031065156 |
| Publication status | Published - 2022 |
| Peer-reviewed | Yes |
| Externally published | Yes |
Publication series
| Series | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 13229 LNCS |
| ISSN | 0302-9743 |
Conference
| Title | 17th International Conference on Design Science Research in Information Systems and Technology |
|---|---|
| Subtitle | The Transdisciplinary Reach of Design Science Research |
| Abbreviated title | DESRIST 2022 |
| Conference number | 17 |
| Duration | 1 - 3 June 2022 |
| Website | |
| Degree of recognition | International event |
| Location | University of South Florida |
| City | St. Petersburg |
| Country | United States of America |
Keywords
ASJC Scopus subject areas
Keywords
- Annotation, Annotation quality, Expert annotation, Gamification