User-Centered Development and Evaluation of an Energy Information System: Incremental System Instantiations using Design Science and Multi-Method Evaluation

Research output: Contribution to journal › Research article › Contributed › peer-review

Abstract

Smart meter rollout activities worldwide generate high-resolution energy consumption data that has so far been underutilized to support private households in measuring their individual energy consumption and improving efficiency. This paper presents the development and evaluation of a user-centered Energy Information System (EIS) for private customers, following the Design Science Research (DSR) paradigm. Based on 119 user-identified requirements, 5 Design Principles, and 17 Design Features, a web probing study (n= 35) validated individual UI elements in a first formative step. Afterwards, three incremental system variants were built and evaluated in a controlled between-subjects experiment (n = 46). Task-based metrics (Task Success Rate, Efficiency), the Technology Acceptance Model (TAM), and the System Usability Scale (SUS) were applied. Looking at the results, the objective performance metrics (TSR and Efficiency) show significant differences between the instantiation variants. Interestingly, TAM and SUS do not replicate these subjectively, their scores are clustered close together across the instantiations. All variants achieve acceptable usability scores, and Actual Use shows a large effect favoring the feature-richer variants (d= 0.87), though with below-threshold internal consistency. This two-stage approach is methodologically relevant and enables the precise identification of potential feature-specific understanding and acceptance barriers, before they are integrated into a full system prototype. The study highlights the need for multi-method evaluation and provides recommendations for stepwise implementation, market launch, or revision of existing Energy Information Systems.

Details

Original languageEnglish
Pages (from-to)94176 - 94192
Number of pages17
JournalIEEE access
Volume14
Early online date18 Jun 2026
Publication statusPublished - 2026
Peer-reviewedYes

External IDs

Scopus 105043149446

Keywords

Research priority areas of TU Dresden

  • Information Technology and Microelectronics - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance

DFG Classification of Subject Areas according to Review Boards

  • Business Administration - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance

Subject groups, research areas, subject areas according to Destatis

  • Business Informatics (for Business Engineers) - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • Business Informatics (for Economics) - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • Business Informatics (for Computer Scientists) - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance

Sustainable Development Goals

  • SDG 8 - Decent Work and Economic Growth - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • SDG 9 - Industry, Innovation, and Infrastructure - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • SDG 7 - Affordable and Clean Energy - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance

ASJC Scopus subject areas

  • General Computer Science - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • General Materials Science - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance
  • General Engineering - Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance

Keywords

  • Between-subjects experiment, Design methodology, Energy, Feedback, Licenses, Nuclear facility regulation, Smart meters, System Usability Scale, Technology Acceptance Model, Technology acceptance model, Timing, Usability, Weighted sum model, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, task success rate, usability evaluation, user acceptance, Between-subjects experiment, decision support systems, design science research, energy conservation, energy feedback systems, energy information systems, human-computer interaction, information systems design, smart energy management, smart metering, system usability scale, task success rate, technology acceptance model, usability evaluation, user acceptance