From Microbenchmarks to LLM Inference an End-To-End Analysis on the Energy Efficiency of the Grace Hopper Superchip

Research output: Contribution to book/Conference proceedings/Anthology/ReportConference contributionContributedpeer-review

Abstract

The Nvidia Grace Hopper Superchip has seen a wide adoption across both HPC sites and AI data centers. This hybrid architecture places the Grace CPU and Hopper GPU on the same board, with a shared and adjustable power budget. We present an end-to-end analysis of its energy efficiency, from a verification of power sensor accuracy using microbenchmarks, an investigation of power knob effects to an energy-efficiency and TCO investigation for selected Large Language Model (LLM) inference use cases. Our study shows that the internal power sensors provide precise readings when compared to an external power meter, unlike contemporary x86 processors. Some power budget configurations are not enforced by the system, the power draw can exceed the configured limit. This behavior is not documented by Nvidia. We analyzed the energy efficiency of LLM inference for prefill and decoding-dominated workloads, with the latter having emerged as the driving factor. Increasing the module power limit while maintaining a constant GPU power did not affect the system's energy efficiency. According to our TCO analysis, no cost savings can be expected from reduced power limits for output lengths of up to 2048 tokens for our LLM setup.

Details

Original languageEnglish
Title of host publicationProceedings - 2026 25th International Symposium on Parallel and Distributed Computing, ISPDC 2026
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages121-129
Number of pages9
ISBN (electronic)979-8-3195-3432-3
ISBN (print)979-8-3195-3433-0
Publication statusPublished - 23 Jul 2026
Peer-reviewedYes

Publication series

SeriesInternational Symposium on Parallel and Distributed Computing

Conference

Title25th International Symposium on Parallel and Distributed Computing
Abbreviated titleISPDC 2026
Conference number25
Duration1 - 3 July 2026
Website
LocationDeutsches Elektronen-Synchrotron (DESY)
CityHamburg
CountryGermany

External IDs

ORCID /0000-0002-8491-770X/work/224856192
ORCID /0000-0002-2730-0308/work/224857327

Keywords

Sustainable Development Goals

Keywords

  • benchmarking, Energy Efficiency, Nvidia Grace Hopper