From Microbenchmarks to LLM Inference an End-To-End Analysis on the Energy Efficiency of the Grace Hopper Superchip

Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/Gutachten › Beitrag in Konferenzband › Beigetragen › Begutachtung

Abstract

The Nvidia Grace Hopper Superchip has seen a wide adoption across both HPC sites and AI data centers. This hybrid architecture places the Grace CPU and Hopper GPU on the same board, with a shared and adjustable power budget. We present an end-to-end analysis of its energy efficiency, from a verification of power sensor accuracy using microbenchmarks, an investigation of power knob effects to an energy-efficiency and TCO investigation for selected Large Language Model (LLM) inference use cases. Our study shows that the internal power sensors provide precise readings when compared to an external power meter, unlike contemporary x86 processors. Some power budget configurations are not enforced by the system, the power draw can exceed the configured limit. This behavior is not documented by Nvidia. We analyzed the energy efficiency of LLM inference for prefill and decoding-dominated workloads, with the latter having emerged as the driving factor. Increasing the module power limit while maintaining a constant GPU power did not affect the system's energy efficiency. According to our TCO analysis, no cost savings can be expected from reduced power limits for output lengths of up to 2048 tokens for our LLM setup.

Details

OriginalspracheEnglisch
TitelProceedings - 2026 25th International Symposium on Parallel and Distributed Computing, ISPDC 2026
Herausgeber (Verlag)Institute of Electrical and Electronics Engineers (IEEE)
Seiten121-129
Seitenumfang9
ISBN (elektronisch)979-8-3195-3432-3
ISBN (Print)979-8-3195-3433-0
PublikationsstatusVeröffentlicht - 23 Juli 2026
Peer-Review-StatusJa

Publikationsreihe

ReiheInternational Symposium on Parallel and Distributed Computing

Konferenz

Titel25th International Symposium on Parallel and Distributed Computing
KurztitelISPDC 2026
Veranstaltungsnummer25
Dauer1 - 3 Juli 2026
Webseite
OrtDeutsches Elektronen-Synchrotron (DESY)
StadtHamburg
LandDeutschland

Externe IDs

ORCID /0000-0002-8491-770X/work/224856192
ORCID /0000-0002-2730-0308/work/224857327

Schlagworte

Ziele für nachhaltige Entwicklung

ASJC Scopus Sachgebiete

Schlagwörter

  • benchmarking, Energy Efficiency, Nvidia Grace Hopper