Performance and Sustainability of Edge AI Accelerator Technologies: In-Memory AIPU Versus TPU, NPU, and GPU SoC

Publikation: Beitrag in Fachzeitschrift › Forschungsartikel › Beigetragen › Begutachtung

Abstract

Graphics Processing Units (GPUs), which have typically high power consumption, are widely used for Artificial Intelligence (AI) inference. Different edge AI accelerator technologies and architectures have recently emerged for AI inference workloads in resource-constrained edge computing. We benchmark four edge AI accelerator technologies, which are represented by corresponding commercial accelerators: Tensor Processing Unit (TPU) represented by ASUS Coral Edge TPU, Neural Processing Unit (NPU) represented by Hailo-8, in-memory AI Processing Unit (AIPU) represented by Axelera AI Metis M.2, and GPU System-on-Chip (GPU SoC) represented by NVIDIA Jetson Orin Nano Super. For common You Only Look Once (YOLO) models applied to a brain tumor data set, we evaluate inference performance, i.e., throughput and detection accuracy, as well as sustainability, i.e., power consumption, energy efficiency, and CO2 emissions. For a pipelined form of single-stream real-time inference, we find that compared to common general-purpose NVIDIA GPUs, the edge AI accelerators have the potential to substantially increase the energy efficiency (inference throughput per Watt of power consumption) and greatly reduce the CO2 emissions. We also find that the in-memory AIPU technology (represented by Axelera AI Metis M.2) achieves a favorable balance between high inference throughput and high detection accuracy, rivaling general-purpose NVIDIA GPUs, while vastly reducing the power consumption and CO2 emissions. The GPU SoC technology (NVIDIA Jetson Orin Nano Super) also achieves very low power consumption and CO2 emissions, as well as high detection accuracy, albeit at a lower inference throughput than in-memory AIPU.

Details

OriginalspracheEnglisch
Seiten (von - bis)145440-145460
Seitenumfang21
FachzeitschriftIEEE access
Jahrgang14
PublikationsstatusVeröffentlicht - Sept. 2026
Peer-Review-StatusJa

Externe IDs

ORCID /0000-0001-8469-9573/work/229890365

Schlagworte

Ziele für nachhaltige Entwicklung

  • SDG 7 – Erschwingliche und saubere Energie - Accelerator architecture, AI accelerator, benchmarking, carbon footprint, edge AI, edge computing, energy efficiency, GPU, green AI, in-memory computing, You Only Look Once (YOLO)

ASJC Scopus Sachgebiete

  • Allgemeine Computerwissenschaft - Accelerator architecture, AI accelerator, benchmarking, carbon footprint, edge AI, edge computing, energy efficiency, GPU, green AI, in-memory computing, You Only Look Once (YOLO)
  • Allgemeine Materialwissenschaften - Accelerator architecture, AI accelerator, benchmarking, carbon footprint, edge AI, edge computing, energy efficiency, GPU, green AI, in-memory computing, You Only Look Once (YOLO)
  • Allgemeiner Maschinenbau - Accelerator architecture, AI accelerator, benchmarking, carbon footprint, edge AI, edge computing, energy efficiency, GPU, green AI, in-memory computing, You Only Look Once (YOLO)

Schlagwörter

  • Accelerator architecture, AI accelerator, benchmarking, carbon footprint, edge AI, edge computing, energy efficiency, GPU, green AI, in-memory computing, You Only Look Once (YOLO)