OmniOpt – A tool for hyperparameter optimization on HPC

Research output: Contribution to book/conference proceedings/anthology/reportConference contributionContributedpeer-review

Abstract

Hyperparameter optimization is a crucial task in numerous applications of numerical modelling techniques. Methods as diverse as classical simulations and the great variety of machine learning techniques used nowadays, require an appropriate choice of their hyperparameters (HPs). While for classical simulations, calibration to measured data by numerical optimization techniques has a long tradition, the HPs of neural networks are often chosen by a mixture of grid search, random search and manual tuning. In the present study the expert tool “OmniOpt” is introduced, which allows to optimize the HPs of a wide range of problems, ranging from classical simulations to different kinds of neural networks. Thereby, the emphasis is on versatility and flexibility for the user in terms of the applications and the choice of its HPs to be optimized. Moreover, the optimization procedure – which is usually a very time-consuming task – should be performed in a highly parallel way on the HPC system Taurus at TU Dresden. To this end, a Bayesian stochastic optimization algorithm (TPE) has been implemented on the Taurus system and connected to a user-friendly graphical user interface (GUI). In addition to the automatic optimization service, there is a variety of tools for analyzing and graphically displaying the results of the optimization. The application of OmniOpt to a practical problem from material science is presented as an example.

Details

Original languageEnglish
Title of host publicationHigh Performance Computing - ISC High Performance Digital 2021 International Workshops, 2021, Revised Selected Papers
EditorsHeike Jagode, Hartwig Anzt, Hatem Ltaief, Piotr Luszczek
PublisherSpringer, Berlin [u. a.]
Pages285-296
Number of pages12
ISBN (electronic)978-3-030-90539-2
ISBN (print)978-3-030-90538-5
Publication statusPublished - 13 Nov 2021
Peer-reviewedYes

Publication series

SeriesLecture Notes in Computer Science, Volume 12761
ISSN0302-9743

Conference

TitleInternational Conference on High Performance Computing, ISC High Performance 2021
Duration24 June - 2 July 2021
CityVirtual, Online

External IDs

ORCID /0000-0002-0169-8602/work/142242244
ORCID /0000-0003-2653-7546/work/142249311

Keywords

Keywords

  • High performance computing, Hyperparameter optimization, Neural networks

Library keywords