Resilient gossip algorithms for collecting online management information in exascale clusters

Research output: Contribution to journalResearch articleContributedpeer-review

Contributors

Abstract

Management of forthcoming exascale clusters requires frequent collection of run-time information about the nodes and the running applications. This paper presents a new paradigm for providing online information to the management system of scalable clusters, consisting of a large number of nodes and one or more masters that manage these nodes. We describe the details of resilient gossip algorithms for sharing local information within subsets of nodes and for sending global information to a master, which holds information on all the nodes. The presented algorithms are decentralized, scalable and resilient, working well even when some nodes fail, without needing any recovery protocol. The paper gives formal expressions for approximating the average ages of the local information at each node and the information collected by the master. It then shows that these results closely match the results of simulations and measurements on a real cluster. The paper also investigates the resilience of the algorithms and the impact on the average age when nodes or masters fail. The main outcome of this paper is that partitioning of large clusters can improve the quality of information available to the management system without increasing the number of messages per node. Copyright © 2015 John Wiley & Sons, Ltd.

Details

Original languageEnglish
Pages (from-to)4797-4818
Number of pages22
JournalConcurrency and Computation: Practice and Experience
Volume27
Issue number17
Publication statusPublished - 2015
Peer-reviewedYes

External IDs

Scopus 84923169613
ORCID /0000-0003-3137-0648/work/142238860

Keywords

Keywords

  • exascale clusters, resource management, gossip algorithm