Cluster-Based Input Weight Initialization for Echo State Networks

Research output: Contribution to journalResearch articleContributedpeer-review

Contributors

Abstract

Echo state networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image, and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the K-means algorithm on the training data. We show that for a large variety of datasets, this initialization performs equivalently or superior than a randomly initialized ESN while needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.

Details

Original languageEnglish
Number of pages12
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume2022
Issue number34(10)
Publication statusPublished - 2022
Peer-reviewedYes

External IDs

Scopus 85124228195
ORCID /0000-0003-0167-8123/work/167214847

Keywords

Keywords

  • Clustering, Clustering algorithms, Mathematical models, Neurons, Reservoirs, Self-organizing feature maps, Task analysis, Training, echo state networks (ESNs), reservoir computing, unsupervised pretraining.

Library keywords