Note onset detection – the detection of the beginning of new note events – is a fundamental task for music analysis that can help to improve Automatic Music Transcription (AMT). The method for onset detection always follows a similar outline: An audio signal is transformed into an Onset Detection Function (ODF), which should have rather low values (i.e. close to zero) for most of the time, and pronounced peaks at onset times, which can then be extracted by applying peak picking algorithms on the ODF. Currently, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) define the state of the art. In this paper, we build upon previous work about onset detection using Echo State Networks (ESNs) that have achieved comparable results to CNNs. We show that unsupervised pre-training of the ESN leads to similar results whilst reducing the model complexity.