The enormous amount of data generated nowadays worldwide is increasingly triggering the search for unconventional and more efficient ways of processing and classifying information, eventually able to transcend the conventional von Neumann-Turing computational central dogma. It is, therefore, greatly appealing to draw inspiration from less conventional but computationally more powerful systems such as the neural architecture of the human brain. This neuromorphic route has the potential to become one of the most influential and long-lasting paradigms in the field of unconventional computing. Memristive and the recently proposed memfractive systems have been shown to display basic features of neural systems such as synaptic-like plasticity and memory features, so that they may offer a diverse playground to implement synaptic connections. In this review, we address various material-based strategies of implementing unconventional computing hardware: (i) electrochemical oscillators based on liquid metals and (ii) mem-devices exploiting Schottky barrier modulation in polycrystalline and disordered structures made of oxide or perovskite-type semiconductors. Both items (i) and (ii) build the two pillars of neuromimetic computing devices, which we will denote as synthetic neural networks. We expect that the current review will be of great interest for scientists aiming at bridging unconventional computing strategies with specific materials-based platforms.