Neural networks enable the solution of various complex problems, by building convoluted structures from simple building blocks. In the past decade that more and more complex neural networks were introduced and resulted higher accuracies on commonly investigated benchmark datasets. As this trend clearly demonstrates, the complexity of networks is typically improved by increasing the number of neurons and layers in their architecture, but higher complexity can also be achieved by enriching the dynamics of the cells.In this work we demonstrate that a simple memristor cellular neural network containing two cells is able to solve the XOR problem, which is not feasible for traditional neural networks with only two cells. We train the parameters of this dynamical system employing modern machine learning methods such as gradient descent optimization. Our case study demonstrates how the employment of complex circuit dynamics can extend the range of solvable problems with a given number of neurons.