Deep recurrent networks can be computed as an unrolled computation graph in a defined time window. In theory, the unrolled network and a continuous time recurrent computation are equivalent. However, we encountered a shift in accuracy for models based on LSTM-/GRU- and SNN-cells during the switch from unrolled computation during training towards a continuous stateful inference without state resets. In this work, we evaluate these time recurrent neural network approaches based on the error created by using a time continuous inference. This error would be small in case of good time domain generalization and we can show that some training setups are favourable for that with the chosen example use case. A real time critical motor position prediction use case is chosen as a reference. This task can be phrased as a time series regression problem. A time continuous stateful inference for time recurrent neural networks benefits an embedded systems by reduced need of compute resources.