In Physical Layer Security, knowing the reciprocal state information of the legitimate terminals' wireless channel is considered a shared secret. Although questioned in recent works, the basic assumption is that an eavesdropper, residing more than half of a wavelength away from the legitimate terminals, is unable to even obtain estimates that are correlated to the state information of the legitimate channel. In this work, we present a Machine Learning based attack that does not require knowledge about the environment or terminal positions, but is solely based on the eavesdropper's measurements. It still successfully infers the legitimate channel state information as represented in impulse responses. We show the effectiveness of our attack by evaluating it on two sets of real world ultra wideband channel impulse responses, for which our attack predictions can achieve higher correlations than even the measurements at the legitimate channel.