Few-mode fibers (FMFs) are promising for advancements in transmission capacity in classical and quantum communications. However, the inherent modal crosstalk limits the practical application of FMF. One reliable way to overcome this obstacle is the measurement of the complex transmission matrix (TM), describing the light propagation behavior of fiber. The TM can be obtained by performing mode decomposition (MD) of the spatial modes at the output of the fiber. MD techniques require the retrieval of both the amplitude and phase components of the detected light field, which is commonly done by using holography. However, the provision of a reference wave is highly unfavorable for the implementation of a holography-based MD in communication technology, especially for long fibers. Using deep neural networks to process intensity-only images, this drawback can be overcome. We introduce the mode transformer network, which can perform MD on 23 modes and has been trained offline using synthetic data. Experimentally, we demonstrate, for the first time, not only the measurement of complex TM of an FMF but also the inversion of the TM using a deep learning-based MD method. For mode transmission, we achieve an average fidelity of 97%. The short duration of the determination of TM allows for overcoming time-varying effects due to, e.g., mechanical stress or temperature fluctuations. The proposed reference-less calibration is promising for fiber communication with classical light and single photons, such as at quantum key distribution.