In sheet metal forming, the quality of a formed part is strongly influenced by the local lubrication conditions on the blank. Fluctuations in lubrication distribution can cause failures such as excessive thinning and cracks. Predicting these failures in real-time for the entire part is still a very challenging task. Machine learning (ML) based digital twins and advanced computing power offer new ways to analyze manufacturing processes inline in the shortest possible time. This study presents a digital twin for simulating a deep drawing process that incorporates an advanced ML model and optimization algorithm. Convolutional neural networks with RES-SE-U-Net architecture, were used to capture the full friction conditions on the blank. The ML model was trained with data from a calibrated finite element model. The ML model establishes a correlation between the local friction conditions across the blank and the quality of the drawn part. It accurately predicts the geometry and thinning of the formed part in real-time by assessing the friction conditions on the blank. A particle swarm optimization algorithm incorporates the ML model and provides tailored recommendations for adjusting local friction conditions to promptly correct detected quality deviations with minimal amount of additional lubricant. Experiments show that the ML model deployed on an industrial control system can predict part quality in real-time and recommend adjustments in case of quality deviation in 1.6 s. The error between prediction and ground truth is on average 0.16 mm for geometric accuracy and 0.02 % for thinning.