Imaging photoplethysmography (iPPG) is a camera-based approach for the remote measurement of superficial tissue perfusion most commonly applied to facial video recordings. Since only tissue contains information about perfusion, skin detection is a necessary processing step. Several approaches for the detection of skin pixels in video recordings have been developed, e.g. using color thresholds. Within this work we present a deep learning based approach capable of combining color and morphology information, which makes the skin detection robust against different illumination conditions. We evaluated our new approach using two datasets with 182 individuals of different gender, age, skin tone and illumination conditions. Our approach outperformed state-of-the-art algorithms or yielded at least comparable results (mean absolute error of estimated pulse rate improved by up to 68 %). The method presented allows more accurate assessment of superficial tissue perfusion with iPPG.