Identification of different cell types is an indispensable part in biomedical research and clinical application. During the last decades, much attention was put onto molecular characterization and many cell types can now be identified and sorted based on established markers. The required staining process is a lengthy and costly treatment, which can cause alterations of cellular properties, contaminate the sample and therefore limit its subsequent use. A promising alternative to molecular markers is the label-free identification of cells using mechanical or morphological features. We introduce a microfluidic device for active label-free sorting of cells based on their bright field image supported by innovative real-time image processing and deep neural networks (DNNs). A microfluidic chip features a standing surface acoustic wave generator for actively pushing up to 100 cells/sec to a determined outlet for collection. This novel method is successfully applied for enrichment of lymphocytes, granulo-monocytes and red blood cells from human blood. Furthermore, we combined the setup with lasers and a fluorescence detection unit, allowing to assign a fluorescence signal to each captured bright-field image. Leveraging this tool and common molecular staining, we created a labelled dataset containing thousands of images of different blood cells. We used this dataset to train a DNN with optimized latency below 1 ms and used it to sort unstained neutrophils from human blood, resulting in a target concentration of 90%. The innovative approach to use deep learning for image-based sorting opens up a wide field of potential applications, for example label-free enrichment of stem-cells for transplantation.