The demand for computer vision systems and algorithms is steadily increasing. However, users often have to deal with different or new languages, architectures and tools. Furthermore, there is often no linkage between vendors, defined standards, or model-based modularization to connect everything. We propose a modularized framework for distributing applications on heterogeneous systems consisting of CPUs, GPUs, and FPGAs. The user builds an OpenVX-compliant application without knowledge of the underlying hardware. The middleend automatically schedules and maps the nodes to the available OpenCL devices. Benefits of FPGA acceleration, such as pipelining and running multiple nodes in parallel, are taken into account. The backend generates a program including memory management, synchronization mechanisms and data transfers, even between vendors. This is executed in our parallelized OpenCL based runtime system with minimal overhead. We achieved speedups of 1.63 for a heterogeneous schedule in comparison to a single GPU design when limiting the FPGA resources. Without this limitation a speedup of 13.39 is achieved for the same application.