The planning phases of industrial surface defect detection systems for machined, automotive workpieces are time-consuming and require project teams comprised of experts from different disciplines in order to achieve optimal systems regarding the prioritized objectives. To master this challenge current research is aiming for flexible inspection system concepts that are not application-specific and thus can be used for a range of industry-dependent specifications without the necessity to complete each planning phase from the beginning. This paper presents a thorough analysis of the influencing factors at the beginning of the inspection planning process of an industrial surface defect detection system for machined, automotive workpieces. Based on this analysis, design requirements for a modular framework for industrial surface defect detection system designs in this sector are derived and contextualized with available system components. Finally, an outlook for future research is presented with the goal to further increase the flexibility of industrial surface defect detection systems.