Inhibitory control represents a central component of executive functions and focuses on the ability to actively inhibit or delay a dominant response to achieve a goal. Although various tasks exist to measure inhibitory control, correlations between these tasks are rather small, partly because of the task impurity problem. To alleviate this problem, a latent variable approach has been previously applied and two closely related yet separable functions have been identified: prepotent response inhibition and resistance to distractor interference. The goal of our study was a) to replicate the proposed structure of inhibitory control and b) to extend previous literature by additionally accounting for speed-accuracy trade-offs, thereby potentially increasing explained variance in the investigated latent factors. To this end, 190 participants completed six inhibitory control tasks (antisaccade task, Stroop task, stop-signal task, flanker task, shape-matching task, word-naming task). Analyses were conducted using standard scores as well as inverse efficiency scores (combining response times and error rates). In line with previous studies, we generally found low zero-order correlations between the six tasks. By applying confirmatory factor analysis using standard reaction time difference scores, we were not able to replicate a satisfactory model with good fit to the data. By using inverse efficiency scores, a two-related-factor and a one-factor model emerged that resembled previous literature, but only four out of six tasks demonstrated significant factor loadings. Our results highlight the difficulty in finding robust inter-correlations between commonly used inhibitory control tasks, even when applying a latent variable analysis and accounting for speed-accuracy trade-offs.