In applied machine learning, model selection often requires compromises between predictive performance and interpretability, especially in application domains where trust, accountability or insight generation are essential. While numerous interpretability methods exist, interpretability is often considered user- and context-dependent and can neither be assumed to be directly measurable nor objectively comparable across heterogeneous models. Existing approaches often assess models in isolation or rely on proxy criteria, providing limited support for comparative reasoning and model selection. This paper presents an interactive, human-centered framework for the comparative assessment of model interpretability, given a portfolio of machine learning models. Rather than prescribing a universal interpretability metric, the framework elicits user-specific interpretability preferences through pairwise model comparisons based on the available interpretability artifacts. These pairwise judgments are aggregated using a preference learning approach to derive user-specific interpretability preference scores. The preference scores are combined with predictive performance estimates in a visual decision support interface that facilitates the exploration of performance-interpretability trade-offs and the identification of Pareto-optimal models. The framework is implemented as an interactive dashboard for (spatio-)temporal regression tasks. Using an exemplary case study on wind power forecasting, we illustrate how the approach supports comparative model analysis and underscores the subjective nature of interpretability assessment.