Revealing and reducing bias when modelling choice behaviour on imbalanced panel datasets

Research output: Contribution to journalResearch articleContributedpeer-review

Contributors

  • Miroslawa Lukawska - , Technical University of Denmark (Author)
  • Laurent Cazor - (Author)
  • Mads Paulsen - (Author)
  • Thomas Kjær Rasmussen - (Author)
  • Otto Anker Nielsen - (Author)

Abstract

The emergence of modern tools and technologies gives a unique opportunity to collect large amounts of data for understanding behaviour. However, the generated datasets are often imbalanced, as individuals might contribute to the datasets at different frequencies and periods. Models based on these datasets are challenging to estimate, and the results are not straightforward to interpret without considering the sample structure. This study investigates the issue of handling imbalanced panel datasets for modelling individual behaviour. It first conducts a simulation experiment to study to which degree mixed logit models with and without panel reproduce the population preferences when using imbalanced data. It then investigates how the application of bias reduction strategies, such as subsampling and likelihood weighting, influences model results and finds that combining these techniques helps to find an optimal trade-off between bias and variance of the estimates. Considering the conclusions from the simulation study, a large-scale case study estimates bicycle route choice models with different correction strategies. These strategies are compared in terms of efficiency, weighted fit measures, and computational burden to provide recommendations that fit the modelling purpose. We find that the weighted panel mixed multinomial logit model, estimated on the entire dataset, performs best in terms of minimising the bias-efficiency trade-off in the estimates. Finally, we propose a strategy that ensures equal contribution of each individual to the estimation results, regardless of their representation in the sample, while reducing the computational burden related to estimating models on large datasets.

Details

Original languageEnglish
Article number100471
JournalJournal of Choice Modelling
Volume50
Publication statusPublished - Mar 2024
Peer-reviewedYes
Externally publishedYes

External IDs

Scopus 85183555721
Mendeley b3e6960c-efaf-3d6c-9fd7-d85b3ec7d5f8

Keywords

Keywords

  • Bias-efficiency trade-off, Imbalanced panel, Panel mixed multinomial logit model, Subsampling, Weighting