Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study

Research output: Contribution to journalResearch articleContributedpeer-review


  • Anna Ostropolets - , Columbia University Irving Medical Center (CUMC) (Author)
  • Yasser Albogami - , King Saud University (Author)
  • Mitchell Conover - , Janssen Research & Development (Author)
  • Juan M Banda - , Middle Georgia State University (Author)
  • William A Baumgartner - , University of Colorado Anschutz Medical Campus (Author)
  • Clair Blacketer - , Janssen Research & Development (Author)
  • Priyamvada Desai - , IBM Research Zurich (Author)
  • Scott L DuVall - , VA Salt Lake City Health Care System (Author)
  • Stephen Fortin - , Janssen Research & Development (Author)
  • James P Gilbert - , Janssen Research & Development (Author)
  • Asieh Golozar - , Odysseus Data Services GmbH (Author)
  • Joshua Ide - , Johnson & Johnson (Author)
  • Andrew S Kanter - , Columbia University Irving Medical Center (CUMC) (Author)
  • David M Kern - , Janssen Research & Development (Author)
  • Chungsoo Kim - , Ajou University Graduate School of Medicine (Author)
  • Lana Y H Lai - , University of Manchester (Author)
  • Chenyu Li - , University of Pittsburgh (Author)
  • Feifan Liu - , University of Massachusetts Medical School (Author)
  • Kristine E Lynch - , VA Salt Lake City Health Care System (Author)
  • Evan Minty - , University of Calgary (Author)
  • Maria Inês Neves - , Real World Solutions (Author)
  • Ding Quan Ng - , University of California at Irvine (Author)
  • Tontel Obene - , Jackson State University (Author)
  • Victor Pera - , Erasmus University Medical Center (Author)
  • Nicole Pratt - , Flinders University (Author)
  • Gowtham Rao - , Janssen Research & Development (Author)
  • Nadav Rappoport - , Ben-Gurion University of the Negev (Author)
  • Ines Reinecke - , Institute for Medical Informatics and Biometry (Author)
  • Paola Saroufim - , Case Western Reserve University (Author)
  • Azza Shoaibi - , Janssen Research & Development (Author)
  • Katherine Simon - , Vanderbilt University (Author)
  • Marc A Suchard - , University of California at Irvine (Author)
  • Joel N Swerdel - , Janssen Research & Development (Author)
  • Erica A Voss - , Janssen Research & Development (Author)
  • James Weaver - , Janssen Research & Development (Author)
  • Linying Zhang - , Columbia University Irving Medical Center (CUMC) (Author)
  • George Hripcsak - , Columbia University Irving Medical Center (CUMC) (Author)
  • Patrick B Ryan - , Columbia University Irving Medical Center (CUMC) (Author)


OBJECTIVE: Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics.

MATERIALS AND METHODS: Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics.

RESULTS: On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.

CONCLUSIONS: Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.


Original languageEnglish
Pages (from-to)859-868
Number of pages10
JournalJournal of the American Medical Informatics Association : JAMIA
Issue number5
Publication statusPublished - 19 Apr 2023

External IDs

PubMedCentral PMC10114120
Scopus 85152976981
ORCID /0000-0003-0154-2867/work/143494695



  • Humans, Research Personnel, Databases, Factual