Scientific Presentations
LabTalks@DCD
Department of Digital and Computational Demography
Max Planck Institute for Demographic Research (MPIDR), Rostock, Germany, November 26, 2025
11:00 AM: Talk with Charles Rahal - On the Responsible use of Pseudo-Random Number Generators in Applied Scientific Research
Auditorium - 057
Abstract
The overwhelming best practice in `Open', `Reproducible' and `Responsible' research is to hide the variation caused by pseudo-random number generators (PRNGs) through the arbitrary use of a `seed' or `random state' in algorithmic pipelines. However -- in the process of doing so -- researchers index scientific results with seeming certainty, when they may be anything from it. Eliminating this source of variation is the opposite of what responsible analysts should be doing. PRNGs are ubiquitous in many research areas, occurring in a large proportion of quantitative and computational research designs. However, it's contribution to variation in the outcome of interest is hitherto significantly under-appreciated. To illustrate the risk of ignoring `seed variability' we undertake a series of simulations and replications of prior and high profile work across descriptive, inferential, and predictive research designs, showing just how large the variation caused by the instantiation of PRNGs can be, and how they dramatically affect scientific findings. Our empirical examples span applied fields such as criminology, economics, education research, medicine, scientometrics, social stratification, and sociology. We also highlight the potential to generate unfair societal outcomes, for example by showing that potentially more than half of parole decisions could be reversed due to seed variability using the canonical ProPublica dataset on recidivism. We also design analysis which shows the effect of seed variability when querying Large Language Models. We conclude with recommendations on how to effectively eliminate the risk that seed variability might lead to false positives (and negatives) in applied empirical work.
About
Charles Rahal is an Associate Professor in Data Science and Informatics at the University of Oxford, and a former British Academy Postdoctoral Fellow. He maintains various open and interactive online projects, such as the GWAS Diversity Monitor and RobustiPy, and he is also active in the Open Science movement. He is a Co-Investigator\Principle Investigator on various projects across the University. He has consulted on a Methods Advisory Group for the ONS (pro-bono), and currently consults for the Banco de la República. He is currently an honorary visiting Professor at Peking University (2025-2030). He is also fortunate enough to be both an Associate Member at Nuffield College and a Researcher at the Gradel Institute, New College. He is an Associate Editor-In-Chief at the Journal of Social Computing, and an Associate Editor at ACM Transactions on Social Computing. He also convene the ‘Metrics and Models’ lab, which has an open seminar series component to it; all are welcome to attend.