Why Reproducibility Became an Infrastructure Problem

When researchers attempt to reproduce computational work, failure is the norm rather than the exception. A 2024 study examining more than 27,000 Jupyter notebooks from biomedical publications found that only 8.5% produced results identical to the originals.¹ A broader analysis of 1.4 million notebooks reported reproducibility rates closer to 4%.² These are not marginal failures or isolated mistakes. They point to something structural.

For a long time, reproducibility has been discussed as a methodological issue. When results cannot be replicated, the explanation is usually sought in missing documentation, incomplete code, or insufficient data sharing.

Those explanations are no longer sufficient.

Across many computational fields, results fail to reproduce even when the logic is clear and the code is available. The problem is that the execution context itself cannot be recovered with enough precision. Dependencies evolve, operating systems change, libraries are deprecated, and infrastructure assumptions disappear. By the time replication is attempted, the environment that produced the original result no longer exists in any usable form.

Reproducibility, increasingly, is constrained less by intent or effort than by infrastructure.

Over the past decade, abstraction worked in researchers’ favor. Cloud platforms lowered entry barriers, made scaling easier, and allowed complex systems to run without deep operational involvement. For many workflows, this was an obvious win.

The trade-off only became visible later.

Even when code is shared, which remains rare in practice, with only about 6% of economics papers providing replication code³, the execution environment is often transient. The details that determine how code actually runs are managed by platforms, hidden behind layers of abstraction, and rarely captured explicitly.

This becomes a problem when the work needs to be evaluated.

During peer review, replication efforts, policy use, or teaching reuse, the inability to recover an execution environment turns into a hard constraint. In education, this failure mode is especially visible: when students cannot run the same code an instructor demonstrated, learning gives way to troubleshooting. What cannot be recovered with sufficient fidelity cannot be validated. What cannot be validated cannot be reused with confidence.

At that point, reproducibility stops being a question of method and becomes a question of control.

Reproducibility rarely fails at the moment results are produced. It fails later, when someone else tries to run the same computation under comparable conditions.

The failure points are familiar to anyone involved in review or replication:

These are not edge cases. They are the predictable outcome of treating execution environments as implementation details rather than as part of the research record.

Once the environment is gone, code alone is not enough to reconstruct what was actually executed.

In computational work, results are produced by more than code. They emerge from the interaction of code, data, dependencies, operating system, hardware assumptions, and runtime configuration.

Together, these elements form the execution environment.

Expecting results to remain reproducible while treating that environment as disposable reflects a mismatch between how research is produced and how it is evaluated later. If results are to be reviewed, validated, or extended, the execution environment must be preserved with the same care as code and data.

This does not require freezing systems or preventing iteration. It requires acknowledging that environments change and designing them to be versioned, referenced, and recoverable.

Institutions have begun to recognize this. 

Nature Human Behaviour introduced routine code review in 2021, noting that reproducibility often fails because code cannot be accessed, compiled, reused, or extended.⁵ Yale University’s Institution for Social and Policy Studies has operated a reproducibility verification program since 2011, one of the longest-running efforts of its kind.⁶

Reproducibility becomes routine only when environments are treated as first-class research objects.

As computational research scales, infrastructure choices increasingly determine what can be inspected at all.

In these settings, reproducibility depends less on willingness to share and more on whether the infrastructure allows inspection without violating legal, ethical, or institutional constraints.

This is where control becomes decisive.

Research organizations face a structural tension.

They need fences: clear boundaries around sensitive data, regulated workloads, and institutional responsibility. At the same time, they need bridges: ways to collaborate, review, and build on work across teams, institutions, and borders.

Most systems force a choice. Lock everything down and collaboration suffers. Open everything up and risk becomes unacceptable.

Reproducible science requires both. It requires infrastructures that allow logic to be shared without forcing full exposure of state, and that preserve execution context without granting unrestricted access. Architectures that can run on institutional infrastructure, on-premises or in controlled cloud environments, offer governance options that purely public cloud platforms cannot.

In this context, sovereignty is not an abstract principle. It is an operational condition.

Execution environments are subject to jurisdiction, audit requirements, and institutional accountability. Whether an environment can be inspected, preserved, or rerun often depends on where it runs, who controls it, and under which legal framework.

As a result, reproducibility is increasingly shaped by infrastructure governance. What can be validated is constrained by what can be legally and technically retained.

Recognizing this does not reduce openness. It clarifies when, how, and under which conditions openness is possible.

The evidence points to a consistent pattern: reproducibility succeeds when environments are captured as complete, versioned artifacts rather than reconstructed after the fact.

Several design choices distinguish infrastructures where reproducibility is routine:

The prevalence of undeclared dependencies shows what happens when environments are treated as external. Effective systems capture code, data, and configuration together as a single snapshot. This is not file versioning. It is versioning of computational state.

Tools like Docker make isolation possible, but adoption remains limited when the burden falls on researchers. Systems that manage containerization transparently remove a major barrier to good practice.

Sharing a repository is not the same as sharing an executable environment. Reviewers, collaborators, and students should be able to run work immediately, without setup. That requires distributing full computational contexts, not just source files.

The bridges-and-fences dilemma is resolved by controlled sharing: logic can be inspected without exposing sensitive data, and institutions retain control over where computation occurs. Hybrid and self-hosted models make this possible.

Adoption follows familiarity. Providing JupyterLab, RStudio, VS Code, or Stata within managed, reproducible environments changes what happens beneath the work without asking researchers to change how they work.

These are not hypothetical requirements. Platforms that embody them are already in use across universities and research institutions.

The remaining challenge is not technical feasibility, but institutional adoption.

Reproducibility does not require perfect foresight. It requires infrastructures that assume change and make it traceable.

When execution environments are preserved alongside code and data, versioned over time, and accessible under controlled conditions, validation becomes part of the normal research lifecycle rather than an exceptional effort.

Reproducibility shifts from a retrospective struggle to a built-in property of how computational work is organized.

Reproducibility has failed at scale. When the vast majority of computational notebooks cannot be reproduced and only a small fraction of studies share executable code, the problem is not individual negligence. It is infrastructure design.

Today, what can be validated, reviewed, and reused depends largely on whether execution environments can be recovered with sufficient precision under real institutional constraints.

Making reproducibility routine again begins with acknowledging this shift. From there, the task is not to ask researchers to do more, but to build infrastructures that reflect how computational science is actually produced, evaluated, and reused.

If this is useful, you can receive short notes like this → Get updates

Sources

1. Samuel, S., & Mietchen, D. (2024). Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience.

2. Pimentel, J. F., et al. (2021). Understanding and improving the quality and reproducibility of Jupyter notebooks. Empirical Software Engineering.

3. Fink, L., & Marcus, J. (2025). Replication code availability over time and across fields. Economic Inquiry.

4. Ankel-Peters, J., Brodeur, A., Dreber, A., Johannesson, M., Neubauer, F., & Rose, J. (2025). A protocol for structured robustness reproductions and replicability assessments. Q Open, 5(3), qoaf004.