The Vanishing Record: Scientific Data Is Disappearing Faster Than Anyone Anticipated

In 2014, a team of researchers at the University of British Columbia published a study with a finding that should have prompted immediate alarm across the scientific community: the probability that a dataset underlying a published study remained accessible declined precipitously with time, dropping to roughly 20 percent for papers more than twenty years old. The study attracted attention, generated citations, and was discussed at several open-science conferences. Then, in a turn that would be darkly comic if the implications were not so serious, some of the contact information listed in that very paper became unreachable within a few years of publication.

The problem of scientific data loss is not new, but its scale, velocity, and structural roots are only now being fully reckoned with. As research becomes increasingly computational, as datasets grow larger and more complex, and as the pace of format and infrastructure change accelerates, the gap between what is nominally preserved and what is practically recoverable has widened into something approaching a systemic crisis.

What Disappearance Actually Looks Like

Data loss in science rarely announces itself dramatically. There is no single catastrophic event, no burning of a library. Instead, loss accumulates through a series of mundane failures: a university server is decommissioned during a budget cycle and its contents are not fully migrated; a graduate student who maintained a dataset graduates, takes a position in industry, and does not respond to emails; a file format that was standard in 2008 requires software that no longer runs on any operating system in active use; a supplementary data file is hosted on a personal faculty webpage that disappears when the faculty member retires.

Each of these events is individually unremarkable. Collectively, they constitute a quiet erosion of the empirical foundation on which subsequent science is built. When a researcher attempting to conduct a meta-analysis or extend a prior finding cannot access the underlying data, the scientific record becomes, in effect, a collection of claims without verifiable substrates.

The quantitative picture is sobering. A 2021 analysis published in Current Biology examined the availability of datasets associated with published ecology and evolutionary biology papers and found that data availability declined at a rate of approximately 17 percent per year after publication. Studies more than five years old had, on average, less than half their associated data accessible through the links or contact information provided. Studies more than a decade old fared considerably worse.

The Infrastructure Problem

The technical dimensions of data decay are real and deserve serious attention. File format obsolescence is among the most underappreciated hazards in long-term data preservation. Formats that were industry standards in the early years of computational research — certain proprietary spreadsheet formats, early genomic data formats, specific image compression schemes — now require emulation software or specialized converters to read, if they can be read at all. Researchers who stored data in formats tied to specific commercial software versions face the prospect that their archives are, in a meaningful sense, already unreadable by anyone who has not maintained legacy systems for precisely this purpose.

Metadata presents an equally serious challenge. A dataset without adequate metadata — documentation of what variables mean, how measurements were taken, what preprocessing steps were applied, what units are in use — is not a dataset in any scientifically useful sense. It is a collection of numbers. The standards governing metadata have evolved substantially over the past two decades, and datasets deposited under earlier conventions frequently cannot be integrated with current systems without manual reconstruction of contextual information that may no longer exist.

Server infrastructure compounds both problems. Many datasets that researchers believe to be preserved are hosted on institutional servers with no formal preservation mandate, maintained by IT departments whose priorities are operational rather than archival, and subject to decommissioning whenever budget pressures or technology refresh cycles demand it. The distinction between a file that exists somewhere on a server and a file that is actively preserved, periodically verified for integrity, and maintained in a format that will remain accessible is one that institutional infrastructure frequently collapses.

The Human Dimension

Behind every lost dataset is a researcher — often several — whose work is thereby rendered partially unverifiable. Conversations with scientists across disciplines reveal a pattern of resignation that itself speaks to how normalized data loss has become.

A computational neuroscientist at a public research university in the Midwest described spending three months attempting to reconstruct a preprocessing pipeline for a dataset she had herself generated during her postdoctoral fellowship, only to conclude that a critical intermediate file had been lost during a server migration she had not been informed of. "The paper is still cited," she noted. "But I couldn't reproduce my own analysis if I had to."

An environmental scientist recounted discovering that a decade of sensor data from a long-term ecological monitoring project had been stored on drives that were no longer readable, because the institution that had hosted the project had changed data management vendors and the migration had been incomplete. The data had never been formally deposited in a public repository because, at the time of collection, no suitable repository existed for that data type.

These are not stories of carelessness. They are stories of researchers operating within an infrastructure that was not designed for long-term preservation and that has not been adequately reformed despite decades of awareness that the problem exists.

Why Mandates Have Not Been Sufficient

Federal data-sharing mandates have proliferated in recent years. The NIH's 2023 Data Management and Sharing Policy requires that researchers funded by the agency submit data management plans and, in most cases, deposit their data in an appropriate repository. Similar requirements apply to NSF-funded research. These mandates represent genuine progress and have unquestionably increased the volume of data deposited in public repositories.

However, deposition and preservation are not synonymous. A dataset deposited in a repository that lacks the resources to maintain it, migrate it through format changes, or verify its ongoing integrity is not preserved in any durable sense. Many of the repositories that researchers are directed to use by funding agency guidance are themselves under-resourced, operating on grant funding that may not be renewed, and lacking the institutional permanence that genuine archival stewardship requires.

Funding structures also create perverse dynamics. Data collection and analysis are fundable activities; data preservation is largely not. When a grant ends, the resources that sustained a dataset's maintenance end with it. The expectation that universities will absorb ongoing preservation costs as an institutional commitment has not been matched by the financial reality of most research institutions, particularly those outside the top tier of research funding.

Toward a Durable Scientific Memory

Constructing a scientific infrastructure capable of preserving the empirical record across decades requires addressing problems at multiple levels simultaneously.

At the technical level, the research community needs agreed-upon, format-agnostic preservation standards that prioritize long-term accessibility over short-term convenience. Open, well-documented formats with broad software support should be the default for data deposition, and repositories should commit to active migration as standards evolve.

At the institutional level, data stewardship needs to be recognized as a professional contribution that carries genuine career weight. Researchers who invest in thorough documentation, clean metadata, and accessible deposition should receive credit proportionate to the scientific value of that work. Institutions that demonstrate strong preservation records should be eligible for the funding incentives that currently reward only publication output.

At the policy level, sustained federal investment in a network of domain-specific, professionally staffed repositories — analogous to the Library of Congress's role in preserving the national documentary record — would provide the institutional permanence that grant-funded repositories cannot. The analogy is instructive: the nation does not rely on individual researchers to preserve historical documents on personal servers. There is no principled reason why the empirical substrate of federally funded science should be treated any differently.

The scientific record is not self-preserving. Without deliberate, sustained, and adequately funded effort, it will continue to dissolve — not all at once, but incrementally, dataset by dataset, until the foundation beneath a significant portion of the published literature is no longer there to stand on.