How to document research code so a stranger can run it

Journals increasingly ask for the code behind a paper, and reviewers increasingly open it. What they find, most of the time, is a repository that its author can navigate perfectly and nobody else can — scripts named for the order they were written in, a notebook that only runs if you already know which cells to skip, and a README containing the project title and nothing else.

The fix is not more documentation. It is a small amount of documentation in the right places, written for a specific reader: a competent stranger, six months from now, with your paper open and no access to you.

The one test

We call it the README-to-run check, and we apply it to every code job that comes through the desk. Take a clean machine, or a fresh container. Follow the README literally, doing only what it says and nothing you happen to know. Stop at the first instruction that fails or requires a guess.

That stopping point is your documentation debt. Everything before it works. Everything after it is untested. Most repositories fail within three steps, and almost always on the same three things: an unlisted dependency, a data file that is not where the code expects it, or a path that only exists on the author's machine.

What a README must contain

In this order, because it is the order a stranger needs it:

• One sentence on what this is and which paper it belongs to, with a citation or DOI. A reader arriving from a reference list needs to confirm they are in the right place.
• Requirements. Language version, operating systems it has actually been run on, any non-Python or non-R system dependencies, and whether a GPU is needed. State what you tested, not what you assume works.
• Installation, as commands that can be copied and pasted in sequence. Not prose describing installation.
• Data. Where it comes from, how to obtain it, where to put it, and how large it is. If the data cannot be shared, say so explicitly and describe the format precisely enough that someone can substitute their own.
• How to reproduce the paper's results — the exact command that regenerates the main figures or tables, and roughly how long it takes. This is the section reviewers look for and the one most often missing.
• Repository layout, a few lines mapping directories to purpose.
• Licence, which determines whether anyone may legally use any of the above.
• How to cite, ideally a CITATION.cff file so the platform can render it.

Two things to leave out: a wall of badges, and aspirational sections describing features that do not exist yet. Both cost trust.

Docstrings a reviewer can use

Pick one convention — NumPy, Google, or reStructuredText for Python; roxygen2 for R — and apply it everywhere. Which one matters far less than consistency, because tooling and readers both key off the pattern.

A useful docstring answers four questions:

• What does this do, in one line, in the imperative. Compute the adjusted odds ratio, not This function computes…
• What goes in: each parameter, its type, its units, and its expected shape or range. Units are the ones people omit and the ones that cause real errors.
• What comes out: type, shape, units, and what the values mean.
• What can go wrong: the exceptions it raises and the assumptions it does not check.

Where a function implements something from the literature, cite it in the docstring. A reader trying to verify your analysis against the published method should not have to reverse-engineer which variant you used.

What a docstring should not do is restate the code. # increment i by one above i += 1 is noise, and noise trains readers to skip comments — including the one comment that mattered.

Comments that earn their place

Good comments explain why, because the what is already on the screen. The ones worth writing:

• Why a non-obvious threshold, constant, or seed was chosen.
• Why an obvious approach was rejected — this saves the next person from trying it.
• Where a workaround exists for a bug or quirk in a dependency, with a version number.
• Where a step is required by the analysis rather than the software: an exclusion, a correction, a filter that came from the protocol.

If a block needs a paragraph of explanation, that is usually a sign it should be a named function instead. Naming is the cheapest documentation there is.

Pinning the environment

The most common reason published code stops working is not the code. It is a dependency that moved.

• Record exact versions — a lock file, requirements.txt with pinned versions, a conda environment.yml, or renv.lock for R.
• Record the language version too, and the platform you ran it on.
• If the analysis is long-lived or the stack is fragile, provide a container image and its definition file. The definition matters more than the image.
• Set and record random seeds, and note anything that remains nondeterministic despite them — thread counts and GPU kernels are the usual culprits.
• Archive the release that corresponds to the paper and cite that version, not the branch. A repository that keeps moving is not a citable artefact.

Notebooks

Notebooks are excellent for showing an argument and poor for reproducing one. Both can be true, and the way to have both is to be deliberate about which each notebook is for.

• Notebooks tell the story; modules do the work. Analysis functions belong in importable files that can be tested; the notebook calls them and narrates.
• The narrative is prose, and prose gets edited. Markdown cells should be full sentences explaining what the next cell does and why — not now we plot.
• Restart and run all before committing. A notebook that only works out of order is a notebook that does not work.
• Decide about outputs. Cleared outputs keep diffs readable; retained outputs let a reader see results without running anything. Either is defensible, but be consistent, and never commit outputs containing identifiable data.
• Number them in execution order if more than one must run in sequence, and say so in the README.

Error messages and user-facing strings

An overlooked category, and one we edit as carefully as prose. An error message is documentation that appears exactly when someone needs it. ValueError: bad input tells a user nothing. ValueError: expected 3 columns (id, dose, response), got 4 — check the delimiter resolves the problem without a support request.

State what was expected, what was received, and what to try. The same applies to help text, log lines, and command-line usage strings.

What we do and do not touch

Our documentation and code editing unit edits programs and their documentation — docstrings, comments, READMEs, API references, notebook narrative, error messages, naming consistency — and does not change what the code does. Logic is out of scope, deliberately and without exception. If a function's behaviour contradicts its docstring, that is a query for the author, not a fix for an editor; we do not know which of the two is wrong, and guessing would put a change into your analysis that you never approved.

Everything comes back tracked, the same as a manuscript, with the README-to-run check reported as part of the return: how far a stranger got, and where they stopped.