Skip to content

How the Project Is Verified

Everything on the other pages of this section is a claim about how the project is built. This page is what makes those claims checkable rather than something you have to believe, and it is also the part of this section that has nothing to do with AI — it would be the same discipline, for the same reasons, if I had written every line by hand.

Scientific verification

One thing I want to underline before any of the machinery: scientific verification is always on the developer. No test suite performs it, and none of what follows on this page should be read as though it does.

A test can confirm that a formula was implemented as written, and that it behaves correctly on data built to a known answer. It cannot tell me the formula was the right one to choose, that its assumptions hold for pooled data, or that the paper it came from says what I think it says. Those are the questions that decide whether a number means anything, and they are answered by a person or not at all.

In practice that is a specific and fairly slow kind of work, and it is worth saying what it looks like rather than leaving it as a principle:

  • Deriving a quantity rather than porting it. Where this pipeline computes something with a published name, the constant in front of it is worked out by hand and checked against a case computed by hand, because transcribing a formula from a paper is exactly how a factor of two travels silently from one codebase into the next.
  • Reading the primary source, not the attribution. Method citations here are checked against the paper before they ship. That is not ceremony: the effective-sample-size correction this pipeline uses was attributed to the wrong source twice before it was right, and the paper it is now cited for never writes the quantity down — it is cited for the result the quantity follows from, with a note saying so.
  • Testing against data whose answer is known by construction, rather than against what the code currently produces. If the expectation comes from the implementation, the test can only tell me the code has not changed.
  • Asking what the estimator assumes, and whether a real experiment provides it. Pooled data breaks assumptions that individual genotypes satisfy — unequal pool sizes, non-homogeneous coverage, no privileged reference allele. Where a standard quantity needs adjusting before it means anything here, the manual says so at the point where the number is produced.

None of that is automatable, and it is the part of the work an AI agent is least able to help with — it will produce a correct implementation of the wrong statistic, and produce a simulation showing the implementation is correct. The implementation being correct is not the question.

So: the suite below tells me I have not broken what I built. It does not tell me I built the right thing. That judgment is mine, and where I have made it the manual says what was chosen and what it assumes, so it is at least visible enough for someone to disagree with.

The shape of it

The suites are organized by seam rather than by file: the wrapper, configuration migration, parameter resolution, the change guards, the helper programs, the dry run, the analysis frame and the static checks each get one. Most ship with the pipeline, and each analysis module ships its own, because a module's tests belong to the module and travel with it when it is published separately. test/run_tests.sh --list prints the current set with what each one costs; there is no case count written here, because it moves with every stage and a number in a manual is a number that goes quietly out of date.

Most of them are not testing what people usually mean by a test. The failures this pipeline can produce are rarely crashes — they are a run that completes, reports success, and gives a number that is quietly wrong, because a filter was applied with the wrong pool's threshold or a step reused an artifact that was produced under different settings. So a large share of the suite exists to catch a plausible result rather than a broken one.

A suite you will not run is not a suite

The whole thing takes about three quarters of an hour, and that is the real problem to solve. A forty-minute gate between me and a one-line change does not make me careful, it makes me skip the check — so the suite is arranged so that the run I actually do is small.

Every suite declares what it costs to run, in its own header:

Class What it needs
static Nothing installed. Reads files, runs the shell and Python helpers directly
jvm A JVM, to build and inspect a workflow without executing it
pipeline A real end-to-end run against the committed fixture data

The static set finishes in seconds on a machine with nothing set up at all, which makes it the loop I develop in. The full run belongs to a release, not to a change.

There is a second axis alongside it. --changed picks the suites for me, from what each suite declares it covers, expanded through the include graph — so editing a helper program selects the suites that exercise it and nothing else. It deliberately errs wide: touching the test library or the selector itself selects everything. That way a narrow answer is trustworthy and a wide one is merely expensive.

The rule I hold myself to is that the cases for a step are written with the step and run on their own, by name. The full suite is a release gate. Treating it as the per-change gate is how you end up not checking anything.

The checks that are not tests

Some things cannot be asserted from inside the suite, so they are separate gates and all of them run before a release:

  • nextflow lint over every workflow file, at zero errors and zero warnings. The strict parser rejects a good deal of ordinary Groovy, and it reports a parse failure in one file as "not defined" at every call site in other files — so a clean lint is worth more than it sounds.
  • The citation check. Every reference is authored once in BibTeX and compiled to the JSON the pipeline reads; the gate regenerates it and fails if the two disagree, so the file a run cites from cannot drift from the file I edit.
  • The manual check. This manual is one file, and every page of the site is generated from it. The gate re-parses it, resolves every cross-reference, and fails on a link to a heading that no longer exists or two headings that would collide — which is what stops the documentation rotting quietly as things are renamed.
  • The version check, which fails when the analysis frame, a module or a library has changed without its own version moving, and the archive check, which builds the release tarball and asserts that everything a user needs is in it and everything they do not need is out — modules included, since none of them ships.

What this does not do

It does not tell me the science is right — that is Scientific verification above, and it is the limit that matters most.

It also does not cover everything. The cross-filesystem paths in the artifact-moving code cannot be reached from the suite, because the sandbox is a single filesystem — the evidence for those is measurement recorded in the development notes, and nothing in the automated run reproduces it. I would rather say that plainly than let a green run imply more than it covers.

And a suite is only as good as the cases in it. When I fix something, I check that the new case actually fails against the unfixed code before I keep it. A test that passes both ways is worse than no test, because it reads as coverage.