Output Layout¶

Everything the analysis layer produces goes under Analysis/ — on mainDir while you are working, and on storageDir once complete has moved it, at the same relative path under either. Nothing it does touches Output/:

Path	Holds	`complete`
`Analysis/Main/`	intermediates derived from your results and shared between modules	moves
`Analysis/Results/<folderName>/`	one analysis, and the verification record that cleared it	moves
`Analysis/Logs/`	one directory per stage, as the pipeline keeps its own	stays
`Analysis/Session/`	this invocation's Nextflow dag, trace, timeline and report	stays
`Analysis/work/`	Nextflow's working directory, removed when a run succeeds	stays

Analysis/Session/ exists so that an analysis run does not overwrite the four session files in Output/Reports, which are the record of the pipeline run that produced the results being read.

Analysis/Main is where a module puts anything it had to derive — a per-position depth file, a frequency matrix, a callable-sites count. Derivations are shared: a second module wanting the same thing finds it already there rather than building it again. It stays empty until you install a module that derives something, and it can then grow large — a full per-position depth file is measured in gigabytes.

Analysis/Results holds one folder per analysis, named by folderName.

The script that produced it¶

Every published analysis carries the script that produced it. That is a guarantee of the layer rather than a convention module authors are asked to follow: an analysis handed over without one is refused, nothing is published, and the folder is left exactly as it was. So a result you find in Analysis/Results can always be regenerated, and the folder also holds the verification record that cleared it — which names the module, its version, the runs it covered and the configuration it was assembled from.

It carries the libraries it used, folded in. A module declares the libraries it needs — effective sample size, gene diversity, the harmonic means — and each is installed alongside it; what is published is those functions and the module's own code in one file, headed by the frame version it ran against. A driver that merely source()s code the reader does not have would satisfy the letter of the guarantee and none of its point — and a library lives in the module store, which you can uninstall from. The file in your results is the copy that ran, whatever the store holds afterward.

And a README.md that says how to read what is in the folder. { #analysis-readme } Every module declares, for each file it publishes, the section of this manual that explains it; the frame renders one table from those declarations and from what every analysis carries anyway:

| File | What it is | Explained at |
|---|---|---|
| `design.tsv`             | one row per pool, the experimental design | [#the-experimental-design](...) |
| `0_verify_analysis.txt`  | the checks that cleared this folder       | [#verification](...)           |
| `CITATIONS.md`           | the software this analysis used           | [#citing-the-tools-it-runs](...) |

It links into the copy of this manual installed with the release that produced the folder, so the explanation you follow is the one that was true at the time. The point is the numbers a table cannot explain for itself: a figure that is a bound rather than a measurement, or an estimate carrying an assumption, reads as neither unless something beside it says so.

A module that declares a file and then does not publish it is refused, and a module whose declared section does not exist in the manual is refused before it starts — a link nobody can follow being worse than none.

One PDF of the whole analysis¶

Every published folder also carries report.pdf: every result in it, in one document, each under the name of the file it came from. Tables are rendered as tables, figures as figures, and the heading above each is the file name — so a number you want to work with can always be traced back to the file that holds it.

It is a capability of the layer, not of any module. A module already declares what it publishes and what each file is, because the README.md is built from those declarations; the report is built from the same ones. So a module gets a report without writing a line of code for it, and so does a module somebody else wrote.

What it is not is a substitute for the files. A table longer than forty rows is shown to forty and says so, and the figures are the same images that sit beside it. Everything in the report is in the folder; the folder is what you compute from.

A report that cannot be built does not stop the analysis being published. The reason is printed and the folder is complete without it — every number was already a file. That is the opposite of how the other publishing checks behave, and deliberately so: a missing script or an undeclared output means the analysis is not what it claims to be, where a missing report means only that a convenience is absent.

Building it needs pandoc and typst, both pinned in the analysis environment. If you are running a module's script by hand rather than through PoolSeqFlow analysis, neither will be on your path and no report is written.

Naming an analysis¶

analysis.folderName decides where an invocation writes, under Analysis/Results:

params {
    analysis {
        folderName = ''                  // the module's own name — mds writes to Results/mds
        // folderName = 'sweep_strict'   // a name of your own
        // folderName = 'MDS/SummerPops' // a path, so related analyses group together
    }
}

Set it in <module>.config instead of analysis.config to have it apply to one module.

Under multiRun, each results directory gets its own folder inside the one you named — Results/sweep/Shared_1, Results/sweep/strict — using the names the pipeline gave them. A single run has no name anywhere, so its analysis sits in the folder directly.

Naming the folder is how two settings of one module are told apart. Nothing is stamped with a date, nothing is versioned, and nothing is overwritten: a folder that already holds an analysis is refused, naming what is in it. To run the same module again with different settings, give the new one a folder of its own; to redo an analysis, move the old one out of the way first.

The one thing allowed to be in the folder already is the verification record from a previous attempt. A module that fails after the check leaves the folder holding nothing else, and that retry has to be allowed — otherwise a single failure would make the name unusable.

Moving finished analyses to permanent storage¶

PoolSeqFlow analysis complete moves Analysis/Main and Analysis/Results from your working directory to storageDir/Analysis/, keeping the same relative path so that a module can still find an intermediate afterwards — it brings back what it needs on its own, a file at a time, and says so when it does. Analysis/Logs, Analysis/Session and Analysis/work stay where they are: they describe invocations rather than results, and the next analysis rewrites Session and appends to Logs.

It asks before it starts, and everything is surveyed before anything moves. A folder name already taken in permanent storage stops the command with nothing moved — two different analyses under one name is exactly what naming a folder exists to prevent. Move the older one out of the way, or give the new one a name of its own, and run it again.

A folder holding nothing but a verification record is passed over and said so, because that is what a failed attempt leaves and archiving it would consume the name the retry needs.

Each item moves on its own, so an interrupted run can simply be run again: what already moved is left alone. There is no command that brings a results folder back to the working directory — move it by hand if you need it there.

Working, archiving, working again¶

You can keep using a project after complete has archived it. Run a module and it finds the intermediates it needs in permanent storage and copies them back to your working directory, a named file at a time. They exist in both places from then on, deliberately: the copy in permanent storage is what makes the next complete cheap.

That is what the second complete is for. An intermediate already in permanent storage is not moved a second time — the working copy is discarded, and the run says which ones and why. What decides that is the provenance record beside each intermediate: a few hundred bytes naming the results it was derived from. When the two records agree the two files are the same file, and only one of them needs keeping.

When they do not agree, the command stops and moves nothing. One of the two was derived from results this project no longer holds, and an analysis you have already published may have come from either — so neither is removed for you. Delete the one you do not want and run it again. The same happens if either copy has lost its provenance record: without it there is no way to tell the two apart, and guessing is not an option the command takes.

The cost of a working cycle is therefore one transfer, not two — which matters when storageDir is billed per operation, as on S3 or a tape-backed archive. What it costs instead is disk on the working volume, and that is what the discard reclaims.

Do not run it while a module is running. The two would be moving the same folders in opposite directions, and neither checks for the other.