Multi-run¶
Sometimes one set of reads needs analyzing more than once: against two reference genomes, under three trimming stringencies, at a range of filter settings. A run table describes those analyses in one file, and one invocation carries them all out.
The reason to do it this way rather than copying the project is that the runs have most of their work in common. Two runs that differ only in filter thresholds share every step up to variant calling, and that work is done once rather than once per run.
Turning it on¶
in a new project, or in an existing one set it yourself:
init_multi also copies multi-run.csv.example into the project. It does not write the table: the runs, and the parameters that differ between them, are the whole content of that file, and only you know them.
The run table¶
Four rules, and they are all of them.
One column must be RunID. It names the run and becomes a directory, so keep it to letters, digits, dot, dash and underscore.
Every other column is a parameter name, spelled exactly as parameters.config spells it, with no params. prefix. Nested ones are dotted: trim_galore.quality, variantCall.maxDepth. Any parameter may be varied, including ones the pipeline normally computes for itself. A column that does not name a parameter is refused rather than ignored — that is a typo, not a preference.
A blank cell means "take it from parameters.config". That is what keeps a table as short as the difference between the runs. One consequence worth knowing: there is no way to set a parameter to an empty string here, because blank already means something else.
A value containing a comma must be quoted — "-a AD,DP,SP,INFO/AD". The default readPattern, *_R{1,2}.fq.gz, is exactly such a value.
Blank lines and lines starting with # are ignored, so a run can be kept in the file with a # in front of it rather than deleted.
RunID,referenceFile,gffFile,trim_galore.quality
reference_a,reference_a.fasta.gz,reference_a.gff.gz,
reference_b,reference_b.fasta.gz,reference_b.gff.gz,30
Two references, the second trimmed harder. The blank cell on the first row takes the configured quality, so the baseline is whatever parameters.config already says rather than a number repeated here.
Setting a value the pipeline would compute¶
Some parameters are derived from others — filterFalsePositives.sensitivity from poolSize and ploidy, variantCall.mpileupOptions from the four bcftools values, the whole cores ladder from threads. Varying either end of that relationship is allowed, and the two behave differently.
Set an input and the computed value follows it. A run with poolSize = 200 gets the sensitivity that a pool of 200 implies; one with trim_galore.quality = 30 gets trimming options built around 30. You do not have to keep the derived values in step by hand.
Set a computed value directly and it is used exactly as written — and nothing is derived from its inputs any more. Pin variantCall.mpileupOptions and variantCall.maxDepth stops meaning anything for that run. That is a legitimate thing to want when you need a command line under your own control, and any column doing it is named in the report, so it is a choice rather than a surprise.
Where the results go¶
There is one results tree, and only divergence gets a name:
storageDir/Output/
├── All_Runs/ ← work every run shared
├── Shared_1/ ← work some of them shared
├── Shared_2/
├── reference_a/ ← work this run did alone
└── reference_b/
Logs/ follows the same shape. Each Shared_<N> directory holds a members.txt naming the runs it belongs to, because a group number on its own says nothing about which runs are in it.
A single run gets none of this and keeps the plain tree. Turning multiRun on for one run is not a different layout, but there is no reason to do it.
Group numbers are assigned in order of appearance in the table, so reordering rows can leave Shared_1 naming a different pair than before. The change guard compares the table you ran against the one recorded beside the results, and row order counts as a change for exactly this reason.
What is shared, and what is not¶
Sharing is decided by comparing the parameters each step actually reads. Two runs share a step when every value that step depends on is the same for both — so runs differing only in vcffilter.minDP share everything up to and including variant calling, and diverge at the filtering.
One thing switches it off entirely. A run that sets its own storageDir shares nothing, because sharing means writing one artifact into one results tree, and a run with its own tree has nowhere to put a shared one. It repeats every step alone. That is allowed and occasionally what you want; the report names any run in that position so it is not discovered from a task count.
Checked before any compute is spent¶
The table is read and validated at the start of the run, and an unusable one stops it outright. A usable one is printed in full: the runs, where each one's results will go, and what differs for each. A mistake in the table costs seconds rather than a night of alignment.
PoolSeqFlow dryrun goes further and creates the directory tree the run would write, empty, so the layout can be looked at and approved before anything is computed. PoolSeqFlow dryclean removes the preview.