Metadata¶

metadata.csv sits in mainDir and carries one row per pair of FASTQ files. It is the file that says what your samples are: which pool each belongs to, how many individuals went into that pool, and whatever else the experiment needs recorded.

Three of its columns change your results. The rest are yours, and the pipeline never interprets them.

SampleID,RG_Sample,RG_Library,RG_Platform,RG_PlatformUnit,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature,sequencing_run
Sample1T1Rep1,Sample1T1,Lib1,ILLUMINA,Unit1,50,Pop1,T1,susceptible,21.5,Run1
Sample1T1Rep2,Sample1T1,Lib1,ILLUMINA,Unit2,50,Pop1,T1,susceptible,21.5,Run2
Sample1T2Rep1,Sample1T2,Lib1,ILLUMINA,Unit1,50,Pop1,T2,resistant,22.1,Run1
Sample1T2Rep2,Sample1T2,Lib1,ILLUMINA,Unit2,50,Pop1,T2,resistant,22.1,Run2
Sample2T1Rep1,Sample2T1,Lib1,ILLUMINA,Unit1,40,Pop2,T1,susceptible,18.0,Run1
Sample2T1Rep2,Sample2T1,Lib1,ILLUMINA,Unit2,40,Pop2,T1,susceptible,18.0,Run2
Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,Unit1,40,Pop2,T2,resistant,19.4,Run1
Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,Unit2,40,Pop2,T2,resistant,19.4,Run2

Eight FASTQ pairs, four distinct RG_Sample values — this file produces a VCF with four columns: Sample1T1, Sample1T2, Sample2T1, Sample2T2. Each pool was sequenced twice, so its two rows share an RG_Sample and repeat everything that belongs to the pool.

Read the columns left to right and they group: what the sample is, what the pipeline should do with it, what was set (exp_), what was measured as the response (pt_), what was measured alongside (cov_), and last sequencing_run — which differs between the two rows of a pool, and so carries no prefix at all.

PoolSeqFlow init does not write this file, because its content is your experiment and a copied one would describe someone else's. It leaves metadata.csv.example beside you to write it from — the same table with every column explained in comments. Blank lines and lines beginning with # are ignored, so those comments can stay in the file you keep. A value containing a comma must be quoted: "Pop1, coastal".

The seven kinds of column¶

The name of a column is what decides how it is treated. There is no second schema to keep in step with it.

Column	What it is
`SampleID`	Required and unique. Matched against the sample name `readPattern` takes from your FASTQ filenames, and becomes the read group's `ID` in the BAM
`RG_*`	A read-group tag. Eight are known, listed below. A blank cell omits that tag rather than writing an empty one
`param_*`	A setting from `parameters.config`, overridden for these samples only. Four are known: `param_poolSize`, `param_capMaxDepth`, `param_adapter1`, `param_adapter2`. A blank cell means "use the global value"
`exp_*`	An experimental variable — something you set. Read by analysis modules and by no pipeline step. Any name you like after the prefix
`pt_*`	A phenotype — something you measured on the pool, and the thing being tested against. Also read only by analysis modules, and also open. See The phenotype
`cov_*`	A covariate — measured on the pool, but neither set nor the response. A cage temperature, an altitude, a collection site. See Covariates
anything else	Yours. A lane, a batch, a note, a measurement from another experiment — whatever this study needs recorded

RG_ and param_ are closed lists, and an unrecognized one is refused rather than ignored. For RG_ that stops a typo quietly losing a tag. For param_ the reason is sharper: a param_ column the pipeline did not recognize would be a setting you had written down, could see in your own file, and that was never applied to anything.

exp_, pt_ and cov_ are three prefixes because only exp_ says what the experiment set up. The analysis layer works out which pools are independent of each other — and, where there is a time course, which are one thing measured repeatedly — from your exp_ columns. A trait value differs from pool to pool, and so does a cage temperature: either one recorded as an exp_ column would make every pool its own unit and split every series into single timepoints, quietly, because a design with no repeated measurements is a legal design. Keeping them apart is what stops that, structurally, rather than by remembering a setting.

All three describe the POOL. What differs between two rows of one pool — the lane, the run, the technician who handled one library — takes no prefix, and not because it does not matter. Once two libraries' reads are merged into one column, nothing downstream can attribute a read to the row it came from, so a row-level factor is not something the analysis layer has yet to support: it is unrecoverable. Record it unprefixed for the record. If every library of a pool does share one — one technician per pool — then it is pool-level after all, and it is a cov_.

Everything else is free. You can add, remove and edit your own columns — exp_ and pt_ ones included — without invalidating results you already have. The change guard does not look at them, so no analysis you have already published stops being valid because you named a variable better.

The experimental design¶

An exp_ column says what an experimental variable was for each pool: exp_population, exp_treatment, exp_generation. Analysis modules read them and the pipeline does not, so adding them changes nothing about your results and makes them describable.

Two conventions and one rule:

exp_time is time, and it is the one variable the analysis layer treats specially — see The time axis. A project that has an exp_time column must say how to read it or every analysis refuses.
The prefix is open. Unlike RG_ and param_ there is no list to check a name against, so exp_tiempoint is a new variable rather than an error. Nothing can catch that for you; the verification report prints the variables it found at the start of every analysis, which is where you will see it.
An exp_ column describes the POOL, not the row. Rows sharing an RG_Sample are one pool — their reads are merged and their depths added into one column of every published table — so they must give the same value for every exp_ column, and an analysis refuses a file where they disagree, naming the column, the pool and each value. A blank cell means no value, which is a third answer rather than agreement with either.

That last rule is the one that catches people, and the fix is always the same: what differs between two rows of one pool — the lane, the batch, the sequencing run, the extraction date — is not an experimental variable. Record it in a column with no prefix.

The refusal is a hard one, and it stops every analysis rather than only the ones that read a design. Each published analysis records the design the project was in at the time; a project whose design contradicts itself has nothing true to record.

The read-group tags¶

Column	SAM tag	Meaning
`RG_Sample`	`SM`	The pool. Decides VCF columns — see below
`RG_Library`	`LB`	Library identifier
`RG_Platform`	`PL`	Sequencing platform, e.g. `ILLUMINA`
`RG_PlatformUnit`	`PU`	Platform unit — flowcell, lane
`RG_Description`	`DS`	Free text
`RG_Center`	`CN`	Sequencing center
`RG_Date`	`DT`	Run date, ISO 8601, e.g. `2024-03-07`
`RG_FlowOrder`	`FO`	Flow order

You never write the two-letter tags yourself; the prefix is what marks a column as one. RG_Sample is optional and defaults to SampleID, which makes every row its own pool — what you want when each sample was sequenced once.

Every SampleID must appear exactly once. A row is looked up by it and only the first match would be read, so a repeat would quietly give a sample the wrong tags and produce a perfectly valid BAM that nothing downstream could flag. The run stops and lists the offending values — along with every other problem in the file, reported together with line numbers rather than one at a time.

Line endings are not your problem. The file is parsed as CSV, so a file saved from Excel on Windows is read correctly as it stands, and nothing rewrites the file you wrote.

`RG_Sample` decides what counts as a sample¶

SampleID identifies each FASTQ pair, but RG_Sample determines the samples in your variant calls. BCFtools names VCF columns after the SM tag, and any read groups sharing a value are pooled into a single column.

`RG_Sample` values in `metadata.csv`	Resulting VCF columns
`Sample1`, `Sample2`, `Sample3`	`Sample1` `Sample2` `Sample3`
`Population1`, `Population1`, `Sample3`	`Population1` `Sample3`

Give every pool its own RG_Sample when you want them analyzed separately. This is what most runs want, and it is the safe default — and it is what you get by leaving the column out, since it falls back to SampleID.

Share an RG_Sample deliberately when several FASTQ pairs are really the same biological pool:

One pool sequenced more than once — split across lanes or runs to reach the depth Pool-seq needs. Each run arrives as its own FASTQ pair, but they describe one set of individuals, and the allele frequencies are only correct once the reads are combined.
Technical replicates of the same library that you want treated as one observation rather than compared with each other.

Because merging happens at variant calling, it changes the numbers: read depths add together and each frequency is computed across the pooled reads. Leaving one pool split across two RG_Sample values instead gives you two under-powered estimates of the same thing — which is easy to do by accident, since the FASTQ files look like two ordinary samples.

The pooling that was worked out is printed before any compute is spent, so a mistake here is visible in seconds rather than in a result months later.

This interacts with the cross-sample filter

Merging changes the sample count, and the false-positive filter requires an allele to appear in a fraction of samples. Eight pairs as eight samples require two supporting samples; the same eight merged into four require one. Deciding RG_Sample is therefore also deciding how strict your filtering is — see The Filter Chain.

Pool size belongs to the pool¶

param_poolSize is how many individuals went into a pool, and it sets that column's detection limit:

\[S = \frac{1}{2 \times \text{ploidy} \times \text{poolSize}}\]

A pool of 10 and a pool of 500 have very different limits, so giving the whole run one number judges the small pool at the large one's resolution. The column exists so each pool can be judged at its own — see Where S comes from.

Because the size describes the pool rather than the row, two rules follow:

Rows sharing an RG_Sample must give the same value. They are one pool; they cannot have been made from two different numbers of individuals.
A blank cell means the global poolSize, not agreement. So a pool with one row filled in and one row blank is a disagreement, and is refused rather than resolved by guessing which was meant.

Leave the column out entirely and every pool uses the global poolSize from parameters.config, which is the right answer when your pools really are all the same size.

The threshold each column was filtered at is recorded in the filtered VCF's header. Those VCFs are intermediates and do not survive a completed run — the durable record is .poolseqflow_metadata, kept beside the results.

A depth ceiling belongs to the library¶

param_capMaxDepth overrides capBAM.maxDepth for one row. Unlike pool size, it is a property of that sequencing run rather than of the pool, so rows sharing an RG_Sample are free to differ: two libraries of one pool can have been amplified quite differently, and each gets its own ceiling.

It takes the same three values as the global setting — -1 to measure a ceiling from that sample's own coverage, a positive number to fix one, 0 to leave the sample uncapped — and a blank cell means the global capBAM.maxDepth. See Depth capping for what the measurement does and when it declines to act.

Reach for this column when you have read a sample's depth report and disagree with it. The usual case is a library where the deep population is too large for the detector to call an artefact, so it reports the sample uncapped: if you know that population is a PCR hill rather than real coverage, pin the ceiling here.

Adapter overrides¶

param_adapter1 and param_adapter2 override the run's adapter settings for one row. Give both or neither. They are covered with the rest of trimming in Per-sample adapters.

Row order decides column order¶

The order of the rows in metadata.csv is the order of the sample columns in the VCF and in the frequency tables. Put the rows in whatever order you want to read your results in — treatment before control, or by time point — and the output follows.

SampleID,RG_Sample,RG_Library,RG_Platform
Sample3,Sample3,Lib1,ILLUMINA          # -> first column
Sample1,Sample1,Lib1,ILLUMINA          # -> second column
Sample2,Sample2,Lib1,ILLUMINA          # -> third column

When several rows share an RG_Sample, the merged column appears where the first of those rows sits.

Reordering rows only moves columns; it never changes a value. Nothing else about the file is positional.

This exists because the alternative is worse. collect() alone emits BAMs in task-completion order, so whichever sample finished first landed first on the bcftools command line — and three consecutive runs on identical input gave three different column orders. Sorting on the file path is no better, since Nextflow's paths begin with a random work-directory hash. Row order is the only ordering that is both stable and meaningful.

Editing `metadata.csv` after a run¶

Completed steps are skipped by looking for their output files, not by checking what produced them. So once this file has been consumed, editing it does not update anything that already exists — the tags are inside the BAMs, the column order is inside the VCF, and the pool sizes are inside the filter that produced the tables.

The analysis-affecting part of the file is recorded the first time it is used and compared on every later run. If it has changed, the run stops before any work happens and names what is now stale. How much that is depends on what you changed:

What you changed	What it invalidates	Delete and rerun
A read-group tag value	The BAMs, and everything called from them	`Output/Ready/`, `Output/VCF/`, `Output/Frequencies/`
Row order only	The VCF sample column order	`Output/VCF/`, `Output/Frequencies/`
`param_poolSize` only	The false-positive filter's output, and nothing before it	`Output/Frequencies/` and the filtered VCFs
One of your own columns	Nothing	—

The pool-size row is the one worth noticing. A changed pool size moves a threshold inside step 7, and step 7's input is the called VCF — so the BAMs and the call set are still valid and you are not asked to delete them. Re-running redoes the filtering and the tables, which is minutes rather than hours.

Deleting what is named is what clears the check; the edit becomes the new baseline on the next run. Or discard everything and start over with PoolSeqFlow reset.

The record is .poolseqflow_metadata, kept beside the results it describes. It holds only the read-group tags, the pool sizes and the row order — never your own columns, which is why the last row of that table says nothing. Line endings and trailing whitespace are ignored when comparing; row order is not.

Checklist¶

Before your first run:

[ ] One row per FASTQ pair, no SampleID repeated
[ ] Every SampleID matches a sample name readPattern will find
[ ] RG_Sample shared only where pairs are genuinely the same pool
[ ] param_poolSize the same on every row of a pool, or absent everywhere
[ ] Rows in the order you want your result columns
[ ] Any column of your own you will want later — adding them now costs nothing, and adding them afterwards costs nothing either