Metadata¶

metadata.csv sits in mainDir and carries one row per pair of FASTQ files. It is the file that says what your samples are: which pool each belongs to, how many individuals went into that pool, and whatever else the experiment needs recorded.

Three of its columns change your results. The rest are yours, and the pipeline never interprets them.

SampleID,RG_Sample,RG_Library,RG_Platform,RG_PlatformUnit,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature,sequencing_run
Sample1T1Rep1,Sample1T1,Lib1,ILLUMINA,Unit1,50,Pop1,T1,susceptible,21.5,Run1
Sample1T1Rep2,Sample1T1,Lib1,ILLUMINA,Unit2,50,Pop1,T1,susceptible,21.5,Run2
Sample1T2Rep1,Sample1T2,Lib1,ILLUMINA,Unit1,50,Pop1,T2,resistant,22.1,Run1
Sample1T2Rep2,Sample1T2,Lib1,ILLUMINA,Unit2,50,Pop1,T2,resistant,22.1,Run2
Sample2T1Rep1,Sample2T1,Lib1,ILLUMINA,Unit1,40,Pop2,T1,susceptible,18.0,Run1
Sample2T1Rep2,Sample2T1,Lib1,ILLUMINA,Unit2,40,Pop2,T1,susceptible,18.0,Run2
Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,Unit1,40,Pop2,T2,resistant,19.4,Run1
Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,Unit2,40,Pop2,T2,resistant,19.4,Run2

Eight FASTQ pairs, four distinct RG_Sample values. This file produces a VCF with four columns: Sample1T1, Sample1T2, Sample2T1, Sample2T2. Each pool was sequenced twice, so its two rows share an RG_Sample and repeat everything that belongs to the pool.

Read the columns left to right and they group: what the sample is, what the pipeline should do with it, what was set (exp_), what was measured as the response (pt_), what was measured alongside (cov_), and last sequencing_run, which differs between the two rows of a pool, and so carries no prefix at all.

PoolSeqFlow init does not write this file, because its content is your experiment and a copied one would describe someone else's. It leaves metadata.csv.example beside you to write it from: the same table with every column explained in comments. Blank lines and lines beginning with # are ignored, so those comments can stay in the file you keep. A value containing a comma must be quoted: "Pop1, coastal".

The seven kinds of column¶

The name of a column is what decides how it is treated. There is no second schema to keep in step with it.

Column	What it is
`SampleID`	Required and unique. The whole name before the read tag, as delivered: any `_S<n>` or `_L00<n>` your sequencer added is part of it. Matched against the sample name `readPattern` takes from your filenames, and becomes the read group's `ID` in the BAM
`RG_*`	A read-group tag. Eight are known, listed below. A blank cell omits that tag rather than writing an empty one
`param_*`	A setting from `parameters.config`, overridden for these samples only. Four are known: `param_poolSize`, `param_capMaxDepth`, `param_adapter1`, `param_adapter2`. A blank cell means "use the global value"
`exp_*`	An experimental variable: something you set. Read by analysis modules and by no pipeline step. Any name you like after the prefix
`pt_*`	A phenotype: something you measured on the pool, and the thing being tested against. Also read only by analysis modules, and also open. See The phenotype
`cov_*`	A covariate: measured on the pool, but neither set nor the response. A cage temperature, an altitude, a collection site. See Covariates
anything else	Yours. A lane, a batch, a note, a measurement from another experiment; whatever this study needs recorded

RG_ and param_ are closed lists, and an unrecognized one is refused rather than ignored. For RG_ that stops a typo quietly losing a tag. For param_ the reason is sharper: a param_ column the pipeline did not recognize would be a setting you had written down, could see in your own file, and that was never applied to anything.

exp_, pt_ and cov_ are three prefixes because only exp_ says what the experiment set up. From your exp_ columns, the analysis layer works out which pools are independent of each other and, where there is a time course, which are one thing measured repeatedly. A trait value differs from pool to pool, and so does a cage temperature: either one recorded as an exp_ column would make every pool its own unit and split every series into single timepoints, quietly, because a design with no repeated measurements is a legal design. Keeping them apart is what stops that, structurally, rather than by remembering a setting.

All three describe the POOL. What differs between two rows of one pool (the lane, the run, the technician who handled one library) takes no prefix, and not because it does not matter. Once two libraries' reads are merged into one column, nothing downstream can attribute a read to the row it came from, so a row-level factor is not something the analysis layer has yet to support: it is unrecoverable. Record it unprefixed for the record. If every library of a pool does share one (one technician per pool), then it is pool-level after all, and it is a cov_.

Everything else is free. You can add, remove and edit your own columns (exp_ and pt_ ones included) without invalidating results you already have. The change guard does not look at them, so no analysis you have already published stops being valid because you named a variable better.

The experimental design¶

An exp_ column says what an experimental variable was for each pool: exp_population, exp_treatment, exp_generation. Analysis modules read them and the pipeline does not, so adding them changes nothing about your results and makes them describable.

Two conventions and one rule:

exp_time is time, and it is the one variable the analysis layer treats specially. See The time axis. A project that has an exp_time column must say how to read it or every analysis refuses.
The prefix is open. Unlike RG_ and param_ there is no list to check a name against, so exp_tiempoint is a new variable rather than an error. Nothing can catch that for you; the verification report prints the variables it found at the start of every analysis, which is where you will see it.
An exp_ column describes the POOL, not the row. Rows sharing an RG_Sample are one pool (their reads are merged and their depths added into one column of every published table), so they must give the same value for every exp_ column, and an analysis refuses a file where they disagree, naming the column, the pool and each value. A blank cell means no value, which is a third answer rather than agreement with either.

That last rule is the one that catches people, and the fix is always the same: what differs between two rows of one pool (the lane, the batch, the sequencing run, the extraction date) is not an experimental variable. Record it in a column with no prefix.

The refusal is a hard one, and it stops every analysis rather than only the ones that read a design. Each published analysis records the design the project was in at the time; a project whose design contradicts itself has nothing true to record.

The read-group tags¶

Column	SAM tag	Meaning
`RG_Sample`	`SM`	The pool. Decides VCF columns. See below
`RG_Library`	`LB`	Library identifier
`RG_Platform`	`PL`	Sequencing platform, e.g. `ILLUMINA`
`RG_PlatformUnit`	`PU`	Platform unit: flowcell, lane
`RG_Description`	`DS`	Free text
`RG_Center`	`CN`	Sequencing center
`RG_Date`	`DT`	Run date, ISO 8601, e.g. `2024-03-07`
`RG_FlowOrder`	`FO`	Flow order

You never write the two-letter tags yourself; the prefix is what marks a column as one.

Fill RG_Sample in on every row, even when it repeats SampleID. The two columns answer different questions and it is worth being deliberate about both:

	answers	comes from
`SampleID`	which pair of files is this row?	your sequencer. Whatever it wrote, in full
`RG_Sample`	what name do I want this under in my results?	you

RG_Sample is what names the VCF column, every frequency table header, and everything an analysis keys on.

The column is optional. However, if left out, it is filled from SampleID. Every such row becomes its own pool, named exactly what its files were named.

Filling it in with the value it was already defaulting to changes nothing and invalidates nothing: the run records the rows as resolved, so a project that adds the column and repeats SampleID in it passes the consistency guards untouched. Only a change that really does move the pooling stops a run, which is the point of them.

Every SampleID must appear exactly once. A row is looked up by it and only the first match would be read, so a repeat would quietly give a sample the wrong tags and produce a perfectly valid BAM that nothing downstream could flag. The run stops and lists the offending values, along with every other problem in the file, reported together with line numbers rather than one at a time.

Line endings are not your problem, and neither is the byte-order mark Excel adds when it saves as "CSV UTF-8". Both are read correctly as they stand, and nothing rewrites the file you wrote. What is not read is a file that is not comma-separated: a spreadsheet saved in a locale where the comma is the decimal mark writes semicolons instead, and the whole header then arrives as a single column. When a file cannot be used, the message lists the column names it actually read, which is what tells that apart from a misspelled heading.

When your file names carry a sequencer suffix¶

Illumina's demultiplexing adds a suffix you did not choose. bcl2fastq and BCL Convert append _S<n>, where n is the sample's row number in the sample sheet, so a sample you call A08781 is delivered as A08781_S58_R1.fq.gz. The sample name readPattern takes from that file is A08781_S58.

That is what SampleID has to say, because SampleID is what joins the row to the files. Put the name you actually use in RG_Sample:

SampleID,RG_Sample
A08781_S58,A08781

The suffix stops at the join. RG_Sample is what names the VCF column and what every result is keyed on, so _S58 never reaches a number you publish.

This is the same mechanism that makes lanes work. A sample split across two lanes arrives as A08781_S58_L001 and A08781_S58_L002, which are two rows sharing one RG_Sample and merging into one pool. A rule that stripped the suffix from SampleID would give both rows the same identifier and lose one of the pairs. That is why nothing here strips anything, and why carrying the suffix into SampleID is yours to do.

Both ways of getting this wrong are in Troubleshooting: a sample with reads but no row when SampleID is too short, and a pool named after your files when RG_Sample is blank.

To save typing the identifiers, list them from the files themselves rather than from the sample sheet:

find Data -name '*_R1.fq.gz' | sed 's|.*/||; s|_R1\.fq\.gz$||' | sort

That prints exactly the SampleID column for the default readPattern, in the form step 0 will check it against.

`RG_Sample` decides what counts as a sample¶

SampleID identifies each FASTQ pair, but RG_Sample determines the samples in your variant calls. BCFtools names VCF columns after the SM tag, and any read groups sharing a value are pooled into a single column.

`RG_Sample` values in `metadata.csv`	Resulting VCF columns
`Sample1`, `Sample2`, `Sample3`	`Sample1` `Sample2` `Sample3`
`Population1`, `Population1`, `Sample3`	`Population1` `Sample3`

Give every pool its own RG_Sample when you want them analyzed separately. This is what most runs want, it is the safe default, and it is what you get by leaving the column out, since it falls back to SampleID.

Share an RG_Sample deliberately when several FASTQ pairs are really the same biological pool:

One pool sequenced more than once: split across lanes or runs to reach the depth Pool-seq needs. Each run arrives as its own FASTQ pair, but they describe one set of individuals, and the allele frequencies are only correct once the reads are combined.
Technical replicates of the same library that you want treated as one observation rather than compared with each other.

Because merging happens at variant calling, it changes the numbers: read depths add together and each frequency is computed across the pooled reads. Leaving one pool split across two RG_Sample values instead gives you two under-powered estimates of the same thing, which is easy to do by accident, since the FASTQ files look like two ordinary samples.

The pooling that was worked out is printed before any compute is spent, so a mistake here is visible in seconds rather than in a result months later.

This interacts with the cross-sample filter

Merging changes the sample count, and the false-positive filter requires an allele to appear in a fraction of samples. Eight pairs as eight samples require two supporting samples; the same eight merged into four require one. Deciding RG_Sample is therefore also deciding how strict your filtering is. See The Filter Chain.

Pool size belongs to the pool¶

param_poolSize is how many individuals went into a pool, and it sets that column's detection limit:

\[S = \frac{1}{2 \times \text{ploidy} \times \text{poolSize}}\]

A pool of 10 and a pool of 500 have very different limits, so giving the whole run one number judges the small pool at the large one's resolution. The column exists so each pool can be judged at its own. See Where S comes from.

Because the size describes the pool rather than the row, two rules follow:

Rows sharing an RG_Sample must give the same value. They are one pool; they cannot have been made from two different numbers of individuals.
A blank cell means the global poolSize, not agreement. So a pool with one row filled in and one row blank is a disagreement, and is refused rather than resolved by guessing which was meant.

Leave the column out entirely and every pool uses the global poolSize from parameters.config, which is the right answer when your pools really are all the same size.

The threshold each column was filtered at is recorded in the filtered VCF's header. Those VCFs are intermediates and do not survive a completed run. The durable record is .poolseqflow_metadata, kept beside the results.

A depth ceiling belongs to the library¶

param_capMaxDepth overrides capBAM.maxDepth for one row. Unlike pool size, it is a property of that sequencing run rather than of the pool, so rows sharing an RG_Sample are free to differ: two libraries of one pool can have been amplified quite differently, and each gets its own ceiling.

It takes the same three values as the global setting (-1 to measure a ceiling from that sample's own coverage, a positive number to fix one, 0 to leave the sample uncapped), and a blank cell means the global capBAM.maxDepth. See Depth capping for what the measurement does and when it declines to act.

Reach for this column when you have read a sample's depth report and disagree with it. The usual case is a library where the deep population is too large for the detector to call an artefact, so it reports the sample uncapped: if you know that population is a PCR hill rather than real coverage, pin the ceiling here.

Adapter overrides¶

param_adapter1 and param_adapter2 override the run's adapter settings for one row. Give both or neither. They are covered with the rest of trimming in Per-sample adapters.

Row order decides column order¶

The order of the rows in metadata.csv is the order of the sample columns in the VCF and in the frequency tables. Put the rows in whatever order you want to read your results in (treatment before control, or by time point), and the output follows.

SampleID,RG_Sample,RG_Library,RG_Platform
Sample3,Sample3,Lib1,ILLUMINA          # -> first column
Sample1,Sample1,Lib1,ILLUMINA          # -> second column
Sample2,Sample2,Lib1,ILLUMINA          # -> third column

When several rows share an RG_Sample, the merged column appears where the first of those rows sits.

Reordering rows only moves columns; it never changes a value. Nothing else about the file is positional.

This exists because the alternative is worse. collect() alone emits BAMs in task-completion order, so whichever sample finished first landed first on the bcftools command line; three consecutive runs on identical input gave three different column orders. Sorting on the file path is no better, since Nextflow's paths begin with a random work-directory hash. Row order is the only ordering that is both stable and meaningful.

Editing `metadata.csv` after a run¶

Completed steps are skipped by looking for their output files, not by checking what produced them. So once this file has been consumed, editing it does not update anything that already exists: the tags are inside the BAMs, the column order is inside the VCF, and the pool sizes are inside the filter that produced the tables.

The analysis-affecting part of the file is recorded the first time it is used and compared on every later run. If it has changed, the run stops before any work happens and names what is now stale. How much that is depends on what you changed:

What you changed	What it invalidates	Delete and rerun
A read-group tag value	The BAMs, and everything called from them	`Output/Ready/`, `Output/VCF/`, `Output/Frequencies/`
Row order only	The VCF sample column order	`Output/VCF/`, `Output/Frequencies/`
`param_poolSize` only	The false-positive filter's output, and nothing before it	`Output/Frequencies/` and the filtered VCFs
One of your own columns	Nothing	Nothing

The pool-size row is the one worth noticing. A changed pool size moves a threshold inside step 7, and step 7's input is the called VCF, so the BAMs and the call set are still valid and you are not asked to delete them. Re-running redoes the filtering and the tables, which is minutes rather than hours.

Deleting what is named is what clears the check; the edit becomes the new baseline on the next run. Or discard everything and start over with PoolSeqFlow reset.

The record is .poolseqflow_metadata, kept beside the results it describes. It holds only the read-group tags, the pool sizes and the row order. Your own columns are never recorded, which is why the last row of that table says nothing. Line endings and trailing whitespace are ignored when comparing; row order is not.

Checklist¶

Before your first run:

[ ] One row per FASTQ pair, no SampleID repeated
[ ] Every SampleID matches a sample name readPattern will find
[ ] RG_Sample shared only where pairs are genuinely the same pool
[ ] param_poolSize the same on every row of a pool, or absent everywhere
[ ] Rows in the order you want your result columns
[ ] Any column of your own you will want later (adding them now costs nothing, and adding them afterwards costs nothing either)