Quick Start¶
This page takes you from an installed PoolSeqFlow to a running pipeline. It assumes only that Install is done; everything to do with your own data starts here.
1. Make your project¶
A project is a directory of your own. Do not put any project into installation directory. The pipeline refuses to start if you point it at the installation.
The paths are yours to choose; the structure inside them is not. A project ready to run looks like this (init makes everything except the reads, the reference and metadata.csv):
/path/to/project/ ← mainDir, and the directory you run from
├── parameters.config ← your settings, read from wherever you launch
├── metadata.csv ← what your samples are; you write this one
├── metadata.csv.example ← every column explained, to write it from
├── Data/ ← dataSource names this folder
│ ├── Sample1_R1.fq.gz
│ ├── Sample1_R2.fq.gz
│ ├── Sample2_R1.fq.gz
│ ├── Sample2_R2.fq.gz
│ └── …
└── Reference/
├── reference.fasta.gz ← referenceFile
└── reference.gff.gz ← gffFile, only if annotate = true
The file names are examples. readPattern is what finds the reads (*_R{1,2}.fq.gz by default, which is what matches the pairs above), and referenceFile and gffFile name the two in Reference/.
The reads may sit in subfolders of Data/ instead, or as well. However your sequencing facility handed them over is how you can leave them: one folder per sample, one per run or lane, nested as deep as you like, or all together as above. All three of these are read the same way:
Data/ Data/ Data/
├── Sample1_R1.fq.gz ├── Sample1/ ├── run_A/
├── Sample1_R2.fq.gz │ ├── Sample1_R1.fq.gz │ ├── Sample1_R1.fq.gz
├── Sample2_R1.fq.gz │ └── Sample1_R2.fq.gz │ └── Sample1_R2.fq.gz
└── Sample2_R2.fq.gz └── Sample2/ └── run_B/
├── Sample2_R1.fq.gz ├── Sample2_R1.fq.gz
└── Sample2_R2.fq.gz └── Sample2_R2.fq.gz
A sample is named by its file, never by its folder. Sample1_R1.fq.gz is sample Sample1 wherever it lies, so rearranging the folders changes neither metadata.csv nor readPattern. A pair whose mates sit in two different folders is still one pair.
What is checked is the pair: every sample has exactly one of each mate. Three ways to break it, none of which would announce itself:
in Data/ |
what it is |
|---|---|
Sample1_R1.fq.gz with no Sample1_R2.fq.gz anywhere |
a sample left out of the run |
run_A/Sample1_R1.fq.gz and run_B/Sample1_R1.fq.gz |
the same mate twice, aligned against itself |
a full pair under both run_A/ and run_B/ |
one sample over again, each copy overwriting the other's results |
Two folders holding two different samples need two names. One sample sequenced twice is two rows in metadata.csv sharing an RG_Sample; see What a unit is.
Hidden folders are skipped, and the run names any that held reads. Nothing beginning with a dot is searched: .snapshot on NetApp storage holds a copy of every file per snapshot, and searching it would find every sample many times over.
Three things are then yours to fill in:
- Edit
parameters.config:mainDir,storageDir,readPattern,referenceFile,poolSizeandploidyat minimum. The reference and the annotation may be gzipped or plain; the pipeline unpacks what it needs intoReference/Dictionaries/. - Point
storageDirat a different directory. Finished results go there rather than into the project. See Requirements. - Write
metadata.csvyourself.initdoes not, because a copied table would describe someone else's experiment. Start from themetadata.csv.exampleit left beside you, which explains every column, and read Metadata before your first run rather than after it.
init never overwrites: run it again in a project you have filled in and it reports what is there and changes nothing.
Analyzing one set of reads under several parameter sets, two reference genomes say? PoolSeqFlow init multi does everything init does, switches multiRun on, and copies multi-run.csv.example in. The run table is yours to write, for the same reason metadata.csv is.
2. Verify it any time¶
There are two types of checks:
PoolSeqFlow check install # the tools and helpers this release is built to run
PoolSeqFlow check project # the configuration and the commands this project names
Both end by reporting how many checks passed, and fail loudly if any did not rather than summarizing.
check install¶
Tools
from ~/.local/opt/miniconda3/envs/PoolSeqFlow-<version>
nextflow nextflow OK 26.04.6 build 12646
cutadapt cutadapt OK 5.2
…
samtools samtools OK samtools 1.24
bcftools bcftools OK bcftools 1.24
…
Pipeline helpers
atomic_mv.sh OK
depth2freq.awk OK
…
Every command the release is built to run, with the version each reports, and every helper in bin/, present and executable; a lost executable bit would otherwise fail mid-run rather than at startup.
Each tool has to come from that environment. All of them are pinned in install/environment.yml, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in, at whatever version it happens to be and only on this machine:
That is the quiet failure worth catching: the pipeline runs, the results look fine, and nothing reproduces anywhere else. Reinstalling with PoolSeqFlow install is the fix.
It reads no parameters.config and needs no project. Run it from anywhere, including straight after installing and before you have a project at all.
check project¶
Run it from your project directory.
Configuration
parameters.config WRITTEN FOR THIS RELEASE
parameters.config PARSES
metadata.csv PARSES
runs.csv not in use
parameters NOTHING TO FLAG
Tools, as this project configures them
nextflow nextflow OK 26.04.6 build 12646
samtools /usr/bin/samtools OK samtools 1.19
bcftools bcftools OK bcftools 1.24
…
tool list from: params.software in parameters.config
Your files parse: parameters.config through Nextflow itself, metadata.csv and the run table through the same parsers a run uses. It also reports whether the config was written for this release, which is the one thing that stops a run before anything else is read.
Your settings are checked by the same rules a run applies, so a project is never told one thing here and another during the run. What is checked lists them. NOTHING TO FLAG is the healthy answer and the common one: a finding appears only when a setting would make the run produce nothing or stop it partway, or would quietly change what a published number means.
Every command, as you configured it. The list comes from params.software, so a command repointed at a system binary is checked the way the run will call it, and the second column shows what will actually be invoked. That override is the setting most likely to be wrong and least likely to announce itself. If params.software cannot be read, no tool is checked and it says so.
A path outside the environment is not a finding here, where check install treats one as a fault. Repointing a command is yours to do, so this check reports what will be invoked and whether it runs, and leaves the judgment to you.
3. Fill in the essentials¶
Open parameters.config. Nine settings need your attention before a first run; everything else has a working default.
params {
mainDir = "/path/to/working/directory" // where you run, and where your inputs live
storageDir = "/path/to/permanent/storage" // where finished results are kept
dataSource = 'Data' // subdirectory of mainDir holding the FASTQs
readPattern = "*_R{1,2}.fq.gz" // glob matching your paired FASTQ files
referenceFile = 'reference.fasta.gz' // reference genome, in mainDir/Reference
gffFile = 'reference.gff.gz' // annotation (only if annotate = true)
poolSize = 50 // individuals per pool
ploidy = 2 // copies of the genome per individual
annotate = true // run SnpEff annotation (step 8)
}
| Setting | How to choose it |
|---|---|
mainDir |
The working directory you ran init in. Holds Data/, Reference/, your configuration and everything actively processed, so it has to persist between runs (not node scratch that is wiped between jobs). Put it on the fastest storage that satisfies that. |
storageDir |
Where finished results are kept. A different directory from mainDir, and the pipeline refuses to start otherwise: the two are storage tiers, and outputs move from the first to the second as each step that needs them completes. |
readPattern |
Must match both mates with a {1,2} group. If your files end _1.fastq.gz/_2.fastq.gz, write "*_{1,2}.fastq.gz". |
poolSize |
Individuals in one pool, not the total across pools. Sets the smallest allele frequency worth believing (see The Filter Chain). If your pools differ in size, give each its own value in metadata.csv; this setting is the default for any that do not. |
ploidy |
Ploidy of the organism: 2 for diploid, 1 for haploid, 4 for tetraploid. |
annotate |
false skips step 8 and makes gffFile unnecessary. |
Set these through the file, never the command line
PoolSeqFlow rejects command-line parameter overrides. Nextflow delivers --param values as strings, so --annotate false sets the string "false", which Groovy evaluates as true, leaving annotation switched on with no warning. Why →
4. Size the run to your machine¶
Set threads to the cores you actually have (on HPC, the size of one node). Every tool's thread count follows from this one number; do not set the per-tool counts by hand. A request larger than the machine fails immediately:
Details and the full ladder: Resources.
5. Write metadata.csv¶
One row per FASTQ pair. SampleID is the whole name before the read tag, exactly as your sequencer delivered it: with *_R{1,2}.fq.gz, the file Sample1T1_S19_L001_R1.fq.gz gives Sample1T1_S19_L001. Do not shorten it, because it is what finds your files. The name you want your results under goes in RG_Sample, and you should always write that one out.
SampleID,RG_Sample,RG_Library,RG_Platform,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature
Sample1T1Rep1,Sample1T1,Lib1,ILLUMINA,50,Pop1,T1,susceptible,21.5
Sample1T1Rep2,Sample1T1,Lib1,ILLUMINA,50,Pop1,T1,susceptible,21.5
Sample1T2Rep1,Sample1T2,Lib1,ILLUMINA,50,Pop1,T2,resistant,22.1
Sample1T2Rep2,Sample1T2,Lib1,ILLUMINA,50,Pop1,T2,resistant,22.1
Sample2T1Rep1,Sample2T1,Lib1,ILLUMINA,40,Pop2,T1,susceptible,18.0
Sample2T1Rep2,Sample2T1,Lib1,ILLUMINA,40,Pop2,T1,susceptible,18.0
Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,40,Pop2,T2,resistant,19.4
Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,40,Pop2,T2,resistant,19.4
Three things this file decides:
RG_Sampledecides what counts as a sample. Rows sharing one are merged into a single VCF column and their read depths add together. The eight rows above produce four columns, not eight: two populations at two timepoints, each sequenced twice.param_poolSizesets that pool's detection limit. It describes the pool rather than the row, so rows sharing anRG_Samplehave to agree on it. Leave the column out and every pool uses the globalpoolSize.- Row order decides column order in the VCF and the frequency tables.
The prefix is what a column means. No pipeline step reads any of the three below; analysis modules do, and they build their whole design from them:
| Prefix | What it holds | Example above |
|---|---|---|
exp_ |
something you set: the experiment's own structure | exp_population, exp_time |
pt_ |
a phenotype you measured on the pool, and are testing against | pt_resistance |
cov_ |
measured too, but neither set nor the thing being tested | cov_temperature |
- Add as many of each as your experiment needs; the names after the prefix are yours.
- A column with no prefix is recorded just as faithfully and read by nothing, so a variable you might analyze later is worth prefixing now.
- Editing any of them never invalidates results you already have.
- All three describe the pool. What differs between two rows of one pool takes no prefix, because once their reads are merged nothing can tell them apart again.
exp_time is the one name treated specially. A project that has it must set analysis.timeVar.kind before any analysis will run; the pipeline itself does not care. Leave the column out if this is not a time course. See The time axis.
Metadata has the full account, and getting RG_Sample wrong is the most common way to end up with results that are valid but not what you meant.
6. Run¶
That is also the resume command. Every step checks whether its outputs already exist and skips itself if they do, so an interrupted run picks up where it left off with no extra flag. There is no -resume.
Run it from your project directory, the one holding parameters.config: that is where the settings are read from, and anywhere else stops with a message saying so.
To see where a run's results would go before spending any compute on it, PoolSeqFlow dryrun builds that directory tree empty and changes nothing else; PoolSeqFlow dryclean removes the preview.
To start genuinely from scratch, use PoolSeqFlow reset first. It requires typing DELETE_MY_ANALYSIS to confirm.
7. Check the output¶
storageDir/
├── Logs/
└── Output/
├── Frequencies/ ← the result: <name>_snp_freq.tsv, <name>_indel_freq.tsv
├── VCF/
├── Ready/ ← cleaned, indexed BAMs
├── Reports/
├── run_parameters.txt ← the settings these results were produced under
└── …
<name> is vcf.fileName, which follows outputPrefix, so it is Test until you change either.
Start with Output/Reports/Depth/ and Output/run_parameters.txt. The first says what depth ceiling each sample was given and why; the second is a read-only record of exactly which settings produced these files.
Your inputs stay where they were, under mainDir. Data/, Reference/ and the dictionaries built from your reference are not copied here.
If you are running a table of several runs, Output/ and Logs/ gain a level: work every run shared sits under All_Runs/, work some of them shared under Shared_<N>/, and whatever one run did alone under its own RunID. A single run keeps the plain tree above.
How to read the tables: Interpreting Results.
Commands¶
| Command | Description |
|---|---|
PoolSeqFlow install |
Create this release's conda environment, install the pipeline, then verify both |
PoolSeqFlow init |
Populate the current directory as a project (what it writes) |
PoolSeqFlow init multi |
The same, for a project running several parameter sets over one set of reads |
PoolSeqFlow check install |
Verify an installation: the tools and helpers it is built to run (what it covers) |
PoolSeqFlow check project |
Verify a project: its configuration, and the commands it names (what it covers) |
PoolSeqFlow run |
Start (or resume) the pipeline |
PoolSeqFlow dryrun |
Create the directory tree the run would write, empty, so the layout can be approved before any compute is spent. Records nothing and changes none of your files |
PoolSeqFlow dryclean |
Remove the preview dryrun made |
PoolSeqFlow migrate_config |
Carry an older parameters.config onto the current template (details) |
PoolSeqFlow clean |
Remove Nextflow work directories |
PoolSeqFlow reset |
Remove all progress and start fresh (requires typed confirmation) |
PoolSeqFlow analysis <command> |
The analysis layer, which carries a word of its own (listed below) |
PoolSeqFlow version |
Print the version of this copy |
PoolSeqFlow cite |
Print how to cite this copy, and which DOI to use (why it matters) |
PoolSeqFlow list |
List the pipelines and conda environments installed on this machine |
PoolSeqFlow uninstall |
Remove one installed version, environment and pipeline together, after confirmation |
PoolSeqFlow uninstall all |
Remove every PoolSeqFlow environment and installation, after confirmation |
Before anything is installed there is no PoolSeqFlow on your PATH, so the first command is ./PoolSeqFlow install, run from the folder you downloaded. Everything after that uses the installed command.
Each subcommand takes no arguments of its own, with one exception: analysis carries a word (the analysis command, or the module to run) and analysis modules carries a second. Anything else is rejected. Two environment variables adjust the wrapper instead: POOLSEQFLOW_PREFIX, where install puts things and where list and uninstall look, and POOLSEQFLOW_HOME, to run a checkout without installing it.
Naming a version. Every installed release is also on your PATH under its own name, so PoolSeqFlow-2.1.0 run uses that release and plain PoolSeqFlow uses the newest. PoolSeqFlow list shows what is installed, marks a version whose analysis layer is installed, and includes the unversioned environment if a release from before per-version environments left one behind:
Installed under /home/you/.local:
1) PoolSeqFlow (unversioned - predates per-version environments)
2) PoolSeqFlow-2.1.0
3) PoolSeqFlow-<version> (this wrapper, analysis installed)
Naming a version matters most when removing one; see Uninstalling.
The analysis layer hangs off analysis, the one subcommand that carries a word of its own. It ships with the same PoolSeqFlow install and is versioned with it:
| Command | Description |
|---|---|
PoolSeqFlow analysis install |
Create this release's analysis conda environment, which carries R, then verify it |
PoolSeqFlow analysis check |
Verify an existing analysis installation: tools, R packages, entry point |
PoolSeqFlow analysis modules available |
List the modules published for this release's table contract |
PoolSeqFlow analysis modules install <module> [version] |
Install one, newest readable version unless you name it. Also installs what it runs on |
PoolSeqFlow analysis modules install all |
Install every module published for this release that is not installed yet. One that cannot be installed is named, and the rest still are |
PoolSeqFlow analysis modules list |
List the modules installed for this release, and name any that are broken |
PoolSeqFlow analysis modules uninstall <module> |
Remove one module, after confirmation. Results it produced are untouched |
PoolSeqFlow analysis complete |
Move finished analyses and shared intermediates from mainDir/Analysis/ to storageDir/Analysis/, after confirmation. A name already taken in permanent storage stops it with nothing moved |
PoolSeqFlow analysis version |
Print the version of this copy |
PoolSeqFlow analysis cite |
Print how to cite this copy, plus R and every package a module runs on |
PoolSeqFlow analysis uninstall |
Remove the analysis environment, after confirmation. The pipeline, its environment, and your results are untouched |
The analysis environment is separate from the pipeline's and is built only when you ask for it. analysis cite reads R's own citation records, so it prints the version of each package actually installed.
PoolSeqFlow analysis also accepts the name of an installed module. Modules are installed separately and each is documented with itself; the table above is what a release ships with.