Skip to content

Quick Start

This page takes you from an installed PoolSeqFlow to a running pipeline. It assumes only that Install is done; everything to do with your own data starts here.

1. Make your project

A project is a directory of your own. Do not put any project into installation directory. The pipeline refuses to start if you point it at the installation.

mkdir -p /path/to/project
cd /path/to/project
PoolSeqFlow init

The paths are yours to choose; the structure inside them is not. A project ready to run looks like this (init makes everything except the reads, the reference and metadata.csv):

/path/to/project/                ← mainDir, and the directory you run from
├── parameters.config            ← your settings, read from wherever you launch
├── metadata.csv                 ← what your samples are; you write this one
├── metadata.csv.example         ← every column explained, to write it from
├── Data/                        ← dataSource names this folder
│   ├── Sample1_R1.fq.gz
│   ├── Sample1_R2.fq.gz
│   ├── Sample2_R1.fq.gz
│   ├── Sample2_R2.fq.gz
│   └── …
└── Reference/
    ├── reference.fasta.gz       ← referenceFile
    └── reference.gff.gz         ← gffFile, only if annotate = true

The file names are examples. readPattern is what finds the reads (*_R{1,2}.fq.gz by default, which is what matches the pairs above), and referenceFile and gffFile name the two in Reference/.

The reads may sit in subfolders of Data/ instead, or as well. However your sequencing facility handed them over is how you can leave them: one folder per sample, one per run or lane, nested as deep as you like, or all together as above. All three of these are read the same way:

Data/                    Data/                      Data/
├── Sample1_R1.fq.gz     ├── Sample1/               ├── run_A/
├── Sample1_R2.fq.gz     │   ├── Sample1_R1.fq.gz   │   ├── Sample1_R1.fq.gz
├── Sample2_R1.fq.gz     │   └── Sample1_R2.fq.gz   │   └── Sample1_R2.fq.gz
└── Sample2_R2.fq.gz     └── Sample2/               └── run_B/
                             ├── Sample2_R1.fq.gz       ├── Sample2_R1.fq.gz
                             └── Sample2_R2.fq.gz       └── Sample2_R2.fq.gz

A sample is named by its file, never by its folder. Sample1_R1.fq.gz is sample Sample1 wherever it lies, so rearranging the folders changes neither metadata.csv nor readPattern. A pair whose mates sit in two different folders is still one pair.

What is checked is the pair: every sample has exactly one of each mate. Three ways to break it, none of which would announce itself:

in Data/ what it is
Sample1_R1.fq.gz with no Sample1_R2.fq.gz anywhere a sample left out of the run
run_A/Sample1_R1.fq.gz and run_B/Sample1_R1.fq.gz the same mate twice, aligned against itself
a full pair under both run_A/ and run_B/ one sample over again, each copy overwriting the other's results

Two folders holding two different samples need two names. One sample sequenced twice is two rows in metadata.csv sharing an RG_Sample; see What a unit is.

Hidden folders are skipped, and the run names any that held reads. Nothing beginning with a dot is searched: .snapshot on NetApp storage holds a copy of every file per snapshot, and searching it would find every sample many times over.

Three things are then yours to fill in:

  • Edit parameters.config: mainDir, storageDir, readPattern, referenceFile, poolSize and ploidy at minimum. The reference and the annotation may be gzipped or plain; the pipeline unpacks what it needs into Reference/Dictionaries/.
  • Point storageDir at a different directory. Finished results go there rather than into the project. See Requirements.
  • Write metadata.csv yourself. init does not, because a copied table would describe someone else's experiment. Start from the metadata.csv.example it left beside you, which explains every column, and read Metadata before your first run rather than after it.

init never overwrites: run it again in a project you have filled in and it reports what is there and changes nothing.

Analyzing one set of reads under several parameter sets, two reference genomes say? PoolSeqFlow init multi does everything init does, switches multiRun on, and copies multi-run.csv.example in. The run table is yours to write, for the same reason metadata.csv is.

2. Verify it any time

There are two types of checks:

PoolSeqFlow check install    # the tools and helpers this release is built to run
PoolSeqFlow check project    # the configuration and the commands this project names

Both end by reporting how many checks passed, and fail loudly if any did not rather than summarizing.

check install

Tools
  from ~/.local/opt/miniconda3/envs/PoolSeqFlow-<version>

  nextflow       nextflow     OK       26.04.6 build 12646
  cutadapt       cutadapt     OK       5.2
  …
  samtools       samtools     OK       samtools 1.24
  bcftools       bcftools     OK       bcftools 1.24
  …

Pipeline helpers

  atomic_mv.sh                 OK
  depth2freq.awk               OK
  …

Every command the release is built to run, with the version each reports, and every helper in bin/, present and executable; a lost executable bit would otherwise fail mid-run rather than at startup.

Each tool has to come from that environment. All of them are pinned in install/environment.yml, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in, at whatever version it happens to be and only on this machine:

  samtools       samtools     OUTSIDE THE ENVIRONMENT  /usr/bin/samtools

That is the quiet failure worth catching: the pipeline runs, the results look fine, and nothing reproduces anywhere else. Reinstalling with PoolSeqFlow install is the fix.

It reads no parameters.config and needs no project. Run it from anywhere, including straight after installing and before you have a project at all.

check project

Run it from your project directory.

Configuration

  parameters.config            WRITTEN FOR THIS RELEASE
  parameters.config            PARSES
  metadata.csv                 PARSES
  runs.csv                     not in use
  parameters                   NOTHING TO FLAG

Tools, as this project configures them

  nextflow       nextflow               OK       26.04.6 build 12646
  samtools       /usr/bin/samtools      OK       samtools 1.19
  bcftools       bcftools               OK       bcftools 1.24
  …

  tool list from: params.software in parameters.config

Your files parse: parameters.config through Nextflow itself, metadata.csv and the run table through the same parsers a run uses. It also reports whether the config was written for this release, which is the one thing that stops a run before anything else is read.

Your settings are checked by the same rules a run applies, so a project is never told one thing here and another during the run. What is checked lists them. NOTHING TO FLAG is the healthy answer and the common one: a finding appears only when a setting would make the run produce nothing or stop it partway, or would quietly change what a published number means.

Every command, as you configured it. The list comes from params.software, so a command repointed at a system binary is checked the way the run will call it, and the second column shows what will actually be invoked. That override is the setting most likely to be wrong and least likely to announce itself. If params.software cannot be read, no tool is checked and it says so.

A path outside the environment is not a finding here, where check install treats one as a fault. Repointing a command is yours to do, so this check reports what will be invoked and whether it runs, and leaves the judgment to you.

3. Fill in the essentials

Open parameters.config. Nine settings need your attention before a first run; everything else has a working default.

params {
    mainDir       = "/path/to/working/directory"  // where you run, and where your inputs live
    storageDir    = "/path/to/permanent/storage"  // where finished results are kept
    dataSource    = 'Data'                        // subdirectory of mainDir holding the FASTQs
    readPattern   = "*_R{1,2}.fq.gz"              // glob matching your paired FASTQ files
    referenceFile = 'reference.fasta.gz'          // reference genome, in mainDir/Reference
    gffFile       = 'reference.gff.gz'            // annotation (only if annotate = true)
    poolSize      = 50                            // individuals per pool
    ploidy        = 2                             // copies of the genome per individual
    annotate      = true                          // run SnpEff annotation (step 8)
}
Setting How to choose it
mainDir The working directory you ran init in. Holds Data/, Reference/, your configuration and everything actively processed, so it has to persist between runs (not node scratch that is wiped between jobs). Put it on the fastest storage that satisfies that.
storageDir Where finished results are kept. A different directory from mainDir, and the pipeline refuses to start otherwise: the two are storage tiers, and outputs move from the first to the second as each step that needs them completes.
readPattern Must match both mates with a {1,2} group. If your files end _1.fastq.gz/_2.fastq.gz, write "*_{1,2}.fastq.gz".
poolSize Individuals in one pool, not the total across pools. Sets the smallest allele frequency worth believing (see The Filter Chain). If your pools differ in size, give each its own value in metadata.csv; this setting is the default for any that do not.
ploidy Ploidy of the organism: 2 for diploid, 1 for haploid, 4 for tetraploid.
annotate false skips step 8 and makes gffFile unnecessary.

Set these through the file, never the command line

PoolSeqFlow rejects command-line parameter overrides. Nextflow delivers --param values as strings, so --annotate false sets the string "false", which Groovy evaluates as true, leaving annotation switched on with no warning. Why →

4. Size the run to your machine

threads = 8          // cores a single task may use
memory  = '24 GB'    // memory ceiling for a single task

Set threads to the cores you actually have (on HPC, the size of one node). Every tool's thread count follows from this one number; do not set the per-tool counts by hand. A request larger than the machine fails immediately:

Process requirement exceeds available CPUs -- req: 12; avail: 8

Details and the full ladder: Resources.

5. Write metadata.csv

One row per FASTQ pair. SampleID is the whole name before the read tag, exactly as your sequencer delivered it: with *_R{1,2}.fq.gz, the file Sample1T1_S19_L001_R1.fq.gz gives Sample1T1_S19_L001. Do not shorten it, because it is what finds your files. The name you want your results under goes in RG_Sample, and you should always write that one out.

SampleID,RG_Sample,RG_Library,RG_Platform,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature
Sample1T1Rep1,Sample1T1,Lib1,ILLUMINA,50,Pop1,T1,susceptible,21.5
Sample1T1Rep2,Sample1T1,Lib1,ILLUMINA,50,Pop1,T1,susceptible,21.5
Sample1T2Rep1,Sample1T2,Lib1,ILLUMINA,50,Pop1,T2,resistant,22.1
Sample1T2Rep2,Sample1T2,Lib1,ILLUMINA,50,Pop1,T2,resistant,22.1
Sample2T1Rep1,Sample2T1,Lib1,ILLUMINA,40,Pop2,T1,susceptible,18.0
Sample2T1Rep2,Sample2T1,Lib1,ILLUMINA,40,Pop2,T1,susceptible,18.0
Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,40,Pop2,T2,resistant,19.4
Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,40,Pop2,T2,resistant,19.4

Three things this file decides:

  • RG_Sample decides what counts as a sample. Rows sharing one are merged into a single VCF column and their read depths add together. The eight rows above produce four columns, not eight: two populations at two timepoints, each sequenced twice.
  • param_poolSize sets that pool's detection limit. It describes the pool rather than the row, so rows sharing an RG_Sample have to agree on it. Leave the column out and every pool uses the global poolSize.
  • Row order decides column order in the VCF and the frequency tables.

The prefix is what a column means. No pipeline step reads any of the three below; analysis modules do, and they build their whole design from them:

Prefix What it holds Example above
exp_ something you set: the experiment's own structure exp_population, exp_time
pt_ a phenotype you measured on the pool, and are testing against pt_resistance
cov_ measured too, but neither set nor the thing being tested cov_temperature
  • Add as many of each as your experiment needs; the names after the prefix are yours.
  • A column with no prefix is recorded just as faithfully and read by nothing, so a variable you might analyze later is worth prefixing now.
  • Editing any of them never invalidates results you already have.
  • All three describe the pool. What differs between two rows of one pool takes no prefix, because once their reads are merged nothing can tell them apart again.

exp_time is the one name treated specially. A project that has it must set analysis.timeVar.kind before any analysis will run; the pipeline itself does not care. Leave the column out if this is not a time course. See The time axis.

Metadata has the full account, and getting RG_Sample wrong is the most common way to end up with results that are valid but not what you meant.

6. Run

PoolSeqFlow run

That is also the resume command. Every step checks whether its outputs already exist and skips itself if they do, so an interrupted run picks up where it left off with no extra flag. There is no -resume.

Run it from your project directory, the one holding parameters.config: that is where the settings are read from, and anywhere else stops with a message saying so.

To see where a run's results would go before spending any compute on it, PoolSeqFlow dryrun builds that directory tree empty and changes nothing else; PoolSeqFlow dryclean removes the preview.

To start genuinely from scratch, use PoolSeqFlow reset first. It requires typing DELETE_MY_ANALYSIS to confirm.

7. Check the output

storageDir/
├── Logs/
└── Output/
    ├── Frequencies/        ← the result: <name>_snp_freq.tsv, <name>_indel_freq.tsv
    ├── VCF/
    ├── Ready/              ← cleaned, indexed BAMs
    ├── Reports/
    ├── run_parameters.txt  ← the settings these results were produced under
    └── …

<name> is vcf.fileName, which follows outputPrefix, so it is Test until you change either.

Start with Output/Reports/Depth/ and Output/run_parameters.txt. The first says what depth ceiling each sample was given and why; the second is a read-only record of exactly which settings produced these files.

Your inputs stay where they were, under mainDir. Data/, Reference/ and the dictionaries built from your reference are not copied here.

If you are running a table of several runs, Output/ and Logs/ gain a level: work every run shared sits under All_Runs/, work some of them shared under Shared_<N>/, and whatever one run did alone under its own RunID. A single run keeps the plain tree above.

How to read the tables: Interpreting Results.


Commands

Command Description
PoolSeqFlow install Create this release's conda environment, install the pipeline, then verify both
PoolSeqFlow init Populate the current directory as a project (what it writes)
PoolSeqFlow init multi The same, for a project running several parameter sets over one set of reads
PoolSeqFlow check install Verify an installation: the tools and helpers it is built to run (what it covers)
PoolSeqFlow check project Verify a project: its configuration, and the commands it names (what it covers)
PoolSeqFlow run Start (or resume) the pipeline
PoolSeqFlow dryrun Create the directory tree the run would write, empty, so the layout can be approved before any compute is spent. Records nothing and changes none of your files
PoolSeqFlow dryclean Remove the preview dryrun made
PoolSeqFlow migrate_config Carry an older parameters.config onto the current template (details)
PoolSeqFlow clean Remove Nextflow work directories
PoolSeqFlow reset Remove all progress and start fresh (requires typed confirmation)
PoolSeqFlow analysis <command> The analysis layer, which carries a word of its own (listed below)
PoolSeqFlow version Print the version of this copy
PoolSeqFlow cite Print how to cite this copy, and which DOI to use (why it matters)
PoolSeqFlow list List the pipelines and conda environments installed on this machine
PoolSeqFlow uninstall Remove one installed version, environment and pipeline together, after confirmation
PoolSeqFlow uninstall all Remove every PoolSeqFlow environment and installation, after confirmation

Before anything is installed there is no PoolSeqFlow on your PATH, so the first command is ./PoolSeqFlow install, run from the folder you downloaded. Everything after that uses the installed command.

Each subcommand takes no arguments of its own, with one exception: analysis carries a word (the analysis command, or the module to run) and analysis modules carries a second. Anything else is rejected. Two environment variables adjust the wrapper instead: POOLSEQFLOW_PREFIX, where install puts things and where list and uninstall look, and POOLSEQFLOW_HOME, to run a checkout without installing it.

Naming a version. Every installed release is also on your PATH under its own name, so PoolSeqFlow-2.1.0 run uses that release and plain PoolSeqFlow uses the newest. PoolSeqFlow list shows what is installed, marks a version whose analysis layer is installed, and includes the unversioned environment if a release from before per-version environments left one behind:

Installed under /home/you/.local:
  1) PoolSeqFlow                   (unversioned - predates per-version environments)
  2) PoolSeqFlow-2.1.0
  3) PoolSeqFlow-<version>             (this wrapper, analysis installed)

Naming a version matters most when removing one; see Uninstalling.

The analysis layer hangs off analysis, the one subcommand that carries a word of its own. It ships with the same PoolSeqFlow install and is versioned with it:

Command Description
PoolSeqFlow analysis install Create this release's analysis conda environment, which carries R, then verify it
PoolSeqFlow analysis check Verify an existing analysis installation: tools, R packages, entry point
PoolSeqFlow analysis modules available List the modules published for this release's table contract
PoolSeqFlow analysis modules install <module> [version] Install one, newest readable version unless you name it. Also installs what it runs on
PoolSeqFlow analysis modules install all Install every module published for this release that is not installed yet. One that cannot be installed is named, and the rest still are
PoolSeqFlow analysis modules list List the modules installed for this release, and name any that are broken
PoolSeqFlow analysis modules uninstall <module> Remove one module, after confirmation. Results it produced are untouched
PoolSeqFlow analysis complete Move finished analyses and shared intermediates from mainDir/Analysis/ to storageDir/Analysis/, after confirmation. A name already taken in permanent storage stops it with nothing moved
PoolSeqFlow analysis version Print the version of this copy
PoolSeqFlow analysis cite Print how to cite this copy, plus R and every package a module runs on
PoolSeqFlow analysis uninstall Remove the analysis environment, after confirmation. The pipeline, its environment, and your results are untouched

The analysis environment is separate from the pipeline's and is built only when you ask for it. analysis cite reads R's own citation records, so it prints the version of each package actually installed.

PoolSeqFlow analysis also accepts the name of an installed module. Modules are installed separately and each is documented with itself; the table above is what a release ships with.