Upgrading¶
Two separate things upgrade here, and it helps to keep them apart. The installation is a tool: a new release installs alongside the ones you already have and disturbs none of them. A project is your results — and a project belongs to one release for as long as it exists.
Installing a new release¶
Download and install it exactly as you did the first time. Nothing is replaced: ~/.local/opt gains a directory, ~/.local/bin gains a PoolSeqFlow-<version> command, and plain PoolSeqFlow starts meaning the new one. Every earlier release still runs, under its own name.
Your projects are untouched by this. parameters.config lives in your project directory, not in the installation, so nothing an install or an uninstall does can reach it.
The new release's module store starts empty. No module ships inside a release, and the store belongs to the installation that runs it — so the old release keeps everything you installed into it and the new one has nothing. Ask the old one what to reinstall, then install each into the new one:
PoolSeqFlow-3.0.0 analysis modules list # what the old release has
PoolSeqFlow analysis modules install mds # and again, into the new one
Nothing is lost by this. A module installs the libraries it needs along with it, and the analyses it has already produced are results — no install or uninstall reaches them.
A project belongs to one release¶
This is the part that changes how upgrading works. Completed steps are skipped by looking for output files, not by checking what produced them, so continuing an existing project under a new release would leave one set of results built by two versions of the pipeline, with nothing on disk to say which is which.
The pipeline refuses. On the first run after upgrading it stops before any work happens:
There are two ways forward, and both are deliberate choices rather than defaults:
- Finish the project on the release that started it —
PoolSeqFlow-2.2.0 run. The version that produced your results is still installed and still works, which is the reason releases sit side by side. - Start the project again under the new one —
PoolSeqFlow reset, then run. This deletes the existing results.
There is deliberately no third option. A project that changed version midway has no answer to the question of which version to cite.
Bringing your configuration forward¶
parameters.config belongs to you and is never touched by an update — an update must not silently change your analysis settings. The consequence is that after installing a new release your file can be missing parameters the newer code expects, and migrate_config is the tool for exactly that. Run it before anything else.
Skip it and the run stops before anything is computed. A configuration from an older release does not set storageDir, the pipeline recognizes that, and it refuses — naming the parameters it found that this release renamed or removed, and telling you to run migrate_config. The refusal covers run, resume, dryrun, reset, running an analysis module, and analysis complete. It never applies to migrate_config itself, which is the fix, nor to clean and dryclean, which read nothing it covers.
The assisted route¶
Run it in your project directory. It backs your file up, rebuilds it from the current template, carries across every setting whose parameter still exists, and reports what happened to each one:
| Report | Meaning |
|---|---|
Kept your value |
The parameter still exists and your setting was carried over |
Renamed this release |
The parameter was renamed and your value followed it to the new name |
Now computed by the pipeline |
This release derives the value; yours was ignored |
Format changed this release |
The value's meaning or format changed, so the template's wins |
New in this release |
The template has a parameter your file did not — review the default |
Still yours to set |
The pipeline works the value out itself now, and your new config carries the parameter commented out — uncomment it to take it back |
No longer used |
Your file had a parameter this release does not use |
Still yours to set is the one people mistake for a loss. Coming from 2.2.0 it covers thirteen parameters — the eight cores values and the five options strings your file already had — none of which is gone. They are computed by default and sit commented out in your new config, so setting one is a matter of removing a //. It is distinct from Now computed by the pipeline, where the value is derived from other parameters and there is no line to uncomment.
It also ends with a list of files to move yourself, and moves none of them. If you are upgrading from 2.2.0 or older, there will be several, because the layout changed: your reads, reference and sample table used to live under the storage directory and now belong on mainDir. It prints the exact mv commands, having checked which files are actually there — read them before running them.
One group in that list matters more than the rest. .poolseqflow_params and its neighbors are the records the change guard compares against — they are how "has anything changed since these results were produced" gets answered. Leave them behind and the next run finds no record, decides the project is new, and writes down your current configuration as though it had produced the results already on disk. The guard would then report that nothing has changed, having quietly stopped guarding.
Treat the migrated config as a starting point, not an answer. Migration can only recognize a parameter that still exists and still means the same thing. A parameter whose behavior changed while its value still looks like an ordinary number or string is carried across and is silently wrong. Always read the report, and compare afterwards.
The manual route¶
Every release adds parameters, and rebuilding by hand is often the safer choice — it is the only way to be certain you have actually looked at the new ones. The template ships inside the installation, and init is what puts a copy of it in front of you:
mv parameters.config parameters.config.bak # keep your settings
PoolSeqFlow init # writes a fresh parameters.config
diff parameters.config.bak parameters.config # see what changed, then re-apply yours
init never overwrites, so move your own file aside first, as above — otherwise it reports the config as already present and leaves it alone.
Coming from 2.2.0¶
3.0 is a major release that brings in a complete new design and more reproducibility options, and the changes are structural rather than a handful of new settings.
Where things live has changed. Before 3.0 there was one storage directory holding your inputs and your results together. There are now two, and they must be different paths: mainDir holds your reads, reference and configuration and is where you run, storageDir holds finished results. migrate_config prints the mv commands for the files that need to move.
projectDir is now storageDir. A straight rename, and your value is carried over — note that projectDir is also a name Nextflow defines for itself, which is why it could not stay.
RGTags.csv is replaced by metadata.csv. This one is not a rename and cannot be migrated: the old file held raw SAM read-group tags and nothing else, while the new one has seven kinds of column and carries your pool sizes as well. migrate_config reports rgTagsFile as no longer used, tells you to move the file, and prints a note saying plainly that the two are not the same file under a new name — but rewriting it into the new schema is yours to do. Start from $POOLSEQFLOW_HOME/metadata.csv.template, which documents every column, and read Metadata. Until metadata.csv exists the run stops at step 0, so this is not a step you can defer. It is also the change that buys the most: the experiment itself — populations, timepoints, replicates — finally has somewhere to live.
The depth ceiling moved, and your old value is deliberately not carried. In 2.2.0 bcftools.maxDepth = 2000 was the only depth control there was: one number for every pileup in the run. From 3.0, step 5 measures a ceiling for each sample from its own depth histogram and step 6 applies it to the BAM before calling, so a sample is capped where its own coverage says to rather than at one number for the whole cohort — see Depth capping. variantCall.maxDepth is now a second ceiling on top of that one and ships as 0, which mpileup reads as no limit at all.
Carrying 2000 across would leave you capped at a number this release never chose, on top of a per-sample cap that cannot see it. So migrate_config reports it under Format changed this release and explains the change in full. Automatic capping is capBAM.maxDepth = -1, which is what you now have. To reproduce results from 2.2.0 exactly, set variantCall.maxDepth back to your old value and capBAM.maxDepth = 0.
The installation is separate from your project now. Earlier releases were run from the folder you unpacked, with parameters.config beside the pipeline. From 3.0 you install once, releases sit side by side, and you run the installed command from your own project directory. If your project is the old unpacked folder, move it out — the pipeline refuses to run inside its own installation.
Some parameters are gone: params.gff, params.dir.scripts and params.dir.output.temp, along with rgTagsFile and rgTagsPath from the section above. migrate_config reports each under No longer used. The cores block and the options strings look gone too and are not — they are computed now and ship commented out, which the report says under Still yours to set.
Multi-run is new, and off by default — multiRun = false changes nothing about how an existing project behaves.
After upgrading¶
Once your configuration is current, the first run will still stop, because your existing results were produced by an older release. That is the version block described above, and migrate_config does not clear it: choose between finishing on the old release and starting again on the new one.
For a project started fresh under 3.0, the ordinary guard applies from then on — a change to an analysis-affecting parameter stops the next run, and the report names the folders to delete.
See The run refuses to mix settings for what is and is not tracked, and the Changelog for what each release changed.