Analysis Layer¶
The pipeline stops at Output/Frequencies. The analysis layer is a second, optional layer that reads what the pipeline published and produces analyses from it. It is reached as PoolSeqFlow analysis <command>, and it has a conda environment carrying R and an entry point of its own — it is not steps 9 and up, and running it never changes, moves or re-runs anything the pipeline made.
Installing it¶
PoolSeqFlow install copies the analysis layer with everything else, so PoolSeqFlow analysis answers as soon as the pipeline is installed. That is not an installation. The scripts weigh nothing and travel with the release so they can never be a version out of step with the pipeline; the weight is the environment, which carries R and which the pipeline install does not create. Until you create it, every module refuses and says so:
PoolSeqFlow analysis install # builds this release's analysis environment
PoolSeqFlow analysis check # what it found: R, the packages, the tools
The pipeline is complete without it, and a machine that only ever analyzes results copied from elsewhere can install this layer and not the pipeline.
Running a module¶
A module is one analysis. You run one at a time, from your project directory — the same one you run the pipeline from:
Only verify comes with the pipeline. It is part of the analysis frame itself, reports what the layer can see and produces nothing. Every other module — basicstats, association and mds among them — is published on its own timetable and installed from the catalogue, so a fresh installation has an empty store and you choose what goes into it. Each has a page of its own under Modules, which is where what it computes and what it assumes is written down.
A module is installed with the libraries it declares: the shared arithmetic more than one module wants, each one published and versioned like a module and installed into analysis/modules/lib/. You never ask for a library by name; it arrives with whatever needs it, and leaves when nothing installed still declares it.
The modules installed here¶
PoolSeqFlow analysis modules available # what is published for this release
PoolSeqFlow analysis modules install mds # install one
PoolSeqFlow analysis modules install mds 1.2.0 # or pin the version
PoolSeqFlow analysis modules list # what is installed for this release
PoolSeqFlow analysis modules uninstall mds # remove one, after confirming
All of them read the installation rather than your project, so they work from anywhere and none needs a parameters.config. list and available need no environment. install does, because installing a module also installs what it runs on; uninstall uses the environment when it is there and says so when it is not.
install pins by name. Without a version it takes the newest one this release can read; with one it takes exactly that. Name the version in your methods section, and install that version to reproduce the analysis — a module carries its own version precisely so it can move without the pipeline moving, which means two runs of "the same module" are not necessarily the same code. Every analysis prints the module version it ran in its header, and installing writes a .source file beside the module recording where it came from and the checksum it matched.
The same list is on the website, at Published modules — the page and the catalogue are generated from the same rows and published in the same deploy, so what it shows is what available will tell you.
available reads a catalogue over the network and a release carries no copy of it, so a module published long after a release is still installable into it. What it lists is filtered to the table contract this release speaks; a module written against a later contract is shown and marked rather than hidden, so being told to install something that cannot work here gives you a reason instead of a blank.
One line per module, and it is the version install would take. The catalogue holds a row for every published version, so several rows name one module — a listing of them would be that module's history rather than something to act on. What you see is the newest version your release can run, which is what install <name> picks, so the two commands cannot tell you different things. Older versions stay published and stay installable by naming one. If your machine has no route to the internet, or your institution keeps a mirror, POOLSEQFLOW_MODULE_INDEX points at a URL or a file instead.
A download is verified before it is unpacked. A module is code that runs on your machine, so the checksum in the catalogue is checked first, and a mismatch stops the install having written nothing.
Every module declares its own license, and the report prints it. PoolSeqFlow is Apache-2.0 and imposes nothing on a module, so a module that builds on a GPL package is GPL itself and the results it produces are produced under those terms. The verification report at the top of every analysis carries the line — published under GPL-3.0-or-later — before anything runs, which is where to look if you are about to redistribute what comes out.
basicstats, association and mds are GPL-3.0-or-later, and the pipeline they run beside is not. Each compiles its hot path with Rcpp, which is GPL, and does so by default — nocpp turns it off for one run but the module ships expecting it. verify is the exception: it belongs to the frame, runs no R at all, and is Apache-2.0 with the rest of the pipeline.
A module also declares the frame and the release it needs, and a module that needs a newer one is refused by name rather than run. The frame is the library a module imports — analysis/frame.version in the installation, versioned separately from the pipeline because a module runs against it and not against the pipeline's own steps. The release is named because the analysis environment belongs to it: there is one R environment per release, shared by every module installed into it.
What installing a module does to your environment¶
There is one analysis environment per release and every module shares it. A module names every R package it needs in its manifest, pinned to an exact version, and modules install puts them in that shared environment. It names them whether or not the release's own environment already carries them: the manifest is a statement of what the module needs, not of what one release happens to provide, and removing a module never takes a package the release itself is built on. This is what you see when you install one:
Only what is missing is installed, so that list is usually shorter than the manifest. A package already there at the version asked for is not work, and conda is not asked about it. Most of what a manifest names is normally already in the environment — the release's own analysis layer carries ggplot2, data.table, Rcpp and the rest — so installing a module that needs nothing new says so and stops:
That is the common case and not a sign that something was skipped: the manifest still states everything the module needs, and uninstalling it still reasons over that whole list.
Nothing already in the environment is allowed to move. A pin naming a package the environment already holds at a different version is refused before conda is asked at all, so a module can never quietly downgrade something another module — or the release itself — is running on:
ERROR: these pins disagree with what 'PoolSeqFlow-
-analysis' already holds: r-poolfstat=2.9.0 (installed 3.0.0)
Beyond that the install is made with conda's --freeze-installed, which lets the solver add whatever the new package needs while refusing to change anything else. So the worst a module can do is fail to install, and it fails having changed nothing:
ERROR: fst v1.0.0 needs packages this release's analysis environment cannot take without moving something already in it. The module was not installed and the environment was not changed.
Either way it is a compatibility question between a release and a module, settled when each is published rather than on your machine. The answer is a build of that module published for the release you have.
Uninstalling takes back only what nothing else asks for. If two modules both name r-poolfstat=3.0.0, removing one leaves it installed for the other. And because conda remove takes everything that depends on what it is given, the removal is planned before it is run: if taking a package out would take something else with it, nothing is removed and the module stays installed.
Two things follow from the store living inside the installation. Reinstalling the pipeline over itself wipes the store, so the packages its modules added are taken out of the environment first, while the manifests declaring them still exist — afterwards both are back to what a fresh installation is, which is empty, and the modules you want are installed again. And analysis uninstall removes the environment while leaving the store, so analysis install puts back what the modules still there need. Neither is something to manage by hand.
list is worth knowing about before you need it. Modules live inside the release's own installation, so each release has its own set and a module installed for one is never picked up by another — reinstalling the same version wipes them, and one command puts each back. More usefully: a module directory that has lost its pipeline stops every analysis run, not only its own, and list is what names the one at fault. It also tells you where the store is, which is the directory a module is installed into.
Versions are not in that listing. A module reports its own version in the header of every analysis it produces, and that is the copy worth reading, because it comes from the same manifest the layer validated before it ran.
verify is also the first thing every other module does. It resolves your configuration, works out which results directories the invocation covers, checks them against the record the pipeline wrote beside them, and refuses before any compute if anything does not line up. Running it on its own is how you check a project is ready without spending anything.