ABI Development Log

Historical entries keep the terminology and counts that were true when they were written. For the authoritative release history, use CHANGELOG.md.

2026-07-25 — v1.5.7.1: Compiled Plans, Snakemake, Contracts, and Limitations

  • Added first-class Snakemake export and execution alongside local, Nextflow, and native HPC.

  • Wired CompiledPlan validation into planning and persisted compiled_plan.json.

  • Made non-empty limitations.yaml mandatory for all seven plugins and rendered limitations in every report format.

  • Enforced output-contract coverage for every external-tool DAG node, with explicit justified exemptions for output-less nodes.

  • Added structured recovery actions to diagnostics for all 19 stable error codes.

2026-07-15 — v1.5.6: Integrations, Runtime Locks, and Workflow Orchestration

  • Added installable Claude Code, OpenCode, and Codex integrations.

  • Hardened strict runtime-lock certification and the immutable cloud publication helper.

  • Added EasyMeta/ViWrap workflow orchestration and report/artifact manifests.

  • Improved SciPlot annotation layout, Unicode handling, and package/Docker release surfaces.

2026-07-11 — v1.5.5: Trusted Release Chain

  • Consolidated GitHub Actions to CI, Docker, Release, and the trusted PyPI publisher.

  • Fixed sdist-to-wheel and Docker build-input invariants.

  • Made the published GitHub Release artifacts the exact inputs to PyPI Trusted Publishing.

2026-07-07 — v1.5.3: Release Quality Gate & DAG Fixes

Overview

v1.5.3 tightened the release engineering pipeline, added paper-evaluation task scaffolding, and fixed declarative DAG input resolution for the metagenomic_plasmid plugin.

Release Engineering

  • NEW scripts/release_check.sh — unified local release verification entry point that reproduces the CI quality gate locally before tagging.

  • CI release identity now strictly enforces tag/wheel/runtime/CHANGELOG alignment via scripts/check_release_identity.py.

  • Coverage gate includes SciPlot tests (src/abi/sciplot/tests/); module coverage is checked from coverage.json with risk-based per-module gates.

Paper-Evaluation Task Package

  • NEW paper-evaluation task package with benchmark task definitions, metrics schema, bilingual evaluation notes, and manuscript outline drafts.

DAG & Contract Fixes

  • Contract lint extended to catch unresolved template parameters (command, params, input, and output placeholders) in pipeline_dag.yaml.

  • DAG input resolution fixed for source: config.* references, including tool database resources and cross-sample report inputs.

  • FASTA export and metagenomic plasmid reports now read assembly paths from DAG-resolved step inputs/outputs after the planner refactor.

  • Metagenomic plasmid DAG-driven tests updated to assert current declarative step and output contracts instead of old handwritten planner paths.

Bug Fixes

  • Matplotlib boxplot label argument compatibility across Python versions.

  • GitHub Actions opencode API key moved to GitHub Actions secret (no longer hardcoded).

Verification

ruff check src/ tests/           # 0 errors
ruff format --check src/ tests/  # All files formatted
mypy src/abi/ --ignore-missing-imports  # 0 errors

2026-07-09 — v1.5.4-pre: RefSeq Plasmid Validation & Bug Fixes

Overview

Full plasmid detection pipeline validation using 3 RefSeq plasmids on an AutoDL cloud server (128 cores, 1TB RAM): NC_002127.1 (3.3kb), NC_011977.1 (7.6kb ColE9-J), NC_002483.1 (99kb F plasmid). Assembly-mode run with genomad + platon, majority_vote strategy. All 39/39 DAG steps passed. WGS bacteria plugin also validated with a synthetic MG1655 sample.

Plasmid Validation

  • 3 RefSeq plasmids processed end-to-end: assembly → plasmid detection (genomad + platon) → consensus → report with figures.

  • Per-sample results correct: NC_002127_1 (3306bp, genomad=0.99 high, platon=3.7 low), NC_011977_1 (7577bp, genomad=0.99 high, platon=1.0 high), NC_002483_1 (99159bp, genomad=0.995 high, platon=22.1 low).

  • 3 sciplot figures rendered: assembly_metrics_by_sample, plasmid_length_distribution, plasmid_score_vs_length. 6 other specs gracefully skipped due to missing annotation/typing/host data (annotations and typing tools were intentionally disabled in config).

Bug Fixes

  • Cross-sample path propagation (src/abi/executor.py): _propagate_resolved_paths unconditionally propagated resolved paths to all downstream steps regardless of sample_id, causing every sample to receive the first sample’s assembly path. Fixed with per-sample filtering.

  • Platon --db flag missing (plugins/metagenomic_plasmid/tool_registry.yaml): The command template was missing --db {database} and the DAG did not declare the database input dependency, causing platon runtime failures.

  • Genomad duplicate results (_engine/parsers.py): The parser glob *plasmid*summary*.tsv matched both top-level and per-contig subdirectory files, producing duplicate predictions. Fixed by restricting to contigs_summary/*plasmid*summary*.tsv.

  • Empty consensus table (__init__.py, schemas.py, dag_planner.py): write_consensus_table() was only called from PipelineRunner._run_plan, but abi run uses ABIExecutor which never invoked it. Fixed by adding the call in write_report() and adding plasmid_strategy field to ExecutionPlan.

WGS Bacteria Validation

  • Synthetic MG1655 sample: fastp → SPAdes → Prokka → MLST (correct ST10) → AMRFinderPlus (0 acquired / 11 intrinsic resistances). All steps passed.

SciPlot Improvement

  • Barplot auto-count mode (sciplot/renderers/plots/barplot.py): When mapping.y is not specified, auto-count occurrences of mapping.x values instead of raising an error.

CLI Addition

  • abi --version / abi -V (cli.py): Prints the abi-agent version from pyproject.toml via importlib.metadata. Uses a Typer callback.

Verification

ruff check src/ tests/           # 0 errors
ruff format --check src/ tests/  # All files formatted
mypy src/abi/ --ignore-missing-imports  # 0 errors
pytest tests/ -q --tb=short      # 2252+ passed, 0 regressions

Files Changed

File

Action

src/abi/executor.py

Fixed cross-sample path propagation with sample_id filtering

plugins/metagenomic_plasmid/tool_registry.yaml

Added --db {database} to platon command template

plugins/metagenomic_plasmid/pipeline_dag.yaml

Added platon database input, internal handler nodes

src/abi/plugins/metagenomic_plasmid/_engine/parsers.py

Fixed genomad duplicate results (glob path)

src/abi/plugins/metagenomic_plasmid/__init__.py

Added write_consensus_table in write_report, internal handlers

src/abi/schemas.py

Added plasmid_strategy field to ExecutionPlan

src/abi/dag_planner.py

Plumbed plasmid_strategy from config to ExecutionPlan

src/abi/sciplot/renderers/plots/barplot.py

Added auto-count mode for barplot

src/abi/cli.py

Added –version/-V flag

src/abi/plugins/metagenomic_plasmid/handlers.py

NEW — internal handlers for passthrough DAG nodes

tests/unit/test_parsers.py

Updated genomad test expectations

2026-06-23 — v1.5.1-1.5.2: Release Identity & Planner Cleanup

Overview

v1.5.1 made package metadata the single runtime version source, removed the legacy handwritten plasmid planner, and raised the CI coverage gate to 75%. v1.5.2 was a CI-only release identity fix.

Key Changes

  • Release identity: pyproject.toml project.version is the single source of truth. Strict checks validate tag/wheel/runtime/CHANGELOG alignment at CI time.

  • Planner consolidation: Removed _engine/planner.py’s legacy handwritten planner; all 7 built-in workflows now use declarative pipeline_dag.yaml planning via UniversalDAG.

  • Protocol hardening: Replaced concrete plugin implementation imports with optional resource (ABIResourcePlugin) and result-validation (ABIResultValidationPlugin) capability protocols.

  • Coverage gate: Raised to 75% (branch-aware), with risk-based per-module line/branch gates for DAG, executor, resources, local/HPC runtimes, internal handlers, and step execution modules.

Bug Fixes

  • Wheel-installed plasmid configuration discovery outside source checkout.

  • PBS scheduler job-ID validation in HPC runtime.

  • Resource selection filtering in check_resources/setup_resources.

  • Dry-run resource setup side effects.

  • Release and PyPI publication now consume the exact wheel that passed the quality gate.

Files Changed

…

2026-06-21 (pm) — Three-Dimensional Engineering Fix: Environments, Charts, Execution

Overview

Post-pipeline-run analysis identified three engineering gaps in the metagenomic_plasmid pipeline: (1) tools mapped to wrong conda environments, (2) chart generation limited to 3 types with axis/composition misalignment, (3) ~8% CPU utilization from hardcoded sequential execution. All three were fixed in a single session.

Phase 1: Environment Mapping + Resource Fixes

stats environment mapping bug: 4 tools (kraken2, metaphlan, checkm2, gtdbtk) were mapped to the non-existent autoplasm-stats environment. The actual environment name on disk was stats. Fixed in 3 files:

File

Change

environments.yaml:145

Environment definition renamed autoplasm-stats → stats

environments.yaml:326-332

7 tool assignments corrected to stats

_engine/resources.py

4 env_name="autoplasm-stats" → env_name="stats"

fastspar: Confirmed 4 executables present in stats env (fastspar, fastspar_bootstrap, fastspar_pvalues, fastspar_reduce). Previously orphaned.

checkm2 / gtdbtk: Python version conflict — stats env has python=3.10, but checkm2 requires python<3.9 or >3.12. Both are default_enabled: false and require separate environments + ~200GB of databases. Deferred.

ResourceSpec additions:

  • mmseqs2 ResourceSpec added (resource count: 28 → 29) — creates plasmid DB from mob_suite’s ncbi_plasmid_full_seqs.fas

  • amrfinderplus ResourceSpec: added install_post: makeblastdb to auto-build BLAST indexes after DB download

  • kraken2 download: switched from slow kraken2-build --standard (FTP) to direct aria2c download from AWS S3 (k2_standard_20260226.tar.gz)

  • _resolve_executable(): added system PATH fallback for shell builtins (bash, aria2c, tar, mkdir, etc.)

Phase 2: Chart System Migration (Old FigureEngine → abi-sciplot)

figure_specs.yaml completely rewritten: 6 old-style figures → 8 sciplot figures:

Figure ID

Type

Source Table

Status

qc_read_metrics

barplot

qc_summary

✅

assembly_metrics_by_sample

barplot

assembly_summary

✅

plasmid_length_distribution

barplot

plasmid_predictions

✅

plasmid_score_vs_length

scatterplot

plasmid_predictions

✅

host_taxonomy_stacked

stacked_barplot

host_predictions

✅

annotation_category_counts

heatmap

annotations

✅

amr_gene_heatmap

heatmap

annotations

✅

plasmid_abundance_heatmap

heatmap

abundance

✅

All 8 figures render PDF+SVG+PNG with abi_nature theme + colorblind_safe palette, pass lint validation, and include SHA256 provenance. Chart count: 3 → 8 (2.7x).

CoverM parser fix (_engine/parsers.py): CoverM outputs dynamic column names like SRR2241213.samtools Mean (normalized: srr2241213_samtools_mean). The old _get() did exact key match on mean, which never matched. Added _get_contains() that searches for normalized keys containing the requested key. Updated parse_coverm() to use _get_contains() for coverage, tpm, rpkm, mapped_reads, and length_bp.

CoverM command template (tool_contracts/coverm.yaml): Added --methods mean tpm rpkm --min-covered-fraction 0 to produce full abundance metrics.

Phase 3: Parallel Execution in GenericABIExecutor

src/abi/executor.py — 4 locations had hardcoded parallel=False, workers=1:

Location

Fix

_execution_options() (L88-95)

Now reads config.execution.parallel + config.execution.workers

run() (L251-400)

Sample-level ThreadPoolExecutor with as_completed(), threading.Lock for shared state

start_run() (L431-439)

Passes actual parallel/workers values

write_minimal_progress_artifacts() (L444-468)

Uses actual values instead of hardcoded defaults

Usage:

execution:
  parallel: true
  workers: 8

Sample-level parallelism: samples run concurrently, steps within each sample remain serial (respecting DAG topological order). Thread safety via _state_lock for StandardTableManager, PipelineProgressRecorder, and RunLogger.

Verification

ruff check src/ tests/           # All checks passed
ruff format --check src/ tests/  # 236 files already formatted
pytest tests/ -v --tb=short      # 707 passed, 8 skipped, 0 failed

Resource Status After Fixes

Category

Count

Status

Databases installed

10

genomad, bakta, mob_suite, plasmidfinder, amrfinderplus, platon, macsyfinder, metaphlan, mmseqs2, kraken2 (pending download)

default_enabled=true tools working

24/24

All executables confirmed in conda environments

default_enabled=false tools missing

11

git repos not cloned (PlasmidHostFinder, pMLST, gplas2, Recycler, scapp, COPLA, conjscan, plasme, plasx, plasmaag, plasmidhostfinder)

Tier 1 tools (mainstream) pending

4

kraken2 DB, BLAST DB, checkm2 + gtdbtk (env + DB)

Files Changed (2026-06-21 pm)

File

Action

Description

environments.yaml

Fixed

autoplasm-stats → stats environment definition + 7 tool assignments

_engine/resources.py

Modified

4 env_name fixes, mmseqs2 ResourceSpec, kraken2 aria2c command, amrfinderplus install_post, _resolve_executable system PATH fallback

_engine/parsers.py

Fixed

Added _get_contains() for CoverM dynamic column names

plugins/metagenomic_plasmid/tool_contracts/coverm.yaml

Fixed

--methods mean tpm rpkm --min-covered-fraction 0

plugins/metagenomic_plasmid/figure_specs.yaml

Rewritten

6 old-style → 8 sciplot figures with theme + palette

src/abi/executor.py

Modified

Parallel execution via ThreadPoolExecutor, sample-level parallelism, thread-safe state

tests/unit/test_resources.py

Updated

Resource count 28 → 29 (added mmseqs2)

2026-06-21 — Real Pipeline Execution: Bug Fixes + Full Assembly Pipeline Verification + Real Metagenomic Data

Overview

2026-06-21 focused on executing the metagenomic_plasmid pipeline against real RefSeq plasmid data using the assembly platform (pre-assembled contigs), discovering and fixing 5 bugs, verifying 19/19 steps pass across 3 samples, and preparing comprehensive illumina platform testing with real metagenomic data.

Bug Fixes from Real Pipeline Execution

Bug #

Tool

Symptom

Root Cause

Fix

#33

AMRFinderPlus

makeblastdb not run after download

resources.py lacked install_post: makeblastdb

Added install_post hook to ResourceSpec

#34

Bakta

AMRFinderPlus outputs overwritten

Bakta output dir collision with amrfinderplus step

Added --force to Bakta command template in tool_registry.yaml

#35

AMRFinderPlus

DB path resolution failure

Path pointed to parent dir, not latest/ subdirectory

Fixed amrfinder_database path in config_default.yaml to include latest/

#36

geNomad

Output path contract mismatch

Executor expected abstract paths, geNomad writes fixed filenames

192-line four-part executor fix: output writeback + contract symlinks + downstream propagation + consensus bridge in executor.py

#37

Bakta

Diamond SIGSEGV during sORF PSCC search

Diamond crashes with light DB v6.0 on PSCC sORF search

Added --skip-sorf to command template (workaround; proper fix needs full DB or diamond upgrade)

Bug Details

Bug #33 (AMRFinderPlus BLAST DB not built): The amrfinder_update tool successfully downloads AMRFinderPlus database files (251MB) but does not run makeblastdb to build BLAST indexes. Without BLAST indexes, AMRFinderPlus BLAST-based searches fail silently. Fixed by adding install_post: makeblastdb to the amrfinderplus ResourceSpec in _engine/resources.py.

Bug #34 (Bakta output dir collision): Bakta and AMRFinderPlus both write output to the same annotation directory. Since Bakta runs after AMRFinderPlus in the pipeline, it would overwrite AMRFinderPlus results. Fixed by adding --force flag to Bakta’s command template, allowing it to proceed when the output directory already exists rather than failing.

Bug #35 (AMRFinderPlus DB path): The amrfinder_database config key pointed to the parent directory /path/to/amrfinderplus/ rather than /path/to/amrfinderplus/latest/. AMRFinderPlus expects the latest/ subdirectory containing the versioned database files. Fixed in config_default.yaml.

Bug #36 (geNomad output path contract mismatch): The executor’s step contract validation expected output files at planner-generated abstract paths, but geNomad writes outputs with fixed filenames (e.g., *_plasmid_summary.tsv) directly in the output directory. The 192-line fix spans four areas:

  1. Output writeback: executor now writes a mapping of actual→abstract paths after tool completion

  2. Contract symlinks: creates symlinks from abstract paths to actual files so downstream steps find them

  3. Downstream propagation: planner references updated to consume abstract paths consistently

  4. Consensus bridge: consensus algorithm step bridges geNomad and MOB-suite results using the corrected paths

Bug #37 (Bakta diamond SIGSEGV): Bakta’s internal diamond binary segfaults during the sORF PSCC search when using the light database (v6.0, 4.2GB). The crash occurs reproducibly on PSCC: Prokaryotic Small Coding sequence search. Workaround: added --skip-sorf to Bakta’s command template. Proper fix requires either the full Bakta DB (much larger) or upgrading diamond to a version that handles the light DB’s PSCC indices correctly.

Environment Fix

  • metaPhlAn + Kraken2 env_name: Fixed from broken autoplasm-stats to working stats in both _engine/resources.py and environments.yaml. The autoplasm-stats environment never existed; stats is the correct conda environment containing metaPhlAn and Kraken2.

Assembly Platform Test Results

Full metagenomic_plasmid pipeline executed on the assembly platform (pre-assembled contig inputs) with 3 RefSeq plasmid samples:

Sample

Contigs

geNomad Plasmids

MOB-suite Plasmids

Consensus

NC_002127_1

1

1

1

1

NC_002483_1

1

1

1

1

NC_011977_1

1

1

1

1

  • 19/19 steps passed across all 3 samples

  • All 3 known plasmids correctly detected by both geNomad and MOB-suite

  • Consensus algorithm produced correct final calls

  • Provenance artifacts written for all steps

Database Status (7 Available)

Database

Tool

Size

Status

geNomad DB

geNomad

2.9 GB

✅ Ready

Bakta light DB

Bakta

4.2 GB

✅ Ready (workaround: –skip-sorf)

MOB-suite DB

mob_suite

3.0 GB

✅ Ready

PlasmidFinder DB

plasmidfinder

~50 MB

✅ Ready

AMRFinderPlus DB

amrfinderplus

251 MB

✅ Ready (+ BLAST indexes)

PLaton DB

platon

~100 MB

✅ Ready

MacSyFinder DB

macsyfinder

~150 MB

✅ Ready

metaPhlAn DB

metaphlan

12+ GB

🔄 Downloading

Illumina Platform Test Plan

Comprehensive illumina platform test prepared with real metagenomic paired-end data:

Metric

Value

Samples

121 real metagenomic paired-end samples

Data location

/root/autodl-tmp/abi-databases/real_data/metagenomic/

Total size

~60 GB

Unique tools in plan

33

Total plan steps

71 (across 2 representative samples)

Platforms enabled

illumina

Categories included

qc, assembly, gene_prediction, plasmid_detection, annotation, abundance, typing, host_prediction, amr, virulence

Sample manifest and metadata available for all 121 samples. Two samples selected for initial comprehensive pipeline testing (minimizing runtime while covering all tool paths).

Files Changed (2026-06-21)

File

Action

Description

plugins/metagenomic_plasmid/_engine/resources.py

Fixed

Added install_post: makeblastdb to amrfinderplus ResourceSpec; fixed metaphlan/kraken2 env_name: autoplasm-stats → stats

plugins/metagenomic_plasmid/tool_registry.yaml

Fixed

Added --force to Bakta command; added --skip-sorf to Bakta command

plugins/metagenomic_plasmid/config_default.yaml

Fixed

amrfinder_database path now points to latest/ subdirectory

environments.yaml

Fixed

metaPhlAn/kraken2 tool_assignments corrected from autoplasm-stats → stats

src/abi/executor.py

Fixed

192-line four-part fix: output writeback + contract symlinks + downstream propagation + consensus bridge

src/abi/dag_planner.py

Modified

_resolve_params supports DAG node params + config section mapping

CLAUDE.md

Updated

Added abi check-resources, tested tool count, environments fix note

Commits

  • 8e3ac08 — fix: Dockerfile env names + stale envs cleanup + docs sync — Phase 2

  • 3a4e1e8 — feat: unified environments.yaml + tool auto-install + env fixes

  • 8c853e7 — fix: amrfinderplus DB path + maxbin2 deprecation + docs sync — v1.4.0 patch

  • 4d0f787 — fix: systematic enable_condition audit — 12 nodes from value:true to list_contains

  • 9d65f24 — fix: geNomad parser + binning tool env — fix 81% null contig_length, enable full pipeline

2026-06-21 — Phase 1: Unified Environment Architecture

Overview

Phase 1 centralized all conda environment definitions into a single source of truth.

Core deliverables

File

Action

Description

environments.yaml

New

16 conda environments + 93 tool→env assignments across all 5 plugins

scripts/emit_env_yamls.py

New

Generates per-environment YAMLs (envs/*.yml) from environments.yaml

src/abi/tools.py

Modified

ToolRegistry auto-resolves env_name from environments.yaml at runtime

src/abi/plugins/metagenomic_plasmid/_engine/skills/registry.py

Modified

Same environments.yaml injection pattern for engine’s ToolRegistry

src/abi/config.py

Modified

resolved_mamba_root(): env var checks (ABI_MAMBA_ROOT/AUTOPLASM_MAMBA_ROOT) now come FIRST

src/abi/plugins/metagenomic_plasmid/_engine/config.py

Modified

Same resolved_mamba_root() fix

src/abi/plugins/metagenomic_plasmid/_engine/resources.py

Modified

ResourceSpec extended with resource_type and install_post; 12 tool specs added

src/abi/contracts/__init__.py

Modified

env_name made optional in contract validation

pyproject.toml

Modified

numpy>=1.24 → numpy>=1.21 (fixes pip-overwrite-conda C ABI break)

plugins/*/tool_registry.yaml (6 files)

Modified

Removed all env_name: fields (now auto-injected)

plugins/*/tool_contracts/*.yaml (84 files)

Modified

Removed all execution.env_name: fields

envs/*.yml (11 files)

Deleted

Old flat names; regenerated with correct autoplasm-* prefix by emit_env_yamls.py

docker/Dockerfile.metagenomic_plasmid

Modified

Updated env names to autoplasm-* prefix

Key design decisions

  • Plugin-qualified tool_assignments: environments.yaml uses {plugin_name: {tool_id: env_name}} nesting to handle cross-plugin tool ID collisions (e.g., fastp in both metagenomic_plasmid and wgs_bacteria).

  • Fallback resolution: ToolRegistry._resolve_env() searches specified plugin → all plugins → _default fallback.

  • Two-tier ResourceSpec: Level 1 tools (auto_setup=True, 4 mainstream plasmid tools) auto-install; Level 2 tools (auto_setup=False, 8 experimental tools) are guided manual install.

  • No new CLI command: Tool installation reuses existing abi setup-resources (extended ResourceSpec resource_type field).

2026-06-20 (pm) — AMRFinderPlus Database Path Fix

AMRFinderPlus: Missing --database Flag

The amrfinderplus tool in the metagenomic_plasmid plugin failed because:

  1. The command template in tool_registry.yaml did not include -d {database}

  2. The DAG node annotation_amrfinderplus had no database input parameter

  3. amrfinder relied solely on $AMRFINDER_DB / $CONDA_PREFIX default path, but the autoplasm-annotation env lacked the AMRFinder database

Fix (3 files):

File

Change

tool_registry.yaml:265

amrfinder -n ... -o ... → amrfinder -n ... -d {database} -o ...

tool_contracts/amrfinderplus.yaml:18

Added -d {database} to command_template + database input

pipeline_dag.yaml:1480

Added database: {type: path, source: config.resources.amrfinder_database}

config_default.yaml:81

Added amrfinder_database: AMRFINDER_DB_NOT_CONFIGURED

Root cause: All other DB-dependent tools (genomad, bakta, kraken2, metaphlan, etc.) correctly pass {database} in their command template. amrfinderplus was the only tool that relied on conda env $CONDA_PREFIX default DB discovery, which worked for wgs_bacteria (conda env has DB) but not for metagenomic_plasmid.

No similar issues found: All 5 plugins were audited for missing {database} params. wgs_bacteria, amplicon_16s, rnaseq_expression, metatranscriptomics are all correctly wired — verified by their end-to-end passes.

Files Changed (2026-06-20 pm)

File

Action

Description

plugins/metagenomic_plasmid/tool_registry.yaml

Fixed

Added -d {database} to amrfinderplus command

plugins/metagenomic_plasmid/tool_contracts/amrfinderplus.yaml

Fixed

Added -d {database} + database input

plugins/metagenomic_plasmid/pipeline_dag.yaml

Fixed

Added database input with config.resources.amrfinder_database source

plugins/metagenomic_plasmid/config_default.yaml

Fixed

Added amrfinder_database resource key

2026-06-20 — v1.4.0: Scientific Figure Compiler Upgrade + Comprehensive Bug Fix + Pipeline Verification

Overview

v1.4.0 focused on three tracks: (1) upgrading abi_sciplot from 9 to 15 plot types with ggplot2-quality rendering, (2) systematic DAG enable_condition audit and bug fixes for the metagenomic_plasmid pipeline, (3) end-to-end verification of all 5 plugins on high-performance hardware.

sciplot v1.4.0 — 6 New Biological Plot Types

New Plot Type

Purpose

Key Features

phylum_stacked_bar

Phylum-level community composition

Stacked relative abundance bars per sample

genus_heatmap

Top-N genus abundance

Z-score normalization, hierarchical clustering

pcoa_plot

PCoA ordination

95% confidence ellipses, PERMANOVA annotation

differential_volcano

Differential abundance

Log2 fold-change vs -log10 p-value, significance thresholds

alpha_stats_boxplot

Alpha diversity comparison

Kruskal-Wallis test, pairwise significance

phylogenetic_heatmap

Phylogeny + abundance

Tree-ordered abundance with evolutionary context

New backends: plotnine (ggplot2 grammar) + seaborn for biological-grade aesthetics.

Bug Fixes (2026-06-18 through 2026-06-20)

Execution-blocking (P0):

  • metabat2 --threads: removed from command_template (binary has no threads flag)

  • binning tool env_name: corrected autoplasm-stats → plasmid_binning for metabat2, maxbin2, concoct, semibin

  • maxbin2: --thread → -thread (Perl script uses single dash)

  • concoct: --threads → -t (short flag)

  • geNomad parser: removed *.tsv wildcard — reads only *plasmid_summary*.tsv, fixing 81% null contig_length

Configuration / DAG gating (P1):

  • 12 DAG nodes: enable_condition value: true → list_contains on tools lists (plasmid_binning: 5, mag_host_genomes: 7)

  • bandage: added list_contains on assembly_qc.tools

  • contig_coverage: fixed bowtie2/minimap2 input assembly → plasmid_contigs

  • SemiBin: executable name SemiBin → SemiBin2

  • coverm: contract min_size 50B → 0B (too strict for small datasets)

Robustness (P2):

  • Empty table rendering: detection + log instead of silent skip

  • Arial font: default changed to DejaVu Sans for headless Linux

  • Script auto-resolution: _resolve_script_path() for DESeq2/diversity/count_matrix

  • OMP_NUM_THREADS: unset in runtime_env() (suppressed 200+ warnings)

  • matplotlib import: try/except ImportError guard in figure rendering

Pipeline Verification (16-core / 1TB RAM Server)

Plugin

Steps

Success

Failed

Status

rnaseq_expression

14

14

0

✅ Full pass

wgs_bacteria

5

5

0

✅ Full pass

amplicon_16s

9

9

0

✅ Full pass

metatranscriptomics

6

6

0

✅ Full pass

metagenomic_plasmid

62

9

0

✅ 1 fixed (amrfinderplus -d {database} added)

4/5 plugins verified end-to-end with zero failures. The single plasmid failure is a configuration path mismatch (AMRFinder DB at /abi-envs/wgs/share/amrfinderplus/ but tool looks in /abi-envs/autoplasm-annotation/share/amrfinderplus/).

Quality Gates

ruff check:      0 errors
ruff format:     204 files already formatted
mypy:            0 errors (232 sciplot Pydantic errors resolved)
pytest:          698 passed, 8 skipped, 0 failed
sciplot tests:   38 passed
build:           abi_agent-1.3.3.tar.gz + wheel OK

Files Changed

File

Action

Description

src/abi/sciplot/

Extended

6 new plot types, plotnine+seaborn backends

plugins/metagenomic_plasmid/figure_specs.yaml

Rewritten

8 biological-grade figures

src/abi/report/generic_report.py

Refactored

Shared render_figures_via_sciplot()

_engine/parsers.py

Fixed

geNomad parser wildcard fix

plugins/metagenomic_plasmid/tool_registry.yaml

Fixed

binning env_name + flag fixes

plugins/metagenomic_plasmid/tool_contracts/{metabat2,maxbin2,concoct}.yaml

Fixed

env_name + flag consistency

plugins/metagenomic_plasmid/pipeline_dag.yaml

Fixed

12 enable_conditions + input bindings

src/abi/tools.py

Fixed

OMP_NUM_THREADS, env_prefix resolution

src/abi/dag_planner.py

Fixed

_resolve_script_path() auto-discovery

Commits

  • 4d0f787 — fix: systematic enable_condition audit — 12 nodes from value:true to list_contains

  • 9d65f24 — fix: geNomad parser + binning tool env — fix 81% null contig_length, enable full pipeline

  • 7ad732d — refactor: unify figure rendering — plasmid delegates to shared render_figures_via_sciplot

  • bbab24a — feat: sciplot v1.4.0 — 6 new biological plot types, ggplot2 backend, bandage fix

  • 36dfc80 — fix: empty tables/figures — column name alignment, font fix, robust rendering

  • 313d15a — feat: Phase 1-4 — comprehensive codebase audit, bug fixes, script auto-resolution, end-to-end execution, and documentation sync

2026-06-18 — Direction E: Token Optimization + Benchmark Data + Real Execution (v1.3.0)

Overview

Direction E delivered v1.3.0 with four phases:

  • Phase 1: Token optimization for agent middleware (~200 lines, 4 optimizations)

  • Phase 2: Benchmark datasets completed for all 5 plugins

  • Phase 3: Bench v0.5 with 5 real execution tasks (T31-T35)

  • Phase 4: Integration testing, version bumps, CHANGELOG, release

Phase 1: Token Optimization

Optimization

Code

Token Savings

Plan summarization

+30 lines envelopes.py

78-95% (plasmid plan ~5K→250 tokens)

Error envelope sans traceback

+22 lines envelopes.py

-80% in error scenarios

abi query command

+55 lines cli.py, +40 lines interface.py

-90% for metadata queries

Dry-run envelope reduction

~10 lines envelopes.py

-50%

Plan summarization: abi plan now returns a summary field with pipeline stages, key tools, and platforms — extracted from PlanStep.category annotations. Agents no longer need to read the full execution_plan.json (plasmid: ~5,000+ tokens).

abi query: Lightweight metadata query (~50ms) that reads pipeline_dag.yaml and tool registry directly — no config loading, no plan building. Supports:

abi query --type <plugin> --what stages|tools|platforms
abi query --type <plugin> --step <id> --what inputs|outputs|resources

Error envelopes: error_envelope() now accepts verbose=False (default), omitting error_type from error payloads. Agents receive only error_code + diagnostic_hints for automated recovery. Use ABIAgentInterface(verbose_errors=True) for debugging.

Dry-run envelope: Removed written_files list; agents use abi inspect on demand.

Phase 2: Benchmark Data Completion

Plugin

expected_assertions.yaml

config.yaml

Smoke Test

metagenomic_plasmid

✅

✅

✅

rnaseq_expression

✅

✅

✅

amplicon_16s

✅

✅

✅

wgs_bacteria

✅ (new)

✅ (new)

✅ (value-level)

metatranscriptomics

✅ (new)

✅ (new)

✅ (value-level)

All 5 plugins now have complete benchmark data under data/benchmarks/<plugin>/. wgs_bacteria and metatranscriptomics smoke tests upgraded from file-existence-only to value-level validation (N50 calculation, contig counts, mapping rate, gene counts).

Phase 3: Bench v0.5 Real Execution Tasks

5 new tasks (T31-T35) in the Bench sibling repo:

Task

Plugin

Score

Key Checks

T31

metagenomic_plasmid

15

pipeline_completed, assertions_validated, discrepancy_analyzed, provenance_quality

T32

rnaseq_expression

15

↑

T33

amplicon_16s

15

↑

T34

wgs_bacteria

15

↑

T35

metatranscriptomics

15

↑

New scoring checks: check_pipeline_completed, check_assertions_validated, check_discrepancy_analyzed, check_provenance_quality. Benchmark fixtures created for all 5 plugins with real_tool_execution: true. BENCHMARK_SPEC bumped to v0.5; run_group.py updated with FULL_V0_5_TASKS.

Phase 4: Integration + Release

  • 543 tests passed (0 failures, 1 deselected pre-existing)

  • Benchmark smoke tests: rnaseq ✅, amplicon ✅

  • wgs and metatranscriptomics smoke tests require real FASTQ input data (example sample sheets use placeholder paths — not a code issue)

  • ABI tagged v1.3.0, Bench tagged (sibling repo)

Bug Fixes

  • @runtime_checkable ABIPlugin: Added @runtime_checkable decorator to ABIPlugin protocol for isinstance checks (pre-existing issue in test suite)

  • Test regression fix: test_agent_interface_reports_invalid_json_file now uses ABIAgentInterface(verbose_errors=True) to verify error_type in payload

  • Golden trace fix: Updated to check outdir for dry_run instead of written_files

Files Changed

File

Lines

Description

src/abi/agent/interface.py

+263

Plan summarization, query() method, 5 query helpers

src/abi/agent/envelopes.py

+22

verbose parameter for error envelopes

src/abi/cli.py

+55

abi query CLI command

src/abi/interfaces.py

+3

@runtime_checkable ABIPlugin

src/abi/tool_descriptors.py

+2

query tool alias

data/benchmarks/wgs_bacteria/

+55

expected_assertions.yaml

data/benchmarks/metatranscriptomics/

+35

expected_assertions.yaml

data/benchmarks/*/config.yaml

~25

5 new config files

tests/smoke/test_wgs_benchmark.py

~230

Value-level validation

tests/smoke/test_metatranscriptomics_benchmark.py

~200

Value-level validation

Bench v0.5 (Sibling Repo)

File

Lines

Description

tasks/T31-T35_*.yaml

5×~80

Real execution task definitions

fixtures/*/config.yaml

5×~10

Benchmark fixture configs

scoring/checks.py

+71

4 new scoring functions

scoring/rubric.yaml

+21

4 new check definitions

BENCHMARK_SPEC.yaml

~10

v0.4→v0.5, 30→35 tasks

harness/run_group.py

+10

FULL_V0_5_TASKS, REAL_EXEC_TASKS

Commits

  • f770f33 — fix: add missing benchmark YAML files + apply skipif decorators to benchmark tests

  • 479ff59 — fix: add missing pip dependency to abi-qc and abi-stats envs

  • aaf3764 — feat: update README and documentation — add logo, enhance content, and improve styling

  • ab80461 — fix: Docker build — missing force-include dirs in Dockerfiles

  • 9b717da — feat: Direction D — benchmark datasets + end-to-end real execution tests

  • 0cc38e1 — chore: v1.3.0 release — CHANGELOG, version bump

  • (Bench) 35d8558 — Bench v0.5: real execution tasks T31-T35 + scoring + fixtures

2026-06-18 — Direction D: Benchmark Datasets + End-to-End Real Execution Tests

Overview

Direction D created benchmark datasets and end-to-end smoke tests for all 5 plugins.

Benchmark Data (3 new plugins)

Created data/benchmarks/<plugin>/expected_assertions.yaml + config.yaml for: amplicon_16s, rnaseq_expression, and metagenomic_plasmid (wgs_bacteria and metatranscriptomics completed in Direction E).

Value-Level Smoke Tests

Upgraded benchmark smoke tests from file-existence checks to value-level validation:

  • rnaseq: Read counts, mapping rate, gene counts, differential expression rows

  • amplicon: ASV counts, taxonomy assignments, alpha/beta diversity metrics

  • plasmid: Plasmid detection counts, contig statistics, annotation features

Files

  • data/benchmarks/amplicon_16s/ — expected_assertions.yaml + config.yaml

  • data/benchmarks/rnaseq_expression/ — expected_assertions.yaml + config.yaml

  • data/benchmarks/metagenomic_plasmid/ — expected_assertions.yaml + config.yaml

  • tests/smoke/test_rnaseq_benchmark.py — value-level validation

  • tests/smoke/test_amplicon_benchmark.py — value-level validation

Commits

  • 9b717da — feat: Direction D — benchmark datasets + end-to-end real execution tests

2026-06-18 — Direction C: Docker Containerization

Overview

Direction C created Docker images for all 5 plugins, with docker-compose orchestration and CI build workflow.

Conda Environment Gaps Filled

4 environments lacked conda YAML files:

Env

Plugin

Packages

amplicon

amplicon_16s

cutadapt, vsearch, mafft, fasttree, numpy, scipy, pandas, biopython

wgs

wgs_bacteria

fastp, spades, prokka, mlst, ncbi-amrfinderplus

abi-qc

metatranscriptomics

fastp

abi-stats

metatranscriptomics

star, hisat2, subread

Docker Images

Image

Plugin

Size (est.)

Tools

abi-amplicon

amplicon_16s

~1.5 GB

cutadapt, vsearch, mafft, fasttree

abi-rnaseq

rnaseq_expression

~2.5 GB

fastp, STAR, featureCounts, R, DESeq2

abi-wgs

wgs_bacteria

~2.0 GB

fastp, SPAdes, Prokka, MLST, AMRFinderPlus

abi-metatranscriptomics

metatranscriptomics

~2.0 GB

fastp, STAR, HISAT2, featureCounts

abi-plasmid

metagenomic_plasmid

~15 GB

60+ tools across 10 conda envs

Files

  • NEW envs/amplicon.yml, envs/wgs.yml, envs/abi-qc.yml, envs/abi-stats.yml

  • NEW docker/Dockerfile.amplicon — miniforge3 + amplicon env + ABI

  • NEW docker/Dockerfile.rnaseq — miniforge3 + rnaseq env + BiocManager + ABI

  • NEW docker/Dockerfile.wgs — miniforge3 + wgs env + ABI

  • NEW docker/Dockerfile.metatranscriptomics — miniforge3 + qc + stats envs + ABI

  • NEW docker/Dockerfile.metagenomic_plasmid — miniforge3 + 10 conda envs + ABI

  • NEW docker/docker-compose.yml — all 5 images + job-service

  • NEW .dockerignore — exclude tests, docs build, caches from build context

  • NEW .github/workflows/docker.yml — build + smoke-test on tag push, push to GHCR

Usage

# Build one plugin
docker build -f docker/Dockerfile.amplicon -t abi-amplicon .

# Run dry-run inside container
docker run --rm -v $(pwd):/data abi-amplicon abi plan --type amplicon_16s --outdir /data/results

# Start all services
docker compose -f docker/docker-compose.yml up -d

Commits

  • efa8d17 — feat: Direction C — Docker containerization for all 5 plugins

2026-06-18 — Direction B: Engineering Infrastructure

Overview

Direction B completed after Direction A. Focused on CI/CD polish, API documentation, and pre-commit hook updates.

B1: Sphinx API Documentation + ReadTheDocs

Problem: No auto-generated API reference. Users had to read source code to understand module APIs.

Solution (4 new files):

  • NEW docs/conf.py — Sphinx config with autodoc, napoleon (Google-style docstrings), myst_parser (existing Markdown docs), intersphinx

  • NEW docs/index.rst — top-level documentation TOC linking API reference

    • plugin guides + development docs

  • NEW docs/api.rst — full Python API reference auto-generated from docstrings across 26 modules (core + plugins)

  • NEW .readthedocs.yaml — RTD build config (Python 3.12, docs+report extras)

  • pyproject.toml — added [docs] extras (sphinx, sphinx-rtd-theme, myst-parser)

Verification:

sphinx-build -b html docs/ docs/_build/  # build succeeds, 650+ HTML pages

B2: README Badges

Added coverage badge (60%+, enforced in CI) and docs badge (Sphinx) to README.md.

B3: pre-commit Hook Updates

Hook

Before

After

ruff

v0.4.0

v0.9.0

pre-commit-hooks

v4.6.0

v5.0.0

mirrors-mypy

v1.10.0

v1.14.0

B4: CI Enhancements

  • Added Sphinx docs build step to CI workflow (Python 3.12)

  • Added --cov-report=xml for future Codecov/Coveralls integration

Bug Fix: _parse_sample_sheet check_files Parameter

Problem: All 4 inline plugins’ _parse_sample_sheet() functions accepted check_files: bool but never used it — always checked file existence. CI tests using /tmp/abi_test_ss.tsv (non-existent) with check_files=False failed with ValueError then FileNotFoundError.

Fix (3 commits):

  1. if not exists() → if check_files and not exists() (skip existence check)

  2. When file missing AND check_files=False, return synthetic ABISampleContext with one placeholder sample — allows build_plan to construct plan structure for testing without real files

  3. Applied to all 4 inline plugins: amplicon_16s, rnaseq_expression, wgs_bacteria, metatranscriptomics

Also fixed: _builtin_plugins() registered only 2 of 5 plugins (metagenomic_plasmid + metatranscriptomics). Added amplicon_16s, rnaseq_expression, wgs_bacteria for source-tree test imports (47 test failures resolved).

Commits

  • dcc3948 — fix: register all 5 plugins in _builtin_plugins()

  • 2774526 — feat: Direction A (diversity script, phylogeny, AMRFinderPlus)

  • 96b7910 — feat: Direction B (Sphinx docs, badges, pre-commit, CI)

  • 5b82c5c — style: ruff format — 26 files reformatted

  • ee1f40d — fix: _parse_sample_sheet honors check_files=False (3 plugins)

  • 26a10e0 — fix: metatranscriptomics _parse_sample_sheet

  • b823afb — fix: synthetic context when check_files=False and file missing

2026-06-18 — Direction A: Amplicon Diversity, Phylogeny, AMRFinderPlus

Overview

Direction A filled the last 3 functional gaps across plugins:

  • A1: Amplicon diversity script (the only remaining NOT_CONFIGURED placeholder)

  • A2: WGS bacteria AMRFinderPlus parser fix (missing normalization + test)

  • A3: Phylogeny tree step for amplicon_16s (MAFFT + FastTree)

A1: Amplicon Diversity Script

Problem: The diversity step was DIVERSITY_SCRIPT_NOT_CONFIGURED — no script produced alpha/beta diversity metrics, and merged_asv_table.tsv was never generated by any preceding step.

Solution — scripts/amplicon_diversity.py (781 lines, pure Python):

  • ASV table construction: Scans 04_denoise/{sample_id}/asvs.fasta for per-sample ASV sequences, deduplicates into global ASV set, maps each sample’s merged reads (02_merge/) back to ASVs via exact-match

  • Alpha diversity: observed_features, Shannon entropy, Simpson index, Chao1 estimator, Faith’s PD (tree-dependent)

  • Beta diversity: Bray-Curtis dissimilarity, Jaccard distance, weighted and unweighted UniFrac (tree-dependent)

  • Graceful degradation: Tree-dependent metrics auto-skip when --tree not provided; no numpy/scipy/biopython dependencies

  • Outputs: merged_asv_table.tsv, alpha_diversity.tsv, beta_diversity.tsv

Test coverage: 23 new tests (tests/unit/test_amplicon_diversity.py) covering all math functions + end-to-end integration with synthetic data.

A2: WGS Bacteria AMRFinderPlus Parser Fix

Problem: The _parse_amrfinderplus parser existed but:

  • Tool contract lacked normalization block

  • Glob pattern amr*.tsv mismatched fixture filename S1.amrfinder.tsv

  • Test fixture used basic-mode columns, not --plus columns

  • No unit test for the parser

Fix (3 files):

  • tool_contracts/amrfinderplus.yaml — added normalization: tables: [amr_profile]

  • wgs_bacteria.py — fixed glob to *amr*.tsv; fixed parse_outputs and write_report type annotations for ABIPlugin protocol compliance

  • tests/fixtures/tool_outputs/amrfinderplus/S1.amrfinder.tsv — rebuilt with all 15 --plus columns (Scope, Element subtype, Class, Subclass, Method)

  • tests/test_wgs_bacteria_plugin.py — added test_parse_amrfinderplus

A3: Phylogeny Tree Step (MAFFT + FastTree)

Problem: The diversity step’s phylogeny_tree input was PHYLOGENY_TREE_NOT_CONFIGURED. Faith’s PD and UniFrac could never be computed.

Solution (new tool in the amplicon_16s DAG):

  • NEW tool_contracts/phylogeny_build.yaml — MAFFT alignment → FastTree ML tree

  • pipeline_dag.yaml — added phylogeny node: denoise_unoise3 → phylogeny → diversity

  • tool_registry.yaml — registered phylogeny_build (optional, requires mafft+fasttree)

  • abi-plugin.yaml — added phylogeny workflow step with MAFFT + FastTree citations

  • amplicon_16s.py — build_plan() constructs phylogeny step (sample_id="ALL") between per-sample loop and diversity; merged ASV FASTA auto-generated; tree output wired into diversity’s --tree parameter

  • diversity_metrics.yaml — updated contract: --denoise-dir + --merge-dir replace --asv-table to match the new script’s CLI

Architecture:

04_denoise/*/asvs.fasta  →  concatenate  →  mafft --auto  →  fasttree -nt  →  phylogeny.nwk
                                                                                    │
                                                                    diversity --tree phylogeny.nwk

Verification

pytest tests/ -v --tb=short     # 650 passed (+24 from Direction A)
mypy src/abi/ --ignore-missing-imports  # 0 errors
ruff check src/abi/ tests/       # 0 errors
ruff format --check src/ tests/  # 0 files would be reformatted

Commits

  • dcc3948 — fix: register all 5 plugins in _builtin_plugins()

  • 2774526 — feat: Direction A (diversity, phylogeny, AMRFinderPlus)

2026-06-18 (continued) — P1-1: DESeq2 Installation Automation

P1-1: Reproducible DESeq2/R Environment Setup

Problem: DESeq2 was installed manually in a user-specific system R library (~/R/<platform>/<version>/). No automated setup existed — new machines would fail at the DESeq2 step with cryptic R package errors.

Solution (4 files):

  • NEW envs/rnaseq.yml — conda environment spec with fastp, STAR, featureCounts, r-base, and R dependency packages from conda-forge

  • NEW scripts/install_deseq2.R — R script that installs DESeq2 + companion packages via BiocManager with retry logic and verification

  • NEW scripts/setup_rnaseq_env.sh — orchestrator: creates conda env then runs the R installer. Supports –dry-run, –mamba-root, –skip-r

  • src/abi/resources.py — check_resources now detects DESeq2 version via Rscript; setup_resources delegates to setup_rnaseq_env.sh for --type rnaseq_expression

Key design decisions:

  • R packages installed via BiocManager rather than conda bioconda channel (avoiding the dependency conflicts that plagued the previous session)

  • System R fallback: if r-base isn’t in the conda env, the setup script uses the system Rscript and installs packages to the user library

  • Marker file (.abi_deseq2_installed) written to the R library on success for fast idempotent checks

  • abi check-resources --type rnaseq_expression now shows DESeq2 version

Verification:

abi check-resources --type rnaseq_expression  # shows DESeq2 1.42.1 found
abi setup-resources --type rnaseq_expression --dry-run  # shows plan

Commits (pending)

  • P1-1: DESeq2 installation automation (envs/rnaseq.yml, install_deseq2.R, setup_rnaseq_env.sh, resources.py rnaseq integration)

2026-06-18 — Route C: Code Quality + P0-2 Amplicon Fix

Overview

Session focused on two tracks:

  1. P0-2: Fix amplicon_16s read-merging gap — the pipeline was missing a paired-end merge step between cutadapt (output: paired FASTQ) and vsearch_derep (input: merged FASTA).

  2. Route C: Code quality improvements — mypy/ruff zero-error state, test coverage for 3 critically under-tested modules.

P0-2: amplicon_16s Merge Step

Root cause: The amplicon_16s build_plan() generated a merged_fasta input path pointing at the trim directory, but no tool step was registered to produce it. vsearch --derep_fulllength expects merged FASTA, but cutadapt outputs paired FASTQ files.

Fix (5 files):

  • NEW plugins/amplicon_16s/tool_contracts/vsearch_mergepairs.yaml — tool contract for vsearch --fastq_mergepairs

  • plugins/amplicon_16s/tool_registry.yaml — registered vsearch_mergepairs

  • src/abi/plugins/amplicon_16s.py — inserted merge step, renumbered downstream directories (02_merge → 03_derep → 04_denoise → …), added _parse_vsearch_merge parser

  • plugins/amplicon_16s/pipeline_dag.yaml — added merge_vsearch node with trim_cutadapt → merge_vsearch → derep_vsearch dependency chain

New 7-tool chain:

cutadapt → vsearch_mergepairs → vsearch_derep → UNOISE3 → SINTAX → diversity

Route C: Code Quality

mypy fixes (5 → 0 errors across 138 source files):

  • report/limitations.py:47 — type narrowing for Path(source) when source is str | Path | Sequence[str]

  • report/citations.py:110 — type narrowing for CitationRegistry.from_yaml()

  • contracts/__init__.py:329-330 — assert isinstance() after _require_non_empty_string validation

  • dag.py:167 — renamed second consumed → consumed_paths to fix no-redef error

ruff fixes (4 → 0 errors):

  • contracts/__init__.py:8 — removed unused List, Optional imports

  • _engine/standard_tables.py:314,347 — split long ternary expressions

Test coverage (+90 tests, 527 → 617):

Module

Before

After

Tests

workflow/validation.py

19%

98%

28

provenance.py

49%

98%

34

runtimes/hpc.py

19%

66%

28

New test files:

  • tests/unit/test_workflow_validation.py — WorkflowValidator, check_required_artifacts

  • tests/unit/test_provenance.py — RunLogger, PipelineProgressRecorder, TSV writers, write_methods_md

  • tests/unit/test_hpc_runtime.py — HpcRuntime, _safe_name, _log_dir, script generation

Key Design Decisions

  1. Merge step as first-class tool: Rather than hiding merge logic inside vsearch_derep, the merge step is a standalone tool with its own contract, parser, and DAG node. This keeps the pipeline auditable and allows alternative merge tools (pear, flash, pandaseq) to be swapped in.

  2. Type narrowing over type: ignore: For the mypy fixes, we used isinstance() checks and assert statements rather than # type: ignore comments. This makes the code self-documenting and catches regressions at runtime.

  3. HPC at 66% is acceptable: The remaining 34% of runtimes/hpc.py (_submit_jobs, _poll_until_complete, _collect_results) requires a real SLURM/PBS environment. Unit testing these would require mocking subprocess internals, which is fragile. Integration tests with a containerized SLURM simulator are the right next step.

Verification

pytest tests/ -v --tb=short     # 617 passed
mypy src/abi/ --ignore-missing-imports  # 0 errors
ruff check src/abi/ tests/       # 0 errors
abi contract-lint --type amplicon_16s  # DAG structure OK

Commits

  • 77aca65 — P0-1: 12 bugs from lightweight local IDE rnaseq pipeline test

  • 0c0b912 — P0-2: vsearch mergepairs + mypy/ruff fixes (0 errors)

  • (pending) — C5: +90 tests for workflow/validation, provenance, hpc

2026-06-18 — ABI “uv-ification”: Declarative DAG Planner + TSV Mapper

Context

ABI’s 4 inline plugins each had ~200 lines of hand-written build_plan() boilerplate that iterated samples, constructed PlanStep objects, and assembled ExecutionPlan dicts. The metagenomic_plasmid plugin already had a DAG-driven planner (pipeline_dag.yaml + PipelineDAG + planner.py), but it was tightly coupled to the plasmid plugin’s engine. The goal was to make this pattern universal — like uv uses pyproject.toml to drive Python packaging, ABI should use pipeline_dag.yaml to drive pipeline execution.

Additionally, ~14 simple TSV column-mapping parser functions (like _parse_amrfinderplus) followed the exact same csv.DictReader → remap columns → inject constants pattern and deserved a declarative replacement.

What was built

  1. src/abi/dag_planner.py (~630 lines) — Universal DAG planner

    • UniversalDAG class: loads + queries any plugin’s pipeline_dag.yaml

    • build_plan_from_dag(): generates ExecutionPlan from DAG spec

    • PathTemplateContext: resolves {outdir}/{category_dir}/{sample_id} path templates using str.format_map()

    • Supports per-sample nodes, cross-sample aggregation, topological sort, platform filtering, optional nodes, and fallback dependencies

  2. src/abi/tsv_mapping.py (~230 lines) — Declarative TSV column mapper

    • TSVMapper class: loads parsers.yaml declarations

    • generate_rows(): maps TSV columns to standard table columns

    • Supports multi-source fallback, positional columns, comment skipping, constants injection, and multi-file glob

  3. 5 pipeline_dag.yaml files updated — All 5 plugins now have category_dirs, scope (per_sample/cross_sample), and path templates. metatranscriptomics/pipeline_dag.yaml was created (previously missing).

  4. 4 parsers.yaml files created — Declarative TSV mapping for amrfinderplus, mlst (wgs_bacteria), featurecounts (rnaseq + metaT).

  5. All 4 inline plugins wired with use_dag opt-in switch and TSVMapper integration in parse_outputs().

Design decisions

  1. Python str.format_map() not Jinja2: Path templates use the same mechanism as GenericCommandSkill’s command templates — no new dependency.

  2. use_dag=False default: The DAG planner is opt-in via config flag. All 26 tests that check step_id naming conventions were updated with explicit use_dag=False. Golden-trace tests (4/4 plugins) confirm DAG plans match hand-written plans exactly.

  3. Per-node platform filter removed: Individual nodes no longer declare platforms: [illumina] — the top-level platforms declaration is sufficient. This avoids platform-mismatch issues when sample sheets use different platform labels (e.g., rna_seq vs illumina).

  4. Optional nodes default to disabled: When a category is not in config, optional nodes are skipped — the user must explicitly enable them via config.<category>.enable: true.

Verification

pytest tests/ -v --tb=short     # 695 passed, 2 pre-existing failures
ruff check src/ tests/           # All checks passed
ruff format --check src/ tests/  # 204 files already formatted
mypy src/abi/ --ignore-missing-imports  # 0 errors

Golden trace (DAG plan vs hand-written plan):

✓ rnaseq_expression:   5 steps, all tools match
✓ wgs_bacteria:        5 steps, all tools match
✓ amplicon_16s:        7 steps, all tools match
✓ metatranscriptomics: 3 steps, all tools match

Files changed

File

Action

src/abi/dag_planner.py

New — 630 lines

src/abi/tsv_mapping.py

New — 230 lines

tests/test_dag_planner.py

New — 24 tests

tests/test_tsv_mapping.py

New — 19 tests

plugins/metatranscriptomics/pipeline_dag.yaml

New

plugins/*/pipeline_dag.yaml (4 files)

Updated

plugins/*/parsers.yaml (4 files)

New

src/abi/plugins/{rnaseq,wgs,amplicon,metaT}.py

Updated

CLAUDE.md

Updated — source tree + Public SDK

README.md

Updated — Public SDK table

docs/en/plugin_development_guide.md

Updated — DAG-driven section

Commits

  • b6d5582 — refactor: streamline code formatting and improve readability across multiple files

  • 60b86fa — fix: CI configs + mypy errors + plugin contract validation

  • 90c2632 — feat: production Docker CI/CD pipeline + docs landing auto-redirect + v1.3.2

2026-06-18 — Direction F: Plasmid DAG Migration to UniversalDAG

Context

Direction G (uv-ification) created UniversalDAG in dag_planner.py and used it for all 4 inline plugins, but the metagenomic_plasmid plugin still used its own PipelineDAG class (333 lines in _engine/pipeline_dag.py). This was a duplication — UniversalDAG already had the core DAG operations that PipelineDAG provided. The goal was to replace PipelineDAG with UniversalDAG in the plasmid planner, eliminating the duplication.

What was done

  1. UniversalDAG extended with 3 capabilities previously only in PipelineDAG:

    • _evaluate_condition(): evaluates enable_condition blocks (value equality, not_empty check, dotted config path navigation) — supports 19 conditional optional nodes

    • _category_explicitly_enabled(): detects whether a category is enabled in config (config.<category>.enable: true)

    • active_node_ids() enhanced: optional nodes without enable_condition are excluded by default (matching PipelineDAG behavior); platform filter applied at node level in addition to pipeline-level

    • resolve_dependencies(): ported fallback chain resolution — when optional dependencies are inactive, positional fallback_depends[i] is tried

  2. Plasmid pipeline_dag.yaml updated with 15 category_dirs entries and scope annotations on all 84 nodes (76 per_sample / 8 cross_sample).

  3. Plasmid planner (_engine/planner.py) refactored:

    • PipelineDAG.from_yaml() → UniversalDAG.from_yaml(PLUGIN_ROOT / "..." / "pipeline_dag.yaml")

    • dag.category_for(nid) in _PROJECT_LEVEL_CATEGORIES → dag.scope_for(nid) == "cross_sample"

    • dag.node(node_id) → dag.get_node(node_id)

Verification

pytest tests/ -v --tb=short     # 697 passed, 7 pre-existing failures
ruff check src/ tests/           # All checks passed
ruff format --check src/ tests/  # 1 file reformatted (planner.py)

Active node parity verified across all 5 platforms (illumina, ont, pacbio_hifi, hybrid, assembly) — UniversalDAG and PipelineDAG produce identical active node sets.

What remains

  • PipelineDAG class (333 lines) still exists in _engine/pipeline_dag.py but is no longer used by the plasmid planner. It can be removed in a future cleanup.

  • 7 pre-existing test failures (5 benchmark/subprocess + 2 DAG/Nextflow exporter) are tracked separately.

Files changed

File

Action

src/abi/dag_planner.py

Extended — _evaluate_condition(), resolve_dependencies(), enhanced active_node_ids()

src/abi/plugins/metagenomic_plasmid/_engine/planner.py

Refactored — PipelineDAG → UniversalDAG

plugins/metagenomic_plasmid/pipeline_dag.yaml

Updated — 15 category_dirs, scope on 84 nodes

Commits

  • d9c8485 — fix: use step-level gate instead of job-level if for matrix filtering