Refactor workflow to DAG architecture with expanded testing, CI modernisation, and validated numerical equivalence

[harryswift01]

Summary

This PR introduces a major architectural refactor that transitions CodeEntropy to a DAG-based execution model while preserving numerical behaviour. The refactor improves modularity, maintainability, and testability, and modernises the project’s testing, CI, and tooling infrastructure.

All systems and component-level contributions match the previous implementation within floating-point tolerance (maximum absolute difference 2.45e-08).

No breaking changes to user-facing behaviour are expected.

Motivation

The previous workflow was primarily procedural, which made it harder to reason about execution order, extend functionality, and test individual stages. Moving to a DAG model improves separation of concerns, enables clearer data flow, and provides a stronger foundation for future development while maintaining numerical equivalence.

Changes

DAG-based workflow architecture

• Refines orchestration separating static setup, per-frame execution, and reducers.
• Decomposes workflow into smaller nodes (detection, bead construction, covariance, reducers).
• Standardises shared context passing between nodes.
• Implements incremental reduction (streaming mean) for covariance accumulation.

Frame-level covariance computation

• Introduces FrameCovarianceNode for per-frame second-moment matrices.
• Adds optional combined force-torque block matrix generation at highest level.
• Standardises axis handling via axes_manager.
• Improves robustness for missing beads and metadata.

CLI and job-folder execution model

• Ensures consistent job folder creation for each run.
• Guarantees output artifacts land in job directories.

ResultsReporter improvements

• Reworks JSON output into grouped structure:
  groups: { "": { components: {...}, total: ... } }
• Groups console tables by Group ID.
• Adds metadata sections (args, provenance).
• Adds utilities for argument serialization and git SHA detection.

Example output JSON.

{
  "args": {
    "top_traj_file": [
      "/home/tdo96567/BioSim/test_data/dna/md_A4_dna.tpr",
      "/home/tdo96567/BioSim/test_data/dna/md_A4_dna_xf.trr"
    ],
    "force_file": null,
    "file_format": null,
    "kcal_force_units": false,
    "selection_string": "all",
    "start": 0,
    "end": 1,
    "step": 1,
    "bin_width": 30,
    "temperature": 298.0,
    "verbose": false,
    "output_file": "/home/tdo96567/BioSim/temp/refactor/1-frame/job001/output_file.json",
    "force_partitioning": 0.5,
    "water_entropy": true,
    "grouping": "molecules",
    "combined_forcetorque": true,
    "customised_axes": true
  },
  "provenance": {
    "python": "3.14.0",
    "platform": "Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",
    "codeentropy_version": "1.0.7",
    "git_sha": "cb22762349b99c149f13392d9280acb4dffec976"
  },
  "groups": {
    "0": {
      "components": {
        "united_atom:Transvibrational": 0.0,
        "united_atom:Rovibrational": 0.002160679012128457,
        "residue:Transvibrational": 0.0,
        "residue:Rovibrational": 3.376800684085249,
        "polymer:FTmat-Transvibrational": 12.341104347192612,
        "polymer:FTmat-Rovibrational": 0.0,
        "united_atom:Conformational": 7.269386795471401,
        "residue:Conformational": 0.0
      },
      "total": 22.989452505761392
    },
    "1": {
      "components": {
        "united_atom:Transvibrational": 0.0,
        "united_atom:Rovibrational": 0.01846427765949586,
        "residue:Transvibrational": 0.0,
        "residue:Rovibrational": 2.3863201082544565,
        "polymer:FTmat-Transvibrational": 11.11037253388596,
        "polymer:FTmat-Rovibrational": 0.0,
        "united_atom:Conformational": 6.410455987098191,
        "residue:Conformational": 0.46183561256411515
      },
      "total": 20.387448519462218
    }
  }

Testing architecture overhaul

• Expands unit test coverage across previously untested branches.
• Adds regression test harness running CLI in isolated temp directories.
• Ensures deterministic job folder creation during tests.
• Adds baseline comparison against stored JSON outputs.

Regression dataset system

• Automatic dataset download from CCPBioSim HTTPS filestore.
• Local caching in .testdata/.
• Intelligent detection of required files.
• No manual setup required.

Quick vs slow regression separation

• Introduces slow marker for long-running systems.
• Quick regression suite excludes slow tests for fast feedback.
• Full regression suite runs in weekly workflows.

Developer and tooling improvements

• Standardises pytest markers and commands.
• Adds --update-baselines workflow.
• Improves debug diagnostics and artifact capture.
• Migrates linting and formatting to Ruff.
• Removes Black, Flake8, and isort.
• Updates pre-commit configuration.
• Simplifies optional dependencies.

CI/CD modernisation

• Multi-OS testing (Linux, macOS, Windows).
• Python matrix (3.12–3.14).
• Quick regression tests on PRs.
• Weekly full regression workflow.
• Weekly docs build across Python versions.
• Daily validation workflow.
• Artifact upload on failures.
• Dataset caching in CI.

Documentation pipeline

• Docs build validation on PRs.
• Weekly docs compatibility checks.
• Updates developer guide reflecting new workflows.

Logging and error handling improvements

• Eliminates duplicate traceback logging.
• Centralises error boundary in CLI.
• Improves exception chaining.
• Adds argument logging on runtime failures.

Impact

• Improves maintainability through DAG decomposition.
• Increases confidence via expanded unit and regression coverage.
• Provides faster CI feedback with quick regression separation.
• Improves reproducibility with provenance metadata.
• Simplifies developer setup with automatic datasets.
• Modernises tooling stack with faster linting.
• Improves cross-platform reliability via expanded CI matrices.
• Provides clearer debugging through improved logging and artifacts.

Regression validation results

CodeEntropy Graph Implementation.xlsx

Entropy outputs from the refactored DAG workflow were compared against the previous implementation across all systems and component types. All values agree within floating-point tolerance.

Maximum absolute difference across all systems: 2.45e-08

This comparison confirms agreement across all individual contributions, not only group totals. Observed differences are consistent with expected floating-point variation introduced by consolidating numerical operations into NumPy.