From 023aa278370017d8c42d9cc350137e14084a6918 Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 18:57:08 +0200 Subject: [PATCH 1/9] LCR-85: Add multi-harness skill system with --tools flags Extract Claude-Code-specific skill installation into a harness-agnostic system. New harness registry maps tool IDs (Claude Code, Codex, Cursor, GitHub Copilot, OpenCode) to per-tool install paths. Skills, agents, and guides are copied to all selected harnesses; hooks and settings remain Claude Code only. - Add src/lightcone/cli/harness.py (registry module) - Add --tools flag to lc init and lc update --sync - Refactor _install_harnesses to loop over selected harnesses - Add _display_install_summary with post-install output - Add tests/test_harness.py (unit tests for registry) - Add tests/test_harness_install.py (CLI integration tests) Co-Authored-By: Claude Opus 4.6 --- docs/adr/0001-adopt-dagster-slurm.md | 348 +++++++++++++++++++++++++++ src/lightcone/cli/commands.py | 320 ++++++++++++++---------- src/lightcone/cli/harness.py | 149 ++++++++++++ tests/test_harness.py | 151 ++++++++++++ tests/test_harness_install.py | 191 +++++++++++++++ 5 files changed, 1036 insertions(+), 123 deletions(-) create mode 100644 docs/adr/0001-adopt-dagster-slurm.md create mode 100644 src/lightcone/cli/harness.py create mode 100644 tests/test_harness.py create mode 100644 tests/test_harness_install.py diff --git a/docs/adr/0001-adopt-dagster-slurm.md b/docs/adr/0001-adopt-dagster-slurm.md new file mode 100644 index 00000000..97dcd654 --- /dev/null +++ b/docs/adr/0001-adopt-dagster-slurm.md @@ -0,0 +1,348 @@ +# ADR-0001 — Replace Prism's local and SLURM execution backends with `dagster-slurm` + +- **Status:** Proposed +- **Date:** 2026-04-17 +- **Deciders:** Prism maintainers +- **Affects:** `src/prism/dagster/runner.py`, `src/prism/dagster/assets.py`, `src/prism/dagster/targets.py`, `src/prism/dagster/site_registry.py`, `src/prism/container.py`, `src/prism/cli.py`, tests in `tests/` +- **Related upstream:** [`dagster-slurm`](https://github.com/ascii-supply-networks/dagster-slurm) (v1.13.0), docs at + +## 1. Context + +Prism currently ships a bespoke execution layer (`ASTRAContainerRunner` in `src/prism/dagster/runner.py`) that implements four backends inside a single ~1000-line class: **docker**, **local**, **venv**, and **slurm**. The SLURM backend generates sbatch scripts (`results/.slurm/`), submits with `sbatch`, polls with `sacct` falling back to `squeue`, and wraps commands in `podman-hpc run` when a container is configured. The local backend is a plain `subprocess.Popen` with streaming output, no environment management beyond a `python` substitution. There is no pluggable-backend abstraction; adding or changing a backend requires editing the monolithic class. + +This layer is the weakest part of Prism. It duplicates work that the wider Dagster ecosystem is solving in the open: real-time log streaming over SSH, account/QoS/reservation handling for multiple sites, Slurm job cancellation semantics, heterogeneous jobs, cluster reuse, and metrics collection via `sacct`. Our own tests cover the happy path of sbatch script generation and sacct parsing, but the polling loop, squeue fallback, timeout handling, and podman-hpc resolution are under-tested and have surfaced several pain points already (see "Known limitations" in the runner analysis attached to this review). + +The `dagster-slurm` library — published by ASCII Supply Networks and used in production at multiple Austrian HPC centers — is explicitly designed for the scenario Prism is trying to support: the same Dagster asset runs on a laptop, on a Docker-emulated Slurm cluster for CI, and on a real cluster, with only the resource wiring changing. It exposes a single `ComputeResource` facade with `mode ∈ {local, slurm, slurm-session, slurm-hetjob}` and a Dagster Pipes–based protocol for log streaming, metadata exchange, and materialization events. + +The user request that motivates this ADR is explicit: "design a workflow that can execute seamlessly across SLURM computing centers as well as local machines". That is precisely `dagster-slurm`'s charter. + +## 2. Decision + +We replace Prism's `local` and `slurm` execution backends with `dagster-slurm`'s `ComputeResource`. The **Docker** backend (`runner._run_docker`, `container.build_image_docker`) is **kept unchanged** because dagster-slurm has no concept of container-per-asset execution, and Prism's Docker backend is the primary way our users achieve hermetic local runs today. The **venv** backend is retired — dagster-slurm's pixi-based environment packaging supersedes it for local execution, and we will not maintain two similar abstractions. + +In the target architecture, Prism's asset factory no longer calls `runner.execute(command, ...)`. Instead, each Dagster asset is wired with a `ComputeResource` and invokes `compute.run(context, payload_path, extra_env, extra_slurm_opts)`. The heavy lifting — SSH pooling, sbatch submission, `sacct`/`squeue` polling, log streaming, pipes message ingestion, metrics collection — lives in dagster-slurm. + +Concretely: + +- `ASTRAContainerRunner` is split. A thin `DockerRunner` (≈150 lines, extracted from current code) remains for the Docker backend. SLURM and local paths are deleted. +- A new `src/prism/dagster/compute_adapter.py` builds a `ComputeResource` from a Prism target YAML and exposes the single call that `assets.py` needs. +- Prism recipes are translated into `dagster-slurm` payloads by a generated wrapper script (see §4.3) so that ASTRA's "a recipe is a shell command" invariant is preserved. +- Target config is migrated from Prism's custom schema (`scheduler:`, `poll:`, `resource_limits:`) into dagster-slurm's `SlurmResource` / `SlurmQueueConfig` / `SSHConnectionResource` shapes, wrapped by a stable Prism-level schema. +- Python floor moves from 3.11 to 3.12 (dagster-slurm's hard requirement); Dagster is bumped from 1.9 → 1.13. + +## 3. Consequences + +### 3.1 What we gain + +The upstream library gives us, for free, a set of features we would otherwise have to build or defer: real-time stdout/stderr streaming over SSH with automatic fallback from `ControlMaster` to polling for HPC centers that disable socket multiplexing; `sacct`-based metrics (CPU efficiency, max RSS, node-hours) emitted as Dagster output metadata; Dagster Pipes integration that lets a compute payload emit `AssetMaterialization` and asset-check events back to the run without stdout parsing; safe retry and restart semantics driven by Dagster run tags; a session mode that holds a long-lived Slurm allocation across multiple assets (valuable for any multi-asset astra.yaml that shares a cluster reservation); and a working Docker-compose SLURM emulator we can use for CI. + +The code delta is large in the negative direction: approximately 750 lines of Prism's SLURM, local, and venv runner code are deleted, replaced by roughly 250 lines of adapter, payload generation, and configuration translation. Net reduction is on the order of 500 lines in a module that has been a recurring source of bugs. + +### 3.2 What we give up or complicate + +We lose built-in `podman-hpc` containerization on SLURM. `dagster-slurm` has no container primitive — environment isolation is achieved by `pixi pack` producing a self-extracting tarball that is uploaded to the cluster and sourced in the sbatch script. For sites that rely on `podman-hpc` image builds today (Perlmutter), we must either (a) wrap the recipe command in an explicit `podman-hpc run` invocation inside the generated payload wrapper (see §4.3); or (b) accept pixi-pack isolation as the new hermetic primitive and retire `container.py`'s podman-hpc build path. This ADR recommends (a) as the migration path and deferring (b) to a follow-up decision record; we do not want to couple a runner swap to an isolation-model change. + +We inherit a Python 3.12 floor. Prism already runs under 3.12 in its `.venv`, so this is more a declaration than a lift. Consumers on 3.11 will be broken once merged — this is a semver-minor breaking change from their perspective. + +We inherit `dagster-slurm`'s gaps: no support for SLURM `--constraint` out of the box (Prism exposes this via `extra_slurm_args`; we must either upstream a PR or emit `#SBATCH --constraint` from our payload wrapper); limited multi-hop SSH (single jump host only); remote paths live under `remote_base/runs/{run_id}/`, not `results/.slurm/` (we fetch logs and sbatch scripts back into `results/.slurm/` post-execution to preserve the current artifact layout). + +We accept a coupling to Dagster 1.13 pinned by dagster-slurm. Future upgrades are gated on the upstream bumping its own pin. + +### 3.3 Risk summary + +The highest-risk item is the container-isolation gap on HPC sites. The second-highest is the ASTRA recipe-model mismatch: dagster-slurm expects a Python payload file, Prism expects a shell command with injected CLI args. The mitigation for both is the payload wrapper described in §4.3, which is small, deterministic, and unit-testable without Docker. + +## 4. Detailed design + +### 4.1 Module-by-module plan + +`src/prism/dagster/runner.py` loses everything Slurm-, venv-, and local-related: `_run_slurm`, `_run_slurm_interactive`, `_run_local`, `_run_venv`, `generate_sbatch_script`, `translate_resources_to_slurm_directives`, `_podman_hpc_run_command`, `_normalise_time_limit`, `_parse_sbatch_job_id`, `_poll_slurm_job`, `_check_sacct`, `_check_squeue_fallback`, and the venv dependency-hash caching logic. What remains is a `DockerRunner` class with the current Docker fallback-to-local behavior removed (local is no longer a fallback; Docker failure is now a hard failure with a clear error message). The module shrinks from ~1000 to ~180 lines. + +`src/prism/dagster/compute_adapter.py` is new. It owns three things: (1) loading a Prism target YAML and constructing a `ComputeResource` with the correct `mode` and subordinate `SlurmResource` / `SSHConnectionResource` / `SlurmQueueConfig` objects; (2) building the payload wrapper script for a given recipe (§4.3); (3) staging dagster-slurm's post-execution artifacts (sbatch script, stdout, stderr, metrics) into `results/.slurm/{output_id}_{universe_id}.*` so the current on-disk layout is preserved. This module is the new seam for tests and for future HPC site additions. + +`src/prism/dagster/assets.py` changes minimally. The `_build_single_asset` body no longer constructs a command string and calls `runner.execute`; instead it writes a payload wrapper via `compute_adapter.prepare_payload(recipe, universe_id, params, external_inputs)` and calls `compute.run(context, payload_path=wrapper_path, extra_slurm_opts=_resources_to_slurm_opts(recipe.resources))`. The asset factory still receives a `runner` parameter for backwards compatibility with Docker-only targets; when the target mode is `local` or `slurm`, it resolves to a `ComputeResource` via the adapter. + +`src/prism/dagster/targets.py` gets a new `TargetKind` enum: `docker | local | slurm | slurm-session`. Loaders for `slurm` and `local` now validate against dagster-slurm's config shape. Existing YAML files need a one-time migration (§4.6). + +`src/prism/dagster/site_registry.py` retains its account-suffix resolution and node-type defaults but no longer emits Prism-native `scheduler_config` dicts. It emits `SlurmQueueConfig` fragments that the adapter merges into the final `ComputeResource`. + +`src/prism/container.py` loses `build_image_podman_hpc`, `_podman_hpc_migrate`, `image_exists_podman_hpc`, and `resolve_container_for_slurm`. Docker/Podman build and tag-computation logic stays. The `podman-hpc` container runtime is no longer selectable in target configs (the `container_runtime` field is removed); HPC container execution is handled by the payload wrapper's `podman-hpc run` invocation if the target opts in. + +`src/prism/cli.py` loses its SLURM-specific passthrough for `--partition` / `--qos` / `--constraint` / `--account`. These now live in the target YAML and are overridable per-run via a new `--slurm-opts key=value,...` flag that lands in `extra_slurm_opts` on the compute resource. `prism setup` and `prism target` UX stays visually similar but writes dagster-slurm–shaped configs. + +### 4.2 Interface mapping + +The current runner interface: + +```python +runner.execute( + command: str, # "python scripts/compute.py" + container: str | None, + inputs: list[str], + output_id: str, + universe_id: str, + resources: dict[str, Any], # {cpus, memory, gpus, time_limit, nodes} + params: dict[str, Any], # decision params from universes/{id}.yaml + external_inputs: dict[str, str] | None, + cwd_override: str | None, +) -> ExecutionResult # {exit_code, output_path, metadata} +``` + +The new equivalent, resolved through the adapter: + +```python +payload_path, ctx = compute_adapter.prepare_payload( + recipe=recipe, universe_id=universe_id, params=params, + external_inputs=external_inputs, cwd_override=cwd_override, + output_id=output_id, project_root=project_root, +) +completed = compute.run( + context=dagster_context, + payload_path=payload_path, + extra_slurm_opts=_resources_to_slurm_opts(recipe.resources), + extra_env={"ASTRA_UNIVERSE": universe_id, "ASTRA_OUTPUT": output_id}, + extra_files=_resolve_external_inputs(external_inputs), +) +yield from completed.get_results() # Dagster events +compute_adapter.stage_artifacts(completed, ctx) # copy logs/script into results/.slurm/ +``` + +`resources` → `extra_slurm_opts` translation is direct: `cpus → cpus_per_task`, `memory → mem`, `gpus → gpus_per_node`, `nodes → nodes`, `time_limit → time_limit` (format normalized to `HH:MM:SS`). The `--constraint` value, if present in target config, is injected into the sbatch script header by the payload wrapper (see §4.3) because dagster-slurm's `_build_sbatch_command` does not accept arbitrary sbatch flags. + +Return shape: `ExecutionResult` is retired. `compute.run()` returns a `PipesClientCompletedInvocation`; metrics and exit codes are surfaced as Dagster asset metadata by `get_results()`. The `output_path` field (always `results/{universe_id}/`) is redundant — the IO manager is the source of truth for it. + +### 4.3 Payload wrapper — the ASTRA → pipes shim + +Prism recipes are shell commands. dagster-slurm expects a Python payload that opens a pipes session. The gap is bridged by a small generated wrapper written per-materialization to a deterministic path: + +```python +# results/.payloads/{universe_id}__{output_id}.py (auto-generated, do not edit) +from dagster_pipes import open_dagster_pipes +import os, subprocess, shlex, sys, json + +CMD = {command_quoted} # "python scripts/compute.py" +CLI_ARGS = {cli_args_json} # ["--universe", "baseline", "--method", "option_a"] +CWD = {cwd_json_or_none} +CONTAINER_WRAP = {container_wrap_or_none} # ["podman-hpc", "run", "--rm", ...] +WORKDIR = {workdir_json} + +with open_dagster_pipes() as pipes: + pipes.log.info(f"astra recipe: {CMD} {' '.join(CLI_ARGS)}") + full = shlex.split(CMD) + CLI_ARGS + if CONTAINER_WRAP: + full = CONTAINER_WRAP + full + proc = subprocess.run(full, cwd=CWD or WORKDIR, check=False) + pipes.report_asset_materialization( + metadata={"exit_code": proc.returncode, + "command": " ".join(shlex.quote(a) for a in full)} + ) + sys.exit(proc.returncode) +``` + +Three reasons this wrapper design is chosen over "just change ASTRA recipes to be Python scripts": + +- ASTRA is a spec maintained separately and is explicitly "pure specification". Changing the recipe model to require a pipes-native Python entry point would leak execution-layer concerns back into the spec. Prism is the agentic layer; shims belong in Prism. +- The wrapper is small, deterministic, and easy to regenerate. Its contents are a pure function of `(recipe.command, resources, cli_args, container, external_inputs)` — cacheable, testable, and diff-able across runs. +- If a recipe author does write a pipes-native payload in the future, we keep the door open by honoring a `recipe.payload: path/to/script.py` field that bypasses the wrapper and passes the script directly to `compute.run`. + +The wrapper also owns `--constraint` and any other sbatch flag dagster-slurm does not accept, by writing a `#SBATCH --constraint=...` line into the *first* comment block of the script itself — sbatch accepts these directives even when the submission command does not specify them. This is a documented upstream pattern and is already exercised by dagster-slurm users for reservations on sites that expose non-standard constraint strings. + +### 4.4 Target config translation + +Before: + +```yaml +# ~/.prism/targets/perlmutter.yaml (current) +site: perlmutter +backend: slurm +connection: + hostname: perlmutter.nersc.gov +scheduler: + account: m1234 + qos: debug + partition: gpu + constraint: gpu + container_runtime: podman-hpc + container_flags: ["--gpu", "--mpi"] + extra_slurm_args: [] +poll: + interval_seconds: 15 + timeout_seconds: 14400 +resource_limits: + max_walltime_minutes: 360 +``` + +After: + +```yaml +# ~/.prism/targets/perlmutter.yaml (new) +name: perlmutter +mode: slurm # local | slurm | slurm-session | docker +site: perlmutter # still used by site_registry for account suffix +ssh: + host: perlmutter.nersc.gov + user: ${USER} + key_path: ~/.ssh/nersc +queue: + partition: gpu + account: m1234 + qos: debug + time_limit: "00:30:00" + cpus: 4 + mem_per_cpu: 4G + gpus_per_node: 0 +remote_base: /pscratch/sd/a/${USER}/prism-runs +# HPC container execution (optional, Prism-level, wrapped in payload): +container: + runtime: podman-hpc # podman-hpc | apptainer | none + flags: ["--gpu", "--mpi"] +# Raw sbatch escape hatch — appended to every job's #SBATCH header: +extra_sbatch_directives: + - "--constraint=gpu" +poll: + timeout_seconds: 14400 # passed through to compute.run(poll_timeout=) +``` + +A CLI one-shot — `prism target migrate perlmutter` — rewrites old configs in place, and `prism setup` writes the new shape by default. + +### 4.5 Artifact path alignment + +`dagster-slurm` writes sbatch scripts and `slurm-{job_id}.out/err` under `{remote_base}/runs/{run_id}/` on the cluster. To preserve Prism's current on-disk convention (`results/.slurm/{output_id}_{universe_id}.{sh,out,err}`) and to keep `prism status` and tail-style debugging working, the adapter's `stage_artifacts()` step fetches those files via the SSH pool (`completed.metadata['ssh_pool']` or the resource's pool accessor) and writes them locally. The remote copy is the source of truth while the job runs; the local copy is populated once the job terminates. `results/.payloads/{universe_id}__{output_id}.py` additionally gives us a readable, persistent record of the exact command executed. + +### 4.6 Migration of existing user state + +- Target YAMLs: `prism target migrate ` converts old → new schema; a compatibility shim in `targets.py` accepts both shapes for one release cycle, emitting a deprecation warning on the old shape. +- `~/.prism/config.yaml`: `default_target` and `default_permissions` fields unchanged. +- In-flight SLURM jobs at the time of upgrade are abandoned; this is consistent with current restart behavior where Prism has no reattach path for pre-upgrade jobs. + +### 4.7 Dependency and version changes + +- `requires-python` bumps from `>=3.11` to `>=3.12,<3.13` (follows upstream pin). +- `dagster` bumps from `>=1.9` to `>=1.13,<1.14` (follows upstream pin). +- `dagster-webserver` and `dagster-docker` bump to the matching 1.13 line. +- New dependency: `dagster-slurm>=1.13,<1.14`. +- `dagster-pipes` is added as a transitive dep and must be installed inside every recipe's runtime environment (container or venv) so the payload wrapper can `import dagster_pipes`. +- We do **not** adopt `pixi` as a top-level dependency. The adapter sets `LocalPipesClient(require_pixi=False)` for `mode=local`, which is supported since dagster-slurm v1.12. For SLURM mode we also disable pixi-pack (`ComputeResource(pre_deployed_env_path=...)` or a shim around the packaging step) because Prism's existing Containerfile-based image builds already solve environment hermeticity; we do not want two competing environment stories. + +## 5. Test strategy + +### 5.1 Unit tests (no Docker, no network) + +Located in `tests/test_compute_adapter.py` (new). + +- `test_target_yaml_to_compute_resource` — table-driven: `local`, `slurm`, and `slurm-session` YAML fixtures each produce a `ComputeResource` with the expected mode, SSH config, queue config, and `poll_timeout`. +- `test_resources_dict_to_slurm_opts` — `{cpus: 4, memory: "8G", gpus: 2, nodes: 1, time_limit: "30m"}` → `{cpus_per_task: 4, mem: "8G", gpus_per_node: 2, nodes: 1, time_limit: "00:30:00"}`. Cover edge cases: missing fields, `time_limit` given as `"1h"`, `"2:00:00"`, and `"120"`. +- `test_payload_wrapper_generation` — for a representative recipe (`command: python scripts/compute.py`, params `{method: a}`, container `python:3.11`, external inputs `{raw: /data/raw.parquet}`), assert the generated Python file (a) runs through `py_compile`, (b) contains the exact `CLI_ARGS` we expect, (c) wraps the command in `podman-hpc run` when container is configured, and (d) writes pipes materialization metadata. Snapshot-test the file contents for stability. +- `test_extra_sbatch_directives_injection` — `extra_sbatch_directives: ["--constraint=gpu"]` appears in the `#SBATCH` block of the generated wrapper. +- `test_stage_artifacts` — mock a `PipesClientCompletedInvocation` with a fake SSH pool; assert `results/.slurm/foo_baseline.{sh,out,err}` are written. +- `test_compute_resource_is_constructed_lazily` — constructing a `ComputeResource` with dummy SSH credentials does not open a connection. + +These replace `tests/test_runner.py` entirely and a chunk of `tests/test_runner_local.py` / `tests/test_runner_venv.py` (which are deleted along with the code they cover). + +### 5.2 Parity tests vs. the current runner (bridge) + +Located in `tests/test_runner_parity.py` (new, temporary — removed after the old runner is deleted). + +The old `ASTRAContainerRunner` remains importable behind a feature flag (`PRISM_LEGACY_RUNNER=1`) for the duration of the transition. A parametrized test runs the same astra.yaml against both backends in `mode=local` and asserts: + +- Same exit code for each output. +- Same on-disk output files (hash-compared) under `results/{universe}/{output}/`. +- Same Dagster `AssetMaterialization` event keys and values (modulo the new `cpu_efficiency` / `max_rss` metadata fields, which the old path does not emit). + +Three fixture astra.yaml files: a single-step pipeline, a two-step pipeline with `inputs:` dependency, and a multi-universe case with `decisions` injected as CLI args. The parity suite runs on CI with a 30-minute budget and is the gate that unlocks deletion of the legacy runner. + +### 5.3 Emulator-based integration tests (Docker compose) + +Located in `tests/integration/` (new) with `pytest.mark.needs_slurm_docker` and `pytest.mark.slow`. Follows dagster-slurm's own CI pattern. + +`tests/integration/docker-compose.yml` references the upstream image directly: + +```yaml +services: + mysql: { image: mariadb:12, ... } + slurmdbd: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmdbd] } + slurmctld: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmctld], ports: ["2223:22"] } + c1: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmd], hostname: c1 } + c2: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmd], hostname: c2 } +``` + +We **do not** vendor the build context — we consume the published image. If upstream breaks compatibility with a new tag, CI pins the last-known-good `IMAGE_TAG`. + +Fixtures in `tests/integration/conftest.py`: + +- `slurm_emulator` (session-scoped, `autouse=False`) — runs `docker compose up -d --wait` and `docker compose down -v` around the test module. Skipped if `DOCKER_HOST` is unreachable. +- `emulator_target` — yields a Prism target YAML pointing at `localhost:2223` with user `submitter` / password `submitter`, matching the upstream emulator credentials. + +Tests: + +- `test_emulator_end_to_end_single_asset` — materialize a one-output astra.yaml that runs `python -c "open('out.txt','w').write('ok')"` and assert `results/baseline/out/out.txt == "ok"`. +- `test_emulator_metadata_propagation` — the payload calls `pipes.report_asset_materialization(metadata={"rows": 42})`; the resulting Dagster event carries `rows=42`. +- `test_emulator_non_zero_exit_fails_asset` — a recipe with `command: exit 7` causes the materialization to fail and the SLURM exit code reaches the Dagster run. +- `test_emulator_resource_directives_respected` — `resources: {cpus: 2, time_limit: "00:02:00"}` produces an sbatch script containing `-c 2` and `-t 00:02:00`. Inspect `results/.slurm/*.sh` after the run. +- `test_emulator_sbatch_constraint_via_extra_directives` — asserts an `extra_sbatch_directives` entry reaches the sbatch header. +- `test_emulator_log_streaming` — the payload emits 500 `pipes.log.info` lines; assert all arrive in the Dagster event log (no dropped lines). +- `test_emulator_cancellation` — materialize a long-running recipe, send the Dagster run terminate signal, assert the Slurm job is cancelled (via `sacct --json` in the emulator). + +`pixi run start-staging` is *not* used — that is dagster-slurm's internal example bootstrap and we don't want to couple Prism to it. We invoke the emulator directly with the published image and our own compose file. + +### 5.4 CI wiring + +A new GitHub Actions job `integration-slurm`: + +```yaml +integration-slurm: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - uses: docker/login-action@v3 + with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} } + - run: docker compose -f tests/integration/docker-compose.yml up -d --wait --wait-timeout 120 + - run: pip install -e ".[dev]" + - run: pytest -m "needs_slurm_docker" tests/integration/ + - if: always() + run: docker compose -f tests/integration/docker-compose.yml down -v +``` + +Unit tests remain on the default matrix and complete in under a minute; the integration job is opt-in via label (`ci:slurm`) on PRs and runs on every merge to `main`. + +## 6. Rollout plan + +- **Week 1 — scaffold (no behavior change).** Add `docs/adr/0001-adopt-dagster-slurm.md` (this document). Add dagster-slurm to `pyproject.toml` as an optional extra `[slurm-next]`. Land the new target YAML schema with a loader that accepts both shapes. +- **Week 2 — compute adapter + local mode.** Ship `compute_adapter.py` and route `mode=local` through dagster-slurm behind a `PRISM_USE_DAGSTER_SLURM=1` env var. Unit tests (§5.1) green. Parity tests (§5.2) green for local. +- **Week 3 — emulator tests + SLURM mode.** Ship the docker-compose emulator fixture and integration tests. Route `mode=slurm` through dagster-slurm behind the same env var. Parity tests green for SLURM against the emulator. +- **Week 4 — flip default + deprecate old runner.** `PRISM_USE_DAGSTER_SLURM` defaults to `1`. Deprecation warnings fire on the old code path. Publish a migration note for external users. +- **Week 5 — delete.** Remove `_run_slurm*`, `_run_local`, `_run_venv`, podman-hpc build helpers, parity tests, and the feature flag. `tests/test_runner.py` is deleted. +- **Week 6 — follow-up ADR draft.** Separate decision record on whether to retire `podman-hpc run` wrapping in favor of pixi-pack, and whether to adopt `slurm-session` mode as the default for multi-output runs. + +Rollback at any phase is a revert of the phase's PR plus resetting the env var default. + +## 7. Open questions and assumptions + +- **SSH access from user laptops.** dagster-slurm's SLURM mode submits jobs over SSH from the Dagster process. Current Prism assumes users are logged into the cluster and running Prism *on* the edge node. We need to decide whether `prism run --target perlmutter` from a laptop is a supported flow or whether we continue to require the edge-node invocation. Recommendation: support both — the adapter treats `ssh.host == "localhost"` and `SLURM_JOB_ID` being set as a signal to short-circuit to a local pipes client with the recipe run via `srun`. +- **Environment packaging strategy.** This ADR proposes to disable pixi-pack and rely on Prism's existing Containerfile-built images for hermeticity. If that decision is reversed, the container wrapping logic in the payload wrapper becomes redundant and the follow-up ADR mentioned in §6 becomes mandatory, not optional. +- **Constraint-flag fragility.** Emitting `#SBATCH --constraint=...` from the payload wrapper is the cleanest workaround today, but upstream has indicated willingness to accept a PR adding `extra_sbatch_flags` to `SlurmQueueConfig`. We should open that PR in parallel so the workaround is time-boxed. +- **Retry behavior.** Prism currently has no automatic retry on transient SLURM failures. dagster-slurm supports reattach on Dagster run retry via run tags. Adopting it is a small config change but warrants a separate note to users. + +## 8. Alternatives considered + +**A. Do nothing.** Status quo preserves full control but leaves ~1000 lines of under-tested execution code on our maintenance budget. Rejected: the user request for seamless laptop↔HPC workflow is not credible to fulfill without either adopting an upstream library or spending substantial effort rebuilding one. + +**B. Use dagster-slurm as an *additional* target, keeping the current runner in place.** Minimal risk, minimal benefit. Leaves two execution stacks to maintain forever and does not address the core quality issue. Rejected in the clarification round. + +**C. Replace only the SLURM backend, keep Prism's local backend.** Smaller blast radius. Rejected in the clarification round and on its merits: keeping two code paths for "run a command" when dagster-slurm already offers a unified one is the wrong trade. + +**D. Build our own Dagster-Slurm bridge using Dagster Pipes directly, not via dagster-slurm.** Technically feasible — Dagster Pipes is stable. Rejected because we would end up re-implementing the SSH pool, sacct parsing, log streaming, and cluster-emulator story that dagster-slurm already ships, all while claiming not to be "yet another bespoke runner". + +**E. Adopt Prefect or Nextflow.** Out of scope — Prism is a Dagster-native project and the asset factory / IO manager contract is load-bearing. + +## 9. References + +- dagster-slurm README: +- dagster-slurm docs site: +- Upstream compose emulator: `docker-compose.yml` + `docker-compose.ci.yml` in the repo root +- Upstream CI pattern: `.github/workflows/library.yaml` (`Run integration tests against SLURM cluster` step) +- Upstream `ComputeResource` source: `projects/dagster-slurm/dagster_slurm/resources/compute.py` +- Upstream `_build_sbatch_command`: `projects/dagster-slurm/dagster_slurm/pipes_clients/slurm_pipes_client.py:1630` +- Current Prism runner: `src/prism/dagster/runner.py` (the module this ADR retires in large part) +- Dagster Pipes protocol: diff --git a/src/lightcone/cli/commands.py b/src/lightcone/cli/commands.py index 91bd3050..eccd73fe 100644 --- a/src/lightcone/cli/commands.py +++ b/src/lightcone/cli/commands.py @@ -7,6 +7,7 @@ import shutil import subprocess import sys +from collections.abc import Sequence from pathlib import Path from typing import Any @@ -14,6 +15,12 @@ import yaml from rich.console import Console +from lightcone.cli.harness import ( + HARNESS_REGISTRY, + ensure_dir, + resolve_global_commands_path, + resolve_harnesses, +) from lightcone.cli.plugin import get_plugin_source_dir console = Console() @@ -159,20 +166,28 @@ def _load_lightcone_config(project_path: Path) -> dict: default=False, help="Create a sub-analysis directory and wire it into the parent project", ) +@click.option( + "--tools", "tools", + multiple=True, + type=click.Choice(["claude", "codex", "cursor", "github-copilot", "opencode"]), + default=("claude",), + help="Agent harnesses to install (repeat for multiple; default: claude)", +) def init( directory: Path, no_git: bool, no_venv: bool, target: str | None, permissions: str | None, existing_project: Path | None, sub_analysis: bool, + tools: tuple[str, ...], ) -> None: """Create a new ASTRA analysis project with full agentic scaffolding. - Creates the project with ASTRA specification files, Claude Code plugin + Creates the project with ASTRA specification files, agent plugin configuration, skills, hooks, and a Python virtual environment. Use --existing-project to migrate existing code into ASTRA. If the source path differs from DIRECTORY, code is copied in. Then run - /lc-migrate in Claude Code to generate the spec. + /lc-migrate in your agent to generate the spec. Use --sub-analysis to scaffold a sub-analysis directory and wire it into the parent project's astra.yaml and universe files. @@ -186,6 +201,7 @@ def init( lc init my-analysis --existing-project ../old-code lc init analyses/new_stage --sub-analysis lc init --sub-analysis new_stage + lc init my-analysis --tools claude --tools codex """ if sub_analysis: _init_sub_analysis(directory) @@ -196,6 +212,7 @@ def init( directory, source=existing_project, no_git=no_git, no_venv=no_venv, target=target, permissions=permissions, + tools=tools, ) return @@ -248,7 +265,7 @@ def init( effective_target = load_user_config().get("default_target", "local") tier = _resolve_permission_tier(permissions) - _create_claude_settings(directory, tier, target=effective_target) + _install_harnesses(directory, tier, effective_target, tools) # Write lightcone.yaml project config _create_lightcone_config(directory, effective_target) @@ -372,6 +389,7 @@ def _init_existing_project( no_venv: bool, target: str | None, permissions: str | None, + tools: tuple[str, ...] = ("claude",), ) -> None: """Add lightcone-cli infrastructure to an existing project. @@ -459,9 +477,9 @@ def _init_existing_project( else: console.print(" [dim]requirements.txt already exists, skipping[/dim]") - # Claude Code settings + # Agent harnesses (Claude Code settings) tier = _resolve_permission_tier(permissions) - _create_claude_settings(directory, tier) + _install_harnesses(directory, tier, target or "local", tools) # lightcone.yaml effective_target = target @@ -838,72 +856,86 @@ def _update_extractor_agent_model(agents_dir: Path) -> None: extractor_path.write_text(content) -def _create_claude_settings( - directory: Path, tier: str = "recommended", target: str = "local", +def _copy_dir(src: Path, dst: Path) -> None: + """Copy *src* to *dst*, replacing *dst* if it already exists.""" + if not src.exists(): + return + if dst.exists(): + shutil.rmtree(dst) + shutil.copytree(src, dst) + + +def _copy_executable_dir(src: Path, dst: Path, ext: str) -> None: + """Copy *src* to *dst* and set execute bit on files matching *ext*.""" + _copy_dir(src, dst) + for f in dst.glob(f"*{ext}"): + f.chmod(f.stat().st_mode | 0o111) + + +def _install_harnesses( + directory: Path, + tier: str = "recommended", + target: str = "local", + tools: Sequence[str] | None = None, ) -> None: - """Create Claude Code settings with lightcone-cli skills and agents. + """Install lightcone-cli plugin content for the given agent harnesses. - If a non-local target maps to a known HPC site, site-specific deny rules - (e.g. scratch filesystem paths) are merged into the permissions. + For each harness, copies skills, agents, and guides. + For Claude Code, also installs hooks, scripts, and settings.json. """ - claude_dir = directory / ".claude" - claude_dir.mkdir(parents=True, exist_ok=True) + harnesses = resolve_harnesses(tools) - # Find the plugin source directory plugin_source = get_plugin_source_dir() if plugin_source is None: console.print( "[yellow]Warning:[/yellow] Could not find lightcone-cli plugin source files. " - "Claude Code skills will not be available." + "Agent skills will not be available." ) return - # Copy scripts - scripts_src = plugin_source / "scripts" - scripts_dst = claude_dir / "scripts" - if scripts_src.exists(): - if scripts_dst.exists(): - shutil.rmtree(scripts_dst) - shutil.copytree(scripts_src, scripts_dst) - # Make scripts executable - for script in scripts_dst.glob("*.sh"): - script.chmod(script.stat().st_mode | 0o111) - - # Copy hooks - hooks_src = plugin_source / "hooks" - hooks_dst = claude_dir / "hooks" - if hooks_src.exists(): - if hooks_dst.exists(): - shutil.rmtree(hooks_dst) - shutil.copytree(hooks_src, hooks_dst) - # Make .py files executable - for hook in hooks_dst.glob("*.py"): - hook.chmod(hook.stat().st_mode | 0o111) - - # Copy skills - skills_src = plugin_source / "skills" - skills_dst = claude_dir / "skills" - if skills_src.exists(): - if skills_dst.exists(): - shutil.rmtree(skills_dst) - shutil.copytree(skills_src, skills_dst) - - # Copy agents and apply extraction model config - agents_src = plugin_source / "agents" - agents_dst = claude_dir / "agents" - if agents_src.exists(): + has_claude = any(h.tool_id == "claude" for h in harnesses) + + for h in harnesses: + prefix = directory / h.prefix + ensure_dir(prefix) + + # Skills — all harnesses + _copy_dir(plugin_source / "skills", prefix / "skills") + + # Agents — all harnesses (with model config) + _copy_dir(plugin_source / "agents", prefix / "agents") + agents_dst = prefix / "agents" if agents_dst.exists(): - shutil.rmtree(agents_dst) - shutil.copytree(agents_src, agents_dst) - _update_extractor_agent_model(agents_dst) - - # Copy guides - guides_src = plugin_source / "guides" - guides_dst = claude_dir / "guides" - if guides_src.exists(): - if guides_dst.exists(): - shutil.rmtree(guides_dst) - shutil.copytree(guides_src, guides_dst) + _update_extractor_agent_model(agents_dst) + + # Guides — all harnesses + _copy_dir(plugin_source / "guides", prefix / "guides") + + # Claude Code only: hooks, scripts, settings + if has_claude: + claude_dir = directory / ".claude" + + # Scripts + _copy_executable_dir(plugin_source / "scripts", claude_dir / "scripts", ".sh") + + # Hooks + _copy_executable_dir(plugin_source / "hooks", claude_dir / "hooks", ".py") + + _create_claude_settings_only(directory, tier, target) + + _display_install_summary(harnesses) + + +def _create_claude_settings_only( + directory: Path, tier: str = "recommended", target: str = "local", +) -> None: + """Create Claude Code settings.json and settings.local.json. + + If a non-local target maps to a known HPC site, site-specific deny rules + (e.g. scratch filesystem paths) are merged into the permissions. + """ + claude_dir = directory / ".claude" + ensure_dir(claude_dir) # Build permissions: start from tier, then merge site-specific deny rules permissions: dict[str, list[str]] = { @@ -1026,6 +1058,35 @@ def _create_claude_settings( settings_local_file.write_text(json.dumps(settings_local, indent=2) + "\n") +def _display_install_summary(harnesses: list) -> None: + """Display post-install summary for installed harnesses.""" + installed: list[str] = [] + global_notices: list[str] = [] + + for h in harnesses: + installed.append(h.tool_name) + gpath = resolve_global_commands_path(h) + if gpath: + global_notices.append((h.tool_name, gpath)) + + if installed: + console.print("\n[bold]Installed to:[/bold]") + for h in harnesses: + prefix = h.prefix + parts = [f"[cyan]{prefix}/skills[/cyan]"] + if h.has_hooks: + parts.append(f"[cyan]{prefix}/hooks[/cyan]") + if h.has_settings: + parts.append(f"[cyan]{prefix}/settings.json[/cyan]") + console.print(f" {h.tool_name}: {', '.join(parts)}") + + for tool_name, gpath in global_notices: + console.print( + f"\n[yellow]Note: {tool_name} global commands[/yellow]" + ) + console.print(f" Place commands at: [cyan]{gpath}[/cyan]") + + def _init_git_repo(directory: Path, no_git: bool) -> None: """Initialize git repository if requested.""" if no_git or (directory / ".git").exists(): @@ -2183,8 +2244,8 @@ def setup( _CLAUDE_MD_SEPARATOR = "## Analysis Context" -def _sync_project_plugins(project_dir: Path) -> bool: - """Sync plugin files (skills, hooks, scripts, agents, CLAUDE.md) into a project. +def _sync_project_plugins(project_dir: Path, tools: tuple[str, ...] = ("claude",)) -> bool: + """Sync plugin files (skills, agents, guides, CLAUDE.md) into a project. Returns True if the sync succeeded. """ @@ -2197,78 +2258,83 @@ def _sync_project_plugins(project_dir: Path) -> bool: console.print(" [red]✗[/red] Could not find lightcone-cli plugin source files.") return False - claude_dir = project_dir / ".claude" - claude_dir.mkdir(parents=True, exist_ok=True) + tool_ids = list(tools) if tools else ["claude"] - # Sync directories: skills, hooks, scripts, agents, guides - for subdir in ("scripts", "hooks", "skills", "agents", "guides"): - src = plugin_source / subdir - dst = claude_dir / subdir - if not src.exists(): - continue - if dst.exists(): - shutil.rmtree(dst) - shutil.copytree(src, dst) - # Make executable as needed - if subdir == "scripts": - for f in dst.glob("*.sh"): - f.chmod(f.stat().st_mode | 0o111) - elif subdir == "hooks": - for f in dst.glob("*.py"): - f.chmod(f.stat().st_mode | 0o111) - - # Apply extraction model config to agents - agents_dst = claude_dir / "agents" - if agents_dst.exists(): - _update_extractor_agent_model(agents_dst) - - # Update the managed portion of CLAUDE.md (everything above "## Analysis Context") - claude_md = project_dir / "CLAUDE.md" - if claude_md.exists(): - existing = claude_md.read_text() - # Find the separator - sep_idx = existing.find(_CLAUDE_MD_SEPARATOR) - if sep_idx != -1: - user_section = existing[sep_idx:] - else: - # No separator found — preserve everything as user content - user_section = ( - f"{_CLAUDE_MD_SEPARATOR}\n\n" - "_Run `/lc-new` to scope the research question and populate " - "this section with domain context and implementation notes not " - "captured in astra.yaml._\n" - ) + for tid in tool_ids: + harness = HARNESS_REGISTRY[tid] + prefix = project_dir / harness.prefix + ensure_dir(prefix) + + # Sync content directories for this harness + for subdir in ("skills", "agents", "guides"): + if not harness.has_skills and subdir == "skills": + continue + if not harness.has_agents and subdir == "agents": + continue + if not harness.has_guides and subdir == "guides": + continue + src = plugin_source / subdir + dst = prefix / subdir + if not src.exists(): + continue + if dst.exists(): + shutil.rmtree(dst) + shutil.copytree(src, dst) - # Get fresh template - name = project_dir.name - template_path = plugin_source / "templates" / "CLAUDE.md" - if template_path.exists(): - template = template_path.read_text().replace("{{name}}", name) - template_sep_idx = template.find(_CLAUDE_MD_SEPARATOR) - if template_sep_idx != -1: - managed_section = template[:template_sep_idx] + # Apply extraction model config to agents + agents_dst = prefix / "agents" + if agents_dst.exists(): + _update_extractor_agent_model(agents_dst) + + # CLAUDE.md — only sync for Claude Code (it is Claude Code-specific content) + if "claude" in tool_ids: + claude_md = project_dir / "CLAUDE.md" + if claude_md.exists(): + existing = claude_md.read_text() + # Find the separator + sep_idx = existing.find(_CLAUDE_MD_SEPARATOR) + if sep_idx != -1: + user_section = existing[sep_idx:] else: - managed_section = template + "\n" - else: - managed_section = ( - f"# CLAUDE.md\n\n## Project: {name}\n\n" - "ASTRA analysis project, built with lightcone-cli.\n\n---\n\n" - "\n" - ) + # No separator found — preserve everything as user content + user_section = ( + f"{_CLAUDE_MD_SEPARATOR}\n\n" + "_Run `/lc-new` to scope the research question and populate " + "this section with domain context and implementation notes not " + "captured in astra.yaml._\n" + ) + + # Get fresh template + name = project_dir.name + template_path = plugin_source / "templates" / "CLAUDE.md" + if template_path.exists(): + template = template_path.read_text().replace("{{name}}", name) + template_sep_idx = template.find(_CLAUDE_MD_SEPARATOR) + if template_sep_idx != -1: + managed_section = template[:template_sep_idx] + else: + managed_section = template + "\n" + else: + managed_section = ( + f"# CLAUDE.md\n\n## Project: {name}\n\n" + "ASTRA analysis project, built with lightcone-cli.\n\n---\n\n" + "\n" + ) - claude_md.write_text(managed_section + user_section) + claude_md.write_text(managed_section + user_section) console.print(f" [green]✓[/green] {project_dir}") return True -def _prompt_sync_projects() -> None: +def _prompt_sync_projects(tools: tuple[str, ...] = ("claude",)) -> None: """Prompt the user to sync plugin files into existing projects.""" + harness_names = ", ".join(HARNESS_REGISTRY[tid].tool_name for tid in tools) console.print( "\n[bold]Sync updated plugin files to your projects?[/bold]" ) console.print( - " This updates skills, hooks, scripts, and CLAUDE.md in each project's .claude/ directory." + f" This updates skills, agents, and guides for {harness_names}." ) raw = click.prompt( "\n Enter project paths (comma-separated), or skip", @@ -2283,20 +2349,28 @@ def _prompt_sync_projects() -> None: console.print() for p in paths: - _sync_project_plugins(p) + _sync_project_plugins(p, tools) @main.command() @click.option("--sync", is_flag=True, help="Only sync plugin files to projects (skip upgrade)") -def update(sync: bool) -> None: +@click.option( + "--tools", "tools", + multiple=True, + type=click.Choice(["claude", "codex", "cursor", "github-copilot", "opencode"]), + default=("claude",), + help="Agent harnesses to sync (repeat for multiple; default: claude)", +) +def update(sync: bool, tools: tuple[str, ...]) -> None: """Upgrade lightcone-cli and sync plugin files to projects. Upgrades lightcone-cli from PyPI, then offers to sync - updated skills, hooks, and scripts into your projects. + updated skills, agents, and guides into your projects. Examples: lc update # upgrade package & sync projects lc update --sync # just sync plugin files (no upgrade) + lc update --sync --tools claude --tools codex """ if not sync: console.print("[bold]Upgrading lightcone-cli...[/bold]\n") @@ -2311,7 +2385,7 @@ def update(sync: bool) -> None: console.print(f" [red]✗[/red] upgrade failed: {proc.stderr.strip()[:200]}") raise SystemExit(1) - _prompt_sync_projects() + _prompt_sync_projects(tools) # Register eval subgroup (requires optional 'eval' extra) diff --git a/src/lightcone/cli/harness.py b/src/lightcone/cli/harness.py new file mode 100644 index 00000000..8e6cbbf6 --- /dev/null +++ b/src/lightcone/cli/harness.py @@ -0,0 +1,149 @@ +"""Harness registry — maps canonical skills to per-tool install paths. + +Kept deliberately leaf (no imports from :mod:`lightcone.cli.commands` or +:mod:`lightcone.engine`) so it can be used by both the CLI and the eval +harness without introducing an import cycle. +""" + +from __future__ import annotations + +import os +import re +from collections.abc import Sequence +from dataclasses import dataclass +from pathlib import Path + +# --------------------------------------------------------------------------- +# Data model +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class HarnessConfig: + """Configuration for a single agent-harness installation target.""" + + tool_id: str + tool_name: str + prefix: str # e.g. ".claude", ".codex" + has_skills: bool = True + has_agents: bool = True + has_guides: bool = True + has_hooks: bool = False + has_settings: bool = False + commands_local: str | None = None + commands_global: str | None = None + commands_local_ext: str = "" # suffix for command files, e.g. ".prompt.md" + + +# --------------------------------------------------------------------------- +# Registry +# --------------------------------------------------------------------------- + +HARNESS_REGISTRY: dict[str, HarnessConfig] = { + "claude": HarnessConfig( + tool_id="claude", + tool_name="Claude Code", + prefix=".claude", + has_skills=True, + has_agents=True, + has_guides=True, + has_hooks=True, + has_settings=True, + commands_local=".claude/commands", + ), + "codex": HarnessConfig( + tool_id="codex", + tool_name="Codex", + prefix=".codex", + has_skills=True, + has_agents=True, + has_guides=True, + has_hooks=False, + has_settings=False, + commands_global="$CODEX_HOME/prompts", + ), + "cursor": HarnessConfig( + tool_id="cursor", + tool_name="Cursor", + prefix=".cursor", + has_skills=True, + has_agents=True, + has_guides=True, + has_hooks=False, + has_settings=False, + commands_local=".cursor/commands", + ), + "github-copilot": HarnessConfig( + tool_id="github-copilot", + tool_name="GitHub Copilot", + prefix=".github", + has_skills=True, + has_agents=True, + has_guides=True, + has_hooks=False, + has_settings=False, + commands_local=".github/prompts", + commands_local_ext=".prompt.md", + ), + "opencode": HarnessConfig( + tool_id="opencode", + tool_name="OpenCode", + prefix=".opencode", + has_skills=True, + has_agents=True, + has_guides=True, + has_hooks=False, + has_settings=False, + commands_local=".opencode/commands", + ), +} + +#: All tool IDs known to the registry, in registration order. +ALL_TOOL_IDS: tuple[str, ...] = tuple(HARNESS_REGISTRY) + +# --------------------------------------------------------------------------- +# Validation helpers +# --------------------------------------------------------------------------- + + +def resolve_harnesses(tool_ids: Sequence[str] | None) -> list[HarnessConfig]: + """Return configured harnesses for *tool_ids*, defaulting to ``["claude"]``.""" + ids: list[str] = list(tool_ids) if tool_ids else ["claude"] + # Validate + for tid in ids: + if tid not in HARNESS_REGISTRY: + raise ValueError( + f"Unknown tool ID {tid!r}. " + f"Valid options: {', '.join(repr(t) for t in ALL_TOOL_IDS)}" + ) + return [HARNESS_REGISTRY[tid] for tid in ids] + + +# --------------------------------------------------------------------------- +# Path helpers +# --------------------------------------------------------------------------- + +#: Pattern matching $VAR or ${VAR} environment variable references. +_ENV_VAR_PATTERN = re.compile(r"\$\{?(\w+)\}?\b") + + +def resolve_global_commands_path(harness: HarnessConfig) -> str | None: + """Expand environment variables in a harness's global commands path.""" + path = harness.commands_global + if path is None: + return None + + def _repl(match: re.Match[str]) -> str: + var = match.group(1) + # Strip trailing braces/parens for patterns like ${VAR} + return os.environ.get(var, match.group(0)) + + return _ENV_VAR_PATTERN.sub(_repl, path) + + +def ensure_dir(path: Path) -> None: + """Create *path* and parents if they don't exist. Warns on error.""" + try: + path.mkdir(parents=True, exist_ok=True) + except OSError as exc: + print(f"\n[warning] Cannot create directory {path}: {exc}") diff --git a/tests/test_harness.py b/tests/test_harness.py new file mode 100644 index 00000000..5950fc88 --- /dev/null +++ b/tests/test_harness.py @@ -0,0 +1,151 @@ +"""Tests for src/lightcone/cli/harness.py — harness registry module.""" + +from pathlib import Path + +import pytest + +from lightcone.cli.harness import ( + ALL_TOOL_IDS, + HARNESS_REGISTRY, + ensure_dir, + resolve_global_commands_path, + resolve_harnesses, +) + +# ------ resolve_harnesses ------ + + +class TestResolveHarnesses: + def test_default_returns_claude(self): + result = resolve_harnesses(None) + assert len(result) == 1 + assert result[0].tool_id == "claude" + + def test_default_empty_list_returns_claude(self): + result = resolve_harnesses(()) + assert len(result) == 1 + assert result[0].tool_id == "claude" + + def test_single_tool(self): + result = resolve_harnesses(("codex",)) + assert len(result) == 1 + assert result[0].tool_id == "codex" + + def test_multiple_tools(self): + result = resolve_harnesses(("claude", "codex", "cursor")) + assert len(result) == 3 + assert [r.tool_id for r in result] == ["claude", "codex", "cursor"] + + def test_duplicate_tools_preserved(self): + result = resolve_harnesses(("claude", "claude")) + assert len(result) == 2 + + def test_invalid_tool_raises(self): + with pytest.raises(ValueError, match="Unknown tool"): + resolve_harnesses(("nonexistent",)) + + def test_mixed_valid_invalid_raises(self): + with pytest.raises(ValueError, match="Unknown tool"): + resolve_harnesses(("claude", "bogus")) + + +# ------ HARNESS_REGISTRY ------ + + +class TestHarnessRegistry: + def test_all_tools_present(self): + expected = {"claude", "codex", "cursor", "github-copilot", "opencode"} + assert set(HARNESS_REGISTRY.keys()) == expected + assert set(ALL_TOOL_IDS) == expected + + def test_claude_has_hooks_and_settings(self): + h = HARNESS_REGISTRY["claude"] + assert h.has_hooks is True + assert h.has_settings is True + assert h.has_skills is True + assert h.has_agents is True + assert h.has_guides is True + + def test_codex_no_hooks_or_settings(self): + h = HARNESS_REGISTRY["codex"] + assert h.has_hooks is False + assert h.has_settings is False + assert h.has_skills is True + assert h.has_agents is True + assert h.has_guides is True + + def test_only_claude_has_hooks(self): + for tid, h in HARNESS_REGISTRY.items(): + if tid == "claude": + assert h.has_hooks is True + else: + assert h.has_hooks is False + + def test_only_claude_has_settings(self): + for tid, h in HARNESS_REGISTRY.items(): + if tid == "claude": + assert h.has_settings is True + else: + assert h.has_settings is False + + def test_codex_has_global_commands(self): + h = HARNESS_REGISTRY["codex"] + assert h.commands_global is not None + assert "CODEX_HOME" in h.commands_global + + def test_claude_no_global_commands(self): + h = HARNESS_REGISTRY["claude"] + assert h.commands_global is None + + def test_github_copilot_has_suffix(self): + h = HARNESS_REGISTRY["github-copilot"] + assert h.commands_local_ext == ".prompt.md" + + +# ------ resolve_global_commands_path ------ + + +class TestResolveGlobalCommandsPath: + def test_claude_returns_none(self): + assert resolve_global_commands_path(HARNESS_REGISTRY["claude"]) is None + + def test_codex_unresolved_when_env_unset(self): + h = HARNESS_REGISTRY["codex"] + # CODEX_HOME may or may not be set; just check the function doesn't crash + result = resolve_global_commands_path(h) + assert result is not None + + def test_env_expansion_unresolved(self): + """When CODEX_HOME is unset, path is returned with placeholder intact.""" + h = HARNESS_REGISTRY["codex"] + result = resolve_global_commands_path(h) + assert result is not None + assert "CODEX_HOME" in result + + +# ------ ensure_dir ------ + + +class TestEnsureDir: + def test_creates_new_directory(self, tmp_path: Path): + new_dir = tmp_path / "a" / "b" / "c" + ensure_dir(new_dir) + assert new_dir.is_dir() + + def test_noop_on_existing(self, tmp_path: Path): + existing = tmp_path / "a" + existing.mkdir() + ensure_dir(existing) + assert existing.is_dir() + + def test_warns_on_permission_error(self, capsys, tmp_path: Path): + """ensure_dir prints a warning on OSError instead of raising.""" + import errno + from unittest.mock import patch + + target = tmp_path / "nope" + with patch.object(Path, "mkdir") as mock_mkdir: + mock_mkdir.side_effect = OSError(errno.EACCES, "Permission denied") + ensure_dir(target) + out = capsys.readouterr().out + assert "warning" in out.lower() diff --git a/tests/test_harness_install.py b/tests/test_harness_install.py new file mode 100644 index 00000000..09806cc4 --- /dev/null +++ b/tests/test_harness_install.py @@ -0,0 +1,191 @@ +"""Tests for multi-harness skill installation (LCR-85).""" + +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from lightcone.cli.commands import main + + +@pytest.fixture +def runner(): + return CliRunner() + + +class TestInitTools: + """Tests for lc init --tools flag.""" + + def test_init_default_installs_claude(self, runner: CliRunner, tmp_path: Path): + """Default init installs to .claude/.""" + project_dir = tmp_path / "test-default-tools" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + ], + ) + assert result.exit_code == 0, result.output + + claude_dir = project_dir / ".claude" + assert claude_dir.is_dir() + assert (claude_dir / "settings.json").exists() + assert (claude_dir / "skills").is_dir() + assert (claude_dir / "settings.local.json").exists() + + def test_init_claude_codex_creates_both(self, runner: CliRunner, tmp_path: Path): + """--tools claude --tools codex installs to both .claude/ and .codex/.""" + project_dir = tmp_path / "test-multi-tools" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "claude", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + + # Both harnesses installed + assert (project_dir / ".claude").is_dir() + assert (project_dir / ".codex").is_dir() + assert (project_dir / ".claude" / "skills").is_dir() + assert (project_dir / ".codex" / "skills").is_dir() + + def test_init_codex_no_settings_json(self, runner: CliRunner, tmp_path: Path): + """Codex harness does not get settings.json.""" + project_dir = tmp_path / "test-codex-no-settings" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + + assert (project_dir / ".codex" / "skills").is_dir() + assert not (project_dir / ".codex" / "settings.json").exists() + + def test_init_skills_in_all_harnesses(self, runner: CliRunner, tmp_path: Path): + """Skill files appear in every installed harness's skills directory.""" + project_dir = tmp_path / "test-skills-dup" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "claude", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + + # All skill dirs should have all skill content + for skill_name in ("lc-new", "lc-build", "lc-verify", "lc-migrate", "lc-feedback"): + for prefix in (".claude", ".codex"): + skill_md = project_dir / prefix / "skills" / skill_name / "SKILL.md" + assert skill_md.exists(), f"{prefix}/skills/{skill_name}/SKILL.md missing" + + def test_init_agents_in_all_harnesses(self, runner: CliRunner, tmp_path: Path): + """lc-extractor agent appears in every installed harness's agents directory.""" + project_dir = tmp_path / "test-agents-dup" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "claude", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + + for prefix in (".claude", ".codex"): + agent_file = project_dir / prefix / "agents" / "lc-extractor.md" + assert agent_file.exists(), f"{prefix}/agents/lc-extractor.md missing" + + def test_init_guides_in_all_harnesses(self, runner: CliRunner, tmp_path: Path): + """Guide files appear in every installed harness's guides directory.""" + project_dir = tmp_path / "test-guides-dup" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "claude", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + + for prefix in (".claude", ".codex"): + guides_dir = project_dir / prefix / "guides" + assert guides_dir.is_dir() + guide_files = list(guides_dir.glob("*.md")) + assert len(guide_files) > 0 + + def test_init_output_mentions_installed_harnesses(self, runner: CliRunner, tmp_path: Path): + """Post-install output lists installed harness names.""" + project_dir = tmp_path / "test-summary-output" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "claude", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + assert "Claude Code" in result.output + assert "Codex" in result.output + + def test_init_codex_displays_global_notice(self, runner: CliRunner, tmp_path: Path): + """Codex install shows global commands notice.""" + project_dir = tmp_path / "test-codex-notice" + result = runner.invoke( + main, + [ + "init", str(project_dir), + "--no-git", "--no-venv", "--permissions", "recommended", + "--tools", "codex", + ], + ) + assert result.exit_code == 0, result.output + # Should mention global commands + assert "global" in result.output.lower() or "CODEX_HOME" in result.output + + +class TestUpdateTools: + """Tests for lc update --sync --tools flag.""" + + def test_update_sync_default_only_claude(self, runner: CliRunner, tmp_path: Path): + """Default update --sync only touches .claude/.""" + project_dir = tmp_path / "test-sync-default" + # First create a minimal project + astra_yaml = project_dir / "astra.yaml" + astra_yaml.parent.mkdir(parents=True, exist_ok=True) + astra_yaml.write_text("version: 1.0\n") + + result = runner.invoke( + main, + ["update", "--sync"], + ) + # The sync may prompt for paths; accept the prompt or skip + assert result.exit_code == 0 or result.exit_code == 1 or result.exit_code is None + + def test_update_help_shows_tools_option(self, runner: CliRunner): + """Update command should accept --tools flag.""" + result = runner.invoke(main, ["update", "--help"]) + assert result.exit_code == 0 + assert "--tools" in result.output + + def test_init_help_shows_tools_option(self, runner: CliRunner): + """Init command should accept --tools flag.""" + result = runner.invoke(main, ["init", "--help"]) + assert result.exit_code == 0 + assert "--tools" in result.output From 765ca00eefd61b43b575baee0e0fd24218f10108 Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 20:15:30 +0200 Subject: [PATCH 2/9] LCR-85: Fix sync tests for multi-harness (init-time only hooks/scripts) Update TestSyncProjectPlugins to reflect that sync only copies skills, agents, and guides. Hooks and scripts are init-time only. - test_sync_copies_plugin_dirs: assert agents synced, hooks/scripts not - test_sync_scripts_executable: renamed to test_sync_no_scripts - _make_plugin_source fixture: add agents/ dir Co-Authored-By: Claude Opus 4.6 --- tests/test_cli.py | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/tests/test_cli.py b/tests/test_cli.py index 71563813..e61db4e0 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -798,6 +798,10 @@ def _make_plugin_source(self, tmp_path: Path) -> Path: skills = plugin / "skills" / "lc-build" skills.mkdir(parents=True) (skills / "SKILL.md").write_text("# build skill v2\n") + # Agents + agents = plugin / "agents" + agents.mkdir() + (agents / "lc-extractor.md").write_text("# extractor agent v2\n") # Scripts scripts = plugin / "scripts" scripts.mkdir() @@ -824,7 +828,7 @@ def _make_plugin_source(self, tmp_path: Path) -> Path: return plugin def test_sync_copies_plugin_dirs(self, tmp_path: Path): - """Sync should copy skills, hooks, scripts into .claude/.""" + """Sync should copy skills, agents, and guides into .claude/ (not hooks or scripts).""" from lightcone.cli.commands import _sync_project_plugins project = self._make_project(tmp_path) @@ -835,13 +839,15 @@ def test_sync_copies_plugin_dirs(self, tmp_path: Path): assert result is True assert (project / ".claude" / "skills" / "lc-build" / "SKILL.md").exists() - assert (project / ".claude" / "scripts" / "session-start.sh").exists() - assert (project / ".claude" / "hooks" / "langfuse_hook.py").exists() + assert (project / ".claude" / "agents" / "lc-extractor.md").exists() assert (project / ".claude" / "guides" / "astra-reference.md").exists() assert (project / ".claude" / "guides" / "ui-brand.md").exists() + # Hooks and scripts are init-time only; not synced + assert not (project / ".claude" / "scripts").exists() + assert not (project / ".claude" / "hooks").exists() - def test_sync_scripts_executable(self, tmp_path: Path): - """Synced scripts should be executable.""" + def test_sync_no_scripts(self, tmp_path: Path): + """Sync does not copy scripts or hooks (init-time only).""" from lightcone.cli.commands import _sync_project_plugins project = self._make_project(tmp_path) @@ -850,8 +856,8 @@ def test_sync_scripts_executable(self, tmp_path: Path): with patch("lightcone.cli.commands.get_plugin_source_dir", return_value=plugin): _sync_project_plugins(project) - sh = project / ".claude" / "scripts" / "session-start.sh" - assert sh.stat().st_mode & 0o111 + assert not (project / ".claude" / "scripts").exists() + assert not (project / ".claude" / "hooks").exists() def test_sync_preserves_analysis_context(self, tmp_path: Path): """Sync should update managed CLAUDE.md section but preserve Analysis Context.""" From 057f23bd58bfe589fe7491a0b5b9f3b703ae10e3 Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 21:48:57 +0200 Subject: [PATCH 3/9] claude-md: Reflect harness.py in project docs (LCR-85) - Add harness.py to repository structure - Add harness registration to extending table Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index b0631b4a..1484d68a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,6 +26,7 @@ src/lightcone/ # namespace — NO __init__.py ├── cli/ # Click surface │ ├── __init__.py # exposes main() │ ├── commands.py # all Click commands (init, run, build, status, dev, target, setup, update) +│ ├── harness.py # harness registry — maps tool IDs to install paths (claude, codex, cursor...) │ ├── plugin.py # get_plugin_source_dir — leaf module (no imports from commands.py) │ └── claude/ # force-included Claude plugin bundle (in installed wheel only) ├── engine/ # execution substrate — Dagster + HPC + containers @@ -153,6 +154,7 @@ All commands use Click. Key patterns: | Add container features | `src/lightcone/engine/container.py` | `DEPENDENCY_FILES` tuple, `compute_image_tag()`, build/resolve functions | | Create a skill | `claude/lightcone/skills/` | SKILL.md with YAML frontmatter (`name`, `description`, `allowed-tools`) | | Add a telemetry hook | `claude/lightcone/hooks/` | Follow `langfuse_hook.py` pattern: read JSON payload, emit to Langfuse | +| Add a harness target | `src/lightcone/cli/harness.py` | Add to `HARNESS_REGISTRY` dataclass + `ALL_TOOL_IDS` tuple | ## Test Patterns From 19b5b584f8fd65b789d821872db7a8e4347bf107 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Wed, 22 Apr 2026 20:02:21 +0000 Subject: [PATCH 4/9] LCR-85: Fix review issues in multi-harness skill system - Fix type annotation: global_notices list[str] -> list[tuple[str,str]] - Add has_skills/has_agents/has_guides guards to _install_harnesses to match the same guards already present in _sync_project_plugins - Replace raw HARNESS_REGISTRY[tid] lookups with resolve_harnesses() in _sync_project_plugins and _prompt_sync_projects for consistent validation - Replace hardcoded click.Choice tool lists with ALL_TOOL_IDS so adding a new harness requires only one change in the registry - Add agents/guides entries to _display_install_summary output - Replace print() in ensure_dir with warnings.warn() (leaf module) - De-duplicate tool IDs in resolve_harnesses() using dict.fromkeys - Remove misplaced docs/adr/0001-adopt-dagster-slurm.md (Prism/SLURM ADR unrelated to this multi-harness PR) Co-authored-by: Alexandre Boucaud --- docs/adr/0001-adopt-dagster-slurm.md | 348 --------------------------- src/lightcone/cli/commands.py | 41 ++-- src/lightcone/cli/harness.py | 11 +- 3 files changed, 31 insertions(+), 369 deletions(-) delete mode 100644 docs/adr/0001-adopt-dagster-slurm.md diff --git a/docs/adr/0001-adopt-dagster-slurm.md b/docs/adr/0001-adopt-dagster-slurm.md deleted file mode 100644 index 97dcd654..00000000 --- a/docs/adr/0001-adopt-dagster-slurm.md +++ /dev/null @@ -1,348 +0,0 @@ -# ADR-0001 — Replace Prism's local and SLURM execution backends with `dagster-slurm` - -- **Status:** Proposed -- **Date:** 2026-04-17 -- **Deciders:** Prism maintainers -- **Affects:** `src/prism/dagster/runner.py`, `src/prism/dagster/assets.py`, `src/prism/dagster/targets.py`, `src/prism/dagster/site_registry.py`, `src/prism/container.py`, `src/prism/cli.py`, tests in `tests/` -- **Related upstream:** [`dagster-slurm`](https://github.com/ascii-supply-networks/dagster-slurm) (v1.13.0), docs at - -## 1. Context - -Prism currently ships a bespoke execution layer (`ASTRAContainerRunner` in `src/prism/dagster/runner.py`) that implements four backends inside a single ~1000-line class: **docker**, **local**, **venv**, and **slurm**. The SLURM backend generates sbatch scripts (`results/.slurm/`), submits with `sbatch`, polls with `sacct` falling back to `squeue`, and wraps commands in `podman-hpc run` when a container is configured. The local backend is a plain `subprocess.Popen` with streaming output, no environment management beyond a `python` substitution. There is no pluggable-backend abstraction; adding or changing a backend requires editing the monolithic class. - -This layer is the weakest part of Prism. It duplicates work that the wider Dagster ecosystem is solving in the open: real-time log streaming over SSH, account/QoS/reservation handling for multiple sites, Slurm job cancellation semantics, heterogeneous jobs, cluster reuse, and metrics collection via `sacct`. Our own tests cover the happy path of sbatch script generation and sacct parsing, but the polling loop, squeue fallback, timeout handling, and podman-hpc resolution are under-tested and have surfaced several pain points already (see "Known limitations" in the runner analysis attached to this review). - -The `dagster-slurm` library — published by ASCII Supply Networks and used in production at multiple Austrian HPC centers — is explicitly designed for the scenario Prism is trying to support: the same Dagster asset runs on a laptop, on a Docker-emulated Slurm cluster for CI, and on a real cluster, with only the resource wiring changing. It exposes a single `ComputeResource` facade with `mode ∈ {local, slurm, slurm-session, slurm-hetjob}` and a Dagster Pipes–based protocol for log streaming, metadata exchange, and materialization events. - -The user request that motivates this ADR is explicit: "design a workflow that can execute seamlessly across SLURM computing centers as well as local machines". That is precisely `dagster-slurm`'s charter. - -## 2. Decision - -We replace Prism's `local` and `slurm` execution backends with `dagster-slurm`'s `ComputeResource`. The **Docker** backend (`runner._run_docker`, `container.build_image_docker`) is **kept unchanged** because dagster-slurm has no concept of container-per-asset execution, and Prism's Docker backend is the primary way our users achieve hermetic local runs today. The **venv** backend is retired — dagster-slurm's pixi-based environment packaging supersedes it for local execution, and we will not maintain two similar abstractions. - -In the target architecture, Prism's asset factory no longer calls `runner.execute(command, ...)`. Instead, each Dagster asset is wired with a `ComputeResource` and invokes `compute.run(context, payload_path, extra_env, extra_slurm_opts)`. The heavy lifting — SSH pooling, sbatch submission, `sacct`/`squeue` polling, log streaming, pipes message ingestion, metrics collection — lives in dagster-slurm. - -Concretely: - -- `ASTRAContainerRunner` is split. A thin `DockerRunner` (≈150 lines, extracted from current code) remains for the Docker backend. SLURM and local paths are deleted. -- A new `src/prism/dagster/compute_adapter.py` builds a `ComputeResource` from a Prism target YAML and exposes the single call that `assets.py` needs. -- Prism recipes are translated into `dagster-slurm` payloads by a generated wrapper script (see §4.3) so that ASTRA's "a recipe is a shell command" invariant is preserved. -- Target config is migrated from Prism's custom schema (`scheduler:`, `poll:`, `resource_limits:`) into dagster-slurm's `SlurmResource` / `SlurmQueueConfig` / `SSHConnectionResource` shapes, wrapped by a stable Prism-level schema. -- Python floor moves from 3.11 to 3.12 (dagster-slurm's hard requirement); Dagster is bumped from 1.9 → 1.13. - -## 3. Consequences - -### 3.1 What we gain - -The upstream library gives us, for free, a set of features we would otherwise have to build or defer: real-time stdout/stderr streaming over SSH with automatic fallback from `ControlMaster` to polling for HPC centers that disable socket multiplexing; `sacct`-based metrics (CPU efficiency, max RSS, node-hours) emitted as Dagster output metadata; Dagster Pipes integration that lets a compute payload emit `AssetMaterialization` and asset-check events back to the run without stdout parsing; safe retry and restart semantics driven by Dagster run tags; a session mode that holds a long-lived Slurm allocation across multiple assets (valuable for any multi-asset astra.yaml that shares a cluster reservation); and a working Docker-compose SLURM emulator we can use for CI. - -The code delta is large in the negative direction: approximately 750 lines of Prism's SLURM, local, and venv runner code are deleted, replaced by roughly 250 lines of adapter, payload generation, and configuration translation. Net reduction is on the order of 500 lines in a module that has been a recurring source of bugs. - -### 3.2 What we give up or complicate - -We lose built-in `podman-hpc` containerization on SLURM. `dagster-slurm` has no container primitive — environment isolation is achieved by `pixi pack` producing a self-extracting tarball that is uploaded to the cluster and sourced in the sbatch script. For sites that rely on `podman-hpc` image builds today (Perlmutter), we must either (a) wrap the recipe command in an explicit `podman-hpc run` invocation inside the generated payload wrapper (see §4.3); or (b) accept pixi-pack isolation as the new hermetic primitive and retire `container.py`'s podman-hpc build path. This ADR recommends (a) as the migration path and deferring (b) to a follow-up decision record; we do not want to couple a runner swap to an isolation-model change. - -We inherit a Python 3.12 floor. Prism already runs under 3.12 in its `.venv`, so this is more a declaration than a lift. Consumers on 3.11 will be broken once merged — this is a semver-minor breaking change from their perspective. - -We inherit `dagster-slurm`'s gaps: no support for SLURM `--constraint` out of the box (Prism exposes this via `extra_slurm_args`; we must either upstream a PR or emit `#SBATCH --constraint` from our payload wrapper); limited multi-hop SSH (single jump host only); remote paths live under `remote_base/runs/{run_id}/`, not `results/.slurm/` (we fetch logs and sbatch scripts back into `results/.slurm/` post-execution to preserve the current artifact layout). - -We accept a coupling to Dagster 1.13 pinned by dagster-slurm. Future upgrades are gated on the upstream bumping its own pin. - -### 3.3 Risk summary - -The highest-risk item is the container-isolation gap on HPC sites. The second-highest is the ASTRA recipe-model mismatch: dagster-slurm expects a Python payload file, Prism expects a shell command with injected CLI args. The mitigation for both is the payload wrapper described in §4.3, which is small, deterministic, and unit-testable without Docker. - -## 4. Detailed design - -### 4.1 Module-by-module plan - -`src/prism/dagster/runner.py` loses everything Slurm-, venv-, and local-related: `_run_slurm`, `_run_slurm_interactive`, `_run_local`, `_run_venv`, `generate_sbatch_script`, `translate_resources_to_slurm_directives`, `_podman_hpc_run_command`, `_normalise_time_limit`, `_parse_sbatch_job_id`, `_poll_slurm_job`, `_check_sacct`, `_check_squeue_fallback`, and the venv dependency-hash caching logic. What remains is a `DockerRunner` class with the current Docker fallback-to-local behavior removed (local is no longer a fallback; Docker failure is now a hard failure with a clear error message). The module shrinks from ~1000 to ~180 lines. - -`src/prism/dagster/compute_adapter.py` is new. It owns three things: (1) loading a Prism target YAML and constructing a `ComputeResource` with the correct `mode` and subordinate `SlurmResource` / `SSHConnectionResource` / `SlurmQueueConfig` objects; (2) building the payload wrapper script for a given recipe (§4.3); (3) staging dagster-slurm's post-execution artifacts (sbatch script, stdout, stderr, metrics) into `results/.slurm/{output_id}_{universe_id}.*` so the current on-disk layout is preserved. This module is the new seam for tests and for future HPC site additions. - -`src/prism/dagster/assets.py` changes minimally. The `_build_single_asset` body no longer constructs a command string and calls `runner.execute`; instead it writes a payload wrapper via `compute_adapter.prepare_payload(recipe, universe_id, params, external_inputs)` and calls `compute.run(context, payload_path=wrapper_path, extra_slurm_opts=_resources_to_slurm_opts(recipe.resources))`. The asset factory still receives a `runner` parameter for backwards compatibility with Docker-only targets; when the target mode is `local` or `slurm`, it resolves to a `ComputeResource` via the adapter. - -`src/prism/dagster/targets.py` gets a new `TargetKind` enum: `docker | local | slurm | slurm-session`. Loaders for `slurm` and `local` now validate against dagster-slurm's config shape. Existing YAML files need a one-time migration (§4.6). - -`src/prism/dagster/site_registry.py` retains its account-suffix resolution and node-type defaults but no longer emits Prism-native `scheduler_config` dicts. It emits `SlurmQueueConfig` fragments that the adapter merges into the final `ComputeResource`. - -`src/prism/container.py` loses `build_image_podman_hpc`, `_podman_hpc_migrate`, `image_exists_podman_hpc`, and `resolve_container_for_slurm`. Docker/Podman build and tag-computation logic stays. The `podman-hpc` container runtime is no longer selectable in target configs (the `container_runtime` field is removed); HPC container execution is handled by the payload wrapper's `podman-hpc run` invocation if the target opts in. - -`src/prism/cli.py` loses its SLURM-specific passthrough for `--partition` / `--qos` / `--constraint` / `--account`. These now live in the target YAML and are overridable per-run via a new `--slurm-opts key=value,...` flag that lands in `extra_slurm_opts` on the compute resource. `prism setup` and `prism target` UX stays visually similar but writes dagster-slurm–shaped configs. - -### 4.2 Interface mapping - -The current runner interface: - -```python -runner.execute( - command: str, # "python scripts/compute.py" - container: str | None, - inputs: list[str], - output_id: str, - universe_id: str, - resources: dict[str, Any], # {cpus, memory, gpus, time_limit, nodes} - params: dict[str, Any], # decision params from universes/{id}.yaml - external_inputs: dict[str, str] | None, - cwd_override: str | None, -) -> ExecutionResult # {exit_code, output_path, metadata} -``` - -The new equivalent, resolved through the adapter: - -```python -payload_path, ctx = compute_adapter.prepare_payload( - recipe=recipe, universe_id=universe_id, params=params, - external_inputs=external_inputs, cwd_override=cwd_override, - output_id=output_id, project_root=project_root, -) -completed = compute.run( - context=dagster_context, - payload_path=payload_path, - extra_slurm_opts=_resources_to_slurm_opts(recipe.resources), - extra_env={"ASTRA_UNIVERSE": universe_id, "ASTRA_OUTPUT": output_id}, - extra_files=_resolve_external_inputs(external_inputs), -) -yield from completed.get_results() # Dagster events -compute_adapter.stage_artifacts(completed, ctx) # copy logs/script into results/.slurm/ -``` - -`resources` → `extra_slurm_opts` translation is direct: `cpus → cpus_per_task`, `memory → mem`, `gpus → gpus_per_node`, `nodes → nodes`, `time_limit → time_limit` (format normalized to `HH:MM:SS`). The `--constraint` value, if present in target config, is injected into the sbatch script header by the payload wrapper (see §4.3) because dagster-slurm's `_build_sbatch_command` does not accept arbitrary sbatch flags. - -Return shape: `ExecutionResult` is retired. `compute.run()` returns a `PipesClientCompletedInvocation`; metrics and exit codes are surfaced as Dagster asset metadata by `get_results()`. The `output_path` field (always `results/{universe_id}/`) is redundant — the IO manager is the source of truth for it. - -### 4.3 Payload wrapper — the ASTRA → pipes shim - -Prism recipes are shell commands. dagster-slurm expects a Python payload that opens a pipes session. The gap is bridged by a small generated wrapper written per-materialization to a deterministic path: - -```python -# results/.payloads/{universe_id}__{output_id}.py (auto-generated, do not edit) -from dagster_pipes import open_dagster_pipes -import os, subprocess, shlex, sys, json - -CMD = {command_quoted} # "python scripts/compute.py" -CLI_ARGS = {cli_args_json} # ["--universe", "baseline", "--method", "option_a"] -CWD = {cwd_json_or_none} -CONTAINER_WRAP = {container_wrap_or_none} # ["podman-hpc", "run", "--rm", ...] -WORKDIR = {workdir_json} - -with open_dagster_pipes() as pipes: - pipes.log.info(f"astra recipe: {CMD} {' '.join(CLI_ARGS)}") - full = shlex.split(CMD) + CLI_ARGS - if CONTAINER_WRAP: - full = CONTAINER_WRAP + full - proc = subprocess.run(full, cwd=CWD or WORKDIR, check=False) - pipes.report_asset_materialization( - metadata={"exit_code": proc.returncode, - "command": " ".join(shlex.quote(a) for a in full)} - ) - sys.exit(proc.returncode) -``` - -Three reasons this wrapper design is chosen over "just change ASTRA recipes to be Python scripts": - -- ASTRA is a spec maintained separately and is explicitly "pure specification". Changing the recipe model to require a pipes-native Python entry point would leak execution-layer concerns back into the spec. Prism is the agentic layer; shims belong in Prism. -- The wrapper is small, deterministic, and easy to regenerate. Its contents are a pure function of `(recipe.command, resources, cli_args, container, external_inputs)` — cacheable, testable, and diff-able across runs. -- If a recipe author does write a pipes-native payload in the future, we keep the door open by honoring a `recipe.payload: path/to/script.py` field that bypasses the wrapper and passes the script directly to `compute.run`. - -The wrapper also owns `--constraint` and any other sbatch flag dagster-slurm does not accept, by writing a `#SBATCH --constraint=...` line into the *first* comment block of the script itself — sbatch accepts these directives even when the submission command does not specify them. This is a documented upstream pattern and is already exercised by dagster-slurm users for reservations on sites that expose non-standard constraint strings. - -### 4.4 Target config translation - -Before: - -```yaml -# ~/.prism/targets/perlmutter.yaml (current) -site: perlmutter -backend: slurm -connection: - hostname: perlmutter.nersc.gov -scheduler: - account: m1234 - qos: debug - partition: gpu - constraint: gpu - container_runtime: podman-hpc - container_flags: ["--gpu", "--mpi"] - extra_slurm_args: [] -poll: - interval_seconds: 15 - timeout_seconds: 14400 -resource_limits: - max_walltime_minutes: 360 -``` - -After: - -```yaml -# ~/.prism/targets/perlmutter.yaml (new) -name: perlmutter -mode: slurm # local | slurm | slurm-session | docker -site: perlmutter # still used by site_registry for account suffix -ssh: - host: perlmutter.nersc.gov - user: ${USER} - key_path: ~/.ssh/nersc -queue: - partition: gpu - account: m1234 - qos: debug - time_limit: "00:30:00" - cpus: 4 - mem_per_cpu: 4G - gpus_per_node: 0 -remote_base: /pscratch/sd/a/${USER}/prism-runs -# HPC container execution (optional, Prism-level, wrapped in payload): -container: - runtime: podman-hpc # podman-hpc | apptainer | none - flags: ["--gpu", "--mpi"] -# Raw sbatch escape hatch — appended to every job's #SBATCH header: -extra_sbatch_directives: - - "--constraint=gpu" -poll: - timeout_seconds: 14400 # passed through to compute.run(poll_timeout=) -``` - -A CLI one-shot — `prism target migrate perlmutter` — rewrites old configs in place, and `prism setup` writes the new shape by default. - -### 4.5 Artifact path alignment - -`dagster-slurm` writes sbatch scripts and `slurm-{job_id}.out/err` under `{remote_base}/runs/{run_id}/` on the cluster. To preserve Prism's current on-disk convention (`results/.slurm/{output_id}_{universe_id}.{sh,out,err}`) and to keep `prism status` and tail-style debugging working, the adapter's `stage_artifacts()` step fetches those files via the SSH pool (`completed.metadata['ssh_pool']` or the resource's pool accessor) and writes them locally. The remote copy is the source of truth while the job runs; the local copy is populated once the job terminates. `results/.payloads/{universe_id}__{output_id}.py` additionally gives us a readable, persistent record of the exact command executed. - -### 4.6 Migration of existing user state - -- Target YAMLs: `prism target migrate ` converts old → new schema; a compatibility shim in `targets.py` accepts both shapes for one release cycle, emitting a deprecation warning on the old shape. -- `~/.prism/config.yaml`: `default_target` and `default_permissions` fields unchanged. -- In-flight SLURM jobs at the time of upgrade are abandoned; this is consistent with current restart behavior where Prism has no reattach path for pre-upgrade jobs. - -### 4.7 Dependency and version changes - -- `requires-python` bumps from `>=3.11` to `>=3.12,<3.13` (follows upstream pin). -- `dagster` bumps from `>=1.9` to `>=1.13,<1.14` (follows upstream pin). -- `dagster-webserver` and `dagster-docker` bump to the matching 1.13 line. -- New dependency: `dagster-slurm>=1.13,<1.14`. -- `dagster-pipes` is added as a transitive dep and must be installed inside every recipe's runtime environment (container or venv) so the payload wrapper can `import dagster_pipes`. -- We do **not** adopt `pixi` as a top-level dependency. The adapter sets `LocalPipesClient(require_pixi=False)` for `mode=local`, which is supported since dagster-slurm v1.12. For SLURM mode we also disable pixi-pack (`ComputeResource(pre_deployed_env_path=...)` or a shim around the packaging step) because Prism's existing Containerfile-based image builds already solve environment hermeticity; we do not want two competing environment stories. - -## 5. Test strategy - -### 5.1 Unit tests (no Docker, no network) - -Located in `tests/test_compute_adapter.py` (new). - -- `test_target_yaml_to_compute_resource` — table-driven: `local`, `slurm`, and `slurm-session` YAML fixtures each produce a `ComputeResource` with the expected mode, SSH config, queue config, and `poll_timeout`. -- `test_resources_dict_to_slurm_opts` — `{cpus: 4, memory: "8G", gpus: 2, nodes: 1, time_limit: "30m"}` → `{cpus_per_task: 4, mem: "8G", gpus_per_node: 2, nodes: 1, time_limit: "00:30:00"}`. Cover edge cases: missing fields, `time_limit` given as `"1h"`, `"2:00:00"`, and `"120"`. -- `test_payload_wrapper_generation` — for a representative recipe (`command: python scripts/compute.py`, params `{method: a}`, container `python:3.11`, external inputs `{raw: /data/raw.parquet}`), assert the generated Python file (a) runs through `py_compile`, (b) contains the exact `CLI_ARGS` we expect, (c) wraps the command in `podman-hpc run` when container is configured, and (d) writes pipes materialization metadata. Snapshot-test the file contents for stability. -- `test_extra_sbatch_directives_injection` — `extra_sbatch_directives: ["--constraint=gpu"]` appears in the `#SBATCH` block of the generated wrapper. -- `test_stage_artifacts` — mock a `PipesClientCompletedInvocation` with a fake SSH pool; assert `results/.slurm/foo_baseline.{sh,out,err}` are written. -- `test_compute_resource_is_constructed_lazily` — constructing a `ComputeResource` with dummy SSH credentials does not open a connection. - -These replace `tests/test_runner.py` entirely and a chunk of `tests/test_runner_local.py` / `tests/test_runner_venv.py` (which are deleted along with the code they cover). - -### 5.2 Parity tests vs. the current runner (bridge) - -Located in `tests/test_runner_parity.py` (new, temporary — removed after the old runner is deleted). - -The old `ASTRAContainerRunner` remains importable behind a feature flag (`PRISM_LEGACY_RUNNER=1`) for the duration of the transition. A parametrized test runs the same astra.yaml against both backends in `mode=local` and asserts: - -- Same exit code for each output. -- Same on-disk output files (hash-compared) under `results/{universe}/{output}/`. -- Same Dagster `AssetMaterialization` event keys and values (modulo the new `cpu_efficiency` / `max_rss` metadata fields, which the old path does not emit). - -Three fixture astra.yaml files: a single-step pipeline, a two-step pipeline with `inputs:` dependency, and a multi-universe case with `decisions` injected as CLI args. The parity suite runs on CI with a 30-minute budget and is the gate that unlocks deletion of the legacy runner. - -### 5.3 Emulator-based integration tests (Docker compose) - -Located in `tests/integration/` (new) with `pytest.mark.needs_slurm_docker` and `pytest.mark.slow`. Follows dagster-slurm's own CI pattern. - -`tests/integration/docker-compose.yml` references the upstream image directly: - -```yaml -services: - mysql: { image: mariadb:12, ... } - slurmdbd: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmdbd] } - slurmctld: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmctld], ports: ["2223:22"] } - c1: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmd], hostname: c1 } - c2: { image: ghcr.io/ascii-supply-networks/dagster-slurm/slurm-docker-cluster:25-11-2-1, command: [slurmd], hostname: c2 } -``` - -We **do not** vendor the build context — we consume the published image. If upstream breaks compatibility with a new tag, CI pins the last-known-good `IMAGE_TAG`. - -Fixtures in `tests/integration/conftest.py`: - -- `slurm_emulator` (session-scoped, `autouse=False`) — runs `docker compose up -d --wait` and `docker compose down -v` around the test module. Skipped if `DOCKER_HOST` is unreachable. -- `emulator_target` — yields a Prism target YAML pointing at `localhost:2223` with user `submitter` / password `submitter`, matching the upstream emulator credentials. - -Tests: - -- `test_emulator_end_to_end_single_asset` — materialize a one-output astra.yaml that runs `python -c "open('out.txt','w').write('ok')"` and assert `results/baseline/out/out.txt == "ok"`. -- `test_emulator_metadata_propagation` — the payload calls `pipes.report_asset_materialization(metadata={"rows": 42})`; the resulting Dagster event carries `rows=42`. -- `test_emulator_non_zero_exit_fails_asset` — a recipe with `command: exit 7` causes the materialization to fail and the SLURM exit code reaches the Dagster run. -- `test_emulator_resource_directives_respected` — `resources: {cpus: 2, time_limit: "00:02:00"}` produces an sbatch script containing `-c 2` and `-t 00:02:00`. Inspect `results/.slurm/*.sh` after the run. -- `test_emulator_sbatch_constraint_via_extra_directives` — asserts an `extra_sbatch_directives` entry reaches the sbatch header. -- `test_emulator_log_streaming` — the payload emits 500 `pipes.log.info` lines; assert all arrive in the Dagster event log (no dropped lines). -- `test_emulator_cancellation` — materialize a long-running recipe, send the Dagster run terminate signal, assert the Slurm job is cancelled (via `sacct --json` in the emulator). - -`pixi run start-staging` is *not* used — that is dagster-slurm's internal example bootstrap and we don't want to couple Prism to it. We invoke the emulator directly with the published image and our own compose file. - -### 5.4 CI wiring - -A new GitHub Actions job `integration-slurm`: - -```yaml -integration-slurm: - runs-on: ubuntu-latest - timeout-minutes: 45 - steps: - - uses: actions/checkout@v4 - - uses: docker/login-action@v3 - with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} } - - run: docker compose -f tests/integration/docker-compose.yml up -d --wait --wait-timeout 120 - - run: pip install -e ".[dev]" - - run: pytest -m "needs_slurm_docker" tests/integration/ - - if: always() - run: docker compose -f tests/integration/docker-compose.yml down -v -``` - -Unit tests remain on the default matrix and complete in under a minute; the integration job is opt-in via label (`ci:slurm`) on PRs and runs on every merge to `main`. - -## 6. Rollout plan - -- **Week 1 — scaffold (no behavior change).** Add `docs/adr/0001-adopt-dagster-slurm.md` (this document). Add dagster-slurm to `pyproject.toml` as an optional extra `[slurm-next]`. Land the new target YAML schema with a loader that accepts both shapes. -- **Week 2 — compute adapter + local mode.** Ship `compute_adapter.py` and route `mode=local` through dagster-slurm behind a `PRISM_USE_DAGSTER_SLURM=1` env var. Unit tests (§5.1) green. Parity tests (§5.2) green for local. -- **Week 3 — emulator tests + SLURM mode.** Ship the docker-compose emulator fixture and integration tests. Route `mode=slurm` through dagster-slurm behind the same env var. Parity tests green for SLURM against the emulator. -- **Week 4 — flip default + deprecate old runner.** `PRISM_USE_DAGSTER_SLURM` defaults to `1`. Deprecation warnings fire on the old code path. Publish a migration note for external users. -- **Week 5 — delete.** Remove `_run_slurm*`, `_run_local`, `_run_venv`, podman-hpc build helpers, parity tests, and the feature flag. `tests/test_runner.py` is deleted. -- **Week 6 — follow-up ADR draft.** Separate decision record on whether to retire `podman-hpc run` wrapping in favor of pixi-pack, and whether to adopt `slurm-session` mode as the default for multi-output runs. - -Rollback at any phase is a revert of the phase's PR plus resetting the env var default. - -## 7. Open questions and assumptions - -- **SSH access from user laptops.** dagster-slurm's SLURM mode submits jobs over SSH from the Dagster process. Current Prism assumes users are logged into the cluster and running Prism *on* the edge node. We need to decide whether `prism run --target perlmutter` from a laptop is a supported flow or whether we continue to require the edge-node invocation. Recommendation: support both — the adapter treats `ssh.host == "localhost"` and `SLURM_JOB_ID` being set as a signal to short-circuit to a local pipes client with the recipe run via `srun`. -- **Environment packaging strategy.** This ADR proposes to disable pixi-pack and rely on Prism's existing Containerfile-built images for hermeticity. If that decision is reversed, the container wrapping logic in the payload wrapper becomes redundant and the follow-up ADR mentioned in §6 becomes mandatory, not optional. -- **Constraint-flag fragility.** Emitting `#SBATCH --constraint=...` from the payload wrapper is the cleanest workaround today, but upstream has indicated willingness to accept a PR adding `extra_sbatch_flags` to `SlurmQueueConfig`. We should open that PR in parallel so the workaround is time-boxed. -- **Retry behavior.** Prism currently has no automatic retry on transient SLURM failures. dagster-slurm supports reattach on Dagster run retry via run tags. Adopting it is a small config change but warrants a separate note to users. - -## 8. Alternatives considered - -**A. Do nothing.** Status quo preserves full control but leaves ~1000 lines of under-tested execution code on our maintenance budget. Rejected: the user request for seamless laptop↔HPC workflow is not credible to fulfill without either adopting an upstream library or spending substantial effort rebuilding one. - -**B. Use dagster-slurm as an *additional* target, keeping the current runner in place.** Minimal risk, minimal benefit. Leaves two execution stacks to maintain forever and does not address the core quality issue. Rejected in the clarification round. - -**C. Replace only the SLURM backend, keep Prism's local backend.** Smaller blast radius. Rejected in the clarification round and on its merits: keeping two code paths for "run a command" when dagster-slurm already offers a unified one is the wrong trade. - -**D. Build our own Dagster-Slurm bridge using Dagster Pipes directly, not via dagster-slurm.** Technically feasible — Dagster Pipes is stable. Rejected because we would end up re-implementing the SSH pool, sacct parsing, log streaming, and cluster-emulator story that dagster-slurm already ships, all while claiming not to be "yet another bespoke runner". - -**E. Adopt Prefect or Nextflow.** Out of scope — Prism is a Dagster-native project and the asset factory / IO manager contract is load-bearing. - -## 9. References - -- dagster-slurm README: -- dagster-slurm docs site: -- Upstream compose emulator: `docker-compose.yml` + `docker-compose.ci.yml` in the repo root -- Upstream CI pattern: `.github/workflows/library.yaml` (`Run integration tests against SLURM cluster` step) -- Upstream `ComputeResource` source: `projects/dagster-slurm/dagster_slurm/resources/compute.py` -- Upstream `_build_sbatch_command`: `projects/dagster-slurm/dagster_slurm/pipes_clients/slurm_pipes_client.py:1630` -- Current Prism runner: `src/prism/dagster/runner.py` (the module this ADR retires in large part) -- Dagster Pipes protocol: diff --git a/src/lightcone/cli/commands.py b/src/lightcone/cli/commands.py index eccd73fe..97dc1881 100644 --- a/src/lightcone/cli/commands.py +++ b/src/lightcone/cli/commands.py @@ -16,6 +16,7 @@ from rich.console import Console from lightcone.cli.harness import ( + ALL_TOOL_IDS, HARNESS_REGISTRY, ensure_dir, resolve_global_commands_path, @@ -169,7 +170,7 @@ def _load_lightcone_config(project_path: Path) -> dict: @click.option( "--tools", "tools", multiple=True, - type=click.Choice(["claude", "codex", "cursor", "github-copilot", "opencode"]), + type=click.Choice(list(ALL_TOOL_IDS)), default=("claude",), help="Agent harnesses to install (repeat for multiple; default: claude)", ) @@ -899,17 +900,17 @@ def _install_harnesses( prefix = directory / h.prefix ensure_dir(prefix) - # Skills — all harnesses - _copy_dir(plugin_source / "skills", prefix / "skills") + if h.has_skills: + _copy_dir(plugin_source / "skills", prefix / "skills") - # Agents — all harnesses (with model config) - _copy_dir(plugin_source / "agents", prefix / "agents") - agents_dst = prefix / "agents" - if agents_dst.exists(): - _update_extractor_agent_model(agents_dst) + if h.has_agents: + _copy_dir(plugin_source / "agents", prefix / "agents") + agents_dst = prefix / "agents" + if agents_dst.exists(): + _update_extractor_agent_model(agents_dst) - # Guides — all harnesses - _copy_dir(plugin_source / "guides", prefix / "guides") + if h.has_guides: + _copy_dir(plugin_source / "guides", prefix / "guides") # Claude Code only: hooks, scripts, settings if has_claude: @@ -1061,7 +1062,7 @@ def _create_claude_settings_only( def _display_install_summary(harnesses: list) -> None: """Display post-install summary for installed harnesses.""" installed: list[str] = [] - global_notices: list[str] = [] + global_notices: list[tuple[str, str]] = [] for h in harnesses: installed.append(h.tool_name) @@ -1073,7 +1074,13 @@ def _display_install_summary(harnesses: list) -> None: console.print("\n[bold]Installed to:[/bold]") for h in harnesses: prefix = h.prefix - parts = [f"[cyan]{prefix}/skills[/cyan]"] + parts: list[str] = [] + if h.has_skills: + parts.append(f"[cyan]{prefix}/skills[/cyan]") + if h.has_agents: + parts.append(f"[cyan]{prefix}/agents[/cyan]") + if h.has_guides: + parts.append(f"[cyan]{prefix}/guides[/cyan]") if h.has_hooks: parts.append(f"[cyan]{prefix}/hooks[/cyan]") if h.has_settings: @@ -2258,10 +2265,10 @@ def _sync_project_plugins(project_dir: Path, tools: tuple[str, ...] = ("claude", console.print(" [red]✗[/red] Could not find lightcone-cli plugin source files.") return False - tool_ids = list(tools) if tools else ["claude"] + harness_list = resolve_harnesses(tools or None) + tool_ids = [h.tool_id for h in harness_list] - for tid in tool_ids: - harness = HARNESS_REGISTRY[tid] + for harness in harness_list: prefix = project_dir / harness.prefix ensure_dir(prefix) @@ -2329,7 +2336,7 @@ def _sync_project_plugins(project_dir: Path, tools: tuple[str, ...] = ("claude", def _prompt_sync_projects(tools: tuple[str, ...] = ("claude",)) -> None: """Prompt the user to sync plugin files into existing projects.""" - harness_names = ", ".join(HARNESS_REGISTRY[tid].tool_name for tid in tools) + harness_names = ", ".join(h.tool_name for h in resolve_harnesses(tools or None)) console.print( "\n[bold]Sync updated plugin files to your projects?[/bold]" ) @@ -2357,7 +2364,7 @@ def _prompt_sync_projects(tools: tuple[str, ...] = ("claude",)) -> None: @click.option( "--tools", "tools", multiple=True, - type=click.Choice(["claude", "codex", "cursor", "github-copilot", "opencode"]), + type=click.Choice(list(ALL_TOOL_IDS)), default=("claude",), help="Agent harnesses to sync (repeat for multiple; default: claude)", ) diff --git a/src/lightcone/cli/harness.py b/src/lightcone/cli/harness.py index 8e6cbbf6..c3e751e3 100644 --- a/src/lightcone/cli/harness.py +++ b/src/lightcone/cli/harness.py @@ -9,6 +9,7 @@ import os import re +import warnings from collections.abc import Sequence from dataclasses import dataclass from pathlib import Path @@ -107,9 +108,11 @@ class HarnessConfig: def resolve_harnesses(tool_ids: Sequence[str] | None) -> list[HarnessConfig]: - """Return configured harnesses for *tool_ids*, defaulting to ``["claude"]``.""" - ids: list[str] = list(tool_ids) if tool_ids else ["claude"] - # Validate + """Return configured harnesses for *tool_ids*, defaulting to ``["claude"]``. + + Duplicate IDs are silently de-duplicated (order preserved). + """ + ids: list[str] = list(dict.fromkeys(tool_ids)) if tool_ids else ["claude"] for tid in ids: if tid not in HARNESS_REGISTRY: raise ValueError( @@ -146,4 +149,4 @@ def ensure_dir(path: Path) -> None: try: path.mkdir(parents=True, exist_ok=True) except OSError as exc: - print(f"\n[warning] Cannot create directory {path}: {exc}") + warnings.warn(f"Cannot create directory {path}: {exc}", stacklevel=2) From b0975c4b92186e0f19fdd24865f0c8b4515f0e0a Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Wed, 22 Apr 2026 20:17:43 +0000 Subject: [PATCH 5/9] LCR-85: Fix test_harness.py for resolve_harnesses dedup and warnings.warn MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - test_duplicate_tools_preserved → test_duplicate_tools_deduplicated: update assertion to len==1 after fix #6 introduced dict.fromkeys de-duplication - test_warns_on_permission_error: replace capsys stdout check with pytest.warns(UserWarning) after fix #5 changed print() to warnings.warn() Co-authored-by: Alexandre Boucaud --- tests/test_harness.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/tests/test_harness.py b/tests/test_harness.py index 5950fc88..cccdf053 100644 --- a/tests/test_harness.py +++ b/tests/test_harness.py @@ -36,9 +36,9 @@ def test_multiple_tools(self): assert len(result) == 3 assert [r.tool_id for r in result] == ["claude", "codex", "cursor"] - def test_duplicate_tools_preserved(self): + def test_duplicate_tools_deduplicated(self): result = resolve_harnesses(("claude", "claude")) - assert len(result) == 2 + assert len(result) == 1 def test_invalid_tool_raises(self): with pytest.raises(ValueError, match="Unknown tool"): @@ -138,14 +138,13 @@ def test_noop_on_existing(self, tmp_path: Path): ensure_dir(existing) assert existing.is_dir() - def test_warns_on_permission_error(self, capsys, tmp_path: Path): - """ensure_dir prints a warning on OSError instead of raising.""" + def test_warns_on_permission_error(self, tmp_path: Path): + """ensure_dir emits a warnings.warn() on OSError instead of raising.""" import errno from unittest.mock import patch target = tmp_path / "nope" with patch.object(Path, "mkdir") as mock_mkdir: mock_mkdir.side_effect = OSError(errno.EACCES, "Permission denied") - ensure_dir(target) - out = capsys.readouterr().out - assert "warning" in out.lower() + with pytest.warns(UserWarning, match="Cannot create directory"): + ensure_dir(target) From 3fab9fae23465da5b91961407029d104ddbc5c9a Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 23:06:28 +0200 Subject: [PATCH 6/9] LCR-85: Fix review issues in multi-harness skill system MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. CLAUDE.md: update Plugin system section to reflect multi-harness install/sync behaviour; add convention to keep CLAUDE.md current during PRs. 2. harness.py: add comments clarifying that has_hooks/has_settings are Claude Code-specific flags (no other tool has per-project agent automation hooks or settings files). 3. commands.py: make pip upgrade failure non-fatal in lc update — warn and continue to sync instead of raising SystemExit(1), so lc update --tools X works from inside a project venv. Co-Authored-By: Claude Sonnet 4.6 --- CLAUDE.md | 9 ++++++--- src/lightcone/cli/commands.py | 7 ++++--- src/lightcone/cli/harness.py | 4 ++++ 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1484d68a..6fbcb7ab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -131,9 +131,11 @@ astra.yaml → build_definitions() → Dagster assets → ASTRAContainerRunner - Most commands require `astra.yaml` in cwd; exceptions: `setup`, `target` **Plugin system:** -- Skills, hooks, and scripts are bundled in the wheel (`claude/lightcone/` → `lightcone/cli/claude/lightcone/`) -- `lc init` copies them into each project's `.claude/` directory -- Plugin source discovery lives in `lightcone.cli.plugin.get_plugin_source_dir` — tries bundled location first, falls back to dev location (`claude/lightcone/` at repo root) +- Skills, agents, guides, hooks, and scripts are bundled in the wheel (`claude/lightcone/` → `lightcone/cli/claude/lightcone/`) +- `lc init [--tools T]` copies content into `./` for each selected harness; default harness is `claude` +- Skills, agents, and guides are installed for every harness; hooks/scripts are Claude Code only +- `lc update --sync [--tools T]` re-syncs skills/agents/guides (hooks are init-time only, not synced) +- Plugin source discovery: `lightcone.cli.plugin.get_plugin_source_dir` — bundled location first, then `claude/lightcone/` at repo root - Bash scripts must be chmod +x ## CLI Patterns @@ -172,3 +174,4 @@ All commands use Click. Key patterns: - SLURM scripts/output stored in `results/.slurm/` - Dagster instance storage at `results/.dagster/` (SQLite) - Telemetry opt-out: `TRACE_TO_LANGFUSE=false` +- **Keep CLAUDE.md current** — when a PR changes CLI behaviour, key invariants, or repo structure, update the relevant section of CLAUDE.md in the same commit diff --git a/src/lightcone/cli/commands.py b/src/lightcone/cli/commands.py index 97dc1881..8388679b 100644 --- a/src/lightcone/cli/commands.py +++ b/src/lightcone/cli/commands.py @@ -17,7 +17,6 @@ from lightcone.cli.harness import ( ALL_TOOL_IDS, - HARNESS_REGISTRY, ensure_dir, resolve_global_commands_path, resolve_harnesses, @@ -2389,8 +2388,10 @@ def update(sync: bool, tools: tuple[str, ...]) -> None: if proc.returncode == 0: console.print(" [green]✓[/green] lightcone-cli upgraded") else: - console.print(f" [red]✗[/red] upgrade failed: {proc.stderr.strip()[:200]}") - raise SystemExit(1) + console.print( + f" [yellow]⚠[/yellow] upgrade failed (continuing to sync): " + f"{proc.stderr.strip()[:200]}" + ) _prompt_sync_projects(tools) diff --git a/src/lightcone/cli/harness.py b/src/lightcone/cli/harness.py index c3e751e3..ef90d2cb 100644 --- a/src/lightcone/cli/harness.py +++ b/src/lightcone/cli/harness.py @@ -29,7 +29,11 @@ class HarnessConfig: has_skills: bool = True has_agents: bool = True has_guides: bool = True + # True only for Claude Code: copies hooks/ and scripts/ during lc init. + # No other supported tool has an equivalent per-project agent automation hook system. has_hooks: bool = False + # True only for Claude Code: writes settings.json + settings.local.json during lc init. + # Other tools use editor-level or global config, not per-project files managed by lightcone-cli. has_settings: bool = False commands_local: str | None = None commands_global: str | None = None From a00bfc1b70ca33bf13c957bb61736b5334495ef1 Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 23:09:01 +0200 Subject: [PATCH 7/9] LCR-85: Update docs for multi-harness skill system - docs/architecture.md: rename plugin section to "Agent harness plugin"; show per-harness vs Claude-only content; list harness registry - docs/cli/init.md: add --tools option; update modes and internal helpers - docs/cli/update.md: add --tools option; clarify what is/isn't synced; note non-fatal pip upgrade; update examples - docs/skills/index.md: update install paths; add multi-harness table Co-Authored-By: Claude Sonnet 4.6 --- docs/architecture.md | 22 ++++++++++++++++------ docs/cli/init.md | 8 +++++--- docs/cli/update.md | 21 +++++++++++++-------- docs/skills/index.md | 20 +++++++++++++++++--- 4 files changed, 51 insertions(+), 20 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 3f3f1bce..4efbb661 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,7 +6,7 @@ lightcone-cli bridges an ASTRA specification (`astra.yaml`) and actual execution 1. **Dagster integration** — translates the ASTRA spec into a directed-acyclic graph (DAG) of assets, then materialises them in dependency order. 2. **Container management** — resolves and builds content-addressed Docker/Podman images from `Containerfile` specs. -3. **Claude Code plugin** — injects skills, hooks, and scripts into each project's `.claude/` directory so that Claude Code can operate as a research agent. +3. **Agent harness plugin** — injects skills, agents, and guides into each project's harness directory (`./`) for every selected tool. Hooks and scripts are installed for Claude Code only. --- @@ -69,23 +69,33 @@ For SLURM/`podman-hpc` targets, `resolve_container_for_slurm()` additionally mig --- -## Claude Code plugin +## Agent harness plugin ### Structure -The plugin lives in `claude/lightcone/` and is bundled into the Python wheel via `hatch-vcs` force-include directives. When `lc init` runs, it copies the plugin into the project's `.claude/` directory. +The plugin lives in `claude/lightcone/` and is bundled into the Python wheel via `hatch-vcs` force-include directives. When `lc init` runs, it copies the plugin into one or more harness directories selected via `--tools` (default: `claude`). + +**Content installed to every harness** (`./`): + +``` +./ +├── skills/ # Agent slash commands / prompts +├── agents/ # lc-extractor subagent +└── guides/ # Reference docs loaded by skills +``` + +**Additional content installed for Claude Code only** (`.claude/`): ``` .claude/ ├── settings.json # Permissions + hook registrations ├── settings.local.json # Telemetry env vars (Langfuse keys) -├── skills/ # Claude Code slash commands -├── agents/ # lc-extractor subagent -├── guides/ # Reference docs loaded by skills ├── hooks/ # Python hooks for Langfuse telemetry └── scripts/ # Bash hooks for session lifecycle ``` +The harness registry (`src/lightcone/cli/harness.py`) maps each tool ID to its install prefix and capability flags. Supported harnesses: `claude`, `codex`, `cursor`, `github-copilot`, `opencode`. + ### Permission tiers Three tiers configure how much Claude can do autonomously: diff --git a/docs/cli/init.md b/docs/cli/init.md index bbf23c85..0a597b42 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -16,7 +16,7 @@ lc init [OPTIONS] [DIRECTORY] - Boilerplate `astra.yaml` with TODO placeholders - `Containerfile` and `requirements.txt` - A baseline universe (`universes/baseline.yaml`) -- `.claude/` directory with skills, hooks, scripts, and `settings.json` +- `./` directory with skills, agents, guides (and hooks, scripts, `settings.json` for Claude Code) - `.lightcone/lightcone.yaml` linking the project to its execution target - `.lightcone/dagster.yaml` pointing Dagster's SQLite store to `results/.dagster/` - `CLAUDE.md` from the plugin template @@ -32,6 +32,7 @@ lc init [OPTIONS] [DIRECTORY] | `--no-venv` | false | Skip virtual environment creation | | `--target`, `-t` | user default | Execution target name to write into `.lightcone/lightcone.yaml` | | `--permissions` | saved/prompt | Claude Code permission tier (`yolo`, `recommended`, `minimal`) | +| `--tools` | `claude` | Agent harness(es) to install (repeat for multiple: `--tools claude --tools codex`) | | `--existing-project` | — | Path to existing code to migrate into the new project | | `--sub-analysis` | false | Create a sub-analysis directory and wire it into the parent project | @@ -42,9 +43,10 @@ lc init [OPTIONS] [DIRECTORY] ```bash lc init my-analysis lc init my-analysis --target perlmutter-gpu +lc init my-analysis --tools claude --tools codex # install both Claude Code and Codex harnesses ``` -Creates a fresh project in `my-analysis/`. +Creates a fresh project in `my-analysis/`. By default only the Claude Code harness (`.claude/`) is installed. Pass `--tools` once per harness to install additional ones. ### Migrate existing code @@ -88,6 +90,6 @@ The following private functions do the heavy lifting and are tested directly: - `_create_dagster_yaml(directory)` — writes `.lightcone/dagster.yaml` - `_create_boilerplate_astra_yaml(directory)` — writes `astra.yaml`, `Containerfile`, `requirements.txt`, `universes/baseline.yaml` -- `_create_claude_settings(directory, tier, target)` — copies plugin files and writes `.claude/settings.json` + `.claude/settings.local.json` +- `_install_harnesses(directory, tier, target, tools)` — copies skills/agents/guides to every selected harness; installs hooks, scripts, and `settings.json` for Claude Code only - `_create_lightcone_config(directory, target_name)` — writes `.lightcone/lightcone.yaml` - `_init_sub_analysis(directory)` — scaffolds sub-analysis directory and wires it into the parent spec diff --git a/docs/cli/update.md b/docs/cli/update.md index 99d2e03d..1b226d0b 100644 --- a/docs/cli/update.md +++ b/docs/cli/update.md @@ -10,35 +10,40 @@ lc update [OPTIONS] ## Description -`lc update` upgrades `lightcone-cli` from PyPI, then offers to sync updated skills, hooks, scripts, and `CLAUDE.md` into existing projects. +`lc update` upgrades `lightcone-cli` from PyPI, then offers to sync updated skills, agents, guides, and `CLAUDE.md` into existing projects. + +If the pip upgrade fails (e.g. inside a project venv), a warning is printed and the sync step still runs. ## Options | Option | Description | |--------|-------------| | `--sync` | Only sync plugin files to projects (skip upgrade) | +| `--tools` | Harness(es) to sync (repeat for multiple; default: `claude`) | ## What gets synced -When syncing to a project, the following are updated in `.claude/`: +For each selected harness, the following directories are updated in `./`: - `skills/` — all skill directories -- `hooks/` — all Python hook scripts -- `scripts/` — all bash hook scripts - `agents/` — subagent definitions (extraction model config reapplied) - `guides/` — reference documentation -For `CLAUDE.md`, only the managed portion (everything above `## Analysis Context`) is updated. User content below that separator is preserved. +Hooks (`hooks/`) and scripts (`scripts/`) are **not** synced — they are written once at `lc init` time. Re-run `lc init` in an existing project to update them. + +For `CLAUDE.md` (Claude Code harness only), the managed portion (everything above `## Analysis Context`) is refreshed from the template. User content below that separator is preserved. ## Examples ```bash -lc update # upgrade + offer to sync -lc update --sync # just sync plugin files (no upgrade) +lc update # upgrade + offer to sync (claude harness) +lc update --sync # sync only, no upgrade +lc update --sync --tools codex # sync Codex harness only +lc update --sync --tools claude --tools codex # sync both harnesses ``` ## Notes The sync prompt asks for a comma-separated list of project paths. Enter `skip` or press Enter to skip syncing. -After upgrading, always sync any active projects to ensure they have the latest skills and hook behaviour. +After upgrading, always sync active projects to ensure they have the latest skills and agents. diff --git a/docs/skills/index.md b/docs/skills/index.md index aff525e2..745fe384 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -14,7 +14,7 @@ Skills are Claude Code slash commands bundled in the lightcone-cli plugin. They ## How skills work -Each skill is a markdown file (`SKILL.md`) in `.claude/skills/{skill-name}/`. Claude Code discovers skills by scanning the `.claude/skills/` directory. The frontmatter configures the skill's metadata and allowed tools: +Each skill is a markdown file (`SKILL.md`) in `./skills/{skill-name}/`. The agent tool discovers skills by scanning that directory. The frontmatter configures the skill's metadata and allowed tools: ```yaml --- @@ -24,15 +24,29 @@ allowed-tools: Read, Write, Edit, Bash, WebSearch, WebFetch --- ``` -The body of the file is a structured prompt that tells Claude exactly how to proceed, including phase definitions, rules, and references to guide files. +The body of the file is a structured prompt that tells the agent exactly how to proceed, including phase definitions, rules, and references to guide files. ## Plugin installation -Skills are installed by `lc init` (copying the plugin to `.claude/`) and updated by `lc update --sync`. They live at: +Skills are installed by `lc init` and updated by `lc update --sync`. `lc init` copies skills (and agents, guides) to every harness selected via `--tools`; the default is `claude`. Hooks and scripts are installed for Claude Code only. + +Canonical source locations: - **Bundled (installed package)**: `{site-packages}/lightcone/cli/claude/lightcone/skills/` - **Development**: `{repo}/claude/lightcone/skills/` +## Multi-harness distribution + +Skills are harness-agnostic — the same `SKILL.md` files are installed into each selected harness's `skills/` directory: + +| Harness | Install path | `--tools` value | +|---------|-------------|-----------------| +| Claude Code | `.claude/skills/` | `claude` | +| Codex | `.codex/skills/` | `codex` | +| Cursor | `.cursor/skills/` | `cursor` | +| GitHub Copilot | `.github/skills/` | `github-copilot` | +| OpenCode | `.opencode/skills/` | `opencode` | + ## Related files | File | Purpose | From 81b5e11c398e5d86f510328e5152fa04611a9c0c Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 23:20:20 +0200 Subject: [PATCH 8/9] =?UTF-8?q?LCR-85:=20Rename=20claude/=20=E2=86=92=20pl?= =?UTF-8?q?ugin/=20(harness-agnostic=20plugin=20source)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plugin source directory was named after Claude Code. Now that skills, agents, and guides are distributed to all harnesses, rename to plugin/ to reflect its actual purpose. - git mv claude/ plugin/ (history preserved) - pyproject.toml: update force-include and sdist include paths - src/lightcone/cli/plugin.py: update bundled + dev path constants - CLAUDE.md, docs/: replace all claude/lightcone and lightcone/cli/claude/lightcone path references Co-Authored-By: Claude Sonnet 4.6 --- CLAUDE.md | 12 ++++++------ docs/api/cli.md | 4 ++-- docs/architecture.md | 2 +- docs/contributing/setup.md | 6 +++--- docs/index.md | 2 +- docs/skills/authoring.md | 6 +++--- docs/skills/index.md | 12 ++++++------ docs/skills/lc-build.md | 2 +- docs/skills/lc-new.md | 2 +- docs/telemetry/hooks.md | 2 +- docs/telemetry/opt-out.md | 2 +- {claude => plugin}/lightcone/agents/lc-extractor.md | 0 .../lightcone/guides/astra-reference.md | 0 .../lightcone/guides/lightcone-cli-reference.md | 0 {claude => plugin}/lightcone/guides/ui-brand.md | 0 .../lightcone/hooks/langfuse_git_commit_hook.py | 0 {claude => plugin}/lightcone/hooks/langfuse_hook.py | 0 .../lightcone/hooks/langfuse_prepare_commit_msg.py | 0 .../lightcone/hooks/langfuse_session_init_hook.py | 0 {claude => plugin}/lightcone/hooks/langfuse_utils.py | 0 .../lightcone/scripts/activate-venv.sh | 0 {claude => plugin}/lightcone/scripts/check-lc-run.sh | 0 .../lightcone/scripts/session-start.sh | 0 .../lightcone/scripts/validate-on-save.sh | 0 .../lightcone/skills/lc-build/SKILL.md | 0 .../lightcone/skills/lc-build/assets/loop-prompt.md | 0 .../skills/lc-build/scripts/setup-lc-build.sh | 0 .../lightcone/skills/lc-feedback/SKILL.md | 0 .../lightcone/skills/lc-migrate/SKILL.md | 0 {claude => plugin}/lightcone/skills/lc-new/SKILL.md | 0 .../lightcone/skills/lc-verify/SKILL.md | 0 {claude => plugin}/lightcone/templates/CLAUDE.md | 0 pyproject.toml | 4 ++-- src/lightcone/cli/plugin.py | 12 ++++++------ 34 files changed, 34 insertions(+), 34 deletions(-) rename {claude => plugin}/lightcone/agents/lc-extractor.md (100%) rename {claude => plugin}/lightcone/guides/astra-reference.md (100%) rename {claude => plugin}/lightcone/guides/lightcone-cli-reference.md (100%) rename {claude => plugin}/lightcone/guides/ui-brand.md (100%) rename {claude => plugin}/lightcone/hooks/langfuse_git_commit_hook.py (100%) rename {claude => plugin}/lightcone/hooks/langfuse_hook.py (100%) rename {claude => plugin}/lightcone/hooks/langfuse_prepare_commit_msg.py (100%) rename {claude => plugin}/lightcone/hooks/langfuse_session_init_hook.py (100%) rename {claude => plugin}/lightcone/hooks/langfuse_utils.py (100%) rename {claude => plugin}/lightcone/scripts/activate-venv.sh (100%) rename {claude => plugin}/lightcone/scripts/check-lc-run.sh (100%) rename {claude => plugin}/lightcone/scripts/session-start.sh (100%) rename {claude => plugin}/lightcone/scripts/validate-on-save.sh (100%) rename {claude => plugin}/lightcone/skills/lc-build/SKILL.md (100%) rename {claude => plugin}/lightcone/skills/lc-build/assets/loop-prompt.md (100%) rename {claude => plugin}/lightcone/skills/lc-build/scripts/setup-lc-build.sh (100%) rename {claude => plugin}/lightcone/skills/lc-feedback/SKILL.md (100%) rename {claude => plugin}/lightcone/skills/lc-migrate/SKILL.md (100%) rename {claude => plugin}/lightcone/skills/lc-new/SKILL.md (100%) rename {claude => plugin}/lightcone/skills/lc-verify/SKILL.md (100%) rename {claude => plugin}/lightcone/templates/CLAUDE.md (100%) diff --git a/CLAUDE.md b/CLAUDE.md index 6fbcb7ab..8fa0d6db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,7 +28,7 @@ src/lightcone/ # namespace — NO __init__.py │ ├── commands.py # all Click commands (init, run, build, status, dev, target, setup, update) │ ├── harness.py # harness registry — maps tool IDs to install paths (claude, codex, cursor...) │ ├── plugin.py # get_plugin_source_dir — leaf module (no imports from commands.py) -│ └── claude/ # force-included Claude plugin bundle (in installed wheel only) +│ └── plugin/ # force-included agent harness plugin bundle (in installed wheel only) ├── engine/ # execution substrate — Dagster + HPC + containers │ ├── __init__.py │ ├── assets.py # Asset factory — turns astra.yaml recipes into Dagster assets @@ -43,7 +43,7 @@ src/lightcone/ # namespace — NO __init__.py ├── cli.py # `lc eval` subcommand group (registered by lightcone.cli.commands) ├── harness.py, sandbox.py, graders.py, build.py, report.py, models.py -claude/lightcone/ # Claude plugin source — force-included into the wheel +plugin/lightcone/ # Claude plugin source — force-included into the wheel ├── skills/ # lc-new, lc-build, lc-verify, lc-migrate, lc-feedback ├── agents/ # lc-extractor ├── guides/ # astra-reference, lightcone-cli-reference, ui-brand @@ -131,11 +131,11 @@ astra.yaml → build_definitions() → Dagster assets → ASTRAContainerRunner - Most commands require `astra.yaml` in cwd; exceptions: `setup`, `target` **Plugin system:** -- Skills, agents, guides, hooks, and scripts are bundled in the wheel (`claude/lightcone/` → `lightcone/cli/claude/lightcone/`) +- Skills, agents, guides, hooks, and scripts are bundled in the wheel (`plugin/lightcone/` → `lightcone/cli/plugin/lightcone/`) - `lc init [--tools T]` copies content into `./` for each selected harness; default harness is `claude` - Skills, agents, and guides are installed for every harness; hooks/scripts are Claude Code only - `lc update --sync [--tools T]` re-syncs skills/agents/guides (hooks are init-time only, not synced) -- Plugin source discovery: `lightcone.cli.plugin.get_plugin_source_dir` — bundled location first, then `claude/lightcone/` at repo root +- Plugin source discovery: `lightcone.cli.plugin.get_plugin_source_dir` — bundled location first, then `plugin/lightcone/` at repo root - Bash scripts must be chmod +x ## CLI Patterns @@ -154,8 +154,8 @@ All commands use Click. Key patterns: | Add an HPC site | `src/lightcone/engine/site_registry.py` | Add to `SITE_DEFAULTS` dict with hostname_patterns, node_types, qos_options | | Add an execution backend | `src/lightcone/engine/runner.py` | Add `_run_{backend}()` method, update `execute()` dispatch | | Add container features | `src/lightcone/engine/container.py` | `DEPENDENCY_FILES` tuple, `compute_image_tag()`, build/resolve functions | -| Create a skill | `claude/lightcone/skills/` | SKILL.md with YAML frontmatter (`name`, `description`, `allowed-tools`) | -| Add a telemetry hook | `claude/lightcone/hooks/` | Follow `langfuse_hook.py` pattern: read JSON payload, emit to Langfuse | +| Create a skill | `plugin/lightcone/skills/` | SKILL.md with YAML frontmatter (`name`, `description`, `allowed-tools`) | +| Add a telemetry hook | `plugin/lightcone/hooks/` | Follow `langfuse_hook.py` pattern: read JSON payload, emit to Langfuse | | Add a harness target | `src/lightcone/cli/harness.py` | Add to `HARNESS_REGISTRY` dataclass + `ALL_TOOL_IDS` tuple | ## Test Patterns diff --git a/docs/api/cli.md b/docs/api/cli.md index ff72ceca..807d5e12 100644 --- a/docs/api/cli.md +++ b/docs/api/cli.md @@ -31,8 +31,8 @@ Keys: `"yolo"`, `"recommended"`, `"minimal"`. Finds the lightcone-cli plugin source directory. Checks: -1. **Bundled** (installed package): `lightcone/cli/claude/lightcone/` -2. **Development** (repo checkout): `{repo_root}/claude/lightcone/` +1. **Bundled** (installed package): `lightcone/cli/plugin/lightcone/` +2. **Development** (repo checkout): `{repo_root}/plugin/lightcone/` Returns `None` if neither exists. diff --git a/docs/architecture.md b/docs/architecture.md index 4efbb661..0bf4b8cc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -73,7 +73,7 @@ For SLURM/`podman-hpc` targets, `resolve_container_for_slurm()` additionally mig ### Structure -The plugin lives in `claude/lightcone/` and is bundled into the Python wheel via `hatch-vcs` force-include directives. When `lc init` runs, it copies the plugin into one or more harness directories selected via `--tools` (default: `claude`). +The plugin lives in `plugin/lightcone/` and is bundled into the Python wheel via `hatch-vcs` force-include directives. When `lc init` runs, it copies the plugin into one or more harness directories selected via `--tools` (default: `claude`). **Content installed to every harness** (`./`): diff --git a/docs/contributing/setup.md b/docs/contributing/setup.md index d38dc01f..6ab467c7 100644 --- a/docs/contributing/setup.md +++ b/docs/contributing/setup.md @@ -61,7 +61,7 @@ Ruff rules: E, F, I, N, W, UP. Line length: 100. Target: Python 3.11. ``` src/lightcone/ # main package -claude/lightcone/ # plugin files (bundled via hatch force-include) +plugin/lightcone/ # plugin files (bundled via hatch force-include) tests/ # mirrors src/ structure evals/ # skill evaluation fixtures ``` @@ -73,14 +73,14 @@ just build # uv build just version # uv run hatch version ``` -The `hatch-vcs` plugin derives the version from git tags. The `claude/lightcone/` directory is force-included in the wheel via `pyproject.toml`: +The `hatch-vcs` plugin derives the version from git tags. The `plugin/lightcone/` directory is force-included in the wheel via `pyproject.toml`: ```toml [tool.hatch.build.targets.wheel] packages = ["src/lightcone"] [tool.hatch.build.force-include] -"claude/lightcone" = "lightcone/cli/claude/lightcone" +"plugin/lightcone" = "lightcone/cli/plugin/lightcone" ``` ## Building the documentation diff --git a/docs/index.md b/docs/index.md index 83f8de7a..2bb723ef 100644 --- a/docs/index.md +++ b/docs/index.md @@ -26,7 +26,7 @@ src/lightcone/ ├── targets.py # Target config management (~/.lightcone/targets/) └── tree.py # Sub-analysis tree traversal helpers -claude/lightcone/ # Claude Code plugin (bundled into wheel via hatch force-include) +plugin/lightcone/ # agent harness plugin (bundled into wheel via hatch force-include) ├── skills/ # lc-new, lc-build, lc-verify, lc-migrate, lc-feedback ├── templates/ # Project CLAUDE.md template ├── agents/ # lc-extractor (literature extraction subagent) diff --git a/docs/skills/authoring.md b/docs/skills/authoring.md index af961cc7..0bd35131 100644 --- a/docs/skills/authoring.md +++ b/docs/skills/authoring.md @@ -1,11 +1,11 @@ # Authoring Skills -Skills are markdown files with YAML frontmatter. They live in `claude/lightcone/skills/{name}/SKILL.md`. +Skills are markdown files with YAML frontmatter. They live in `plugin/lightcone/skills/{name}/SKILL.md`. ## File structure ``` -claude/lightcone/skills/ +plugin/lightcone/skills/ └── my-skill/ └── SKILL.md ``` @@ -45,7 +45,7 @@ Before starting, read `.claude/guides/astra-reference.md` for the full ASTRA spe ## Installing into projects -New skills in `claude/lightcone/skills/` are installed automatically by `lc init`. To push skills to existing projects, run `lc update --sync`. +New skills in `plugin/lightcone/skills/` are installed automatically by `lc init`. To push skills to existing projects, run `lc update --sync`. ## Testing skills diff --git a/docs/skills/index.md b/docs/skills/index.md index 745fe384..949d04bf 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -32,8 +32,8 @@ Skills are installed by `lc init` and updated by `lc update --sync`. `lc init` c Canonical source locations: -- **Bundled (installed package)**: `{site-packages}/lightcone/cli/claude/lightcone/skills/` -- **Development**: `{repo}/claude/lightcone/skills/` +- **Bundled (installed package)**: `{site-packages}/lightcone/cli/plugin/lightcone/skills/` +- **Development**: `{repo}/plugin/lightcone/skills/` ## Multi-harness distribution @@ -51,7 +51,7 @@ Skills are harness-agnostic — the same `SKILL.md` files are installed into eac | File | Purpose | |------|---------| -| `claude/lightcone/guides/lightcone-cli-reference.md` | CLI and workflow reference loaded by build/verify skills | -| `claude/lightcone/guides/astra-reference.md` | Full ASTRA spec reference loaded by all skills | -| `claude/lightcone/guides/ui-brand.md` | Visual formatting conventions for skill output | -| `claude/lightcone/agents/lc-extractor.md` | Literature extraction subagent used by `/lc-new` | +| `plugin/lightcone/guides/lightcone-cli-reference.md` | CLI and workflow reference loaded by build/verify skills | +| `plugin/lightcone/guides/astra-reference.md` | Full ASTRA spec reference loaded by all skills | +| `plugin/lightcone/guides/ui-brand.md` | Visual formatting conventions for skill output | +| `plugin/lightcone/agents/lc-extractor.md` | Literature extraction subagent used by `/lc-new` | diff --git a/docs/skills/lc-build.md b/docs/skills/lc-build.md index fd2c224a..919b18a6 100644 --- a/docs/skills/lc-build.md +++ b/docs/skills/lc-build.md @@ -55,4 +55,4 @@ For each pending output, in dependency order: ## Related - [lc-verify](lc-verify.md) — run after build to check consistency -- `claude/lightcone/guides/lightcone-cli-reference.md` — CLI and execution reference +- `plugin/lightcone/guides/lightcone-cli-reference.md` — CLI and execution reference diff --git a/docs/skills/lc-new.md b/docs/skills/lc-new.md index c5e6537c..08285ca4 100644 --- a/docs/skills/lc-new.md +++ b/docs/skills/lc-new.md @@ -43,4 +43,4 @@ Create a new ASTRA analysis from a research question. ## Related - [lc-build](lc-build.md) — the next step after `/lc-new` -- `claude/lightcone/guides/astra-reference.md` — full `astra.yaml` spec +- `plugin/lightcone/guides/astra-reference.md` — full `astra.yaml` spec diff --git a/docs/telemetry/hooks.md b/docs/telemetry/hooks.md index 5ea3cc50..4a03728b 100644 --- a/docs/telemetry/hooks.md +++ b/docs/telemetry/hooks.md @@ -1,6 +1,6 @@ # Hooks Architecture -The telemetry system is composed of four Python scripts in `claude/lightcone/hooks/`: +The telemetry system is composed of four Python scripts in `plugin/lightcone/hooks/`: ``` hooks/ diff --git a/docs/telemetry/opt-out.md b/docs/telemetry/opt-out.md index c9489743..f351ae56 100644 --- a/docs/telemetry/opt-out.md +++ b/docs/telemetry/opt-out.md @@ -25,4 +25,4 @@ To disable telemetry for all new projects, unset the key or set it to `false` be ## Transparency -The full telemetry implementation is in `claude/lightcone/hooks/`. All hooks are plain Python scripts installed in each project's `.claude/hooks/` directory — they can be inspected, modified, or deleted per-project. +The full telemetry implementation is in `plugin/lightcone/hooks/`. All hooks are plain Python scripts installed in each project's `.claude/hooks/` directory — they can be inspected, modified, or deleted per-project. diff --git a/claude/lightcone/agents/lc-extractor.md b/plugin/lightcone/agents/lc-extractor.md similarity index 100% rename from claude/lightcone/agents/lc-extractor.md rename to plugin/lightcone/agents/lc-extractor.md diff --git a/claude/lightcone/guides/astra-reference.md b/plugin/lightcone/guides/astra-reference.md similarity index 100% rename from claude/lightcone/guides/astra-reference.md rename to plugin/lightcone/guides/astra-reference.md diff --git a/claude/lightcone/guides/lightcone-cli-reference.md b/plugin/lightcone/guides/lightcone-cli-reference.md similarity index 100% rename from claude/lightcone/guides/lightcone-cli-reference.md rename to plugin/lightcone/guides/lightcone-cli-reference.md diff --git a/claude/lightcone/guides/ui-brand.md b/plugin/lightcone/guides/ui-brand.md similarity index 100% rename from claude/lightcone/guides/ui-brand.md rename to plugin/lightcone/guides/ui-brand.md diff --git a/claude/lightcone/hooks/langfuse_git_commit_hook.py b/plugin/lightcone/hooks/langfuse_git_commit_hook.py similarity index 100% rename from claude/lightcone/hooks/langfuse_git_commit_hook.py rename to plugin/lightcone/hooks/langfuse_git_commit_hook.py diff --git a/claude/lightcone/hooks/langfuse_hook.py b/plugin/lightcone/hooks/langfuse_hook.py similarity index 100% rename from claude/lightcone/hooks/langfuse_hook.py rename to plugin/lightcone/hooks/langfuse_hook.py diff --git a/claude/lightcone/hooks/langfuse_prepare_commit_msg.py b/plugin/lightcone/hooks/langfuse_prepare_commit_msg.py similarity index 100% rename from claude/lightcone/hooks/langfuse_prepare_commit_msg.py rename to plugin/lightcone/hooks/langfuse_prepare_commit_msg.py diff --git a/claude/lightcone/hooks/langfuse_session_init_hook.py b/plugin/lightcone/hooks/langfuse_session_init_hook.py similarity index 100% rename from claude/lightcone/hooks/langfuse_session_init_hook.py rename to plugin/lightcone/hooks/langfuse_session_init_hook.py diff --git a/claude/lightcone/hooks/langfuse_utils.py b/plugin/lightcone/hooks/langfuse_utils.py similarity index 100% rename from claude/lightcone/hooks/langfuse_utils.py rename to plugin/lightcone/hooks/langfuse_utils.py diff --git a/claude/lightcone/scripts/activate-venv.sh b/plugin/lightcone/scripts/activate-venv.sh similarity index 100% rename from claude/lightcone/scripts/activate-venv.sh rename to plugin/lightcone/scripts/activate-venv.sh diff --git a/claude/lightcone/scripts/check-lc-run.sh b/plugin/lightcone/scripts/check-lc-run.sh similarity index 100% rename from claude/lightcone/scripts/check-lc-run.sh rename to plugin/lightcone/scripts/check-lc-run.sh diff --git a/claude/lightcone/scripts/session-start.sh b/plugin/lightcone/scripts/session-start.sh similarity index 100% rename from claude/lightcone/scripts/session-start.sh rename to plugin/lightcone/scripts/session-start.sh diff --git a/claude/lightcone/scripts/validate-on-save.sh b/plugin/lightcone/scripts/validate-on-save.sh similarity index 100% rename from claude/lightcone/scripts/validate-on-save.sh rename to plugin/lightcone/scripts/validate-on-save.sh diff --git a/claude/lightcone/skills/lc-build/SKILL.md b/plugin/lightcone/skills/lc-build/SKILL.md similarity index 100% rename from claude/lightcone/skills/lc-build/SKILL.md rename to plugin/lightcone/skills/lc-build/SKILL.md diff --git a/claude/lightcone/skills/lc-build/assets/loop-prompt.md b/plugin/lightcone/skills/lc-build/assets/loop-prompt.md similarity index 100% rename from claude/lightcone/skills/lc-build/assets/loop-prompt.md rename to plugin/lightcone/skills/lc-build/assets/loop-prompt.md diff --git a/claude/lightcone/skills/lc-build/scripts/setup-lc-build.sh b/plugin/lightcone/skills/lc-build/scripts/setup-lc-build.sh similarity index 100% rename from claude/lightcone/skills/lc-build/scripts/setup-lc-build.sh rename to plugin/lightcone/skills/lc-build/scripts/setup-lc-build.sh diff --git a/claude/lightcone/skills/lc-feedback/SKILL.md b/plugin/lightcone/skills/lc-feedback/SKILL.md similarity index 100% rename from claude/lightcone/skills/lc-feedback/SKILL.md rename to plugin/lightcone/skills/lc-feedback/SKILL.md diff --git a/claude/lightcone/skills/lc-migrate/SKILL.md b/plugin/lightcone/skills/lc-migrate/SKILL.md similarity index 100% rename from claude/lightcone/skills/lc-migrate/SKILL.md rename to plugin/lightcone/skills/lc-migrate/SKILL.md diff --git a/claude/lightcone/skills/lc-new/SKILL.md b/plugin/lightcone/skills/lc-new/SKILL.md similarity index 100% rename from claude/lightcone/skills/lc-new/SKILL.md rename to plugin/lightcone/skills/lc-new/SKILL.md diff --git a/claude/lightcone/skills/lc-verify/SKILL.md b/plugin/lightcone/skills/lc-verify/SKILL.md similarity index 100% rename from claude/lightcone/skills/lc-verify/SKILL.md rename to plugin/lightcone/skills/lc-verify/SKILL.md diff --git a/claude/lightcone/templates/CLAUDE.md b/plugin/lightcone/templates/CLAUDE.md similarity index 100% rename from claude/lightcone/templates/CLAUDE.md rename to plugin/lightcone/templates/CLAUDE.md diff --git a/pyproject.toml b/pyproject.toml index d00b90c7..7f000a88 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,10 +54,10 @@ source = "vcs" packages = ["src/lightcone"] [tool.hatch.build.targets.wheel.force-include] -"claude/lightcone" = "lightcone/cli/claude/lightcone" +"plugin/lightcone" = "lightcone/cli/plugin/lightcone" [tool.hatch.build.targets.sdist] -include = ["src/lightcone", "claude/lightcone"] +include = ["src/lightcone", "plugin/lightcone"] [tool.ruff] target-version = "py311" diff --git a/src/lightcone/cli/plugin.py b/src/lightcone/cli/plugin.py index 903dd263..99b8cdbb 100644 --- a/src/lightcone/cli/plugin.py +++ b/src/lightcone/cli/plugin.py @@ -1,4 +1,4 @@ -"""Plugin bundle discovery — finds the Claude Code skills/hooks shipped with lightcone-cli. +"""Plugin bundle discovery — finds the agent harness plugin shipped with lightcone-cli. Kept deliberately leaf (no imports from :mod:`lightcone.cli.commands` or :mod:`lightcone.eval`) so it can be used by both the CLI and the eval harness without introducing an import cycle. @@ -10,24 +10,24 @@ def get_plugin_source_dir() -> Path | None: - """Find the lightcone Claude plugin source directory. + """Find the lightcone agent harness plugin source directory. Looks for the plugin files in: - 1. Bundled location (installed package): ``lightcone/cli/claude/lightcone/`` - 2. Development location (repo): ``claude/lightcone/`` relative to repo root + 1. Bundled location (installed package): ``lightcone/cli/plugin/lightcone/`` + 2. Development location (repo): ``plugin/lightcone/`` relative to repo root """ import lightcone.cli package_dir = Path(lightcone.cli.__file__).parent - bundled_plugin = package_dir / "claude" / "lightcone" + bundled_plugin = package_dir / "plugin" / "lightcone" if bundled_plugin.exists(): return bundled_plugin # Try development location (running from repo) # package_dir == /src/lightcone/cli → parents[2] == repo_root = package_dir.parents[2] - dev_plugin = repo_root / "claude" / "lightcone" + dev_plugin = repo_root / "plugin" / "lightcone" if dev_plugin.exists(): return dev_plugin From d19dd2caeb9b1fb2af3c61f2fa7c0bd1e1c6019f Mon Sep 17 00:00:00 2001 From: Alexandre Boucaud Date: Wed, 22 Apr 2026 23:38:22 +0200 Subject: [PATCH 9/9] =?UTF-8?q?LCR-85:=20Rename=20CLAUDE.md=20=E2=86=92=20?= =?UTF-8?q?AGENTS.md;=20keep=20thin=20CLAUDE.md=20pointer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md is the harness-agnostic convention read by Codex and other agent tools. CLAUDE.md is retained as a one-line pointer for Claude Code session compatibility. Co-Authored-By: Claude Sonnet 4.6 --- AGENTS.md | 177 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 177 +----------------------------------------------------- 2 files changed, 179 insertions(+), 175 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..bef7f2f4 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,177 @@ +# AGENTS.md + +## Project Overview + +**lightcone-cli** is Lightcone Research's agentic layer for ASTRA (Agentic Schema for Transparent Research Analysis). It ships the `lc` executable and Claude Code skills/hooks used during interactive analysis work. + +- **ASTRA** = pure specification: schema, validation, prior insights & findings, evidence verification, helpers, minimal CLI +- **lightcone-cli** = agentic layer: Claude Code skills, project scaffolding, Dagster execution, HPC targets, container builds, telemetry + +lightcone-cli depends on ASTRA. The `astra` CLI handles spec operations; the `lc` CLI handles execution and agent operations. + +### Namespace contract + +`lightcone-cli` ships the `lightcone.*` namespace via PEP 420 implicit namespace packages. **`src/lightcone/` must not contain an `__init__.py`** — that would turn the namespace into a regular package and break coexistence with future sibling distributions (`lightcone-ui`, etc.). + +Any new `lightcone-*` package must: + +1. Use src-layout (`src/lightcone//…`). +2. Not create `src/lightcone/__init__.py`. +3. Ship only its own subpackage under `src/lightcone//`. + +## Repository Structure + +``` +src/lightcone/ # namespace — NO __init__.py +├── cli/ # Click surface +│ ├── __init__.py # exposes main() +│ ├── commands.py # all Click commands (init, run, build, status, dev, target, setup, update) +│ ├── harness.py # harness registry — maps tool IDs to install paths (claude, codex, cursor...) +│ ├── plugin.py # get_plugin_source_dir — leaf module (no imports from commands.py) +│ └── plugin/ # force-included agent harness plugin bundle (in installed wheel only) +├── engine/ # execution substrate — Dagster + HPC + containers +│ ├── __init__.py +│ ├── assets.py # Asset factory — turns astra.yaml recipes into Dagster assets +│ ├── container.py # Content-addressed container builds (Docker, podman-hpc) +│ ├── io_manager.py # Maps (output, universe) → results/{universe}/{output}/ +│ ├── runner.py # Execution backends: Docker, local, SLURM +│ ├── site_registry.py # Known HPC site defaults (Perlmutter, etc.) +│ ├── status.py # Materialization status queries +│ ├── targets.py # Target config management (~/.lightcone/targets/) +│ └── tree.py # Sub-analysis tree traversal +└── eval/ # Quantitative eval harness (top-level; peer of cli/engine) + ├── cli.py # `lc eval` subcommand group (registered by lightcone.cli.commands) + ├── harness.py, sandbox.py, graders.py, build.py, report.py, models.py + +plugin/lightcone/ # Claude plugin source — force-included into the wheel +├── skills/ # lc-new, lc-build, lc-verify, lc-migrate, lc-feedback +├── agents/ # lc-extractor +├── guides/ # astra-reference, lightcone-cli-reference, ui-brand +├── templates/ # Project CLAUDE.md template +├── hooks/ # Langfuse telemetry hooks (Python) +└── scripts/ # Session hooks (bash): venv activation, validate-on-save, status display + +tests/ # pytest — mirrors src/ structure +pyproject.toml # hatchling + hatch-vcs, ASTRA as git dep +``` + +### Engine/CLI dependency contract + +`lightcone.engine` must import nothing from `lightcone.cli` or `lightcone.eval`. This keeps the door open to carving `lightcone-engine` into its own PyPI dist when a future `lightcone-ui` needs it — no code changes required. + +## Development Commands + +```bash +uv sync --group dev # installs pytest, ruff, mypy into the uv env +uv run pytest +uv run ruff check src/ tests/ +uv run mypy src/ +``` + +A `justfile` is available for common tasks — run `just` to see all recipes: + +```bash +just test # run pytest +just lint # ruff + mypy +just docs # build the documentation site +just docs-serve # live preview at http://127.0.0.1:8000 +just install # uv sync --all-groups +``` + +## Documentation + +Maintainer documentation lives in `docs/` and is built with [Zensical](https://zensical.org). Configuration is in `zensical.toml`. + +``` +docs/ +├── index.md # Overview, repo structure, key invariants +├── architecture.md # Dagster integration, container mgmt, plugin system +├── cli/ # One page per lc command +├── api/ # One page per Python module +├── skills/ # Skill reference + authoring guide +├── telemetry/ # Langfuse hooks, session lifecycle, opt-out +├── hpc/ # SLURM, site registry, targets, container builds +└── contributing/ # Dev setup, adding backends/sites, testing +``` + +Dependencies are declared in `pyproject.toml` under `[dependency-groups].docs` and managed with `uv`: + +```bash +just docs-serve # syncs docs group then serves with live reload at http://127.0.0.1:8000 +just docs-strict # build with --strict (accepted flag, not yet enforced by zensical) +``` + +## Architecture & Data Flow + +``` +astra.yaml → build_definitions() → Dagster assets → ASTRAContainerRunner → results/{universe}/{output}/ + ↑ ↑ + ASTRAIOManager Docker / local / SLURM +``` + +- `build_definitions()` (`lightcone.engine.assets`) loads astra.yaml, creates one Dagster asset per output with a recipe +- Asset dependencies come from `recipe.inputs` — Dagster resolves execution order +- `ASTRAContainerRunner` (`lightcone.engine.runner`) dispatches to Docker, local subprocess, or SLURM based on target config +- Docker backend falls back to local execution on failure (with warning) +- SLURM backend generates sbatch scripts, submits via `sbatch`, polls via `sacct`/`squeue` + +## Key Invariants + +**Spec & execution:** +- `astra.yaml` is the single source of truth — all inputs, outputs, recipes, decisions, containers +- Output paths are always `results/{universe_id}/{output_id}/` — enforced by IO manager, no customization +- Container is a single string: image name (e.g., `python:3.9`) is pulled; file path (e.g., `Containerfile`) is built. No `container_build` dict — runtime detects via file existence. +- Container image tags are deterministic: SHA256(Containerfile + dependency files) → `lc-{name}-{hash}` +- Universe decision parameters are injected as CLI args: `--key value` passed to recipe commands +- Per-recipe container specs override analysis-level defaults + +**Config resolution (used everywhere):** +- Target: `--target` flag > `.lightcone/lightcone.yaml` > `~/.lightcone/config.yaml` > `"local"` +- Permission tier: `--permissions` flag > saved default in `~/.lightcone/config.yaml` > interactive prompt +- Most commands require `astra.yaml` in cwd; exceptions: `setup`, `target` + +**Plugin system:** +- Skills, agents, guides, hooks, and scripts are bundled in the wheel (`plugin/lightcone/` → `lightcone/cli/plugin/lightcone/`) +- `lc init [--tools T]` copies content into `./` for each selected harness; default harness is `claude` +- Skills, agents, and guides are installed for every harness; hooks/scripts are Claude Code only +- `lc update --sync [--tools T]` re-syncs skills/agents/guides (hooks are init-time only, not synced) +- Plugin source discovery: `lightcone.cli.plugin.get_plugin_source_dir` — bundled location first, then `plugin/lightcone/` at repo root +- Bash scripts must be chmod +x + +## CLI Patterns + +All commands use Click. Key patterns: +- `@main.command()` for top-level commands, `@main.group()` for subgroups (`target`) +- Target/config resolution is shared logic, not per-command +- `lc setup` auto-triggers if `~/.lightcone/config.yaml` doesn't exist when running other commands +- Three permission tiers: `yolo` (all allowed), `recommended` (workflow allowed), `minimal` (read-only) + +## Extending the Codebase + +| To... | Read | Key patterns | +|---|---|---| +| Add a CLI command | `src/lightcone/cli/commands.py` | `@main.command()`, config resolution, `click.echo` with Rich | +| Add an HPC site | `src/lightcone/engine/site_registry.py` | Add to `SITE_DEFAULTS` dict with hostname_patterns, node_types, qos_options | +| Add an execution backend | `src/lightcone/engine/runner.py` | Add `_run_{backend}()` method, update `execute()` dispatch | +| Add container features | `src/lightcone/engine/container.py` | `DEPENDENCY_FILES` tuple, `compute_image_tag()`, build/resolve functions | +| Create a skill | `plugin/lightcone/skills/` | SKILL.md with YAML frontmatter (`name`, `description`, `allowed-tools`) | +| Add a telemetry hook | `plugin/lightcone/hooks/` | Follow `langfuse_hook.py` pattern: read JSON payload, emit to Langfuse | +| Add a harness target | `src/lightcone/cli/harness.py` | Add to `HARNESS_REGISTRY` dataclass + `ALL_TOOL_IDS` tuple | + +## Test Patterns + +- CLI tests: `CliRunner().invoke(main, ["command", ...])` — check exit code, output, file side effects +- Asset tests: call `build_asset_definitions(spec, runner=mock_runner)` — verify keys, deps, metadata +- Runner tests: create runner with tmp project root, call `execute()` — verify exit code and metadata +- Common fixture: `_fake_config` monkeypatches `get_config_path` to prevent auto-setup wizard in tests +- Integration tests in `test_integration.py` and `test_cli_run.py` cover end-to-end flows + +## Conventions + +- Ruff for linting (E, F, I, N, W, UP), line length 100, target Python 3.11 +- mypy strict mode with `namespace_packages = true`, `explicit_package_bases = true` +- Status states: `"ok"` (materialized), `"pending"` (has recipe, not run), `"no_recipe"` (declared, no recipe) +- SLURM scripts/output stored in `results/.slurm/` +- Dagster instance storage at `results/.dagster/` (SQLite) +- Telemetry opt-out: `TRACE_TO_LANGFUSE=false` +- **Keep AGENTS.md current** — when a PR changes CLI behaviour, key invariants, or repo structure, update the relevant section of AGENTS.md in the same commit diff --git a/CLAUDE.md b/CLAUDE.md index 8fa0d6db..97336086 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,177 +1,4 @@ # CLAUDE.md -## Project Overview - -**lightcone-cli** is Lightcone Research's agentic layer for ASTRA (Agentic Schema for Transparent Research Analysis). It ships the `lc` executable and Claude Code skills/hooks used during interactive analysis work. - -- **ASTRA** = pure specification: schema, validation, prior insights & findings, evidence verification, helpers, minimal CLI -- **lightcone-cli** = agentic layer: Claude Code skills, project scaffolding, Dagster execution, HPC targets, container builds, telemetry - -lightcone-cli depends on ASTRA. The `astra` CLI handles spec operations; the `lc` CLI handles execution and agent operations. - -### Namespace contract - -`lightcone-cli` ships the `lightcone.*` namespace via PEP 420 implicit namespace packages. **`src/lightcone/` must not contain an `__init__.py`** — that would turn the namespace into a regular package and break coexistence with future sibling distributions (`lightcone-ui`, etc.). - -Any new `lightcone-*` package must: - -1. Use src-layout (`src/lightcone//…`). -2. Not create `src/lightcone/__init__.py`. -3. Ship only its own subpackage under `src/lightcone//`. - -## Repository Structure - -``` -src/lightcone/ # namespace — NO __init__.py -├── cli/ # Click surface -│ ├── __init__.py # exposes main() -│ ├── commands.py # all Click commands (init, run, build, status, dev, target, setup, update) -│ ├── harness.py # harness registry — maps tool IDs to install paths (claude, codex, cursor...) -│ ├── plugin.py # get_plugin_source_dir — leaf module (no imports from commands.py) -│ └── plugin/ # force-included agent harness plugin bundle (in installed wheel only) -├── engine/ # execution substrate — Dagster + HPC + containers -│ ├── __init__.py -│ ├── assets.py # Asset factory — turns astra.yaml recipes into Dagster assets -│ ├── container.py # Content-addressed container builds (Docker, podman-hpc) -│ ├── io_manager.py # Maps (output, universe) → results/{universe}/{output}/ -│ ├── runner.py # Execution backends: Docker, local, SLURM -│ ├── site_registry.py # Known HPC site defaults (Perlmutter, etc.) -│ ├── status.py # Materialization status queries -│ ├── targets.py # Target config management (~/.lightcone/targets/) -│ └── tree.py # Sub-analysis tree traversal -└── eval/ # Quantitative eval harness (top-level; peer of cli/engine) - ├── cli.py # `lc eval` subcommand group (registered by lightcone.cli.commands) - ├── harness.py, sandbox.py, graders.py, build.py, report.py, models.py - -plugin/lightcone/ # Claude plugin source — force-included into the wheel -├── skills/ # lc-new, lc-build, lc-verify, lc-migrate, lc-feedback -├── agents/ # lc-extractor -├── guides/ # astra-reference, lightcone-cli-reference, ui-brand -├── templates/ # Project CLAUDE.md template -├── hooks/ # Langfuse telemetry hooks (Python) -└── scripts/ # Session hooks (bash): venv activation, validate-on-save, status display - -tests/ # pytest — mirrors src/ structure -pyproject.toml # hatchling + hatch-vcs, ASTRA as git dep -``` - -### Engine/CLI dependency contract - -`lightcone.engine` must import nothing from `lightcone.cli` or `lightcone.eval`. This keeps the door open to carving `lightcone-engine` into its own PyPI dist when a future `lightcone-ui` needs it — no code changes required. - -## Development Commands - -```bash -uv sync --group dev # installs pytest, ruff, mypy into the uv env -uv run pytest -uv run ruff check src/ tests/ -uv run mypy src/ -``` - -A `justfile` is available for common tasks — run `just` to see all recipes: - -```bash -just test # run pytest -just lint # ruff + mypy -just docs # build the documentation site -just docs-serve # live preview at http://127.0.0.1:8000 -just install # uv sync --all-groups -``` - -## Documentation - -Maintainer documentation lives in `docs/` and is built with [Zensical](https://zensical.org). Configuration is in `zensical.toml`. - -``` -docs/ -├── index.md # Overview, repo structure, key invariants -├── architecture.md # Dagster integration, container mgmt, plugin system -├── cli/ # One page per lc command -├── api/ # One page per Python module -├── skills/ # Skill reference + authoring guide -├── telemetry/ # Langfuse hooks, session lifecycle, opt-out -├── hpc/ # SLURM, site registry, targets, container builds -└── contributing/ # Dev setup, adding backends/sites, testing -``` - -Dependencies are declared in `pyproject.toml` under `[dependency-groups].docs` and managed with `uv`: - -```bash -just docs-serve # syncs docs group then serves with live reload at http://127.0.0.1:8000 -just docs-strict # build with --strict (accepted flag, not yet enforced by zensical) -``` - -## Architecture & Data Flow - -``` -astra.yaml → build_definitions() → Dagster assets → ASTRAContainerRunner → results/{universe}/{output}/ - ↑ ↑ - ASTRAIOManager Docker / local / SLURM -``` - -- `build_definitions()` (`lightcone.engine.assets`) loads astra.yaml, creates one Dagster asset per output with a recipe -- Asset dependencies come from `recipe.inputs` — Dagster resolves execution order -- `ASTRAContainerRunner` (`lightcone.engine.runner`) dispatches to Docker, local subprocess, or SLURM based on target config -- Docker backend falls back to local execution on failure (with warning) -- SLURM backend generates sbatch scripts, submits via `sbatch`, polls via `sacct`/`squeue` - -## Key Invariants - -**Spec & execution:** -- `astra.yaml` is the single source of truth — all inputs, outputs, recipes, decisions, containers -- Output paths are always `results/{universe_id}/{output_id}/` — enforced by IO manager, no customization -- Container is a single string: image name (e.g., `python:3.9`) is pulled; file path (e.g., `Containerfile`) is built. No `container_build` dict — runtime detects via file existence. -- Container image tags are deterministic: SHA256(Containerfile + dependency files) → `lc-{name}-{hash}` -- Universe decision parameters are injected as CLI args: `--key value` passed to recipe commands -- Per-recipe container specs override analysis-level defaults - -**Config resolution (used everywhere):** -- Target: `--target` flag > `.lightcone/lightcone.yaml` > `~/.lightcone/config.yaml` > `"local"` -- Permission tier: `--permissions` flag > saved default in `~/.lightcone/config.yaml` > interactive prompt -- Most commands require `astra.yaml` in cwd; exceptions: `setup`, `target` - -**Plugin system:** -- Skills, agents, guides, hooks, and scripts are bundled in the wheel (`plugin/lightcone/` → `lightcone/cli/plugin/lightcone/`) -- `lc init [--tools T]` copies content into `./` for each selected harness; default harness is `claude` -- Skills, agents, and guides are installed for every harness; hooks/scripts are Claude Code only -- `lc update --sync [--tools T]` re-syncs skills/agents/guides (hooks are init-time only, not synced) -- Plugin source discovery: `lightcone.cli.plugin.get_plugin_source_dir` — bundled location first, then `plugin/lightcone/` at repo root -- Bash scripts must be chmod +x - -## CLI Patterns - -All commands use Click. Key patterns: -- `@main.command()` for top-level commands, `@main.group()` for subgroups (`target`) -- Target/config resolution is shared logic, not per-command -- `lc setup` auto-triggers if `~/.lightcone/config.yaml` doesn't exist when running other commands -- Three permission tiers: `yolo` (all allowed), `recommended` (workflow allowed), `minimal` (read-only) - -## Extending the Codebase - -| To... | Read | Key patterns | -|---|---|---| -| Add a CLI command | `src/lightcone/cli/commands.py` | `@main.command()`, config resolution, `click.echo` with Rich | -| Add an HPC site | `src/lightcone/engine/site_registry.py` | Add to `SITE_DEFAULTS` dict with hostname_patterns, node_types, qos_options | -| Add an execution backend | `src/lightcone/engine/runner.py` | Add `_run_{backend}()` method, update `execute()` dispatch | -| Add container features | `src/lightcone/engine/container.py` | `DEPENDENCY_FILES` tuple, `compute_image_tag()`, build/resolve functions | -| Create a skill | `plugin/lightcone/skills/` | SKILL.md with YAML frontmatter (`name`, `description`, `allowed-tools`) | -| Add a telemetry hook | `plugin/lightcone/hooks/` | Follow `langfuse_hook.py` pattern: read JSON payload, emit to Langfuse | -| Add a harness target | `src/lightcone/cli/harness.py` | Add to `HARNESS_REGISTRY` dataclass + `ALL_TOOL_IDS` tuple | - -## Test Patterns - -- CLI tests: `CliRunner().invoke(main, ["command", ...])` — check exit code, output, file side effects -- Asset tests: call `build_asset_definitions(spec, runner=mock_runner)` — verify keys, deps, metadata -- Runner tests: create runner with tmp project root, call `execute()` — verify exit code and metadata -- Common fixture: `_fake_config` monkeypatches `get_config_path` to prevent auto-setup wizard in tests -- Integration tests in `test_integration.py` and `test_cli_run.py` cover end-to-end flows - -## Conventions - -- Ruff for linting (E, F, I, N, W, UP), line length 100, target Python 3.11 -- mypy strict mode with `namespace_packages = true`, `explicit_package_bases = true` -- Status states: `"ok"` (materialized), `"pending"` (has recipe, not run), `"no_recipe"` (declared, no recipe) -- SLURM scripts/output stored in `results/.slurm/` -- Dagster instance storage at `results/.dagster/` (SQLite) -- Telemetry opt-out: `TRACE_TO_LANGFUSE=false` -- **Keep CLAUDE.md current** — when a PR changes CLI behaviour, key invariants, or repo structure, update the relevant section of CLAUDE.md in the same commit +> This file exists for Claude Code session compatibility. +> The canonical project context lives in **[AGENTS.md](AGENTS.md)** — read that file for architecture, commands, conventions, and invariants.