Skip to content

feat(registry): tag-based versioning with # ref syntax and CDN-bypass #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jrob5756 merged 2 commits into from
May 6, 2026
Merged

feat(registry): tag-based versioning with # ref syntax and CDN-bypass #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e4aedcb
feat(registry): tag-based versioning with # ref syntax and CDN-bypass
jrob5756 May 5, 2026
2defc4b
fix(registry): apply PR review feedback
jrob5756 May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,8 +231,9 @@ conductor registry add official myorg/conductor-workflows --default
conductor registry list official

# Run a workflow from the registry
conductor run qa-bot # latest from default registry
conductor run qa-bot@official@1.2.3 # specific version
conductor run qa-bot # latest from default registry
conductor run 'qa-bot@official#v1.2.3' # specific tag (quote the #)
conductor run 'qa-bot@official#main' # branch HEAD (re-resolved on fetch)
```

See [docs/design/registry.md](docs/design/registry.md) for the full design.
Expand Down
31 changes: 23 additions & 8 deletions docs/cli-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,12 +224,12 @@ conductor registry <subcommand> [OPTIONS]

| Subcommand | Description |
|------------|-------------|
| `list [NAME]` | List registries, or list workflows in a specific registry |
| `list [NAME]` | List configured registries, or list workflows in a specific registry. For GitHub registries, the per-registry listing also prints a "Latest tags:" footer with up to 5 newest tags. |
| `add <NAME> <SOURCE>` | Add a new registry (GitHub `owner/repo` or local path) |
| `remove <NAME>` | Remove a registry |
| `set-default <NAME>` | Set the default registry |
| `update [NAME]` | Update cached registry index (all or specific) |
| `show <NAME>` | Show registry details and status |
| `update [NAME]` | Refresh the cached index for one or all registries. For GitHub registries, the index is re-fetched via a SHA-pinned raw URL that bypasses Fastly's CDN, so updates always reflect the current state of the registry repo. |
| `show <NAME>` | Show details for a single configured registry: type, source, default status, and (for GitHub registries) a "Latest tags:" footer listing up to 5 newest tags discovered on the registry repo. Use `list <NAME>` to inspect the workflows it contains. |

### Options

Expand Down Expand Up @@ -267,19 +267,34 @@ conductor registry remove local

### Running Workflows from a Registry

Once a registry is configured, `conductor run` accepts short workflow names:
Once a registry is configured, `conductor run` accepts short workflow names
of the form `<workflow>[@<registry>][#<ref>]`. `@` selects the registry;
`#` selects a git ref (tag, branch, or commit SHA). Quote the reference in
shell commands so `#` isn't treated as a comment.

```bash
# Run from default registry (latest version)
# Run from default registry (latest tag, or default-branch HEAD if no tags)
conductor run qa-bot

# Run from a specific registry
# Run from a specific registry (latest)
conductor run qa-bot@official

# Run a specific version
conductor run qa-bot@official@1.2.3
# Pin a specific tag
conductor run 'qa-bot@official#v1.2.3'

# Pin the default-branch HEAD or any other branch
conductor run 'qa-bot@official#main'

# Pin a specific commit SHA
conductor run 'qa-bot@official#a1b2c3d'

# Pin a tag in the default registry (empty registry segment)
conductor run 'qa-bot@#v1.2.3'
```

Path-type registries do not support `#<ref>` and will reject any reference
that includes one.

See [design/registry.md](./design/registry.md) for the full design.

## Environment Variables
Expand Down
108 changes: 76 additions & 32 deletions docs/design/registry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,25 +92,41 @@ The `type` flag is optional; `add` infers `github` if `<source>` matches
### Reference syntax

```
<workflow>[@<registry>][@<version>]
<workflow>[@<registry>][#<ref>]
```

`@` separates the workflow name from the registry name. `#` separates the
ref (a git tag, branch name, or commit SHA) from the rest.

Resolution rules, in order:

1. If the argument exists as a file on disk, treat it as a local path.
2. Otherwise parse as a registry reference.
3. Missing `@<registry>` → use the configured default registry.
4. Missing `@<version>` → use `latest` (highest version listed in the
registry index).
4. An empty registry between `@` and `#` (e.g. `name@#ref`) is allowed and
means "use the default registry at this ref".
5. Missing `#<ref>` → use `latest`. `latest` resolves to the newest
semver-sorted git tag (with a leading `v` stripped for parsing). If the
registry repo has no tags, `latest` falls back to the default branch HEAD.
6. An empty ref after `#` (e.g. `name@reg#`) is a hard error.
7. Multiple `@` or multiple `#` in a single reference are hard errors.
8. Path-type registries do not support `#<ref>`. Passing
`name@local#anything` against a path registry is a hard error.

Note: `#` is significant to most shells. Quote registry references in
shell commands (`conductor run 'qa-bot@team#v1.2.3'`) and avoid spaces
around the `#`.

Examples:

```
conductor run ./my-workflow.yaml # local file (unchanged)
conductor run qa-bot # latest from default registry
conductor run qa-bot@team # latest from `team` registry
conductor run qa-bot@team@1.2.3 # exact version from `team`
conductor run qa-bot@@1.2.3 # exact version from default registry
conductor run 'qa-bot@team#v1.2.3' # tag v1.2.3 from `team`
conductor run 'qa-bot@#v1.2.3' # tag v1.2.3 from default registry
conductor run 'qa-bot@team#main' # default-branch HEAD of `team`
conductor run 'qa-bot@team#a1b2c3d' # specific commit SHA
```

### Registry index
Expand All @@ -122,35 +138,63 @@ workflows:
qa-bot:
description: "Simple Q&A workflow"
path: workflows/qa-bot.yaml # path relative to registry root
versions: ["1.0.0", "1.1.0", "2.0.0"]
code-review:
description: "Multi-agent code review"
path: workflows/code-review.yaml
versions: ["0.3.0"]
```

For GitHub registries, versions correspond to git tags on the registry repo.
For local registries, the maintainer maintains the version list directly.

The index is the single source of truth for what workflows exist and what
versions are available. Conductor does not auto-discover YAML files in a
registry — the maintainer curates the index.
The index is the single source of truth for what workflows exist and where
they live in the repo. Available versions are **not** listed in the index —
for GitHub registries they are auto-discovered from the registry repo's git
tags; for path registries no versioning exists. Conductor does not
auto-discover YAML files in a registry — the maintainer curates the index.

### Versioning

Versioning is automatic and tag-driven for GitHub registries:

- **Auto-discovery**: available versions are the registry repo's git tags,
fetched on demand via the GitHub API. Maintainers do not list versions in
`index.yaml`.
- **`latest` resolution**: `latest` resolves to the newest semver-sorted tag
(a leading `v` is stripped before parsing, so `v1.2.3` and `1.2.3` sort
identically). If the repo has no tags, `latest` falls back to the
default-branch HEAD.
- **Flexible refs**: any tag, branch, or commit SHA can be pinned via
`#<ref>`. Branch refs are re-resolved to their current commit SHA at
fetch time, so a branch ref always refers to the latest commit on that
branch when a fresh fetch is performed.
- **SHA-based caching**: workflows are cached by the resolved commit SHA
(`<cache>/<reg>/<workflow>/<sha[:12]>/`). When a branch advances, the
cache key changes automatically — no manual invalidation needed for the
next fresh fetch.
- **CDN bypass**: index fetches resolve the ref to a commit SHA via the
GitHub API, then download from
`raw.githubusercontent.com/<owner>/<repo>/<sha>/index.yaml`. The unique
per-SHA URL bypasses Fastly's CDN cache, so you always see the current
index for a given ref without needing a `--force` flag.
- **Path registries**: do not support refs at all. Local registries are
always read directly from disk.

### Caching

Fetched workflows are cached at:

```
~/.conductor/cache/registries/<registry>/<workflow>/<version>/
~/.conductor/cache/registries/<registry>/<workflow>/<sha[:12]>/
```

- Cache is keyed by `(registry, workflow, version)`.
- Explicit versions are immutable: once cached, never re-fetched.
- `latest` is re-resolved on `conductor registry update`. Each resolved
version is cached in its own directory.
- Cache is keyed by `(registry, workflow, resolved-commit-sha)`.
- A given commit SHA is immutable, so cached entries are never re-fetched
for the same SHA. Branch refs re-resolve to a (possibly new) SHA on each
fresh fetch, which transparently invalidates the cache.
- Workflow files are first downloaded into a temp directory and then renamed
atomically into the final cache path, so a partial fetch never leaves a
half-populated entry visible to other commands.
- Index files are cached separately at
`~/.conductor/cache/registries/<registry>/index.<yaml|json>` and refreshed
on `update`.
on `update`. Index fetches always go through a SHA-pinned raw URL to
bypass the CDN.

This produces a stable on-disk path for every registry-fetched workflow,
which is required by:
Expand All @@ -168,17 +212,17 @@ operation. Registry maintainers should keep a workflow and its assets in the
same directory.

For GitHub registries, sibling fetch uses the Git Trees API to enumerate the
directory and `raw.githubusercontent.com` to download files at the pinned
ref (tag).
directory and SHA-pinned `raw.githubusercontent.com` URLs to download files
at the resolved commit SHA.

### Run / resume / validate

These commands accept either a local path or a registry reference:

```
conductor run qa-bot@team@1.2.3 --input question="What is X?"
conductor resume qa-bot@team@1.2.3
conductor validate qa-bot@team@1.2.3
conductor run 'qa-bot@team#v1.2.3' --input question="What is X?"
conductor resume 'qa-bot@team#v1.2.3'
conductor validate 'qa-bot@team#v1.2.3'
```

The resolver runs first, returns a concrete `Path` to the cached file, and
Expand All @@ -203,10 +247,10 @@ recent, undocumented in many places, and supersede-able by a 5-line
| Module | Responsibility |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `config.py` | Pydantic models for `registries.toml`. Atomic load/save. Handles missing-file case (returns empty config). |
| `resolver.py` | Parses `name[@registry][@version]`. Decides file-vs-ref. Returns a `ResolvedRef` with registry name, workflow name, version, and the registry config. |
| `index.py` | Loads and parses `index.yaml`/`index.json`. Validates structure. Resolves `latest` to a concrete version. Backed by either the local FS or `github.py`. |
| `cache.py` | Manages `~/.conductor/cache/registries/`. `get_or_fetch(ref) -> Path`. Idempotent. Fetches sibling files. Knows when to refetch (`latest`) vs. reuse (explicit version). |
| `github.py` | Public-only GitHub helpers: fetch a file at a ref via raw URL, list tags via the REST API for `latest`, list directory contents via Git Trees API for sibling enumeration. Uses `httpx`, no auth. |
| `resolver.py` | Parses `name[@registry][#ref]`. Decides file-vs-ref. Returns a `ResolvedRef` with registry name, workflow name, ref, and the registry config. Rejects multiple `@`/`#`, empty `#`, and `#ref` against path registries. |
| `index.py` | Loads and parses `index.yaml`/`index.json`. Validates structure. Resolves `latest` to a concrete tag (or default-branch HEAD if no tags). Backed by either the local FS or `github.py`. |
| `cache.py` | Manages `~/.conductor/cache/registries/`. `get_or_fetch(ref) -> Path`. Idempotent. Fetches sibling files. Cache is keyed by resolved commit SHA; writes are staged in a temp dir and renamed atomically. |
| `github.py` | Public-only GitHub helpers: resolve a ref to a commit SHA via the GitHub API, fetch files at a SHA via SHA-pinned raw URLs (bypassing the CDN), list tags via the REST API for `latest`, list directory contents via Git Trees API for sibling enumeration. Uses `httpx`, no auth. |

### CLI: `src/conductor/cli/registry.py`

Expand Down Expand Up @@ -244,10 +288,10 @@ file of the same name.
| Decision | Choice | Why |
| ----------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Named registries vs always-inline source | Named, configured once | Mirrors npm/cargo. Short refs in `run` commands. Default registry makes the common case zero-config. |
| Versioning | Explicit, npm-style | Reproducibility. Lockfile-friendly later. Matches existing GitHub tag conventions. |
| Local-registry layout | Directory + `index.yaml` | Consistent with GitHub registries. Maintainer controls what's exposed and at what versions. |
| Caching strategy | Local cache, refresh on `registry update` or new version | Avoids per-run network. Stable on-disk paths needed for `!file` and checkpoints. |
| Reference syntax | `name@registry@version` | Visually unambiguous. `@` parses cleanly. Supports either-or-both omissions. |
| Versioning | Auto-discovered from git tags; pin any tag/branch/SHA via `#ref` | Reproducibility without forcing maintainers to maintain a parallel version list. `latest` follows newest semver tag, falling back to default-branch HEAD. Branches and SHAs are first-class refs. |
| Local-registry layout | Directory + `index.yaml` | Consistent with GitHub registries. Maintainer controls what's exposed. Local registries do not support refs. |
| Caching strategy | Local cache keyed by resolved commit SHA, atomic writes | Avoids per-run network. SHA-based keys make branch refs self-invalidate on a fresh fetch. SHA-pinned raw URLs bypass the CDN, so no `--force` flag is needed. |
| Reference syntax | `name@registry#ref` | Visually unambiguous: `@` selects the registry, `#` selects a git ref (tag, branch, or SHA). Both segments are independently optional. |
| Publish / publish validation | Dropped | Distribution is `git push` + tag. Validation belongs in user CI, not the CLI. |
| Authenticated/private registries | Out of scope v1 | Public raw URLs cover the common case. Token support can come later via a registry config field. |
| SemVer ranges | Out of scope v1 | Adds resolver complexity for marginal benefit until ecosystems exist. |
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ dependencies = [
"uvicorn>=0.30.0",
"websockets>=12.0",
"httpx>=0.27.0",
"packaging>=21.0",
]

[project.urls]
Expand Down
12 changes: 6 additions & 6 deletions src/conductor/cli/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ def run(
registry_name=ref.registry_name,
registry_entry=ref.registry_entry,
workflow_name=ref.workflow,
version=ref.version,
ref=ref.ref,
)
except RegistryError as e:
print_error(e)
Expand Down Expand Up @@ -518,7 +518,7 @@ def validate(
registry_name=ref.registry_name,
registry_entry=ref.registry_entry,
workflow_name=ref.workflow,
version=ref.version,
ref=ref.ref,
)
except RegistryError as e:
print_error(e)
Expand Down Expand Up @@ -577,7 +577,7 @@ def show(
registry_name=ref.registry_name,
registry_entry=ref.registry_entry,
workflow_name=ref.workflow,
version=ref.version,
ref=ref.ref,
)
except RegistryError as e:
print_error(e)
Expand All @@ -600,8 +600,8 @@ def show(

if ref.kind == "registry":
output_console.print(f"[bold]Registry:[/bold] {ref.registry_name}")
if ref.version:
output_console.print(f"[bold]Version:[/bold] {ref.version}")
if ref.ref:
output_console.print(f"[bold]Version:[/bold] {ref.ref}")

from rich.table import Table

Expand Down Expand Up @@ -773,7 +773,7 @@ def resume(
registry_name=ref.registry_name,
registry_entry=ref.registry_entry,
workflow_name=ref.workflow,
version=ref.version,
ref=ref.ref,
)
except RegistryError as e:
print_error(e)
Expand Down
Loading
Loading