From 42a65b7bd9af04ad079b2f0176924d221e208ff6 Mon Sep 17 00:00:00 2001 From: Chris Logue Linux <17660199+omniscoder@users.noreply.github.com> Date: Mon, 12 Jan 2026 19:34:31 -0500 Subject: [PATCH] docs: refresh core kit docs --- CONTRIBUTING.md | 37 ++++++++++++ README.md | 109 ++++++++++++++++++++++++++++++---- sdk/python/README.md | 10 +++- spec/v1/conformance/README.md | 6 +- spec/v1/jobspec.md | 9 ++- 5 files changed, 152 insertions(+), 19 deletions(-) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..bde5654 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,37 @@ +# Contributing to OGN Core Kit + +Thanks for helping improve the OGN Core Kit adoption surface (runner, contracts, SDKs, workflow glue). + +## Quick dev setup (Python) + +```bash +python -m pip install -e sdk/python[dev] +pytest -q +``` + +## Contract / conformance changes + +Job Spec v1 is **additive-only**. Do not make optional fields required or change meanings in-place. + +When editing anything under `spec/v1/`: + +```bash +python tools/validate_conformance.py +``` + +If you add or modify fixtures, keep them minimal and deterministic. + +## Public repo hygiene + +Before opening a PR: + +```bash +./public_audit.sh +git ls-files -s | awk '$1==120000{print}' +``` + +## Pull requests + +- Keep PRs small and contract-focused. +- If you change runner behavior, add or update a focused test in `tests/`. +- Avoid adding proprietary engine references, hard-coded machine paths, or secrets. diff --git a/README.md b/README.md index 712d7f9..d653c41 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,125 @@ # OGN Core Kit -OGN Core Kit is the open adoption surface for OGN: +OGN Core Kit is the open adoption surface for Omnis Genome Nexus (OGN). -- Job Spec JSON v1 contract (docs + fixtures) -- `ogn-runner`: a stable worker entrypoint for executing an OGN engine and uploading artifacts -- SDKs and workflow adapters (Nextflow, etc.) +It contains: -This repository does **not** include the proprietary OGN engine implementation or GPU kernels. -The engine is expected to be provided separately as a binary (e.g. `ogn_run`) or container image. +- Job Spec JSON v1 contract (`spec/v1/jobspec.md` + `spec/v1/jobspec.schema.json` + fixtures) +- `ogn-runner`: stable worker entrypoint that executes an OGN engine and uploads artifacts +- Python package `ogn-sdk` (CLI `ogn`, SDK `ogn_sdk`, runner `ogn_runner`) +- Workflow adapters (e.g. `pipelines/ogn_variant.nf`) -## Quickstart (local runner) +It does **not** include the proprietary OGN engine implementation, GPU kernels, or engine container images. The engine +must be provided separately as a binary (e.g. `ogn_run`) or container image. -1) Install the Python package: +## Install + +### Python (CLI + runner + SDK) + +Requires Python 3.9+. + +From PyPI: ```bash -pip install ./sdk/python +python -m pip install ogn-sdk ``` -2) Point the runner at your engine binary (optional; defaults to `ogn_run` on PATH): +From source (dev / editable): + +```bash +python -m pip install -e sdk/python[dev] +``` + +This installs: + +- `ogn` (CLI wrapper; requires access to an engine binary such as `ogn_variant_runner`) +- `ogn-runner` (worker entrypoint for Job Spec v1) +- `ogn_sdk` / `ogn_runner` Python modules + +### Rust (SDK) + +The Rust crate lives at `sdk/rust/ogn-sdk`. + +- If/when published, install via crates.io. +- Until then, consume it via a path dependency in a clone: + +```toml +ogn-sdk = { path = "sdk/rust/ogn-sdk" } +``` + +## Quickstart: `ogn-runner` (Job Spec v1) + +`ogn-runner` accepts a jobspec path or `-` for stdin (the gateway worker contract is `ogn-runner -`). + +Engine selection: ```bash export OGN_ENGINE_BIN=/path/to/ogn_run +# or: ogn-runner -E /path/to/ogn_run job_spec.json ``` -3) Run a Job Spec v1: +Run from a file: ```bash ogn-runner job_spec.json ``` +Run from stdin: + +```bash +cat job_spec.json | ogn-runner - +``` + +### Upload semantics + +For each `outputs.*` artifact: + +- If `put_url` is present, `ogn-runner` uploads via HTTP PUT to the presigned URL. +- Otherwise it uploads to `uri` directly when supported: + - `s3://...` via `aws s3 cp` (AWS CLI required) + - `file://...` or a plain path via a local copy + +If an output is marked `optional: true`, upload failures are non-fatal. + ## Contracts -The canonical pilot contract lives in: +Canonical artifacts: - `spec/v1/jobspec.md` +- `spec/v1/jobspec.schema.json` - `spec/v1/conformance/` +Schema-level validation: + +```bash +python tools/validate_conformance.py +``` + Unknown fields are ignored (additive-only). Breaking changes require a new schema version (v2+). +## Development + +Run tests: + +```bash +python -m pip install -e sdk/python[dev] +pytest -q +``` + +Public repo hygiene: + +```bash +./public_audit.sh +``` + +## License + +Apache 2.0; see `LICENSE` and `NOTICE`. + +## Security + +See `SECURITY.md`. + +## Contributing + +See `CONTRIBUTING.md`. diff --git a/sdk/python/README.md b/sdk/python/README.md index ded5aa5..27fb4ed 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -1,6 +1,14 @@ # ogn-sdk (Python) -`ogn-sdk` is a tiny convenience layer around the `ogn_run` binary. Install it with: +`ogn-sdk` is a small set of Python wrappers and CLIs for integrating with the OGN engine. + +It includes: + +- `ogn_sdk`: a thin convenience layer around the `ogn_run` binary +- `ogn` CLI: local orchestration helpers (doctor/setup/run/results) +- `ogn-runner`: the stable Job Spec v1 worker entrypoint (`ogn-runner -`) + +Install it with: ```bash python3 -m pip install ogn-sdk diff --git a/spec/v1/conformance/README.md b/spec/v1/conformance/README.md index e74499d..0a4203c 100644 --- a/spec/v1/conformance/README.md +++ b/spec/v1/conformance/README.md @@ -1,6 +1,6 @@ # Job Spec v1 conformance fixtures -These fixtures exist to keep `schema_version: "v1"` stable and enforceable. +These fixtures exist to keep `schema_version: "v1"` stable and enforceable across the OGN ecosystem. Each `*.json` file is a single conformance case with this shape: @@ -16,5 +16,5 @@ Each `*.json` file is a single conformance case with this shape: } ``` -The runner lives in `internal/jobspec/conformance_test.go`. - +In this repo, `tools/validate_conformance.py` performs schema-level validation for `kind: "jobspec"` cases. +Timeline/state-machine semantics (`kind: "timeline"`) are enforced by the control plane implementation. diff --git a/spec/v1/jobspec.md b/spec/v1/jobspec.md index 0a16394..62df00e 100644 --- a/spec/v1/jobspec.md +++ b/spec/v1/jobspec.md @@ -1,6 +1,7 @@ # Job Spec JSON v1 (frozen pilot contract) -Job Spec v1 is the payload published on `ogn.run.submit.v1` (gateway → worker). It is validated by `internal/jobspec/v1.go` and used end-to-end in the pilot. +Job Spec v1 is the payload produced by the OGN control plane (gateway) and consumed by workers (typically via +`ogn-runner -`). It is intentionally strict and boring: this contract is the integration moat. ## Versioning policy @@ -14,7 +15,10 @@ Job Spec v1 is the payload published on `ogn.run.submit.v1` (gateway → worker) - You MUST NOT remove fields. - Any breaking change requires a new `schema_version` (v2+) and continued support for v1 during the pilot. -Conformance fixtures live in `spec/v1/conformance/` and are enforced by `internal/jobspec/conformance_test.go`. +Conformance fixtures live in `spec/v1/conformance/`. + +- This repo includes schema-level checks via `tools/validate_conformance.py`. +- The control plane (gateway) is responsible for enforcing additional semantic rules (e.g. timeline/state machines). ## Schema (high level) @@ -44,4 +48,3 @@ Optional fields: - `outputs.*.uri` is the **stable artifact identity** (e.g. `s3://bucket/key`). - `outputs.*.put_url` is a short-lived presigned URL used only for upload. - Finalization events MUST reference stable URIs (never presigned URLs). -