diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/README.md b/conformance-tests/2023-09/WRAP_ACTIONS/README.md new file mode 100644 index 0000000..b52ea33 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/README.md @@ -0,0 +1,179 @@ +# WRAP_ACTIONS Conformance Tests + +Conformance tests for the `WRAP_ACTIONS` extension defined in +[RFC 0008 — Environment Wrap Actions](../../../rfcs/0008-environment-wrap-actions.md). + +## What this extension does + +`WRAP_ACTIONS` adds three new fields to ``: + +```yaml + ::= the object: + onEnter: # optional + onWrapEnter: # NEW — optional + onWrapTaskRun: # NEW — optional + onWrapExit: # NEW — optional + onExit: # optional +``` + +When an active environment defines the wrap hooks, the runtime runs each +hook *instead of* the corresponding lifecycle action of every inner +environment and task. Each hook receives the wrapped action's fields via +the `WrappedAction.*` template variables: + +| Variable | Type | Description | +|-------------------------------|----------------|---------------------------------------------------------------------| +| `WrappedAction.Command` | `string` | The `command` from the wrapped action | +| `WrappedAction.Args` | `list[string]` | The `args` from the wrapped action | +| `WrappedAction.Environment` | `list[string]` | `openjd_env`-defined variables as `["KEY=value", ...]` | +| `WrappedAction.Timeout` | `int` | Timeout of the wrapped action in seconds, or `0` if none | +| `Env.Wrapped.Name` | `string` | Name of the inner environment (only in `onWrapEnter`/`onWrapExit`) | + +## RFC rules these tests verify + +- **All-or-nothing**: defining any wrap hook requires defining all three + (`onWrapEnter`, `onWrapTaskRun`, `onWrapExit`). +- **Single wrap layer**: at most one environment in the session stack may + define wrap hooks. +- **Variable scope**: `WrappedAction.*` may be referenced only inside the + three wrap hooks; `Env.Wrapped.*` only inside `onWrapEnter`/`onWrapExit`. +- **Extension gating**: wrap hooks are only valid when `WRAP_ACTIONS` is + declared in `extensions:`. +- **Interception**: the wrap hooks run instead of the corresponding + lifecycle actions of inner environments and tasks. +- **Safe forwarding**: with `repr_sh()` from the EXPR extension, arbitrary + user input (quotes, metacharacters, globs, Unicode) reaches the wrapped + process verbatim. + +## How these tests are designed + +The RFC motivates the wrap hooks with Docker/Apptainer use cases, but +those containers aren't required to *test* the mechanism. These tests use +trivial wrap scripts — typically `echo`, `sh -c`, or `printf` — that +observe the injected `WrappedAction.*` variables and write sentinel +markers to stdout. + +A test passes if: +- Each wrap hook runs in place of its corresponding lifecycle action + (verified by sentinels the original action would never emit). +- The injected `WrappedAction.*` variables carry the exact bytes of the + wrapped action. +- The original action does **not** execute. + +## Layout + +``` +WRAP_ACTIONS/ +├── README.md (this file) +├── env_templates/ # Env template validation tests +│ ├── 4.2--minimal-wrap-env-template.yaml +│ ├── 4.2--wrap-only-environment.yaml +│ ├── 4.2--wrap-with-timeout.yaml +│ ├── 4.2--empty-actions-no-wrap-no-enter-no-exit.invalid.yaml +│ ├── 4.3--wrap-only-onwrap-task-run.invalid.yaml +│ ├── 4.3--wrap-missing-onwrap-exit.invalid.yaml +│ └── 4.3--wrappedaction-outside-wrap-hook.invalid.yaml +└── jobs/ # End-to-end execution tests + ├── wrap-intercepts-simple-echo.test.yaml + ├── wrap-original-onrun-does-not-run.test.yaml + ├── wrap-enter-and-exit-three-hooks.test.yaml + ├── wrap-enter-intercepts-inner-on-enter.test.yaml + ├── wrap-exit-intercepts-inner-on-exit.test.yaml + ├── wrap-task-command-injected.test.yaml + ├── wrap-task-args-preserved.test.yaml + ├── wrap-task-environment-injected.test.yaml + ├── wrap-task-action-timeout-injected.test.yaml + ├── wrap-preserves-shell-metacharacters.test.yaml + ├── wrap-preserves-nested-quotes.test.yaml + ├── wrap-preserves-unicode-cjk.test.yaml + ├── wrap-preserves-empty-arg.test.yaml + ├── wrap-glob-characters-literal.test.yaml + ├── wrap-path-traversal-literal.test.yaml + ├── wrap-outer-env-without-wrap-ignored.test.yaml + ├── wrap-runs-for-every-task.test.yaml + ├── wrap-ignores-wrapped-action-vars.test.yaml + ├── wrap-without-extension-fails.invalid.test.yaml + ├── wrap-multiple-wrap-envs-rejected.invalid.test.yaml + └── wrap-partial-hooks-rejected.invalid.test.yaml +``` + +All execution tests use POSIX shell commands (`sh`, `echo`, `printf`), +so they are gated to `runOn: [posix]`. + +## Test matrix — RFC §"Recommended test cases for implementation" + +The RFC's container-implementation companion lists scenarios that any +implementation should validate. This matrix maps the security-related +ones to the job bundles in this directory: + +| # | Scenario | Test file | +|---|-----------------------------------|-------------------------------------------------| +| 1 | Nested quoting | `wrap-preserves-nested-quotes.test.yaml` | +| 2 | Shell metacharacters | `wrap-preserves-shell-metacharacters.test.yaml` | +| 3 | Path traversal | `wrap-path-traversal-literal.test.yaml` | +| 4 | Shell globbing | `wrap-glob-characters-literal.test.yaml` | +| 5 | Unicode paths | `wrap-preserves-unicode-cjk.test.yaml` | +| 6 | Empty / whitespace-only arguments | `wrap-preserves-empty-arg.test.yaml` | +| 7 | Newlines in arguments | Rejected by the 2023-09 `ArgString` regex — | +| | | no dedicated conformance test needed. | +| 8 | Near-limit command length | Covered by unit tests in the implementation. | +| | | Conformance runners vary in stdout buffer | +| | | limits, so this is left to implementation tests | +| | | rather than conformance. | + +## Additional semantics tested + +- **Three-hook interception**: each hook runs in place of its + corresponding lifecycle action + (`wrap-enter-intercepts-inner-on-enter`, + `wrap-exit-intercepts-inner-on-exit`, + `wrap-enter-and-exit-three-hooks`). +- **Variable injection**: `WrappedAction.Command/Args/Environment/Timeout` + reach the wrap script with the right values. +- **Outer-non-wrap-env coexistence**: a non-wrapping environment can + appear in the stack alongside the single wrapping environment + (`wrap-outer-env-without-wrap-ignored`). +- **Per-task repeat**: variables re-inject cleanly per task + (`wrap-runs-for-every-task`). +- **Invalid templates**: + - `wrap-without-extension-fails`: hooks require the extension. + - `wrap-multiple-wrap-envs-rejected`: at most one wrap env per session. + - `wrap-partial-hooks-rejected`: all-or-nothing rule (job side). + - `4.3--wrap-only-onwrap-task-run`: all-or-nothing rule (env side). + - `4.3--wrap-missing-onwrap-exit`: all-or-nothing rule (env side). + - `4.3--wrappedaction-outside-wrap-hook`: variable scope rule. + +## Running the tests + +From the repo root: + +```bash +# Run just the WRAP_ACTIONS extension tests +uv run conformance-tests/run_openjd_cli_tests.py 2023-09/WRAP_ACTIONS + +# Run only the job execution tests +uv run conformance-tests/run_openjd_cli_tests.py 2023-09/WRAP_ACTIONS/jobs + +# Pattern-match a single scenario +uv run conformance-tests/run_openjd_cli_tests.py '*wrap*unicode*' +``` + +## Writing your own runner + +See the top-level [conformance README](../../README.md) for the standard +runner contract. The behavior specific to `WRAP_ACTIONS`: + +1. When the env template declares `extensions: [WRAP_ACTIONS]`, the + runner must accept `onWrapEnter`, `onWrapTaskRun`, and `onWrapExit` + under `environment.script.actions`, and must reject the template if + any one of the three is defined without all three. +2. When an environment with wrap hooks is entered, every subsequent + inner-environment lifecycle action and task `onRun` must execute the + corresponding wrap hook in place of the original action, with + `WrappedAction.*` variables populated from the wrapped action's fields. +3. When two or more environments in the session stack define any wrap + hook, the runner must reject the session before entering any + environment. +4. When wrap-hook fields are used without `WRAP_ACTIONS` in the + `extensions` list, the runner must reject the template at validation + time. diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--empty-actions-no-wrap-no-enter-no-exit.invalid.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--empty-actions-no-wrap-no-enter-no-exit.invalid.yaml new file mode 100644 index 0000000..a070251 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--empty-actions-no-wrap-no-enter-no-exit.invalid.yaml @@ -0,0 +1,9 @@ +# An environment whose `actions` block is empty is invalid: at least one of +# onEnter, onWrapEnter, onWrapTaskRun, onWrapExit, or onExit must be defined. +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +environment: + name: EmptyActions + script: + actions: {} diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--minimal-wrap-env-template.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--minimal-wrap-env-template.yaml new file mode 100644 index 0000000..7730678 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--minimal-wrap-env-template.yaml @@ -0,0 +1,23 @@ +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +- EXPR +environment: + name: MinimalWrapEnv + script: + actions: + onEnter: + command: echo + args: ["entered"] + onWrapEnter: + command: echo + args: ["wrap-enter", "{{Env.Wrapped.Name}}"] + onWrapTaskRun: + command: echo + args: ["wrap-task", "{{WrappedAction.Command}}"] + onWrapExit: + command: echo + args: ["wrap-exit", "{{Env.Wrapped.Name}}"] + onExit: + command: echo + args: ["exited"] diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-only-environment.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-only-environment.yaml new file mode 100644 index 0000000..c872bf6 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-only-environment.yaml @@ -0,0 +1,18 @@ +# RFC 0008: the three wrap hooks alone (without onEnter/onExit) are sufficient. +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +- EXPR +environment: + name: WrapOnlyEnv + script: + actions: + onWrapEnter: + command: echo + args: ["{{Env.Wrapped.Name}}"] + onWrapTaskRun: + command: echo + args: ["{{WrappedAction.Command}}"] + onWrapExit: + command: echo + args: ["{{Env.Wrapped.Name}}"] diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-with-timeout.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-with-timeout.yaml new file mode 100644 index 0000000..fedecda --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.2--wrap-with-timeout.yaml @@ -0,0 +1,22 @@ +# RFC 0008: each wrap hook supports the same fields as any other Action, +# including timeout. The all-or-nothing rule requires all three hooks. +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +- EXPR +environment: + name: WrapWithTimeout + script: + actions: + onWrapEnter: + command: echo + args: ["{{Env.Wrapped.Name}}"] + timeout: 60 + onWrapTaskRun: + command: echo + args: ["{{WrappedAction.Command}}"] + timeout: 300 + onWrapExit: + command: echo + args: ["{{Env.Wrapped.Name}}"] + timeout: 60 diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-missing-onwrap-exit.invalid.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-missing-onwrap-exit.invalid.yaml new file mode 100644 index 0000000..aac2b8f --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-missing-onwrap-exit.invalid.yaml @@ -0,0 +1,16 @@ +# RFC 0008 all-or-nothing rule: defining onWrapEnter and onWrapTaskRun +# without onWrapExit is invalid. +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +- EXPR +environment: + name: MissingWrapExit + script: + actions: + onWrapEnter: + command: echo + args: ["{{Env.Wrapped.Name}}"] + onWrapTaskRun: + command: echo + args: ["{{WrappedAction.Command}}"] diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-only-onwrap-task-run.invalid.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-only-onwrap-task-run.invalid.yaml new file mode 100644 index 0000000..2ce0d38 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/env_templates/4.3--wrap-only-onwrap-task-run.invalid.yaml @@ -0,0 +1,14 @@ +# RFC 0008 all-or-nothing rule: an environment that defines any wrap hook +# must define all three (onWrapEnter, onWrapTaskRun, onWrapExit). Defining +# only onWrapTaskRun is invalid. +specificationVersion: environment-2023-09 +extensions: +- WRAP_ACTIONS +- EXPR +environment: + name: PartialWrap + script: + actions: + onWrapTaskRun: + command: echo + args: ["{{WrappedAction.Command}}"] diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-empty-environment-list.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-empty-environment-list.test.yaml new file mode 100644 index 0000000..652a11f --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-empty-environment-list.test.yaml @@ -0,0 +1,54 @@ +# RFC 0008: WrappedAction.Environment is an empty list when no openjd_env +# was emitted in the session. Verifies the empty case doesn't trip the +# wrap script's iteration logic. +template: + specificationVersion: jobtemplate-2023-09 + name: WrappedActionEnvironmentEmpty + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["placeholder"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # Print env count + a sentinel so we can confirm empty-list + # handling. With no openjd_env emissions, the loop body should + # not execute at all and only the sentinel appears. + - | + for e in {{ repr_sh(WrappedAction.Environment) }}; do + echo ENV_PRESENT=$e + done + echo ENV_DUMP_COMPLETE + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - ENV_DUMP_COMPLETE + forbidden: + - ENV_PRESENT= diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-and-exit-three-hooks.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-and-exit-three-hooks.test.yaml new file mode 100644 index 0000000..98eb0fb --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-and-exit-three-hooks.test.yaml @@ -0,0 +1,62 @@ +# RFC 0008: a single wrap environment declares all three hooks +# (onWrapEnter, onWrapTaskRun, onWrapExit). This test validates that +# all three intercept correctly in one session, with the inner env's +# onEnter/onExit replaced by the wrap scripts. +template: + specificationVersion: jobtemplate-2023-09 + name: ThreeHooks + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: echo + args: ["INNER_ENTER_NORMAL"] + onExit: + command: echo + args: ["INNER_EXIT_NORMAL"] + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["ORIGINAL_TASK"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "echo WRAP_ENTER_NAME={{Env.Wrapped.Name}}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo WRAP_TASK_CMD={{WrappedAction.Command}}" + onWrapExit: + command: sh + args: + - "-c" + - "echo WRAP_EXIT_NAME={{Env.Wrapped.Name}}" + +runOn: +- posix + +expected: + output: + - WRAP_ENTER_NAME=InnerEnv + - WRAP_TASK_CMD=echo + - WRAP_EXIT_NAME=InnerEnv + forbidden: + - INNER_ENTER_NORMAL + - INNER_EXIT_NORMAL + - ORIGINAL_TASK diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-intercepts-inner-on-enter.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-intercepts-inner-on-enter.test.yaml new file mode 100644 index 0000000..61b1ef4 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-intercepts-inner-on-enter.test.yaml @@ -0,0 +1,58 @@ +# RFC 0008: an outer environment's onWrapEnter must run instead of an +# inner environment's onEnter, with `Env.Wrapped.Name` resolving to the +# inner environment's name. The wrap script writes a marker line that +# the inner env's onEnter would never produce. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapEnterIntercepts + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: echo + args: ["INNER_ON_ENTER_NORMAL"] + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK_RAN"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "echo WRAP_ENTER_FOR={{Env.Wrapped.Name}}" + # All-or-nothing rule: the other two hooks must be defined. + # They forward the wrapped action verbatim via repr_sh. + onWrapTaskRun: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - WRAP_ENTER_FOR=InnerEnv + - TASK_RAN + forbidden: + - INNER_ON_ENTER_NORMAL diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-receives-wrapped-action.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-receives-wrapped-action.test.yaml new file mode 100644 index 0000000..3f0a553 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-enter-receives-wrapped-action.test.yaml @@ -0,0 +1,60 @@ +# RFC 0008: WrappedAction.* is in scope inside onWrapEnter, not just +# onWrapTaskRun. The wrap hook receives the inner env's onEnter command, +# args, and timeout via the same WrappedAction namespace. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapEnterReceivesWrappedAction + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: inner-enter-cmd + args: + - "--flag" + - "value" + timeout: 99 + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "echo NAME={{Env.Wrapped.Name}}; echo CMD={{WrappedAction.Command}}; echo TIMEOUT={{WrappedAction.Timeout}}; printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - NAME=InnerEnv + - CMD=inner-enter-cmd + - TIMEOUT=99 + - "ARG=--flag" + - ARG=value + - TASK diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-env-own-lifecycle-not-wrapped.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-env-own-lifecycle-not-wrapped.test.yaml new file mode 100644 index 0000000..4d9e159 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-env-own-lifecycle-not-wrapped.test.yaml @@ -0,0 +1,55 @@ +# RFC 0008: a wrapping environment's own onEnter and onExit are NEVER +# wrapped by its own wrap hooks. The wrap-enter / wrap-exit must not +# fire for the wrapping env itself, only for its inner environments. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapEnvOwnLifecycleNotWrapped + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK_RAN"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + # Distinct sentinels so we can verify these run NORMALLY (not via + # the wrap hooks below). + onEnter: + command: echo + args: ["WRAP_ENV_OWN_ENTER"] + onExit: + command: echo + args: ["WRAP_ENV_OWN_EXIT"] + # If these fire for the wrapping env's own onEnter/onExit, the + # forbidden sentinels below would appear. + onWrapEnter: + command: echo + args: ["FORBIDDEN_WRAP_ENTER_FIRED"] + onWrapTaskRun: + command: echo + args: ["WRAP_TASK_FIRED"] + onWrapExit: + command: echo + args: ["FORBIDDEN_WRAP_EXIT_FIRED"] + +runOn: +- posix + +expected: + output: + - WRAP_ENV_OWN_ENTER + - WRAP_ENV_OWN_EXIT + - WRAP_TASK_FIRED + forbidden: + - FORBIDDEN_WRAP_ENTER_FIRED + - FORBIDDEN_WRAP_EXIT_FIRED + - TASK_RAN diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-intercepts-inner-on-exit.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-intercepts-inner-on-exit.test.yaml new file mode 100644 index 0000000..a70a8c0 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-intercepts-inner-on-exit.test.yaml @@ -0,0 +1,60 @@ +# RFC 0008: an outer environment's onWrapExit must run instead of an +# inner environment's onExit, with `Env.Wrapped.Name` resolving to the +# inner environment's name. The wrap script writes a marker that the +# inner env's onExit would never produce. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapExitIntercepts + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: echo + args: ["inner-enter"] + onExit: + command: echo + args: ["INNER_ON_EXIT_NORMAL"] + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK_RAN"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + # All-or-nothing rule: the other two hooks forward verbatim. + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "echo WRAP_EXIT_FOR={{Env.Wrapped.Name}}" + +runOn: +- posix + +expected: + output: + - WRAP_EXIT_FOR=InnerEnv + - TASK_RAN + forbidden: + - INNER_ON_EXIT_NORMAL diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-receives-wrapped-action.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-receives-wrapped-action.test.yaml new file mode 100644 index 0000000..dbb314e --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-receives-wrapped-action.test.yaml @@ -0,0 +1,59 @@ +# RFC 0008: WrappedAction.* is in scope inside onWrapExit. Verifies the +# inner env's onExit command and args are surfaced via WrappedAction.*. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapExitReceivesWrappedAction + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: echo + args: ["inner-enter"] + onExit: + command: inner-exit-cmd + args: + - "--cleanup" + - "/tmp/work" + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "echo NAME={{Env.Wrapped.Name}}; echo CMD={{WrappedAction.Command}}; printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - NAME=InnerEnv + - CMD=inner-exit-cmd + - "ARG=--cleanup" + - ARG=/tmp/work diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-status-propagates.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-status-propagates.test.yaml new file mode 100644 index 0000000..6161b01 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-exit-status-propagates.test.yaml @@ -0,0 +1,56 @@ +# RFC 0008 Failure semantics: wrap scripts MUST propagate the wrapped +# process's exit status as their own. When the wrapped command exits +# non-zero, the wrap action — and therefore the task — must fail. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapExitStatusPropagates + steps: + - name: Step1 + script: + actions: + onRun: + # exit 7: a workload failure. The wrap script forwards via + # `exec`, so the wrap script's exit code is the wrapped process's. + command: sh + args: + - "-c" + - "echo TASK_BODY_RAN; exit 7" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo WRAP_BEFORE; exec {{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +# This test expects the runner to surface the wrapped action's failure. +# The .invalid suffix would be stronger, but we want to also assert the +# specific output lines, which the `expected:` block enables for both +# success and failure runs. +expected: + output: + - WRAP_BEFORE + - TASK_BODY_RAN + forbidden: + - "Process exited with code: 0" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-glob-characters-literal.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-glob-characters-literal.test.yaml new file mode 100644 index 0000000..9d44664 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-glob-characters-literal.test.yaml @@ -0,0 +1,50 @@ +# RFC 0008 "Recommended test case" #4 — shell globbing. +# Glob characters (*, ?, [abc]) must reach the wrap script unexpanded. +# repr_sh() quotes them so the shell does not expand them at wrap time. +template: + specificationVersion: jobtemplate-2023-09 + name: GlobCharactersLiteral + steps: + - name: Step1 + script: + actions: + onRun: + command: ls + args: + - "*.txt" + - "?.log" + - "[abc]*" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "ARG=*.txt" + - "ARG=?.log" + - "ARG=[abc]*" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-ignores-wrapped-action-vars.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-ignores-wrapped-action-vars.test.yaml new file mode 100644 index 0000000..42b177d --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-ignores-wrapped-action-vars.test.yaml @@ -0,0 +1,45 @@ +# The wrap action is not required to reference WrappedAction.* at all — it +# can run its own logic. This test verifies the wrap runs successfully +# even when it ignores every injected variable. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapIgnoresWrappedActionSymbols + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["ORIGINAL"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: echo + args: ["SELF_CONTAINED"] + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - SELF_CONTAINED + forbidden: + - ORIGINAL diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-intercepts-simple-echo.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-intercepts-simple-echo.test.yaml new file mode 100644 index 0000000..4a78132 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-intercepts-simple-echo.test.yaml @@ -0,0 +1,45 @@ +# Minimal wrap test — confirms the runtime runs `onWrapTaskRun` in place of +# the step's `onRun`. The wrap action prints a sentinel that the step +# itself would never emit, so the sentinel's presence proves the wrap ran. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapIntercepts + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["ORIGINAL_ONRUN"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: echo + args: ["WRAP_SENTINEL"] + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - WRAP_SENTINEL + forbidden: + - ORIGINAL_ONRUN diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-job-and-step-env-rejected.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-job-and-step-env-rejected.invalid.test.yaml new file mode 100644 index 0000000..d5fc323 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-job-and-step-env-rejected.invalid.test.yaml @@ -0,0 +1,49 @@ +# RFC 0008 single-wrap-layer rule: even across job environments and step +# environments, at most one wrapping environment may exist in the session +# stack. A job env with wrap hooks combined with a step env with wrap +# hooks must be rejected. +template: + specificationVersion: jobtemplate-2023-09 + name: JobAndStepWrapEnvsRejected + extensions: + - WRAP_ACTIONS + - EXPR + jobEnvironments: + - name: JobWrap + script: + actions: + onEnter: + command: echo + args: ["enter"] + onWrapEnter: + command: echo + args: ["job-wrap-enter"] + onWrapTaskRun: + command: echo + args: ["JOB_WRAP"] + onWrapExit: + command: echo + args: ["job-wrap-exit"] + steps: + - name: Step1 + stepEnvironments: + - name: StepWrap + script: + actions: + onWrapEnter: + command: echo + args: ["step-wrap-enter"] + onWrapTaskRun: + command: echo + args: ["STEP_WRAP"] + onWrapExit: + command: echo + args: ["step-wrap-exit"] + script: + actions: + onRun: + command: echo + args: ["TASK"] + +runOn: +- posix diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-multiple-wrap-envs-rejected.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-multiple-wrap-envs-rejected.invalid.test.yaml new file mode 100644 index 0000000..ddc20ac --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-multiple-wrap-envs-rejected.invalid.test.yaml @@ -0,0 +1,58 @@ +# RFC 0008: at most one environment in the session stack may define any +# wrap hook. A job template that stacks two wrapping environments in +# `jobEnvironments` must be rejected before any environment is entered. +template: + specificationVersion: jobtemplate-2023-09 + name: MultipleWrapEnvsRejected + extensions: + - WRAP_ACTIONS + - EXPR + jobEnvironments: + - name: OuterWrap + script: + actions: + onEnter: + command: echo + args: ["outer-enter"] + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: echo + args: ["OUTER_WRAP"] + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + - name: InnerWrap + script: + actions: + onEnter: + command: echo + args: ["inner-enter"] + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: echo + args: ["INNER_WRAP"] + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["ORIGINAL"] + +runOn: +- posix diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-no-args.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-no-args.test.yaml new file mode 100644 index 0000000..54ff39e --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-no-args.test.yaml @@ -0,0 +1,51 @@ +# RFC 0008: WrappedAction.Args is a list[string]. When the wrapped action +# has no args, it must surface as an empty list (not null / not missing). +template: + specificationVersion: jobtemplate-2023-09 + name: WrappedActionNoArgs + steps: + - name: Step1 + script: + actions: + # No args: just a command. + onRun: + command: "true" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # Iterate WrappedAction.Args; the loop should run zero times. + - | + for a in {{ repr_sh(WrappedAction.Args) }}; do + echo SAW_ARG=$a + done + echo CMD={{WrappedAction.Command}} + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "CMD=true" + forbidden: + - SAW_ARG= diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-openjd-env-from-wrapped-onenter-propagates.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-openjd-env-from-wrapped-onenter-propagates.test.yaml new file mode 100644 index 0000000..f1ebc6f --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-openjd-env-from-wrapped-onenter-propagates.test.yaml @@ -0,0 +1,63 @@ +# RFC 0008: openjd_env emitted by a wrapped action propagates to +# WrappedAction.Environment for subsequent wrapped actions in the same +# session — this is what makes wrap composition useful. +# +# Here the inner env's onEnter (running via onWrapEnter) emits +# `openjd_env: WRAPPED_VAR=set-from-wrap`. The subsequent task wrap must +# see that variable in WrappedAction.Environment. +template: + specificationVersion: jobtemplate-2023-09 + name: OpenjdEnvPropagatesThroughWrap + jobEnvironments: + - name: InnerEnv + script: + actions: + onEnter: + command: sh + args: + - "-c" + - "echo 'openjd_env: WRAPPED_VAR=set-from-wrap'" + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + # Forward the inner env's onEnter verbatim so its openjd_env + # macro is captured (the scheduler scans the wrap script's stdout + # and recognizes the macro emitted by the wrapped process). + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + # In the task wrap, dump WrappedAction.Environment to verify the + # variable propagated. + onWrapTaskRun: + command: sh + args: + - "-c" + - "for e in {{ repr_sh(WrappedAction.Environment) }}; do echo ENV=$e; done" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - ENV=WRAPPED_VAR=set-from-wrap diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-original-onrun-does-not-run.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-original-onrun-does-not-run.test.yaml new file mode 100644 index 0000000..2d59fd5 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-original-onrun-does-not-run.test.yaml @@ -0,0 +1,50 @@ +# Complementary to wrap-intercepts-simple-echo: the original onRun must NOT +# execute when a wrap action is present. The original command here writes to a +# file which the wrap action deliberately does not — so if the file contains +# SHOULD_NOT_APPEAR anywhere in the output, the implementation is wrong. +template: + specificationVersion: jobtemplate-2023-09 + name: OriginalOnRunSuppressed + steps: + - name: Step1 + script: + actions: + onRun: + command: sh + args: + - "-c" + - "echo SHOULD_NOT_APPEAR; exit 7" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo WRAP_OK; exit 0" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - WRAP_OK + forbidden: + - SHOULD_NOT_APPEAR diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-outer-env-without-wrap-ignored.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-outer-env-without-wrap-ignored.test.yaml new file mode 100644 index 0000000..d30e639 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-outer-env-without-wrap-ignored.test.yaml @@ -0,0 +1,66 @@ +# RFC 0008: a wrapping environment is not required to be the outermost. +# Here PlainEnterEnv (outer, not a wrapping env) runs normally, and +# InnerWrap (inner, the wrapping env) wraps the task's onRun. Variables +# emitted via openjd_env from PlainEnterEnv are visible to the wrap script +# through the normal session environment. +template: + specificationVersion: jobtemplate-2023-09 + name: OuterWithoutWrapIgnored + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["ORIGINAL"] + +environments: +# Outer environment: only onEnter, no wrap. This env sets an env var to +# prove it ran — the inner wrap later echoes that value. +- specificationVersion: environment-2023-09 + environment: + name: PlainEnterEnv + script: + actions: + onEnter: + command: sh + args: + - "-c" + - "echo 'openjd_env: OUTER_VAR=outer_value'" + +# Inner environment: defines onWrapTaskRun. +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: InnerWrap + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo WRAP_RAN; echo OUTER=$OUTER_VAR" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - WRAP_RAN + # The wrap action runs with the env vars from earlier environments, + # proving both envs were active. + - OUTER=outer_value + forbidden: + - ORIGINAL diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-partial-hooks-rejected.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-partial-hooks-rejected.invalid.test.yaml new file mode 100644 index 0000000..d35d131 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-partial-hooks-rejected.invalid.test.yaml @@ -0,0 +1,30 @@ +# RFC 0008 all-or-nothing rule: an environment that defines any wrap hook +# must define all three. Defining only onWrapTaskRun must be rejected by +# the runtime before the session begins. +template: + specificationVersion: jobtemplate-2023-09 + name: PartialWrapHooksRejected + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK_RAN"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + environment: + name: PartialWrap + script: + actions: + # Only one of the three wrap hooks is defined. The all-or-nothing + # rule requires onWrapEnter, onWrapTaskRun, and onWrapExit together. + onWrapTaskRun: + command: echo + args: ["WRAP_RAN"] + +runOn: +- posix diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-path-traversal-literal.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-path-traversal-literal.test.yaml new file mode 100644 index 0000000..1e10972 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-path-traversal-literal.test.yaml @@ -0,0 +1,49 @@ +# RFC 0008 "Recommended test case" #3 — path traversal. +# The runtime must not resolve, normalize, or reject `../` paths. The wrap +# script + container boundary is what prevents escape; the injection layer +# simply forwards the bytes. +template: + specificationVersion: jobtemplate-2023-09 + name: PathTraversalLiteral + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: + - "../../../etc/passwd" + - "/tmp/../etc/shadow" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "ARG=../../../etc/passwd" + - "ARG=/tmp/../etc/shadow" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-empty-arg.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-empty-arg.test.yaml new file mode 100644 index 0000000..8dcfa62 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-empty-arg.test.yaml @@ -0,0 +1,55 @@ +# RFC 0008 "Recommended test case" #6 — empty and whitespace-only args. +# An empty string is a distinct argument and must not be dropped. Whitespace +# arguments must be preserved verbatim. +template: + specificationVersion: jobtemplate-2023-09 + name: EmptyAndWhitespaceArgs + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: + - "before" + - "" + - "after" + - " " + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # Use a sentinel wrap so empty lines in the output are still + # visible in the log. `` delimits each argv entry. + - "for a in {{ repr_sh(WrappedAction.Args) }}; do echo \"<$a>\"; done" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "" + - "" + - "< >" + # An empty argv entry renders as `<>` after the shell splits. + - "<>" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-nested-quotes.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-nested-quotes.test.yaml new file mode 100644 index 0000000..0257711 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-nested-quotes.test.yaml @@ -0,0 +1,51 @@ +# RFC 0008 "Recommended test case" #1 — nested quoting. +# Arguments containing both single and double quotes must survive injection +# and be quoted safely by repr_sh() in the wrap script. +template: + specificationVersion: jobtemplate-2023-09 + name: NestedQuotes + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: + - "O'Reilly's Guide" + - 'say "hi"' + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # repr_sh() must quote each argument so the shell sees them as + # literal tokens. `printf '%s\n'` then prints each argv on its + # own line. + - "printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "ARG=O'Reilly's Guide" + - 'ARG=say "hi"' diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-shell-metacharacters.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-shell-metacharacters.test.yaml new file mode 100644 index 0000000..7dc1129 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-shell-metacharacters.test.yaml @@ -0,0 +1,59 @@ +# RFC 0008 "Recommended test case" #2 — shell metacharacters. +# Characters like $, `, |, ;, &&, (, ) must be treated as literals and not +# interpreted by the wrap shell. repr_sh() prevents interpretation. +template: + specificationVersion: jobtemplate-2023-09 + name: ShellMetacharacters + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: + # This string, if interpolated unsafely, would execute + # `cat /etc/passwd` and then `id` via command substitution. With + # repr_sh() it must be printed verbatim. + - "$(cat /etc/passwd); `id`; || echo pwned" + - "a|b" + - "c&&d" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "ARG=$(cat /etc/passwd); `id`; || echo pwned" + - "ARG=a|b" + - "ARG=c&&d" + forbidden: + # If command substitution leaked, we'd likely see this line from + # /etc/passwd. Its absence is evidence that repr_sh() held. + - "root:x:0:0" + # `id` output would contain "uid=" — forbidden likewise. + - "uid=" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-unicode-cjk.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-unicode-cjk.test.yaml new file mode 100644 index 0000000..abc7224 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-preserves-unicode-cjk.test.yaml @@ -0,0 +1,53 @@ +# RFC 0008 "Recommended test case" #5 — Unicode paths. +# Multi-byte characters (CJK ~3 bytes/char, emoji ~4 bytes/char in UTF-8) +# must survive injection byte-for-byte. On Linux/macOS, ARG_MAX is enforced +# in bytes, not characters, so this is both a correctness and a limits test. +template: + specificationVersion: jobtemplate-2023-09 + name: UnicodePaths + steps: + - name: Step1 + script: + actions: + onRun: + command: render + args: + - "--scene" + - "/projects/映画/シーン01/レンダー.exr" + - "--emoji" + - "🎬" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "printf 'ARG=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - "ARG=--scene" + - "ARG=/projects/映画/シーン01/レンダー.exr" + - "ARG=--emoji" + - "ARG=🎬" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-runs-for-every-task.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-runs-for-every-task.test.yaml new file mode 100644 index 0000000..3ec6a9a --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-runs-for-every-task.test.yaml @@ -0,0 +1,53 @@ +# The wrap action runs per task — symbols re-inject cleanly each iteration. +# Two tasks are generated from a task parameter range; the wrap action emits +# a distinct sentinel per task so we can verify both ran. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapRunsForEveryTask + steps: + - name: Step1 + parameterSpace: + taskParameterDefinitions: + - name: N + type: INT + range: [1, 2] + script: + actions: + onRun: + command: echo + args: ["ORIGINAL", "{{Task.Param.N}}"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo WRAP_RAN={{WrappedAction.Command}}-{{WrappedAction.Args[1]}}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - WRAP_RAN=echo-1 + - WRAP_RAN=echo-2 + forbidden: + - ORIGINAL diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-step-environment.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-step-environment.test.yaml new file mode 100644 index 0000000..3108d0a --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-step-environment.test.yaml @@ -0,0 +1,36 @@ +# RFC 0008: wrap hooks work in stepEnvironments, not just jobEnvironments. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapInStepEnvironment + steps: + - name: Step1 + stepEnvironments: + - name: StepWrapEnv + script: + actions: + onWrapEnter: + command: echo + args: ["wrap-enter"] + onWrapTaskRun: + command: echo + args: ["WRAP_FROM_STEP_ENV"] + onWrapExit: + command: echo + args: ["wrap-exit"] + script: + actions: + onRun: + command: echo + args: ["ORIGINAL_TASK"] + extensions: + - WRAP_ACTIONS + - EXPR + +runOn: +- posix + +expected: + output: + - WRAP_FROM_STEP_ENV + forbidden: + - ORIGINAL_TASK diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-injected.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-injected.test.yaml new file mode 100644 index 0000000..b462f74 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-injected.test.yaml @@ -0,0 +1,45 @@ +# WrappedAction.Timeout surfaces the timeout of the original onRun action so +# the wrap script can propagate it (e.g., `docker stop --timeout N`). +template: + specificationVersion: jobtemplate-2023-09 + name: WrappedActionTimeoutInjected + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["placeholder"] + timeout: 123 + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo TIMEOUT={{WrappedAction.Timeout}}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - TIMEOUT=123 diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-zero-when-unset.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-zero-when-unset.test.yaml new file mode 100644 index 0000000..e3dd2ae --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-action-timeout-zero-when-unset.test.yaml @@ -0,0 +1,45 @@ +# RFC 0008: when the wrapped action specifies no timeout, WrappedAction.Timeout +# evaluates to 0 (the spec sentinel — `` cannot represent absence). +template: + specificationVersion: jobtemplate-2023-09 + name: WrappedActionTimeoutZeroWhenUnset + steps: + - name: Step1 + script: + actions: + # Note: no timeout field on this onRun. + onRun: + command: echo + args: ["placeholder"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo TIMEOUT={{WrappedAction.Timeout}}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - TIMEOUT=0 diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-args-preserved.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-args-preserved.test.yaml new file mode 100644 index 0000000..51e44e0 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-args-preserved.test.yaml @@ -0,0 +1,57 @@ +# WrappedAction.Args is a list[string]; with the EXPR extension, repr_sh(list) joins +# the elements with spaces and shell-quotes each individually. This test +# verifies that the list contents reach the wrap script intact. +template: + specificationVersion: jobtemplate-2023-09 + name: TaskArgsPreserved + steps: + - name: Step1 + script: + actions: + onRun: + command: mytool + args: + - "--scene" + - "scene01.ma" + - "--frame" + - "42" + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # repr_sh(list[string]) joins elements with spaces and shell-quotes + # each individually. Plain strings with no metacharacters render as + # bare tokens. + - "printf 'ARGS=%s\\n' {{ repr_sh(WrappedAction.Args) }}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + # Each element becomes one argv entry for `printf '%s\n'`, so the output is + # one line per arg. Order is preserved. + - "ARGS=--scene" + - "ARGS=scene01.ma" + - "ARGS=--frame" + - "ARGS=42" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-command-injected.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-command-injected.test.yaml new file mode 100644 index 0000000..63fb282 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-command-injected.test.yaml @@ -0,0 +1,50 @@ +# WrappedAction.Command must reach the wrap action as the exact command from the +# step's onRun. The wrap script echoes `CMD=` so a simple +# substring check can verify the value. +template: + specificationVersion: jobtemplate-2023-09 + name: TaskCommandInjected + steps: + - name: Step1 + script: + actions: + onRun: + command: maya-batch + args: ["-render", "scene.ma"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + - "echo CMD={{WrappedAction.Command}}" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - CMD=maya-batch + forbidden: + # The real command must never actually run — it's not on PATH, which is + # also a useful indicator: if the wrap didn't intercept, the runtime + # would try to exec `maya-batch` and error with a command-not-found. + - "No such file or directory" diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-environment-injected.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-environment-injected.test.yaml new file mode 100644 index 0000000..fd787f1 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-task-environment-injected.test.yaml @@ -0,0 +1,56 @@ +# WrappedAction.Environment is a list[string] of KEY=value entries collected from +# `openjd_env:` lines emitted by active environments. The wrap script can +# feed this list directly to `docker exec -e` style flags. This test checks +# the list contents without needing Docker. +template: + specificationVersion: jobtemplate-2023-09 + name: TaskEnvironmentInjected + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["placeholder"] + +environments: +- specificationVersion: environment-2023-09 + extensions: + - WRAP_ACTIONS + - EXPR + environment: + name: WrapEnv + script: + actions: + # Set two env vars via openjd_env during onEnter. These should land + # in WrappedAction.Environment as KEY=value strings. + onEnter: + command: sh + args: + - "-c" + - "echo 'openjd_env: FOO=foo_value'; echo 'openjd_env: BAR=bar_value'" + onWrapEnter: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + onWrapTaskRun: + command: sh + args: + - "-c" + # Iterate over WrappedAction.Environment, emitting one entry per line so we + # can verify both are present. + - "for e in {{ repr_sh(WrappedAction.Environment) }}; do echo ENV=$e; done" + onWrapExit: + command: sh + args: + - "-c" + - "{{ repr_sh(WrappedAction.Command) }} {{ repr_sh(WrappedAction.Args) }}" + +runOn: +- posix + +expected: + output: + - ENV=FOO=foo_value + - ENV=BAR=bar_value diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-expr-extension.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-expr-extension.invalid.test.yaml new file mode 100644 index 0000000..c527e61 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-expr-extension.invalid.test.yaml @@ -0,0 +1,33 @@ +# RFC 0008: WRAP_ACTIONS has a hard prerequisite on EXPR. A template that +# lists WRAP_ACTIONS without EXPR must be rejected. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapWithoutExpr + extensions: + - WRAP_ACTIONS + jobEnvironments: + - name: WrapEnv + script: + actions: + onEnter: + command: echo + args: ["enter"] + onWrapEnter: + command: echo + args: ["wrap-enter"] + onWrapTaskRun: + command: echo + args: ["wrap-task"] + onWrapExit: + command: echo + args: ["wrap-exit"] + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK"] + +runOn: +- posix diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-extension-fails.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-extension-fails.invalid.test.yaml new file mode 100644 index 0000000..d5d2059 --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-without-extension-fails.invalid.test.yaml @@ -0,0 +1,32 @@ +# An environment template that uses wrap hooks without declaring the +# WRAP_ACTIONS extension must be rejected by the runtime. +template: + specificationVersion: jobtemplate-2023-09 + name: WrapWithoutExtension + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["SHOULD_NOT_RUN"] + +environments: +- specificationVersion: environment-2023-09 + # NOTE: the WRAP_ACTIONS extension is deliberately NOT declared. + environment: + name: WrapEnv + script: + actions: + onWrapEnter: + command: echo + args: ["wrap-enter"] + onWrapTaskRun: + command: echo + args: ["SHOULD_NOT_RUN"] + onWrapExit: + command: echo + args: ["wrap-exit"] + +runOn: +- posix diff --git a/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-wrappedaction-outside-wrap-hook-rejected.invalid.test.yaml b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-wrappedaction-outside-wrap-hook-rejected.invalid.test.yaml new file mode 100644 index 0000000..91e862c --- /dev/null +++ b/conformance-tests/2023-09/WRAP_ACTIONS/jobs/wrap-wrappedaction-outside-wrap-hook-rejected.invalid.test.yaml @@ -0,0 +1,36 @@ +# RFC 0008: WrappedAction.* must not be referenced outside a wrap hook. +# Here the wrap env's onEnter references WrappedAction.Command, which has +# no meaning outside the three wrap hooks. The validator must reject the +# template with an "Undefined variable" error. +template: + specificationVersion: jobtemplate-2023-09 + name: WrappedActionOutOfScope + extensions: + - WRAP_ACTIONS + - EXPR + jobEnvironments: + - name: WrapEnv + script: + actions: + onEnter: + command: echo + args: ["{{WrappedAction.Command}}"] + onWrapEnter: + command: echo + args: ["{{Env.Wrapped.Name}}"] + onWrapTaskRun: + command: echo + args: ["{{WrappedAction.Command}}"] + onWrapExit: + command: echo + args: ["{{Env.Wrapped.Name}}"] + steps: + - name: Step1 + script: + actions: + onRun: + command: echo + args: ["TASK"] + +runOn: +- posix diff --git a/rfcs/0008-environment-wrap-actions.md b/rfcs/0008-environment-wrap-actions.md new file mode 100644 index 0000000..e41aa05 --- /dev/null +++ b/rfcs/0008-environment-wrap-actions.md @@ -0,0 +1,795 @@ +--- + +* Feature Name: Environment Wrap Actions +* Author(s): David Leong <[leongdl](https://github.com/leongdl)> +* RFC Tracking Issue: https://github.com/OpenJobDescription/openjd-specifications/issues/132 +* Start Date: 2026-04-16 +* Specification Version: 2023-09 extension WRAP_ACTIONS +* Accepted On: (pending) +* Depends On: + * RFC 0002, Model Extensions (https://github.com/OpenJobDescription/openjd-specifications/issues/57) + * RFC 0005, Expression Language (https://github.com/OpenJobDescription/openjd-specifications/pull/93) + * RFC 0006, Expression Function Library (https://github.com/OpenJobDescription/openjd-specifications/pull/104) + +## Summary + +This RFC proposes extending `` with three new session actions +(`onWrapEnter`, `onWrapTaskRun`, and `onWrapExit`) that let an environment +template intercept and wrap the lifecycle actions of *inner* environments +and tasks. The +runtime supplies each wrap action with the wrapped action's fields (command, args, +timeout, environment) via template variables. + +The RFC defines a general-purpose wrapping mechanism. Container execution is the +motivating use case and the focus of the examples, but the same mechanic supports +remote execution, session-wide instrumentation, privilege shifts, and any other form +of redirecting *how* an action runs without modifying the action itself. + +## Basic Example + +This environment template runs inner environments and tasks inside a Docker +container. The container starts once in `onEnter`, every inner action is forwarded +into the container via the three wrap hooks, and the container stops in `onExit`. +The example is intentionally short. A production-ready version with registry +authentication, image pull policies, and bind-mount parity, along with +Apptainer, dry-run, SSH, and profiling variants, is checked in as conformance-test +fixtures in future examples. + +```yaml +specificationVersion: "environment-2023-09" +extensions: + - WRAP_ACTIONS + - EXPR +parameterDefinitions: + - name: ContainerImage + type: STRING + default: "ubuntu:latest" + +environment: + name: Docker + script: + actions: + onEnter: + command: bash + args: ["{{Env.File.Enter}}"] + onWrapEnter: + command: bash + args: ["{{Env.File.Wrap}}"] + timeout: "{{WrappedAction.Timeout}}" + onWrapTaskRun: + command: bash + args: ["{{Env.File.Wrap}}"] + timeout: "{{WrappedAction.Timeout}}" + onWrapExit: + command: bash + args: ["{{Env.File.Wrap}}"] + timeout: "{{WrappedAction.Timeout}}" + onExit: + command: bash + args: ["{{Env.File.Exit}}"] + + embeddedFiles: + - name: Enter + filename: enter.sh + type: TEXT + data: | + #!/usr/bin/env bash + set -euo pipefail + DOCKER_CONTAINER_ID=$(docker container run --rm --detach \ + --mount 'type=bind,src={{Session.WorkingDirectory}},dst={{Session.WorkingDirectory}}' \ + {{ repr_sh(Param.ContainerImage) }} \ + bash -c 'sleep infinity') + echo "openjd_env: DOCKER_CONTAINER_ID=$DOCKER_CONTAINER_ID" + + # Shared wrap script for all three hooks: repr_sh() produces safely + # quoted argv tokens for every wrapped field. + - name: Wrap + filename: wrap.sh + type: TEXT + data: | + #!/usr/bin/env bash + set -euo pipefail + docker container exec \ + "$DOCKER_CONTAINER_ID" \ + {{ repr_sh(flatten([['-e', e] for e in WrappedAction.Environment])) }} \ + {{ repr_sh(WrappedAction.Command) }} \ + {{ repr_sh(WrappedAction.Args) }} + + - name: Exit + filename: exit.sh + type: TEXT + data: | + #!/usr/bin/env bash + set -euo pipefail + docker container stop --timeout 30 "$DOCKER_CONTAINER_ID" +``` + +The job template that runs under this wrapping environment is unchanged from one +that runs without wrapping. See the existing +[samples repository](https://github.com/OpenJobDescription/openjd-specifications/tree/mainline/samples/job_templates) +for portable job templates. + +### Execution order with a wrapping environment + +For a session with a wrapping Docker environment `A` and a non-wrapping inner +environment `B`, execution proceeds: + +``` +A.onEnter (wrapping env's own onEnter is never wrapped) + B.onEnter (via A.onWrapEnter) + Task 1: onRun (via A.onWrapTaskRun) + Task 2: onRun (via A.onWrapTaskRun) + B.onExit (via A.onWrapExit) +A.onExit (wrapping env's own onExit is never wrapped) +``` + +The wrapping environment is not required to be the outermost. If `B` were the +wrapping environment instead, `A` would run normally and `B` would wrap its +own inner actions and tasks. + +When no wrapping environment is present, every action runs normally. The job +template and the inner environments are unchanged. + +## Motivation + +Environment templates today can prepare the context in which a job runs: +installing software, setting environment variables, starting background daemons. +They cannot change *how* the actions within the session are executed. The existing +[bash-in-docker sample](https://github.com/OpenJobDescription/openjd-specifications/tree/mainline/samples/job_templates/bash-in-docker) +demonstrates a workaround: the step environment starts a container and the task +execs a script inside it. This approach requires the job template to be written +specifically for Docker, and inner step environments cannot install software inside +the container. Swap the environment from Docker to Conda, and the job template +breaks. + +The [Design Tenets](https://github.com/OpenJobDescription/openjd-specifications/wiki/Design-Tenets) +call for portability: + +Job templates should be portable in a way to run them, unmodified, with either a +Conda, Rez, Docker, or Apptainer environment template that provides the software +environment to run in. + +The three wrap hooks close this gap. Two composability properties make the mechanic +work without new coupling: + +1. An outer environment template can modify the execution of inner + environments that it does not know about. The wrap hooks operate on + whatever `` fields the inner environment supplies. +2. An inner environment template does not need to know a wrapper exists. The + same inner template runs unchanged with or without a wrapping environment. + +### Use cases + +1. **Container execution.** Apply a Docker or Apptainer container as the outer + environment, and let inner environments install plugins, activate Conda + environments, or stage dependencies inside it. The job template and the + inner environments are unchanged when the wrapping environment is swapped + or removed. + +2. **Remote execution.** An environment template can SSH into a remote host, or + submit actions to a cloud API, via the three wrap hooks. Inner environment setup + and task runs then execute on the remote host rather than locally. + +3. **Session-wide instrumentation.** Wrap every action with profiling, tracing, or + resource-accounting tools without modifying the job template or inner + environments. Task-only profiling (which exists via onRun substitution today) + misses the setup phases that often contain performance issues. + +4. **Privilege shifts.** Run inner actions as a different user or with reduced + capabilities by wrapping the command with `sudo -u`, `unshare`, or similar. + +### Backward compatibility + +This RFC is additive and gated by the `WRAP_ACTIONS` extension name declared under +RFC 0002. Specifically: + +- Schedulers that do not implement `WRAP_ACTIONS` MUST reject templates that list + it in `extensions:`, per RFC 0002's extension-handling rules. +- Schedulers that do implement `WRAP_ACTIONS` MUST ignore the three wrap hooks on + templates that do not list `WRAP_ACTIONS` in `extensions:`, so that existing + templates continue to behave exactly as before. +- No existing field changes meaning; `onEnter` and `onExit` continue to behave as + they do today when no wrap hook is active. + +**`EXPR` is a hard prerequisite of `WRAP_ACTIONS`.** Templates that list +`WRAP_ACTIONS` MUST also list `EXPR` (RFC 0005). Schedulers MUST reject +templates that list `WRAP_ACTIONS` without also listing `EXPR`. The +function library defined in RFC 0006 (including `repr_sh`/`repr_cmd`/ +`repr_pwsh` and `flatten`) is part of the same `EXPR` extension, so no +separate extension name is required. + +The rationale: safe reconstruction of a wrapped command line from +`WrappedAction.*` requires shell-aware quoting (see +[Security Considerations](#security-considerations)), and `EXPR` plus its +function library is the specification-provided mechanism for that. + +## Specification + +### Terminology + +The following terms are used throughout this specification. + +| Term | Definition | +|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Host** | The OS environment of the worker where the OpenJD session runtime runs and launches actions. | +| **Session runtime** | The OpenJD implementation running on the worker that launches actions, scans their stdout for `openjd_*` macros, enforces timeouts, and delivers signals. Distinct from the (cloud-side) scheduler that accepts templates and dispatches sessions. | +| **Wrap hook** | One of the three new `` fields: `onWrapEnter`, `onWrapTaskRun`, `onWrapExit`. | +| **Wrap script** | The program supplied by a wrap hook's ``, and the process the session runtime launches to execute it. Always runs on the host. | +| **Wrapping environment**| The environment that defines wrap hooks. Its own `onEnter` and `onExit` run normally and are never wrapped. | +| **Inner environment** | Any environment in the session stack that is inner to the wrapping environment. Its lifecycle actions (`onEnter`, `onExit`) are intercepted by wrap hooks. | +| **Wrapped action** | An inner environment's `onEnter`/`onExit`, or a task's `onRun`, whose execution is intercepted by a wrap hook. The wrap script receives the action's fields via `WrappedAction.*` template variables. | +| **Wrapped process** | The OS process the wrap script invokes to perform the wrapped action (e.g., `docker container exec ...`, `ssh ...`). A grand-child of the session runtime. | + +### Schema modifications + +> Changes to [the template schema](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas). + +> A modification to [``](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas#43-environmentactions) + +```diff + ::= the object: + onEnter: ++ onWrapEnter: # @optional @extension WRAP_ACTIONS ++ onWrapTaskRun: # @optional @extension WRAP_ACTIONS ++ onWrapExit: # @optional @extension WRAP_ACTIONS + onExit: # @optional +``` + +1. *onEnter*: the action to run when entering the environment. +2. *onWrapEnter*: if provided, runs instead of the `onEnter` action of every + *inner* environment that enters the session while this environment is + active. +3. *onWrapTaskRun*: if provided, runs instead of the task's `onRun` action + for every task that runs while this environment is active. +4. *onWrapExit*: if provided, runs instead of the `onExit` action of every + *inner* environment that exits while this environment is active. +5. *onExit*: the action to run when exiting the environment. + +Each wrap hook receives the wrapped action's context via the +`WrappedAction.*` template variables (see +[Template variables](#template-variables)). + +**All-or-nothing rule.** An environment that defines any of `onWrapEnter`, +`onWrapTaskRun`, or `onWrapExit` MUST define all three, ensuring execution +context parity across every wrapped lifecycle phase. Schedulers MUST reject +templates that define only a subset of the wrap hooks before the session +begins. + +**No changes to ``.** This RFC does *not* modify the `` schema. All +changes are additive at the `` level, and the new template +variables are read-only context supplied to wrap actions by the runtime. + +### Wrap ordering with multiple environments + +A session MUST have at most one wrapping environment. Schedulers MUST reject +sessions whose stack contains two or more environments that define any wrap +hook, before entering any environment. + +A wrapping environment's wrap hooks intercept only the actions of inner +environments and tasks. The wrapping environment's own `onEnter` and `onExit` +are never wrapped. + +### Template variables + +The following variables are read-only context supplied by the runtime to wrap +actions. They have identical names and semantics across all three hooks, so +helper scripts can be reused unchanged. + +**Available in `onWrapEnter`, `onWrapTaskRun`, and `onWrapExit`:** + +| Variable | Type | Description | +|-----------------------------------|----------------|-------------| +| `WrappedAction.Command` | `string` | The `command` from the wrapped action. | +| `WrappedAction.Args` | `list[string]` | The `args` from the wrapped action. | +| `WrappedAction.Environment` | `list[string]` | Environment variables defined by `openjd_env` earlier in the session, as `["KEY=value", ...]`. See [Host environment variables and embedded file paths](#host-environment-variables-and-embedded-file-paths). | +| `WrappedAction.Timeout` | `int` | The timeout in seconds specified on the wrapped action, or `0` if none. See [Timeout behavior](#timeout-behavior). | + +**Additionally available in `onWrapEnter` and `onWrapExit`:** + +| Variable | Type | Description | +|-----------------------------------|----------------|-------------| +| `Env.Wrapped.Name` | `string` | The `name` of the inner environment whose action is being wrapped. Not present in `onWrapTaskRun`, since tasks have no equivalent. | + +Templates MUST NOT reference `WrappedAction.*` outside the three wrap hooks, +or `Env.Wrapped.*` outside `onWrapEnter` and `onWrapExit`. Schedulers MUST +reject templates that violate this scope rule. + +### Host environment variables and embedded file paths + +`WrappedAction.Environment` carries only `openjd_env`-defined variables from +the current session. Host-inherited variables (`HOME`, `PATH`, `OPENJD_*`, +etc.) and host filesystem paths referenced in `WrappedAction.Command` or +`WrappedAction.Args` (for example, `{{Env.File.X}}` resolves to a host path) +are not surfaced and are the wrap environment's responsibility to forward or +path-map into the wrapped execution context. Failure manifests as +`command not found` or `No such file or directory` from the wrapped +process. + +### Forwarding future `` fields + +The `WrappedAction.*` namespace forwards every field of the wrapped +`` object. When a future specification adds a field to `` +(for example, the `logMessageTimeout` idea from +[Discussion #118](https://github.com/OpenJobDescription/openjd-specifications/discussions/118)), +it MUST appear automatically under this namespace (e.g., +`WrappedAction.LogMessageTimeout`) without modification to this RFC. + +Wrap scripts MAY reference the new variable to propagate the field to the +wrapped execution context, or ignore it. Wrapped actions that rely on a field +without wrap-script support behave as if the field were absent. This is +consistent with how new `@extension`-gated fields are handled elsewhere in +the specification. + +### Modifications to How Jobs Are Run + +> Modifications to [How Jobs Are Run](https://github.com/OpenJobDescription/openjd-specifications/wiki/How-Jobs-Are-Run) + +```diff + Once the Environments have been created and entered for a Session, a series of Tasks are + run within that Session and the Environments. Tasks from any Step in a Job can run within + the same Session provided that the set of Environments that are required to run those + Tasks are identical. + ++ If exactly one Environment in the session stack defines the wrap hooks (onWrapEnter, ++ onWrapTaskRun, onWrapExit), then the lifecycle actions of inner Environments and ++ tasks are intercepted: ++ ++ - An inner Environment's onEnter is replaced by the wrapping Environment's onWrapEnter. ++ - A task's onRun is replaced by the wrapping Environment's onWrapTaskRun. ++ - An inner Environment's onExit is replaced by the wrapping Environment's onWrapExit. ++ ++ The wrapping Environment's own onEnter and onExit are never wrapped; they always run ++ normally. If more than one Environment in the session stack defines any wrap hook, ++ the session is invalid and the scheduler must reject it before entering any ++ Environment. If an environment defines any wrap hook, it must define all three. +``` + +### Stdout forwarding and macro propagation + +- Wrap scripts MUST forward the wrapped process's stdout and stderr to their + own stdout and stderr verbatim, without buffering, filtering, or + transformation. `docker container exec`, `apptainer exec`, and `ssh` + satisfy this by default; authors of custom wrappers MUST preserve it. +- OpenJD session runtimes MUST scan the wrap script's stdout for OpenJD stdout macros + (`openjd_env`, `openjd_fail`, `openjd_progress`, `openjd_status`, and any + future macros) and MUST NOT scan the grand-child's stdout directly. A + macro emitted by a wrapped process is recognized identically to one + emitted by the wrap script itself. +- OpenJD session runtimes MUST include in `WrappedAction.Environment` every + `openjd_env`-defined variable emitted by any earlier action in the same + session, regardless of whether that action ran normally or via a wrap + hook. This is what makes wrap composition useful: an inner Conda + environment's `onWrapEnter` can emit + `openjd_env: CONDA_PREFIX=/opt/conda/env`, and the wrap script for every + subsequent wrapped task can forward it into the wrapped execution + context. +- A wrap script MAY emit stdout macros directly. The + [Basic Example](#basic-example) uses this to publish + `openjd_env: DOCKER_CONTAINER_ID=...` from `onEnter` for subsequent wrap + hooks. + +Alternatives for `openjd_env` propagation (runtime scans grand-child; +wrap script re-emits; dual-scan) are documented in +[Appendix B](#appendix-b-alternatives-for-openjd_env-propagation-through-wrap-hooks). + +### Failure semantics + +A wrap hook is treated as the action it replaces. A failed `onWrapEnter` is +an inner `onEnter` failure; a failed `onWrapTaskRun` is a task `onRun` +failure; a failed `onWrapExit` is an inner `onExit` failure. The session +handles each failure exactly as it would in the unwrapped case. + +**Exit status propagation.** Wrap scripts MUST propagate the wrapped +process's exit status as their own. Idioms that satisfy this: + +- Shell: `set -euo pipefail` with the wrapped command as the last statement, + or `exec` of the wrapped command in hooks that run exactly one child. +- Programmatic wrappers: return the wrapped process's exit code directly + (e.g., `sys.exit(proc.returncode)` in Python). + +### Lifecycle and cleanup guarantees + +OpenJD already guarantees that when a session fails or is canceled, every +environment the session entered (or attempted to enter) has its `onExit` run +before the session terminates (see +[How Jobs Are Run](https://github.com/OpenJobDescription/openjd-specifications/wiki/How-Jobs-Are-Run)). +Wrap hooks extend this guarantee symmetrically: if `onWrapEnter` starts for +an inner environment, that inner environment's `onWrapExit` MUST run, and +the wrapping environment's own `onExit` MUST run on top of that. + +| Scenario | What runs | +|-----------------------------------------------|--------------------------------------------------------------------------| +| `onWrapEnter` fails for inner env B | `onWrapExit` for B; wrapping environment's `onExit`. | +| `onWrapTaskRun` fails or is canceled | `onWrapExit` for every inner env entered; wrapping environment's `onExit`. | +| `onWrapExit` fails | Wrapping environment's `onExit` still runs. | +| Wrapping environment's `onEnter` fails | Wrapping environment's `onExit` runs (existing OpenJD guarantee). | + +Resources allocated by the wrapping environment's `onEnter` (e.g., the +container) MUST remain available until every `onWrapExit` returns. + +### Cancelation behavior + +The wrap hook's own `` governs cancelation of any wrapped +action; the wrapped action's own `` is not surfaced to the +wrap script and is not honored (a candidate future extension; see +[Future Work](#future-work)). + +When the wrap script receives a termination signal (SIGTERM on POSIX, +platform-equivalent on Windows), it MUST cause the wrapped process to +receive a termination signal within the `` grace period. +The simplest way to satisfy this is to `exec` the wrapped command so the +wrapped process inherits the wrap script's signal lineage; container +runtimes that proxy signals by default (`docker container exec`, +`apptainer exec`) inherit propagation automatically. Detached patterns +(`docker container run --rm --detach`, backgrounded SSH) do not share the +wrap script's signal lineage and MUST implement explicit propagation. + +After the grace period the session runtime delivers SIGKILL, which cannot +be trapped; any resources still owned by the wrap script at that point +may be orphaned. + +### Timeout behavior + +Two independent timeouts apply to a wrapped action: + +| Timeout | Field source | Bounds | Purpose | +|--------------------------------------------------------------------|---------------------------------------|--------------------------------------------|----------------------------------------------------------------------| +| **Hook timeout** | `timeout` on the wrap `` | The wrap script on the host | Limits the wrapper's own execution (infrastructure-side). | +| **Wrapped timeout** | `WrappedAction.Timeout` | The wrapped process (in container, on remote host, etc.) | Limits the inner workload's logic (workload-side). | + +Wrap scripts MUST propagate the wrapped timeout to the underlying execution +runtime where the runtime exposes a timeout mechanism (e.g., +`docker container stop --timeout {{WrappedAction.Timeout}}`), and SHOULD +enforce it in-script (`timeout {{WrappedAction.Timeout}}s ...`) when the +runtime exposes none. + +When the wrapped action specified no timeout, `WrappedAction.Timeout` +evaluates to `0`; wrap scripts MUST treat `0` as "no timeout" and omit +the underlying runtime's timeout flag in that case (for example, omit +`--timeout` from `docker container stop` rather than passing +`--timeout 0`, which Docker interprets as "kill immediately"). +The sentinel `0` was chosen because the OpenJD `` schema cannot +represent absent values. + +Templates that omit `timeout` on a wrap hook inherit OpenJD's default for +that hook position. The session runtime enforces the hook timeout by +applying the wrap hook's `` policy (see +[Cancelation behavior](#cancelation-behavior)). + +## Design Choice Rationale + +### Three separate hooks rather than a single unified hook + +An alternative design uses a single `onWrapAction` hook with a +`WrappedAction.Type` discriminator (`TASK_RUN`, `ENV_ENTER`, `ENV_EXIT`). It is +more DRY and lets the wrapper express action-type-specific logic via EXPR +conditionals. However: + +1. **Schema explicitness.** Three hooks make the environment's capabilities + visible directly in the schema. A template reader sees at a glance that + `onWrapEnter` is defined, which means inner-environment `onEnter` actions are + intercepted. With a single `onWrapAction`, the same information is buried + inside the wrap script. +2. **Per-phase logic without a discriminator.** Three hooks let authors write + dedicated scripts per action type without needing a `WrappedAction.Type` + switch inside a single hook. Under the three-hook model, the phase is + already determined by which hook is being invoked. +3. **Debug story.** A stack trace or log line citing `onWrapTaskRun` is + unambiguous. A stack trace citing `onWrapAction` requires the reader to know + which branch the script took. + +A future extension could add `onWrapAction` as an alternative to the three-hook +model that relaxes the all-or-nothing rule (an environment could define +`onWrapAction` alone and skip the three separate hooks). See +[Future Work](#single-unified-onwrapaction-shorthand). + +### Three hooks on `` rather than a new template type + +Adding the three wrap hooks to the existing `` keeps the +environment template model intact. Environments already have `onEnter` and +`onExit` for setup and teardown of their own state. The wrap hooks are a natural +extension: setup, wrap each inner lifecycle phase, teardown. A new template type +would fragment the model and require new plumbing in every implementation. + +### All-or-nothing rule for wrap hooks + +Requiring all three hooks to be defined together prevents partial wrapping, +where some inner-action phases are intercepted and others run normally. +The schema-level cost is a small amount of additional YAML; the gain is +explicit, mechanically-checkable execution context parity. + +Wrap environments whose enter or exit phases have no meaningful work MAY set +them to shell no-ops (for example, `bash -c "true"`); the rule requires the +hooks to be *defined*, not to do substantive work. The +[`onWrapAction` shorthand in Future Work](#single-unified-onwrapaction-shorthand) +is the anticipated relaxation for wrappers whose three phases share identical +logic. + +### Unified `WrappedAction.*` namespace across all hooks + +All three wrap hooks see the same `WrappedAction.*` variables with identical +semantics. This has three concrete benefits: + +1. **Helper script reuse.** A single wrap script can be written once and + referenced from `onWrapEnter`, `onWrapTaskRun`, and `onWrapExit` without + per-hook variable substitutions. The [Basic Example](#basic-example) + demonstrates this with a single `Wrap` embedded file shared by all three + hooks. +2. **Forward compatibility with `` evolution.** Any new field added + to `` in a later RFC surfaces automatically under the same + namespace — no bespoke per-field plumbing. +3. **Forward compatibility with action self-introspection.** A future + extension that lets an action reference its own fields would naturally + live at `Action.*`; `WrappedAction.*` reads as "the `Action` being + wrapped" and composes cleanly with that. + +`Env.Wrapped.Name` is the only environment-specific variable; it remains +under `Env.Wrapped.*` because tasks have no equivalent. + +### Template variables rather than environment variables + +Each wrap hook receives its context as template variables rather than +environment variables. Template variables are type-safe (with the EXPR +extension, `WrappedAction.Args` is `list[string]` that can be shell-quoted +with `repr_sh()`), available at template expansion time, and consistent +with how other context is provided in OpenJD. Environment variables would +require parsing and escaping in every wrap script. + +### `list[string]` for environment variables + +Environment variables are provided as a flat list of `"KEY=value"` strings rather +than a dictionary. This matches the format expected by container runtimes +(`docker exec -e KEY=VALUE`, `apptainer exec --env KEY=VALUE`) and is +straightforward to iterate over in a list comprehension: + +```yaml +{{ repr_sh(flatten([['-e', e] for e in WrappedAction.Environment])) }} +``` + +A dictionary type would require additional syntax for iteration and would not map +directly to CLI-flag conventions. + +### Command escaping via `repr_sh` rather than raw interpolation + +The wrap script must reconstruct the wrapped action's command line inside a shell +script. This is inherently dangerous: if `WrappedAction.Command` or +`WrappedAction.Args` contain shell metacharacters (`"`, `'`, `` ` ``, `&`, +`|`, `>`, `<`, `*`, `?`, `(`, `)`, `\`, newlines), raw interpolation +breaks the script or, worse, executes unintended commands. + +The RFC requires the EXPR extension and its `repr_sh()` function (from RFC 0006) +for safe command construction. `repr_sh(string)` applies `shlex.quote` semantics: +it wraps the value in single quotes and escapes embedded single quotes. +`repr_sh(list[string])` applies this per element and joins with spaces, so each +element becomes exactly one argv entry when the shell parses the line. + +For Windows shells, the equivalent functions are `repr_cmd()` and `repr_pwsh()` +from RFC 0006, which handle each shell's different escaping rules. + +## Security Considerations + +### Command injection + +Wrap scripts MUST NOT pass raw command strings to a shell parser. Patterns +like `bash -c "{{WrappedAction.Command}} {{WrappedAction.Args}}"` create a +command injection vector when the wrapped command contains nested quotes, +backticks, semicolons, or subshell expressions like `$(...)`. + +Wrap scripts MUST treat `Command` and `Args` as distinct tokens and use +`repr_sh()` (or `repr_cmd()`/`repr_pwsh()` on Windows) to produce safely +quoted strings. Programmatic wrappers SHOULD use argument arrays rather +than shell strings: + +- **Python**: `subprocess.Popen([command] + args)`, never + `subprocess.Popen(shell_string, shell=True)`. +- **Rust**: `std::process::Command::new(command).args(args)`. + +### Wrapping overhead and command length + +Wrap hooks are subject to the host OS maximum command-line length. A wrap +hook effectively doubles the command string (once in the wrapper's +invocation and once in the wrapped action). A job with many `openjd_env`-set +variables plus a `docker exec` prefix can exceed `ARG_MAX` on a command that +would fit directly. An `onRun` near the 256 KB macOS `ARG_MAX`, for example, +may fail only when wrapped. + +Implementations SHOULD validate command-line length before exec and produce +an error that names the limit, rather than letting the OS return `E2BIG`. + +## Prior Art + +### Pydantic WrapValidator + +The naming and concept are inspired by +[Pydantic's WrapValidator](https://docs.pydantic.dev/latest/concepts/validators/#wrap-validators), +which wraps a validation step with before/after logic. The wrap hooks wrap the +lifecycle actions of inner environments and tasks with before/after logic +provided by the outer environment. + +### Workflow DSLs with container abstraction + +Nextflow, CWL, Snakemake, and WDL each provide a way to run a rule or process +inside a container without baking the container invocation into the workflow +code. They differ in *where* the runtime choice lives: + +- **Nextflow** uses a global `nextflow.config` toggle. +- **CWL** declares `DockerRequirement` as a hint; runners translate at execution. +- **Snakemake** declares `container:` per rule and converts at runtime. +- **WDL** declares `docker:` per task; engines may substitute. + +OpenJD's approach uses composable environment templates. The distinction from the +workflow-DSL prior art is that OpenJD separates the outer execution-context layer +(the wrap environment) from the job-logic layer (the job template), rather than +annotating individual processes with a runtime identifier. + +### Conda and Rez run commands + +[Conda](https://docs.conda.io/projects/conda/en/latest/commands/run.html) +(`conda run -n `) and +[Rez](https://rez.readthedocs.io/en/stable/commands/rez-env.html) +(`rez-env -- `) both expose a "run a command inside this +environment" pattern. A wrap environment that exec's `conda run` or `rez-env` +in its wrap hooks reuses that idea at the session level. + +## Rejected Ideas + +### Extending `` with a `wrapper` field + +We rejected adding a `wrapper` field to `` that specifies a command +to prepend. This is simpler than wrap hooks but less flexible: it gives no +access to the task's environment variables, no way to wrap inner +environments' `onEnter`/`onExit`, and no way to transform the command and +args (only prepend to them). + +### Container-specific session action + +We rejected a dedicated `onRunInContainer` action because it would hard-code +container semantics into the specification. The general wrap-hook mechanism +is more flexible and supports use cases beyond containers. + +### Implicit container support in the runtime + +We rejected approaches that bake container semantics into the runtime: a +`containerImage` field that the runtime acts on automatically, or a global +`docker.enabled` / `apptainer.enabled` toggle. Both couple the specification +to specific container runtimes, remove the user's ability to customize the +invocation (mount points, network mode, GPU flags, security options), and +cannot express per-environment runtime choices within a single OpenJD +deployment. Per-environment wrap hooks provide the same single-toggle +simplicity when attached as an environment template, and additionally support +inner-environment and per-job scoping. + +### Nested wrap composition + +An earlier iteration allowed multiple environments in the session stack to +define wrap hooks, composing as nested wrappers (outermost wraps innermost). +We rejected this for the initial RFC because it adds significant +implementation complexity (symbol-table layering, cancelation propagation +across multiple wrap layers) and makes debugging substantially harder. This +RFC restricts the session stack to a single wrap layer. See +[Future Work](#future-work). + +## Future Work + +The following ideas are deferred to follow-up RFCs when concrete demand emerges. + +### Escape hatches for inner actions to bypass wrapping + +Some inner actions cannot run inside a wrapped context: mounting an NFS +share that the container will bind-mount, fetching short-lived host +credentials, or cleaning up after a crashed container. The idiomatic solution +is to nest these as a separate environment outside the wrap environment. Two +candidate escape hatches are deferred for the less idiomatic case: + +1. **`runOnHost: true` on ``.** A declarative field that bypasses + the active wrap hook for a specific action. Visible in the schema; clear + intent. Tradeoff: lets an inner-template author override the wrapping + environment's policy. +2. **`openjd_cmd: ` stdout line emitted by a wrap hook.** + Symmetric with `openjd_env: KEY=value`. A wrap hook could emit + `openjd_cmd: run-on-host` to instruct the session runtime to run a + specific action normally, bypassing the wrap. This places the decision + inside the wrap environment rather than the inner action. + +A future RFC should pick one or both and specify their interaction with +cancelation, timeout, and environment-variable propagation. + +### Single unified `onWrapAction` shorthand + +A future extension could add `onWrapAction` (with a `WrappedAction.Type` +discriminator) as an alternative to the three-hook model. Defining +`onWrapAction` alone would satisfy the wrap-hook requirement and relax the +current all-or-nothing rule, benefiting wrap environments whose three phases +share identical logic (e.g., a profiling wrapper applying the same tool to +every action). The three-hook form would remain available for authors who +want per-phase schema explicitness; the two forms would be mutually exclusive +per environment. Reasons for rejecting it from the initial RFC are in +[Design Choice Rationale](#three-separate-hooks-rather-than-a-single-unified-hook). + +### Nested wrap composition + +Stacking multiple wrap environments (for example, a profiling wrapper outside a +container wrapper) is valuable but adds implementation complexity. Deferred. + +### Wrap-aware cancelation + +Surfacing the wrapped action's `` to the wrap script as a template +variable would allow wrap scripts to honor the inner action's cancelation +semantics. Deferred until a concrete case arises. + +### Distinguishing wrapper failures from workload failures + +A future extension could reserve a sentinel exit status (e.g., `125`, +following the Docker CLI convention) so wrap scripts can signal that the +wrapping substrate failed (Docker daemon unreachable, image pull failed, +SSH connection refused) as distinct from the wrapped action itself +exiting non-zero — useful if schedulers want to retry infrastructure +failures on a different worker without retrying genuine workload +failures. + +### Session health monitoring + +Periodic health-checking of wrapped processes (detecting stalled containers, +unresponsive remote hosts) is a useful adjacent feature, but is cleanly separable +from the wrap mechanic. A separate RFC for environment health monitoring is the +appropriate venue. + +### Cross-OS wrapping + +This RFC does not prohibit cross-OS wrapping (a Windows host launching a Linux +container via WSL, for example). Whether it works is the environment +implementer's problem; the specification provides the same `WrappedAction.*` +variables regardless of host/target OS. + +## Appendix B: Alternatives for `openjd_env` propagation through wrap hooks + +Referenced from +[Specification › Stdout forwarding and macro propagation](#stdout-forwarding-and-macro-propagation). + +The chosen model: the session runtime scans the wrap script's stdout as it +does for any action, and the wrap script forwards the wrapped process's +stdout verbatim. + +### Alternative 1: Runtime scans the wrapped process's stdout directly + +The session runtime bypasses the wrap script and scans the grand-child +process's stdout for `openjd_env:` lines. Rejected because: + +- The runtime would need to locate the grand-child, which different + container runtimes expose differently and which remote-exec wrappers may + not expose at all. +- The existing contract (the runtime reads stdout from the action defined + in the template) breaks for wrapped actions only, creating a special case. +- It presumes exactly one grand-child, which is not true for wrap scripts + that invoke multiple commands. + +### Alternative 2: Wrap script explicitly re-emits `openjd_env` lines + +The wrap script greps the wrapped process's output for `openjd_env:` lines +and re-emits them on its own stdout. Rejected because every wrap-script +author would need to implement this correctly (easy to miss, hard to test), +and it duplicates work the session runtime already performs. + +### Alternative 3: Dual-scan both streams + +The session runtime scans both the wrap script's stdout and the wrapped +process's stdout (where locatable). Rejected because duplicate lines become +possible (the wrap script forwards, then the runtime also scans the +grand-child directly), requiring de-duplication, and it inherits the +locatability problem from Alternative 1. + +### Why the chosen model + +The chosen model preserves the existing OpenJD contract, does not require +the session runtime to reason about the wrapped process's shape, and is +satisfied by default for every container runtime in scope +(`docker container exec`, `apptainer exec`, `podman exec`, `enroot start`) +without additional wrap script code. Authors of exotic wrappers (custom +remote-exec, privilege shifts) must preserve stdout forwarding, a localized +constraint affecting only the small population of non-container +wrap-environment authors. + +## Copyright + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. diff --git a/wiki/2023-09-Template-Schemas.md b/wiki/2023-09-Template-Schemas.md index be12440..d11792a 100644 --- a/wiki/2023-09-Template-Schemas.md +++ b/wiki/2023-09-Template-Schemas.md @@ -50,7 +50,8 @@ Where: * Extensions available for specification version 2023-09: [TASK_CHUNKING](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0001-task-chunking.md), [REDACTED_ENV_VARS](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0003-redacted-env-vars.md), [FEATURE_BUNDLE_1](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0004-enhanced-limits-and-capabilities.md), - [EXPR](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0005-expression-language.md) + [EXPR](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0005-expression-language.md), + [WRAP_ACTIONS](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md) 4. *name* — The name to give to a Job that is created from the template. See: [<JobName>](#111-jobname). 5. *description* — A description to apply to all Jobs that are created from the template. It has no functional purpose, but may appear in UI elements. See: [<Description>](#72-description). @@ -1502,6 +1503,13 @@ The format string scopes available to format strings within an Environment are: 3. `Env.*` — Scope of the environment entity itself. Values such as the embedded files defined within the Environment entity. 4. Names bound by `let` in the enclosing ``. Available with the `EXPR` extension. +5. `WrappedAction.*` — Available within `onWrapEnter`, `onWrapTaskRun`, and `onWrapExit` only. Carries + the wrapped ``'s fields. Available with the `WRAP_ACTIONS` extension. See [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). +6. `Env.Wrapped.Name` — Available within `onWrapEnter` and `onWrapExit` only. The name of the inner + environment whose action is being wrapped. Available with the `WRAP_ACTIONS` extension. + +Templates must not reference `WrappedAction.*` outside the three wrap hooks, or `Env.Wrapped.*` outside +`onWrapEnter` and `onWrapExit`. Schedulers must reject templates that violate this scope rule. Implementations of this specfication must watch STDOUT when running the `onEnter` action for any line matching: @@ -1550,17 +1558,46 @@ An `` is the object: ```yaml onEnter: +onWrapEnter: # @optional @extension WRAP_ACTIONS +onWrapTaskRun: # @optional @extension WRAP_ACTIONS +onWrapExit: # @optional @extension WRAP_ACTIONS onExit: # optional ``` 1. *onEnter* — The action run when the environment is being entered on a host. -2. *onExit* — The action run when the environment is being exited on a host. +2. *onWrapEnter* — When provided, runs instead of the `onEnter` action of every *inner* environment that + enters the session while this environment is active. Available only when using the `WRAP_ACTIONS` + extension. See [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). +3. *onWrapTaskRun* — When provided, runs instead of the task's `onRun` action for every task that runs + while this environment is active. Available only when using the `WRAP_ACTIONS` extension. See + [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). +4. *onWrapExit* — When provided, runs instead of the `onExit` action of every *inner* environment that + exits while this environment is active. Available only when using the `WRAP_ACTIONS` extension. See + [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). + +Each wrap hook receives the wrapped ``'s fields via the `WrappedAction.*` template variables. +An environment that defines any wrap hook must define all three. See +[RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). +5. *onExit* — The action run when the environment is being exited on a host. > **NOTE:** When *onExit* action does not define a *timeout* the action will default to 300 seconds, or five minutes. Job schedulers may provide the ability to cancel jobs/steps/tasks. A reasonable default expectation should be that OpenJobDescription sessions are able to end and cleanup within a bound amount of time. +> **`WRAP_ACTIONS` extension constraints.** +> +> 1. *All-or-nothing rule.* An environment that defines any of `onWrapEnter`, `onWrapTaskRun`, or +> `onWrapExit` must define all three. Schedulers must reject templates that define only a subset of +> the wrap hooks before the session begins. +> 2. *Single-layer rule.* A session must contain at most one environment that defines wrap hooks. +> Schedulers must reject sessions whose stack contains two or more such environments before +> entering any environment. +> 3. *EXPR prerequisite.* A template that lists `WRAP_ACTIONS` in `extensions:` must also list `EXPR`. +> Schedulers must reject templates that list `WRAP_ACTIONS` without also listing `EXPR`. +> 4. The wrapping environment's own `onEnter` and `onExit` are never wrapped; they always run on the +> host. + ### 4.4. `` An `` is a map from ``s to ``s: @@ -1618,12 +1655,21 @@ positive integer value in base-10, and: | --- | --- | --- | | `` | `onRun` | *no timeout* | | `` | `onEnter` | *no timeout* | + | `` | `onWrapEnter` 2 | *no timeout* | + | `` | `onWrapTaskRun` 2 | *no timeout* | + | `` | `onWrapExit` 2 | 300 seconds (five minutes) 1 | | `` | `onExit` | 300 seconds (five minutes) 1 | 1 Environment exit actions are treated specially. Job schedulers may provide the ability to cancel jobs, steps, and tasks. A reasonable default expectation should be that OpenJobDescription sessions are able to end and cleanup within a bound duration of time. + 2 Available with the `WRAP_ACTIONS` extension. The wrap hook's `timeout` field bounds + the wrap script on the host (infrastructure-side); the `WrappedAction.Timeout` template variable + carries the wrapped action's timeout (workload-side). When the wrapped action specified no + timeout, the variable evaluates to `0`. + See [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). + 4. *cancelation* — If defined, provides details regarding how this action should be canceled. If not provided, then it is treated as though provided with ``. @@ -2067,6 +2113,9 @@ positive integer value in base-10, and: | --- | --- | --- | | `` | `onRun` | *no timeout* | | `` | `onEnter` | *no timeout* | + | `` | `onWrapEnter` 2 | *no timeout* | + | `` | `onWrapTaskRun` 2 | *no timeout* | + | `` | `onWrapExit` 2 | 300 seconds (five minutes) 1 | | `` | `onExit` | 300 seconds (five minutes) 1 | 1 Environment exit actions are treated specially. Job schedulers @@ -2074,6 +2123,8 @@ positive integer value in base-10, and: default expectation should be that OpenJobDescription sessions are able to end and cleanup within a bound duration of time. + 2 Available with the `WRAP_ACTIONS` extension. See [RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). + 5. *cancelation* — If defined, provides details regarding how this action should be canceled. If not provided, then it is treated as though provided with ``. diff --git a/wiki/How-Jobs-Are-Run.md b/wiki/How-Jobs-Are-Run.md index 7ff819c..568d19a 100644 --- a/wiki/How-Jobs-Are-Run.md +++ b/wiki/How-Jobs-Are-Run.md @@ -31,6 +31,16 @@ runs Chunks for that Step instead of individual Tasks. A Chunk is a set of Tasks values identical except for the chunked Task parameter. It takes values from an integer range expression like "1-3" or 1-3,5,7" depending on whether the chunks are constrained to be contiguous or not. +If exactly one Environment in the session stack defines the wrap hooks (`onWrapEnter`, `onWrapTaskRun`, +`onWrapExit`, part of the WRAP_ACTIONS extension), the lifecycle actions of inner Environments and tasks +are intercepted: an inner Environment's `onEnter` is replaced by the wrapping Environment's +`onWrapEnter`, a task's `onRun` is replaced by the wrapping Environment's `onWrapTaskRun`, and an inner +Environment's `onExit` is replaced by the wrapping Environment's `onWrapExit`. The wrapping Environment's +own `onEnter` and `onExit` are never wrapped; they always run on the host. If more than one Environment +in the session stack defines any wrap hook, the session is invalid and the scheduler must reject it +before entering any Environment. If an Environment defines any wrap hook, it must define all three. See +[RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). + When the EXPR extension is used, format strings evaluated on the Worker Host support the full [expression language](2026-02-Expression-Language) including arithmetic, conditionals, function calls, path manipulation, and script embedding functions like `repr_sh()` for safe shell quoting. @@ -111,6 +121,12 @@ messages to convey information about the **Action** to the render management sys **Action** for entering an **Environment**. This unsets the given environment variable for all subsequent **Action**s in the **Session** until the **Environment** that emitted it is exited. +When the WRAP_ACTIONS extension is in use, schedulers must scan the stdout of the wrap script (`onWrapEnter`, +`onWrapTaskRun`, or `onWrapExit`) for these macros, not the stdout of the wrapped process. Wrap scripts must +forward the wrapped process's stdout and stderr verbatim, which causes macros emitted by the wrapped process +to be recognized identically to macros emitted by the wrap script itself. See +[RFC 0008](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0008-environment-wrap-actions.md). + ## Path Mapping An artists' workstation's filesystem may not match that of the Worker Host that will be running a Task. Shared