feat: structured JSON logging + K8s probe docs (COW-994)

[lgahdl]

Grant Review Finding

[F19] Document K8s liveness/readiness probes and add JSON structured logging with chainId — MAJOR (production observability / handoff) (COW-994)

Two related gaps: (1) K8s probe documentation — the indexer exposes /healthz (liveness — always 200 once up) and /ready (readiness — 200 only when fully synced). Using /ready for liveness (as the Docker healthcheck does) means a still-indexing pod with a multi-hour cold start gets killed before it ever serves traffic. The probe mapping isn't documented anywhere. (2) JSON structured logging — logs are pretty-printed in production; for a multi-chain product whose logs will be ingested into a log aggregator and filtered by chainId, JSON-structured logging with chainId/event/block labels per line is needed.

---

Summary

Structured JSON logging:

• New src/application/helpers/cowLogger.ts — thin cowLog(level, msg, fields) utility that emits one newline-delimited JSON line per call
• All console.log/console.warn calls in blockHandler.ts (18 call sites across C1–C5) and settlement.ts migrated to cowLog(); each line now carries chainId and block as proper JSON fields, enabling log aggregator filtering by chain without regex
• pnpm start updated with --log-format json so Ponder's own internal log lines are also JSON in production; pnpm dev keeps pretty format for local readability

K8s probe documentation (docs/deployment.md):

• Documents /healthz (liveness — always 200 once up) vs /ready (readiness — 200 only when synced) distinction
• Adds sample K8s manifest snippet with correct livenessProbe/readinessProbe mapping
• Explains why /ready must NOT be used as liveness probe (would kill indexing pod before it finishes cold-start sync)
• Adds structured logging section explaining the --log-format json behavior

Test plan

• pnpm typecheck — passes
• pnpm test — 19/19 passing
• No console.log/console.warn calls remain in blockHandler.ts or settlement.ts
• Each cowLog call includes chainId and block fields
• pnpm start script includes --log-format json

🤖 Generated with Claude Code

[linear-code]

COW-994

[lgahdl]

The new src/application/helpers/cowLogger.ts has a 7-line multi-paragraph JSDoc block at the top. The project convention (from agent_docs/code-patterns.md) is: no multi-paragraph docstrings or multi-line comment blocks — one short line max.

/**
 * Structured JSON logger for handler code. Outputs one JSON line per call so
 * log aggregators (Datadog, CloudWatch, etc.) can filter by chainId, handler,
 * block number, or any other field without regex parsing.
 *
 * Ponder's own log lines are controlled by --log-format (pretty|json) on the
 * CLI. These handler lines are always JSON so they remain parseable regardless
 * of Ponder's format setting.
 */

This should be one line, e.g.:

/** Structured JSON logger — emits one ndjson line per call for log aggregators. */

The deployment doc already explains the JSON logging setup; the inline comment doesn't need to repeat it.

[lgahdl]

Documentation / CI review

cowLogger.ts multi-line comment block — not a violation.
The file opens with a 9-line /** ... */ JSDoc block. The project's agent_docs/code-patterns.md does not define a "max one short comment line" rule — that convention does not appear in any project docs. The existing helpers (orderbookClient.ts, withTimeout.ts, pollResultErrors.ts) all use multi-line JSDoc file headers. The comment in cowLogger.ts is consistent with the established pattern and accurately explains why the handler lines stay JSON regardless of Ponder's --log-format. No change needed.

K8s probe documentation in docs/deployment.md — accurate and complete.
The new "Kubernetes Probes" section correctly distinguishes /healthz (liveness — always 200 once up) from /ready (readiness — only 200 when fully synced), provides a concrete YAML snippet with sensible periodSeconds/failureThreshold values, and includes the important warning against using /ready as liveness. This is exactly what ops needs.

One gap: initialDelaySeconds not set on the liveness probe.
The YAML snippet sets livenessProbe.periodSeconds: 30 and failureThreshold: 3 but omits initialDelaySeconds. On a cold-start pod the process can take a few seconds to bind the port. Without initialDelaySeconds (or a startupProbe), the liveness probe can fire before the server is up and kill the pod immediately. Recommend adding at least initialDelaySeconds: 10 to the liveness probe snippet, or noting that a startupProbe is the preferred K8s pattern for long-starting containers.

Structured logging section — accurate. The pnpm start → --log-format json change and the per-line chainId/block field guarantee are correctly documented.

--log-warn / log level flag not documented.
pnpm start now passes --log-format json but there is no mention of how to control verbosity in production (e.g. --log-level warn). The existing PINO_LOG_LEVEL entry in the env-var table covers Ponder's log level already — just worth confirming those two interact correctly and the table entry is still accurate with the new --log-format json flag. Low priority, but worth a quick check.

Overall: Two minor items — add initialDelaySeconds to the liveness probe YAML, and confirm PINO_LOG_LEVEL still works alongside --log-format json. No blockers.

[lgahdl]

Review: Structured logging + K8s probe docs (COW-994)

cowLogger.ts — multi-line JSDoc block

The file opens with an 8-line /** … */ JSDoc comment. Other helper files in this project (e.g. withTimeout.ts, pollResultErrors.ts) use similar multi-line blocks, so this is consistent with the actual codebase convention — not a deviation. No issue here.

cowLog uses console.log for all levels including warn and error

The function always calls console.log(JSON.stringify({level, …})). The level field in the JSON payload lets aggregators filter by severity. However, warn and error level messages do not go to stderr — they go to stdout. This matters if any log routing in the deployment pipeline distinguishes stdout/stderr streams (e.g. Docker, some Kubernetes logging drivers, systemd journal). If you want warn/error to reach stderr, change the implementation to:

const log = level === "info" ? console.log : level === "warn" ? console.warn : console.error;
log(JSON.stringify({ time: Date.now(), level, msg, ...fields }));

K8s probe documentation accuracy

/healthz is confirmed in src/api/index.ts:18 as a real endpoint returning { status: "ok" }. /ready is referenced in the Docker health check comment; verifying it exists as a Ponder built-in is left to the deployer, but the documentation correctly distinguishes liveness vs readiness semantics. The warning against using /ready for liveness is accurate and important given the hours-long cold-start sync time.

--log-format json in pnpm start

Correct placement. pnpm dev intentionally left without --log-format json for local readability.

Migration completeness

All 18 console.log/console.warn call sites in blockHandler.ts and settlement.ts are migrated. No missed call sites visible in the diff.

One actionable item: consider routing warn/error to console.warn/console.error in cowLog. Everything else looks good.

[yvesfracari]

Anxo mentioned only the blockHandler and settlement logs but there are others that are other files that emit logs and should be using the new logger.

Would actually save this on an ADR.

[yvesfracari]

isn't /health a better fit here?

[lgahdl]

Good point — updated to /health (Ponder's built-in). Simpler and more idiomatic than our /healthz wrapper for a liveness probe.

[lgahdl]

Updated to /health — Ponder's built-in liveness endpoint, always 200 once the server starts. Simpler and more idiomatic than our /healthz wrapper for a liveness probe.

[yvesfracari]

time to deploy would be way bigger, right?

[lgahdl]

You're right, the comment was misleading. A readiness probe failure does NOT kill the pod — it just removes it from load-balancer rotation. On a cold start the pod will be NotReady for hours; that's expected. Updated the comment and added a paragraph clarifying this so operators aren't surprised.

[lgahdl]

You're right — a cold-start pod can be NotReady for hours. Clarified that readiness failure doesn't kill the pod (just removes it from rotation) and that hours-long NotReady is expected on a cold start. Also removed the specific failureThreshold/periodSeconds values per your follow-up comment — those depend on the cluster's SLOs.

[yvesfracari]

Please check my latest review comment about not using the new logger on other parts of the code. Also, this log should also live on a ADR so it is used later as well. Maybe it even worth to create a lint/rule that no console.log is used or something like this.

[yvesfracari]

I know that this is just an example but I am not sure if all values here make sense for the app context. I would prefer to remove it since the k8s operator should know how to set it properly, we just have to give him the app information so he can decide it.

[lgahdl]

Removed all the specific timing values (periodSeconds, failureThreshold, initialDelaySeconds) — the YAML now just shows the path and port, and a note explains that the operator picks the timing based on their cluster SLOs.

[lgahdl]

Migrated all remaining console.log/warn/error call sites in src/application/ to log() from logger.ts:

• composableCow.ts — 4 calls (unknown handler, created, decoded, decode failed)
• setup.ts — 1 call (cache ready)
• orderbookClient.ts — 9 calls (no API URL, fetch, fetch result, timeouts, errors)
• uidPrecompute.ts — 5 calls (missing params, invalid math, too many parts, all terminal, StopLoss missing params)

No console.* calls remain in src/application/ outside logger.ts itself. src/api/ (Hono routes) is intentionally left as-is.

Also expanded the Structured Logging section in docs/deployment.md with the convention (scope, usage example, stdout/stderr routing) instead of a separate ADR.

[yvesfracari]

I don't think this part is necessary actually

[yvesfracari]

A lint or pre-commit routine would be more effective