docs(observability): Gravsearch trace runbook, TraceQL + instrumentation recipes (DEV-6636)

[mpro7]

DEV-6636

Summary

• New top-level Observability docs section (publishes to docs.dasch.swiss) covering the per-stage Gravsearch tracing added in DEV-6635 (feat(Observability): instrument SearchResponderV2 with per-stage Gravsearch spans (DEV-6635) #4155).
• The anti-Faro usability layer: makes the trace data readable and queryable, and documents the instrumentation pattern so the next vertical can reuse it.

Changes

New docs/observability/ section (4 pages, wired into the mkdocs.yml nav under a new top-level "Observability" node):

• index.md — observability / tracing overview and entry point.
• gravsearch-trace-runbook.md (REQ-3.3) — find a slow trace, the span tree, per-stage time decomposition, root shape attributes, and the four absent-data topologies (empty-result, parse-failure, interruption/timeout, shape-less early interrupt) so none is misread as broken instrumentation.
• traceql-recipes.md (REQ-3.1/3.2) — TraceQL queries for slow Gravsearch traces (threshold parameterised on the baseline p95), stage drill-down, query-shape filtering, errors/interrupts, plus a Grafana Explore starter link.
• instrumentation-recipe.md (REQ-4.1) — the abstract-tracing-member pattern, stageSpan helper, span naming, bounded gravsearch.query.shape with the metric-label vs drill-down cardinality split, and the error → UNSET + sanitized-ERROR + exit_reason=interrupted handling — enough to instrument a second vertical without re-deriving it.

Test Plan

• Span/attribute names verified against the merged DEV-6635 code (PR feat(Observability): instrument SearchResponderV2 with per-stage Gravsearch spans (DEV-6635) #4155): root gravsearch + gravsearch.{parse,type_inspection,prequery.generate,prequery.execute,mainquery.generate,mainquery.execute,result_transform}; attributes gravsearch.query.shape, gravsearch.shape.*, gravsearch.schema_predicates, gravsearch.exit_reason.
• TraceQL syntax validated against Tempo's own TraceQL docs (intrinsics span:/trace:, dotted custom attributes span.<key>, structural >>).
• Markdown validated by hand against .markdownlint.yml (nav targets, internal links/anchors, MD033/MD007); local mkdocs/npm were unavailable, so CI mkdocs build --strict is the build gate.
• ⚠️ Span names were provisional until DEV-6635 was reviewed; a final validation pass is due once feat(Observability): instrument SearchResponderV2 with per-stage Gravsearch spans (DEV-6635) #4155 lands on main.

Screenshots

N/A — docs only (the mermaid span-tree diagram and admonitions render on docs.dasch.swiss).

[linear-code]

DEV-6636

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.98%. Comparing base (a978c50) to head (2403cee).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4161   +/-   ##
=======================================
  Coverage   85.98%   85.98%           
=======================================
  Files         538      538           
  Lines       30571    30571           
  Branches     3559     3559           
=======================================
  Hits        26286    26286           
  Misses       4285     4285           

Flag	Coverage Δ
e2e	85.98% <ø> (ø)
integration	85.98% <ø> (ø)
unit	85.98% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.