conv-update crash: subgroup-k-smoother is missing the smoothed-k clamp that #2536 added to group-k-smoother (empty subgroup clustering)

[lgelauff]

Summary

#2536 fixed a conv-update crash where a stale smoothed-k in group-k-smoother fell outside the current group-clusterings k-range, so (get group-clusterings stale-k) returned nil and crashed conv-repness with Don't know how to create ISeq from: clojure.core$map$fn. The identical bug remains one level down, in :subgroup-k-smoother — which #2536 did not clamp. A group's subgroup clustering can still come back empty exactly the same way, and conv-update still fails.

I arrived at this independently — reproducing the crash across a large corpus of public conversations and tracing it to the unclamped subgroup smoothed-k — and only afterwards found #2536, which corroborates the mechanism and the fix at the group level. This is the same fix, applied to the subgroup smoother. (As of 2026-06-22 no open PR addresses :subgroup-k-smoother; #2536's clamp was applied only at the group level.) This also emphasizes the importance of documenting the warm path assumptions.

Expected behavior

:subgroup-k-smoother should clamp smoothed-k to the keys present in that group's current subgroup clusterings — exactly as group-k-smoother now does after #2536.

Actual behavior

A group's subgroup clustering comes back empty when the carried smoothed-k is no longer a computed key, and conv-update fails (re-queues votes, emits math.pca.compute.fail, writes errorconv.<nanoTime>.edn, and freezes that conversation's math at its last good tick — not reliably self-healing; a math restart often clears it). The surface error depends on version:

• Before Fix stale group-k-smoother crash on large conversations #2536 (e.g. fd440c3e): the cryptic java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$map$fn__5931 (the (apply map f []) transducer reaching select-rep-comments/utils/zip).
• On current master (post-Fix stale group-k-smoother crash on large conversations #2536): Fix stale group-k-smoother crash on large conversations #2536's conv-repness guard turns it into a clear IllegalArgumentException ("group-clusters is nil or empty") — but conv-update still fails; the guard makes the error legible, it doesn't prevent the empty clustering. The root cause (unclamped subgroup smoothed-k) is unaddressed.

Root cause (the same chain as #2536, in :subgroup-k-smoother):

1. Per group, :subgroup-clusterings runs k-means for k ∈ (range 2 (inc (max-k-fn group-proj max-k))). max-k-fn is count-based: M = min(max-k, 2 + ⌊(#base-clusters in the group)/12⌋). So a group's available subgroup counts are {2 … M} — a coarse step of group size (≤11→2, 12–23→3, 24–35→4, 36–47→5).
2. :subgroup-k-smoother carries smoothed-k forward (anti-flap) without clamping to the current keys — it still has the pre-Fix stale group-k-smoother crash on large conversations #2536 logic (if (>= this-k-count count-buffer) this-k (if smoothed-k smoothed-k this-k)), with no (contains? …) check (whereas group-k-smoother now clamps).
3. When a group's membership drops below a /12 boundary, M falls; the carried smoothed-k can exceed M; (get group-subgroup-clusterings smoothed-k) → nil → (map f nil) → () — an empty subgroup clustering → conv-repness :stats = (apply map f []) → the crash (cryptic ISeq pre-Fix stale group-k-smoother crash on large conversations #2536, or Fix stale group-k-smoother crash on large conversations #2536's clear IAE on master).

(Empty k-means clusters are filtered out — clusters.clj — so a computed clustering always has ≥1 cluster; the empty here is purely the lookup miss, exactly as #2536 described for the group level. The group is not unclusterable — it's clusterable at a valid k like 2; the engine just requested a k it didn't compute this tick.)

Why it's nondeterministic / "transient" (matching #2536's "PCA rotation after a batch of new votes"): M depends only on how many base-clusters land in the group, and that membership comes from the group k-means over the unseeded cold-start PCA projection. A restart or a marginal vote re-rolls the projection → repartitions base-clusters → a knife-edge group's count crosses a /12 boundary → M flips → the carried smoothed-k is in-range (completes) or out-of-range (crashes). The same conversation can crash on one run and complete on the next (cf. #2358).

To Reproduce

1. A conversation with a group whose base-cluster count sits at/just above a /12 boundary (12, 24, 36) while its subgroup smoothed-k equals that group's M. A marginal vote — or a restart that re-rolls the projection — that drops the group below the boundary lowers M beneath the carried smoothed-k → empty subgroup clustering → crash.
2. Reproduced deterministically by warm-path replay (one vote per conv-update tick) of public openData conversations — e.g. scoop-hivemind.ubi, american-assembly.bowling-green — across 50+ distinct conversations. In every captured pre-crash state the subgroup smoothed-k was ≤ M (in range); the crash follows on the tick where M steps down.

Minimal logical repro (REPL): (apply map (fn [& xs] xs) []) returns a transducer, not (); equivalently call select-rep-comments on conv-repness computed with empty group-clusters.

Screenshots

N/A (backend math worker). Artifacts: the stack trace above and the errorconv.<nanoTime>.edn dump (:error field).

Device information

Clojure math worker (polismath), self-hosted. Reproduced on compdemocracy/polis@fd440c3e (live backend errorconv.*.edn + warm-path replays of public openData). The unclamped :subgroup-k-smoother persists on current master — re-verify line numbers against your commit.

Suggested fix

Mirror #2536's group-k-smoother clamp into :subgroup-k-smoother — one line, in the per-group smoothed-k computation:

;; clamp the carried smoothed-k to a key that exists in THIS group's current clusterings
smoothed-k (if (contains? group-subgroup-clusterings smoothed-k) smoothed-k this-k)

this-k is always a valid key (it's chosen from the current keys), and when M drops the group genuinely supports fewer subclusters, so smoothed-k should follow it down. #2536's conv-repness guard already gives the clear error message; this clamp prevents the empty in the first place. (The M = 1 / empty-menu case — a group too small for even k = 2, where (range 2 2) is empty and there's no key to clamp to — should yield no subgroups for that group, per the maintainer TODO at conversation.clj:~502–505.)