Meta: benchmark coverage unlock plan for GFQL and GFQL/Cypher

[lmeyerov]

Summary

This meta-issue tracks benchmark-impacting GFQL/Cypher compiler and planner gaps for official SNB lanes.

Current Queue (as of 2026-05-06 PT)

Foundational

1. Meta: #1046 native type-system staged decomposition program #1262 decomposition program for [FEA] GFQL native type system: schemas, inference, validation, and Arrow representation #1046 (active child: GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311)

Recently completed foundational meta

• Meta: GFQL monolith shrinkdown (lowering.py + CompiledCypher cleanup) #1260 monolith shrinkdown track closed via S5: add CI growth guards for lowering/CompiledCypher surfaces (#1306) #1310 (GFQL monolith shrinkdown S5: add CI growth guards for lowering/CompiledCypher surfaces #1306 S5)

Recently Landed / Progress

• GFQL/Cypher: add validate-only gfql preflight API #1321 merged: closes GFQL/Cypher: expose validate-only API (compile/bind checks without execution) #1320 (validate-only API)
• GFQL T3.b: consolidate nullable helper call sites (#1309) #1317 merged: closes GFQL native types T3.b: migrate inline nullable patterns onto ir.metadata helpers #1309 (T3.b nullable helper consolidations)
• GFQL native types T4: Arrow/type bridge contracts #1313 merged: closes GFQL native types T4: Arrow/type bridge contracts and coercion semantics #1312 (T4 Arrow/type bridge contracts)
• T3 #1300: type/null metadata propagation contract for BoundIR/LogicalPlan seams #1304 merged: closes GFQL native types T3: propagate nullability/type metadata through bound/logical plan seams #1300 (T3 nullability/type metadata propagation)
• GFQL native types T2: binder-time schema validation in strict mode (#1302) #1308 merged: closes GFQL native types T2: binder-time schema validation in strict mode #1302 (T2 binder-time schema validation)
• S5: add CI growth guards for lowering/CompiledCypher surfaces (#1306) #1310 merged: closes GFQL monolith shrinkdown S5: add CI growth guards for lowering/CompiledCypher surfaces #1306 (S5 CI growth guards) and completes Meta: GFQL monolith shrinkdown (lowering.py + CompiledCypher cleanup) #1260
• S4: reduce CompiledCypher compatibility surface (#1305) #1307 merged: closes GFQL monolith shrinkdown S4: reduce CompiledCypher compatibility surface #1305 (S4 CompiledCypher surface reduction)
• GFQL monolith shrinkdown S3: split lowering.py projection + reentry concerns #1303 merged: closes GFQL monolith shrinkdown S3: split lowering.py by concern boundaries #1301 (S3 split of lowering.py concerns)
• Refactor #1295: extract bounded reentry glue from lowering.py #1299 merged: closes GFQL monolith shrinkdown S2: extract bounded reentry glue from lowering.py #1295 (S2 extraction under Meta: GFQL monolith shrinkdown (lowering.py + CompiledCypher cleanup) #1260)
• feat(gfql-ir): stabilize schema catalog contract surface (#1296) #1298 merged: closes GFQL type-system T1: schema contract + canonical naming alignment #1296 (T1 schema contract + canonical naming alignment)
• feat(cypher): admit multi-prefix-row free-form intermediate MATCH (#1285) #1289 merged: closes Cypher/GFQL: admit multi-prefix-row free-form intermediate MATCH runtime path #1285 (IC3 multi-prefix-row free-form admit)
• fix(cypher): admit free-form carried-property bridge refs (#1275) #1287 merged: closes Cypher/GFQL: free-form trailing WHERE bridge refs across carried + fresh aliases #1275 (IC3 carried-property bridge)
• feat(cypher): admit simple free-form intermediate MATCH (#1263) #1279 merged: lands simple free-form admit for Cypher/GFQL: free-form intermediate MATCH after WITH (LDBC SNB IC3 endpoint) #1263
• Cypher/GFQL: free-form intermediate MATCH after WITH (LDBC SNB IC3 endpoint) #1263 closed (IC3 endpoint lane complete)
• Cypher/GFQL rejects multi-stage MATCH ... WITH ... MATCH ... WITH ... MATCH read queries #999 closed (IC3/IC5 child gates complete)
• #1273: Admit shape A WITH multi-alias scalar projection + convert tests #1277 merged: closes Cypher/GFQL: admit multi-MATCH-source row projection without hidden carries (IC5 sub-residual B) #1273 (IC5 sub-B)
• Fix #1272: forward reentry carry through trailing WITH narrowing #1278 merged: closes Cypher/GFQL: forward scalar carry columns through trailing WITH narrowing in reentry suffix (IC5 sub-residual A) #1272 (IC5 sub-A)
• Cypher WHERE: static validation gap on row-boolean shapes (OR / NOT / XOR among row predicates) — emergent from #1217 Earley swap #1219 closed

Priority & Parallelization

2-worker mode

• Worker A: GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311 (T5 under Meta: #1046 native type-system staged decomposition program #1262)
• Worker B: Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics #989 follow-through

3-worker mode

• Worker A: GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311 (T5 under Meta: #1046 native type-system staged decomposition program #1262)
• Worker B: Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics #989 follow-through
• Worker C: next queued native-types follow-up after GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311 lands

4-worker mode

• Worker A: GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311 (T5 under Meta: #1046 native type-system staged decomposition program #1262)
• Worker B: Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics #989 follow-through
• Worker C: [FEA] GFQL native type system: schemas, inference, validation, and Arrow representation #1046 follow-up scoping/research
• Worker D: CI-consolidation follow-up if split from GFQL native types T5: adoption gates, strict/permissive rollout, and CI/docs receipts #1311

Related

• Active execution DAG: Meta: GFQL compiler/planner post-M4 execution DAG (active wave) #1259
• Row-carrier umbrella: Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics #989
• Native type-system meta: Meta: #1046 native type-system staged decomposition program #1262

[lmeyerov]

New measured benchmark state from pyg-bench as of March 31, 2026:

• official SNB query floor measured: 7 (interactive-short-1/2/3/4/5/6 + interactive-complex-2)
• GFQL: ok=2, supported_with_workaround=5
• Cypher: ok=3, blocked=3, semantic_mismatch=1

New issue from that floor:

• #994 direct Cypher undirected peer binding mismatch on incoming edges
  • blocks interactive-short-3 correctness even though the query now executes

Current local next queue from the translation analyzer:

1. interactive-complex-8
2. interactive-short-7
3. interactive-complex-13
4. interactive-complex-4

[lmeyerov]

Priority recommendation from the current measured SNB benchmark floor (8 official queries measured on March 31, 2026):

Coverage unlock order:

1. #981 + #880 together
   • biggest unlock by far: each touches 13 official SNB queries
   • current measured manifestations include:
     • interactive-complex-2 blocked on direct Cypher multi-binding projection
     • interactive-complex-8 semantic corruption on direct Cypher row lowering
     • multiple GFQL paths still relying on local join workarounds instead of direct row materialization
2. #973
   • next official-query unlock for connected varlen patterns
   • currently blocks the official interactive-short-2 / interactive-short-6 family
3. #982
   • narrower than the row-binding work, but still needed to remove the current direct GFQL edge-projection workaround on interactive-short-3

Run in parallel as a correctness fix:

• #994
  • narrower query impact than the three unlocks above, but it is a wrong-answer bug in a query shape that already executes
  • current measured manifestation: interactive-short-3 returns the seed node as the peer alias on incoming undirected KNOWS edges

So the benchmark recommendation is:

• main unlock track: #981/#880 -> #973 -> #982
• correctness fast-path: #994 in parallel

[lmeyerov]

New measured benchmark state from pyg-bench as of March 31, 2026 after adding interactive-short-7:

• official SNB query floor measured: 9
  • interactive-short-1/2/3/4/5/6/7
  • interactive-complex-2/8
• GFQL: ok=2, supported_with_workaround=7
• Cypher: ok=3, blocked=4, semantic_mismatch=2

Important change: interactive-short-7 is the first measured query that clearly opens a new post-row-shaping feature lane.

Measured result:

• GFQL: workaround-backed partial
• direct Cypher: primitive_gap via new issue #996
  • MATCH ... OPTIONAL MATCH ... RETURN CASE ...
  • current error: Only node-only pre-binding MATCH clauses are supported before the final connected MATCH in this phase

Updated benchmark priority recommendation:

1. #981 + #880
2. #973
3. #982
4. #996 as the first clearly new feature-lowering lane beyond row shaping
5. #994 in parallel as the narrower wrong-answer bug

Current next translation queue:

1. interactive-complex-13
2. interactive-complex-4
3. interactive-complex-7
4. interactive-complex-11

[lmeyerov]

Updated benchmark coverage status from pyg-bench on March 31, 2026 using explicit pygraphistry master snapshot 7095102df:

• SNB Interactive official query texts inventoried: 29
• Measured today: 12
• Current measured floor: interactive-short-1/2/3/4/5/6/7 plus interactive-complex-2/4/8/11/13
• Dataset-backed GFQL status on the measured floor: ok=2, supported_with_workaround=10
• Dataset-backed Cypher status on the measured floor: ok=3, blocked=7, semantic_mismatch=2

Newly added measured queries in this pass:

• interactive-complex-4 / new-topics: GFQL workaround passes; direct Cypher blocked by #981
• interactive-complex-11 / job-referral: GFQL workaround passes; direct Cypher blocked by #981

Updated next translation queue from the analyzer:

1. interactive-complex-3
2. interactive-complex-7
3. interactive-complex-5
4. interactive-complex-1
5. interactive-complex-10

Priority order still holds: #981/#880, then #973, then #982, with #996, #997, and #994 as the next measured side lanes.

[lmeyerov]

Updated measured benchmark state from pyg-bench on March 31, 2026 after adding interactive-complex-3 and reproducing the expanded floor on dgx-spark CPU:

• SNB Interactive official query texts inventoried: 29
• measured today: 14
• current measured floor:
  • interactive-short-1/2/3/4/5/6/7
  • interactive-complex-2/3/4/7/8/11/13
• dataset-backed GFQL status on the measured floor: ok=2, supported_with_workaround=12
• dataset-backed Cypher status on the measured floor: ok=3, blocked=9, semantic_mismatch=2
• latest remote reproduction: dgx-spark CPU run snb-interactive-358480dd77
  • status mix: ok=5, partial=12, planner_gap=3, primitive_gap=4, result_mismatch=2, parser_gap=2

New measured query in this pass:

• interactive-complex-3 / cross-country-messages
  • GFQL: workaround-backed partial
  • direct Cypher: new issue #999
    • multi-stage MATCH ... WITH ... MATCH ... WITH ... MATCH read-query reentry
    • current error: Cypher WITH after post-WITH MATCH is not yet supported in the current GFQL Cypher compiler

Updated benchmark unlock order from the current floor:

1. #981 + #880
2. #999
3. #973
4. #982
5. #996
6. #997
7. #998
8. #994 in parallel as the narrower wrong-answer fix

Updated next translation queue from the analyzer:

1. interactive-complex-1
2. interactive-complex-10
3. interactive-complex-12
4. interactive-complex-5
5. interactive-complex-9
6. interactive-complex-6

[lmeyerov]

Benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 15/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v11/
• Current remote status mix on the 15-query floor: ok=5, partial=13, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=2.

New measured blocker from interactive-complex-9 (Recent messages by friends or friends of friends):

• direct Cypher is blocked by WITH collect(...) -> UNWIND -> MATCH continuation after graph aliases are introduced
• tracked separately in #1000
• GFQL still covers the query via a 1..2-hop friend plus local message-join workaround under #880

The current measured unlock order is now:

1. #981 + #880
2. #1000 + #999
3. #973
4. #982
5. #996 / #997 / #998
6. #994

[lmeyerov]

Another benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 16/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v12/
• Current remote status mix on the 16-query floor: ok=5, partial=14, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=3.

New measured query added:

• interactive-complex-6 (Tag co-occurrence)
  • GFQL: partial via 1..2-hop friend plus local tag co-occurrence aggregation workaround under #880
  • Cypher: parser_gap under #1000

Analyzer state after adding complex-6 and complex-9:

• #1000 now affects 6 official queries with 2 already measured
• next lowest-novelty queue is now interactive-complex-12, interactive-complex-5, interactive-complex-1, interactive-complex-10

[lmeyerov]

Benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 17/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v13/
• Current remote status mix on the 17-query floor: ok=5, partial=15, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=4.

New measured query added:

• interactive-complex-12 (Expert search)
  • GFQL: partial via direct-friend plus local tag-class-expanded reply aggregation workaround under #880
  • Cypher: parser_gap under #983

Important correction:

• #984 improved the earlier REPLY_OF*0.. manifestation, but #983 remains active for relationship-alternation zero-hop forms such as [:HAS_TYPE|IS_SUBCLASS_OF*0..].

Next queue from the analyzer is now:

• interactive-complex-5
• interactive-complex-1
• interactive-complex-10
• interactive-complex-14

[lmeyerov]

Benchmark coverage update from pyg-bench on March 31, 2026:

• official SNB Interactive query inventory: 29
• measured floor: 18
• backend status on the measured floor:
  • GFQL: ok=2, supported_with_workaround=16
  • direct Cypher: ok=3, blocked=13, semantic_mismatch=2
• latest remote conformance artifact: dgx-spark-snb-interactive-master-7095102df-v14
  • status mix: ok=5, partial=16, planner_gap=3, primitive_gap=6, result_mismatch=2, parser_gap=4

New measured query this pass:

• interactive-complex-5 / new-groups
  • GFQL: partial via friend-membership + forum-post-count workaround
  • direct Cypher: blocked by existing #999 lane, no new issue family needed

Current read-side next queue from the translation analyzer:

1. interactive-complex-1
2. interactive-complex-10
3. interactive-complex-14

[lmeyerov]

Benchmark coverage update from pyg-bench on March 31, 2026:

• official SNB Interactive query inventory: 29
• measured floor: 19
• backend status on the measured floor:
  • GFQL: ok=2, supported_with_workaround=17
  • direct Cypher: ok=3, blocked=14, semantic_mismatch=2
• latest remote conformance artifact: dgx-spark-snb-interactive-master-7095102df-v15
  • status mix: ok=5, partial=17, planner_gap=3, primitive_gap=6, result_mismatch=2, parser_gap=5

New measured query this pass:

• interactive-complex-1 / named-shortest-friends
  • GFQL: partial via bounded-shortest-path + profile-history join workaround
  • direct Cypher: blocked by confirmed #997 shortestPath(...) parser/front-end lane

Current read-side next queue from the translation analyzer:

1. interactive-complex-10
2. interactive-complex-14

[lmeyerov]

Benchmark status update from pyg-bench after rerunning the measured #1000 lane against isolated ambient graphistry v0.53.11.

Artifact:

• results/runs/local-snb-interactive-v05311-rerun/

Outcome split:

• interactive-complex-9 / recent-network-messages now clears the old WITH collect(distinct friend) -> UNWIND -> MATCH blocker and fails later at #981
• interactive-complex-6 / tag-cooccurrence still reproduces #1000, but on a narrower carried-scalar shape:
  • WITH knownTagId, collect(distinct friend) AS friends
  • UNWIND friends AS f
  • MATCH ...

Practical priority implication for benchmark coverage:

1. #981/#880 stay the main unlocks
2. #999 remains the broader MATCH/WITH/MATCH continuation family
3. #1000 is now a narrower residual benchmark blocker, not a blanket blocker for every measured WITH -> UNWIND -> MATCH query

[lmeyerov]

Benchmark-driven priority update from pyg-bench as of 2026-03-31.

Measured state:

• DGX CPU sf1 core slice is still badly behind current baselines:
  • PyGraphistry GFQL: 13037.569 ms
  • PyGraphistry Cypher: 6081.106 ms
  • Kuzu: 38.894 ms
  • Neo4j: 342.673 ms
• DGX GPU now has a completed SNB-shaped subset, but it is still conformance-limited:
  • single-query GFQL seed-lookup: 221.807 ms
  • core 3-query GFQL total: 612.941 ms
  • core 3-query Cypher total: 565.969 ms
  • still not apples-to-apples because one-hop-expand mismatches and expand-order-limit still lands on #973

What the language/runtime team should prioritize first:

1. #880 + #981
   • not just for coverage; the current benchmark path is forced into multi-pass local row joins and that is killing both correctness and performance.
2. #994
   • wrong-answer bug on interactive-short-3 / one-hop-expand; until this is fixed, the core comparison slice is semantically dirty.
3. #973
   • still blocks official interactive-short-2 / expand-order-limit on the direct Cypher path.
4. #982
   • still forces the GFQL side into the edge-row workaround path for interactive-short-3.

Important nuance: not all current slowness is a PyGraphistry issue.

• pyg-bench is still doing too much CPU-side fixture prep before queries.
• We fixed one harness bug where narrow GPU runs still instantiated unrelated probe specs.
• The remaining big bench-side task is reducing CPU-heavy fixture prep and keeping GPU slices narrow until semantics are clean.

So the short version is:

• for PyGraphistry, prioritize #880/#981, then #994, then #973, then #982
• for pyg-bench, we will keep reducing adapter/harness overhead so runtime numbers are less polluted by bench-side workarounds

[lmeyerov]

Warmup-aware DGX sf1 core slice update from pyg-bench.

Methodology:

• load/setup split from query timing
• untimed warmup rounds = 1
• PyGraphistry rows remain official-query-grounded custom adapter
• Kuzu and Neo4j rows remain custom reduced baseline

Current steady-state query-only numbers on the reduced 3-intent slice:

• PyGraphistry CPU: 9350.831 ms
  • direct query: 2252.492 ms
  • result materialization: 0.193 ms
  • adapter overhead: 7098.146 ms
• PyGraphistry GPU: 530.804 ms
  • direct query: 178.301 ms
  • result materialization: 4.293 ms
  • adapter overhead: 348.209 ms
• Kuzu CPU: 32.996 ms
• Neo4j CPU: 41.937 ms

Important read:

• GPU is materially better than PyGraphistry CPU on this slice.
• The current PyGraphistry timed path is still dominated by workaround cost, not direct engine execution.
• On the current 3-intent slice, the only direct GFQL timing contribution is still the reduced expand-order-limit lane.
• The core blockers are still product-side: #880, #981, #982, plus correctness gap #994. The benchmark harness is cleaner now, so the remaining gap is harder to dismiss as measurement noise.

Latest artifacts in pyg-bench:

• CPU: results/runs/dgx-spark-snb-interactive-core-sf1-timing-v5/
• GPU: results/runs/dgx-spark-snb-interactive-core-sf1-gpu-timing-v9/
• Kuzu: results/runs/dgx-spark-kuzu-snb-conformance-sf1-timing-v4/
• Neo4j: results/runs/dgx-spark-neo4j-snb-conformance-sf1-timing-v4/

[lmeyerov]

Update: v0.53.15 shipped (#994, #997, #998)

PR #1009 merged and tagged v0.53.15.

Resolved

• Cypher undirected MATCH can bind the seed node as the peer alias on incoming edges #994 (correctness): Undirected MATCH wrong-answer fix — orient_edges now correctly doubles undirected edges in both directions for connected bindings.
• Cypher parser rejects shortestPath((a)-[:REL*]-(b)) official shortest-path syntax #997 (parser): shortestPath() / allShortestPaths() now parse and fail-fast with clear validation error. Execution tracked in feat(cypher): implement shortestPath execution with igraph/cugraph backends #1010.
• Cypher parser rejects boolean pattern existence expressions in RETURN #998 (parser): Pattern existence expressions (not((a)-[:R]-(b)), exists { ... }) now detected and fail-fast with clear validation error.

Test coverage added

• 14 new tests including undirected edge cases (incoming, outgoing, bidirectional, multi-edge, star, self-loop), shortestPath/allShortestPaths fail-fast, pattern existence fail-fast (RETURN + WHERE), and historic regression for parameterized LIMIT in reentry.
• GPU validated on dgx-spark (pandas + cudf).

Impact on measured floor

• interactive-short-3 (Cypher undirected MATCH can bind the seed node as the peer alias on incoming edges #994) should now return correct rows on direct Cypher path
• interactive-complex-13 (Cypher parser rejects shortestPath((a)-[:REL*]-(b)) official shortest-path syntax #997) now fails with clear message instead of generic syntax error
• interactive-complex-7 (Cypher parser rejects boolean pattern existence expressions in RETURN #998) now fails with clear message instead of generic syntax error

Remaining top blockers (unchanged)

1. Direct Cypher row lowering rejects scalar multi-alias projections from a single MATCH #981/[FEA] GFQL bindings_table (row bindings) from chain using Yannakakis prune #880 — multi-binding row materialization (13 queries)
2. Cypher multihop: expand direct variable-length support beyond endpoint-only patterns #973 — connected-pattern variable-length lowering (4 queries) — partially addressed by feat(cypher): support variable-length rels in connected patterns (#973) #995
3. AST row select cannot reference edge alias properties after traversal #982 — native AST edge alias projection (5 queries)
4. feat(cypher): implement shortestPath execution with igraph/cugraph backends #1010 — shortestPath execution (blocked by feat(cypher): support comma-separated MATCH patterns (cartesian product) #990 path variable support)