Subqueries blog post

website/blog/posts/2026-04-27-subqueries.md

@@ -0,0 +1,178 @@
+---
+title: "Subqueries\u2014making sync work in practice"
+description: >-
+  Subqueries let Electric shapes express relational filtering in SQL.
+  Electric 1.6 keeps complex AND/OR/NOT expressions incremental too, so
+  large shapes stay fast.
+excerpt: >-
+  Sync only works in real apps if it can follow relationships.
+  Subqueries let Electric express relational filters for each user in SQL,
+  and Electric 1.6 keeps complex expressions incremental too.
+authors: [rob, icehaunter]
+image: /img/blog/subqueries/header.jpg
+tags: [shapes, postgres-sync, release]
+outline: [2, 3]
+post: true
+published: true
+---
+
+Sync is what makes apps feel instant. The data is already there when a screen renders. Another user changes something and your UI stays current. You can refresh, reconnect, switch devices, and keep going.
+
+That is the broad pitch. We have written more about how [sync replaces data fetching](/blog/2025/04/22/untangling-llm-spaghetti) and why it is the right foundation for [collaborative, real-time apps](/blog/2025/04/09/building-ai-apps-on-sync).
+
+But there is a more practical question underneath all of it:
+
+Which rows should this client actually receive?
+
+In simple demos, a column filter is enough. In real systems, the rule usually lives in other tables. A document is visible because you own it, or because it was shared with you, or because you belong to the workspace that contains it. Comments sync because their issue belongs to a project you can access. Invoice line items sync because their parent invoice does.
+
+This is where subqueries matter.
+
+:::info Subqueries are now available for everyone in Electric 1.6
+[Try subqueries in Cloud](https://dashboard.electric-sql.cloud/).
+:::
+
+## Query-driven sync
+
+Shapes are Electric's primitive for partial replication: a table and a `WHERE` clause. Define the subset once and Electric keeps that subset synced.
+
+For flat cases, the filter is simple:
+
+```sql
+owner_id = $1
+```
+
+And here is how that looks in TanStack DB:
+
+```ts
+const documentsCollection = createCollection(
+  electricCollectionOptions({
+    id: 'my-documents',
+    shapeOptions: {
+      url: `${ELECTRIC_URL}/v1/shape`,
+      params: {
+        table: 'documents',
+        where: 'owner_id = $1',
+        params: { '1': currentUserId },
+      },
+    },
+  })
+)
+```
+
+Parameters (`$1`) are bound per client, so the same shape definition can serve different data to different users.
+
+But real apps do not stay flat for long. Access control, tenant membership, and parent-child data all pull in related tables. Subqueries let you express those rules directly in SQL.
+
+Sync documents for workspaces this user belongs to:
+
+```sql
+workspace_id IN (
+  SELECT workspace_id FROM workspace_members
+  WHERE user_id = $1
+)
+```
+
+
+You can combine relational checks with ordinary predicates. For example, sync documents that I own, plus documents shared with me:
+
+```sql
+owner_id = $1
+OR id IN (
+  SELECT document_id FROM document_shares
+  WHERE shared_with = $1
+)
+```
+
+You can also traverse multiple hops. Sync comments for a project by walking from comments to issues to tasks:
+
+```sql
+issue_id IN (
+  SELECT id FROM issues WHERE task_id IN (
+    SELECT id FROM tasks WHERE project_id = $1
+  )
+)
+```
+
+This is mundane SQL. That is the point.
+
+The rule stays close to the data, where you already reason about memberships, shares, and relationships. Electric evaluates it server-side and keeps only the matching rows on each client.
+
+See the [WHERE clause docs](/docs/guides/shapes#where-clause) for the full reference on supported operators and subquery patterns.
+
+:::info
+Subqueries are available on [Electric Cloud](/cloud) and are included in the [Pro, Scale, and Enterprise plans](/pricing).
+:::
+
+## What changed in Electric 1.6
+
+Subqueries are not new. We have supported them for a while, behind feature flags, and they have already been battle tested by customers in production.
+
+Electric 1.6 is the release that closes one of the last awkward cases.
+
+Subqueries already supported incremental sync for simple expressions. Complex expressions using `AND`, `OR`, and `NOT` also worked, but when the subquery result changed Electric could fall back to a full resync. On small shapes you might never notice. On large ones you would feel it as lag between a write and the UI catching up.
+
+With 1.6, those complex expressions stay incremental too. When memberships change, shares are granted, or related rows move in or out of scope, Electric now syncs only the affected rows. Large shapes keep the low-latency behavior that makes sync useful in the first place.
+
+That is why we now consider subqueries suitable for general use.
+
+This release also includes a client protocol update needed for the new incremental behavior. The feature flags are unchanged for now and we will remove them once we are confident clients have moved onto the newer protocol.
+
+## Using it now
+
+Here is the shared-documents example wired into a TanStack DB collection:
+
+```ts
+import { electricCollectionOptions } from '@tanstack/electric-db-collection'
+import { createCollection } from '@tanstack/react-db'
+
+const documentsCollection = createCollection(
+  electricCollectionOptions({
+    id: 'my-documents',
+    shapeOptions: {
+      url: `${ELECTRIC_URL}/v1/shape`,
+      params: {
+        table: 'documents',
+        where: `
+          owner_id = $1
+          OR id IN (
+            SELECT document_id FROM document_shares
+            WHERE shared_with = $1
+          )
+        `,
+        params: { '1': currentUserId },
+      },
+    },
+  })
+)
+```
+
+Update to the latest packages:
+
+```sh
+npm install @tanstack/db@latest @tanstack/electric-db-collection@latest
+```
+
+Subqueries remain behind the same feature flags as before:
+
+```sh
+ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries
+```
+
+See the [WHERE clause docs](/docs/guides/shapes#where-clause) for the full reference.
+
+## Summary
+
+The interesting part of sync is not a nicer `fetch()`. It is what you get once the right data is already local: live UIs, collaboration, resilient apps, instant navigation, fewer loading states.
+
+But none of that survives contact with production unless sync can follow your actual data model. The moment you have shared documents, org membership, private projects, child records, or exclusions, a simple column filter stops being enough.
+
+Subqueries are what make shapes fit real applications. They let you describe who can see a row, which child rows come along with a parent, how multiple access paths compose, and how exclusions or overrides work.
+
+We now consider them ready for general use, and they are available to everyone in Electric 1.6.
+
+If you want to use them on Cloud today, get in touch and we can enable them on your project. The remaining rollout flags are temporary, and we expect to lift them over the next few weeks.
+
+***
+
+[Docs](/docs/guides/shapes#where-clause) · [Cloud](/cloud) · [Discord](https://discord.electric-sql.com)

website/data/blog/authors.yaml

@@ -51,3 +51,9 @@ tdrz:
   title: Founding Engineer
   image: /img/team/tudor.jpg
   url: /about/team#tudor
+
+rob:
+  name: Rob A'Court
+  title: Founding Engineer
+  image: /img/team/rob.jpg
+  url: /about/team#rob

website/docs/api/config.md

@@ -453,7 +453,7 @@ Consumer processes are partitioned across some number of supervisors to improve
 
 ## Feature Flags
 
-Feature flags enable experimental or advanced features that are not yet enabled by default in production.
+Feature flags enable advanced features and staged rollouts for capabilities that are not yet enabled by default in production.
 
 ### ELECTRIC_FEATURE_FLAGS
 
@@ -465,10 +465,14 @@ Feature flags enable experimental or advanced features that are not yet enabled
 **Available flags:**
 
 - `allow_subqueries` - Enables subquery support in shape WHERE clauses
-- `tagged_subqueries` - Enables improved multi-level dependency handling
+- `tagged_subqueries` - Enables incremental subquery move handling, including compound boolean expressions with compatible clients
 
 </EnvVarConfig>
 
+:::warning Client compatibility
+Electric 1.6's incremental handling for compound subquery expressions changes the client protocol. Upgrade clients before enabling the server rollout. TanStack DB clients need `@tanstack/db >= 0.6.2` and `@tanstack/electric-db-collection >= 0.3.0`.
+:::
+
 ### allow_subqueries
 
 Enables support for subqueries in the WHERE clause of [shape](/docs/guides/shapes) definitions. When enabled, you can use queries in the form:
@@ -479,15 +483,17 @@ WHERE id IN (SELECT user_id FROM memberships WHERE org_id = 'org_123')
 
 This allows creating shapes that filter based on related data in other tables, enabling more complex data synchronization patterns.
 
-**Status:** Experimental. Disabled by default in production.
+**Status:** General use. Disabled by default in production until enabled with `ELECTRIC_FEATURE_FLAGS`.
 
 ### tagged_subqueries
 
-Subqueries create dependency trees between shapes. Without this flag, when data moves into or out of a dependent shape, the shape is invalidated (returning a 409). With this flag enabled, move operations are handled correctly without invalidation.
+Subqueries create dependency trees between shapes. This flag enables incremental move handling when dependency rows change, including compound `WHERE` expressions that combine subqueries with `AND`, `OR`, and `NOT`.
+
+Before Electric 1.6, complex boolean combinations around subqueries could still invalidate the shape and return a `409` on a move. With this flag enabled and compatible clients, those changes are reconciled in-stream instead.
 
 See [discussion #2931](https://github.com/electric-sql/electric/discussions/2931) for more details about this feature.
 
-**Status:** Experimental. Disabled by default in production. Requires `allow_subqueries` to be enabled.
+**Status:** Rollout flag for subquery move handling. Disabled by default in production. Requires `allow_subqueries` to be enabled.
 
 ## Caching
 

website/docs/guides/shapes.md

@@ -51,7 +51,7 @@ Shapes are defined by:
 A shape contains all of the rows in the table that match the where clause, if provided. If a columns clause is provided, the synced rows will only contain those selected columns.
 
 > [!Warning] Limitations
-> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries-experimental) to filter based on related data. Shape definitions are [immutable](#immutable).
+> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries) to filter based on related data. Shape definitions are [immutable](#immutable).
 
 > [!Warning] Security
 > Production apps should request shapes through your backend API for authorization and security. See the [auth guide](/docs/guides/auth).
@@ -116,9 +116,12 @@ Where clauses have the following constraints:
 
 1. can't use non-deterministic SQL functions like `count()` or `now()`
 
-#### Subqueries (experimental)
+<a id="subqueries-experimental"></a>
+#### Subqueries
 
-Electric supports subqueries in where clauses, allowing you to filter rows based on data in other tables. This enables cross-table filtering patterns—for example, syncing only users who belong to a specific organization:
+Electric supports subqueries in where clauses, allowing you to filter rows based on data in other tables. This makes shapes suitable for general-use relational filtering, such as memberships, sharing rules, parent-child traversal, and exclusions.
+
+For example, you can sync only users who belong to a specific organization:
 
 ```ts
 import { electricCollectionOptions } from '@tanstack/electric-db-collection'
@@ -139,6 +142,30 @@ const usersCollection = createCollection(
 )
 ```
 
+Or combine subqueries with boolean logic to express more realistic access rules:
+
+```ts
+const documentsCollection = createCollection(
+  electricCollectionOptions({
+    id: 'visible-documents',
+    shapeOptions: {
+      url: 'http://localhost:3000/v1/shape',
+      params: {
+        table: 'documents',
+        where: `
+          owner_id = $1
+          OR id IN (
+            SELECT document_id FROM document_shares
+            WHERE shared_with = $1
+          )
+        `,
+        params: { '1': 'user_123' },
+      },
+    },
+  })
+)
+```
+
 Or with `ShapeStream` directly:
 
 ```ts
@@ -154,10 +181,12 @@ const stream = new ShapeStream({
 const shape = new Shape(stream)
 ```
 
-When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (e.g., a project becomes archived), rows will automatically move in or out of the shape without the row itself being modified.
+When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (for example, a project becomes archived or a membership row is added), rows will automatically move in or out of the shape without the root row itself being modified.
+
+Electric 1.6 keeps these moves incremental even for compound expressions that use `AND`, `OR`, and `NOT` around subqueries. In older releases those cases could return `409` and force a full resync of the shape.
 
-:::warning Experimental feature
-Subqueries require enabling feature flags. Set `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries` to enable. See the [configuration docs](/docs/api/config#allow_subqueries) for details.
+:::info Feature flags
+Subqueries are currently enabled using `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries`. The flags are unchanged; they gate rollout rather than a separate syntax or API. See the [configuration docs](/docs/api/config#allow_subqueries) for details.
 :::
 
 When constructing a where clause with user input as a filter, it's recommended to use a positional placeholder (`$1`) to avoid
@@ -399,8 +428,9 @@ With non-optimized where clauses, throughput is inversely proportional to the nu
 
 With optimized where clauses, Electric can evaluate millions of clauses at once and maintain a consistent throughput of ~5,000 row changes per second **no matter how many shapes you have**. If you have 10 shapes, Electric can process 5,000 changes per second. If you have 1,000 shapes, throughput remains at 5,000 changes per second.
 
-For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimized-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine.
+For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimised-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine.
 
+<a id="optimised-where-clauses"></a>
 ### Optimized where clauses
 
 We currently optimize the evaluation of the following clauses:
@@ -409,8 +439,17 @@ We currently optimize the evaluation of the following clauses:
   We optimize this by indexing shapes by their constant, allowing a single lookup to retrieve all
   shapes for that constant instead of evaluating the where clause for each shape.
   Note that this index is internal to Electric and unrelated to Postgres indexes.
-- `field = constant AND another_condition` - the `field = constant` part of the where clause is optimized as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however `field = const` matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through.
-- `a_non_optimized_condition AND field = constant` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first).
+- `constant = field` - as above, the order of the field and constant is not important (Electric will index by the field either way).
+- `array_field @> array_constant` - for array fields, we optimize the "contains" operator.
+- `array_constant <@ array_field` - as above, the reverse notation is also optimized.
+- `field IN list_constant` - so for example `status IN ('backlog', 'todo')` is optimized
+- `const = ANY(array_field)` - so for example `'todo' = ANY(statuses)` is optimized
+- `field IN (subquery)` - so for example `project_id IN (SELECT id FROM projects WHERE archived = false)` is optimized
+- `field_list IN (subquery)` - so for example `(project_id, status) IN (SELECT project_id, status FROM projects WHERE archived = false)` is optimized
+- `optimized_condition AND another_optimized_condition` - any of the optimized conditions can be combined with `AND` and still be optimised.
+- `optimized_condition OR another_optimized_condition` - the same for `OR`. This combining with `AND` and `OR` can be repeated to create complex binary expressions.
+- `optimized_condition AND non_optimized_condition` - the optimized condition will as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however if the optimized condition matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through.
+- `non_optimized_condition AND optimized_condition` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first).
 
 > [!Warning] Need additional where clause optimization?
 > We plan to optimize a much larger subset of Postgres where clauses. If you need a particular clause optimized, please [raise an issue on GitHub](https://github.com/electric-sql/electric) or [let us know on Discord](https://discord.electric-sql.com).
@@ -419,7 +458,7 @@ We currently optimize the evaluation of the following clauses:
 
 ### Single table
 
-Shapes sync data from a single table. While you can use [subqueries](#subqueries-experimental) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself.
+Shapes sync data from a single table. While you can use [subqueries](#subqueries) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself.
 
 For syncing related data across tables, you currently need to use multiple shapes. In the [old version of Electric](https://legacy.electric-sql.com/docs/usage/data-access/shapes), Shapes had an include tree that allowed you to sync nested relations. The new Electric has not yet implemented support for include trees.