Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions website/docs/api/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ Consumer processes are partitioned across some number of supervisors to improve

## Feature Flags

Feature flags enable experimental or advanced features that are not yet enabled by default in production.
Feature flags enable advanced features and staged rollouts for capabilities that are not yet enabled by default in production.

### ELECTRIC_FEATURE_FLAGS

Expand All @@ -464,11 +464,15 @@ Feature flags enable experimental or advanced features that are not yet enabled

**Available flags:**

- `allow_subqueries` - Enables subquery support in shape WHERE clauses
- `tagged_subqueries` - Enables improved multi-level dependency handling
- `allow_subqueries` - Enables preview subquery support in shape WHERE clauses
- `tagged_subqueries` - Enables preview incremental subquery move handling, including compound boolean expressions with compatible clients

</EnvVarConfig>

:::warning Client compatibility
Electric 1.6's incremental handling for compound subquery expressions changes the client protocol. Upgrade clients before enabling the server rollout. TanStack DB clients need `@tanstack/db >= 0.6.2` and `@tanstack/electric-db-collection >= 0.3.0`.
:::

### allow_subqueries

Enables support for subqueries in the WHERE clause of [shape](/docs/guides/shapes) definitions. When enabled, you can use queries in the form:
Expand All @@ -479,15 +483,17 @@ WHERE id IN (SELECT user_id FROM memberships WHERE org_id = 'org_123')

This allows creating shapes that filter based on related data in other tables, enabling more complex data synchronization patterns.

**Status:** Experimental. Disabled by default in production.
**Status:** Preview. Disabled by default in production until enabled with `ELECTRIC_FEATURE_FLAGS`.

### tagged_subqueries

Subqueries create dependency trees between shapes. Without this flag, when data moves into or out of a dependent shape, the shape is invalidated (returning a 409). With this flag enabled, move operations are handled correctly without invalidation.
Subqueries create dependency trees between shapes. This flag enables incremental move handling when dependency rows change, including compound `WHERE` expressions that combine subqueries with `AND`, `OR`, and `NOT`.

Before Electric 1.6, complex boolean combinations around subqueries could still invalidate the shape and return a `409` on a move. With this flag enabled and compatible clients, those changes are reconciled in-stream instead.

See [discussion #2931](https://github.com/electric-sql/electric/discussions/2931) for more details about this feature.

**Status:** Experimental. Disabled by default in production. Requires `allow_subqueries` to be enabled.
**Status:** Preview rollout flag for subquery move handling. Disabled by default in production. Requires `allow_subqueries` to be enabled.

## Caching

Expand Down
59 changes: 49 additions & 10 deletions website/docs/guides/shapes.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Shapes are defined by:
A shape contains all of the rows in the table that match the where clause, if provided. If a columns clause is provided, the synced rows will only contain those selected columns.

> [!Warning] Limitations
> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries-experimental) to filter based on related data. Shape definitions are [immutable](#immutable).
> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries) to filter based on related data. Shape definitions are [immutable](#immutable).

> [!Warning] Security
> Production apps should request shapes through your backend API for authorization and security. See the [auth guide](/docs/guides/auth).
Expand Down Expand Up @@ -116,9 +116,12 @@ Where clauses have the following constraints:

1. can't use non-deterministic SQL functions like `count()` or `now()`

#### Subqueries (experimental)
<a id="subqueries-experimental"></a>
#### Subqueries (preview)

Electric supports subqueries in where clauses, allowing you to filter rows based on data in other tables. This enables cross-table filtering patterns—for example, syncing only users who belong to a specific organization:
Electric has preview support for subqueries in where clauses, allowing you to filter rows based on data in other tables. This enables relational filtering patterns such as memberships, sharing rules, parent-child traversal, and exclusions while the feature remains gated behind flags.

For example, you can sync only users who belong to a specific organization:

```ts
import { electricCollectionOptions } from '@tanstack/electric-db-collection'
Expand All @@ -139,6 +142,30 @@ const usersCollection = createCollection(
)
```

Or combine subqueries with boolean logic to express more realistic access rules:

```ts
const documentsCollection = createCollection(
electricCollectionOptions({
id: 'visible-documents',
shapeOptions: {
url: 'http://localhost:3000/v1/shape',
params: {
table: 'documents',
where: `
owner_id = $1
OR id IN (
SELECT document_id FROM document_shares
WHERE shared_with = $1
)
`,
params: { '1': 'user_123' },
},
},
})
)
```

Or with `ShapeStream` directly:

```ts
Expand All @@ -154,10 +181,12 @@ const stream = new ShapeStream({
const shape = new Shape(stream)
```

When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (e.g., a project becomes archived), rows will automatically move in or out of the shape without the row itself being modified.
When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (for example, a project becomes archived or a membership row is added), rows will automatically move in or out of the shape without the root row itself being modified.

Electric 1.6 keeps these moves incremental even for compound expressions that use `AND`, `OR`, and `NOT` around subqueries. In older releases those cases could return `409` and force a full resync of the shape.

:::warning Experimental feature
Subqueries require enabling feature flags. Set `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries` to enable. See the [configuration docs](/docs/api/config#allow_subqueries) for details.
:::info Preview feature
Subqueries are currently in preview and are enabled using `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries`. The flags gate availability rather than a separate syntax or API. See the [configuration docs](/docs/api/config#allow_subqueries) for details.
:::

When constructing a where clause with user input as a filter, it's recommended to use a positional placeholder (`$1`) to avoid
Expand Down Expand Up @@ -399,8 +428,9 @@ With non-optimized where clauses, throughput is inversely proportional to the nu

With optimized where clauses, Electric can evaluate millions of clauses at once and maintain a consistent throughput of ~5,000 row changes per second **no matter how many shapes you have**. If you have 10 shapes, Electric can process 5,000 changes per second. If you have 1,000 shapes, throughput remains at 5,000 changes per second.

For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimized-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine.
For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimised-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine.

<a id="optimised-where-clauses"></a>
### Optimized where clauses

We currently optimize the evaluation of the following clauses:
Expand All @@ -409,8 +439,17 @@ We currently optimize the evaluation of the following clauses:
We optimize this by indexing shapes by their constant, allowing a single lookup to retrieve all
shapes for that constant instead of evaluating the where clause for each shape.
Note that this index is internal to Electric and unrelated to Postgres indexes.
- `field = constant AND another_condition` - the `field = constant` part of the where clause is optimized as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however `field = const` matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through.
- `a_non_optimized_condition AND field = constant` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first).
- `constant = field` - as above, the order of the field and constant is not important (Electric will index by the field either way).
- `array_field @> array_constant` - for array fields, we optimize the "contains" operator.
- `array_constant <@ array_field` - as above, the reverse notation is also optimized.
- `field IN list_constant` - so for example `status IN ('backlog', 'todo')` is optimized
- `const = ANY(array_field)` - so for example `'todo' = ANY(statuses)` is optimized
- `field IN (subquery)` - so for example `project_id IN (SELECT id FROM projects WHERE archived = false)` is optimized
- `field_list IN (subquery)` - so for example `(project_id, status) IN (SELECT project_id, status FROM projects WHERE archived = false)` is optimized
- `optimized_condition AND another_optimized_condition` - any of the optimized conditions can be combined with `AND` and still be optimised.
- `optimized_condition OR another_optimized_condition` - the same for `OR`. This combining with `AND` and `OR` can be repeated to create complex binary expressions.
- `optimized_condition AND non_optimized_condition` - the optimized condition will as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however if the optimized condition matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through.
- `non_optimized_condition AND optimized_condition` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first).

> [!Warning] Need additional where clause optimization?
> We plan to optimize a much larger subset of Postgres where clauses. If you need a particular clause optimized, please [raise an issue on GitHub](https://github.com/electric-sql/electric) or [let us know on Discord](https://discord.electric-sql.com).
Expand All @@ -419,7 +458,7 @@ We currently optimize the evaluation of the following clauses:

### Single table

Shapes sync data from a single table. While you can use [subqueries](#subqueries-experimental) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself.
Shapes sync data from a single table. While you can use [subqueries](#subqueries) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself.

For syncing related data across tables, you currently need to use multiple shapes. In the [old version of Electric](https://legacy.electric-sql.com/docs/usage/data-access/shapes), Shapes had an include tree that allowed you to sync nested relations. The new Electric has not yet implemented support for include trees.

Expand Down
7 changes: 4 additions & 3 deletions website/docs/reference/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Each shape in this benchmark is independent, ensuring that a write operation aff
The two graphs differ based on the type of where clause used for the shapes:

- **Top Graph:** The where clause is in the form `field = constant`, where each shape is assigned a unique constant. These types of where clause, along with
[other patterns](/docs/guides/shapes#optimised-where-clauses),
[other patterns](/docs/guides/shapes#optimized-where-clauses),
are optimised for high performance regardless of the number of shapes — analogous to having an index on the field. As shown in the graph, the latency remains consistently
flat at 6ms as the number of shapes increases. This 6ms latency includes 3ms for PostgreSQL to process the write operation and 3ms for Electric to propagate it.
We are actively working to optimise additional where clause types in the future.
Expand Down Expand Up @@ -182,6 +182,7 @@ In this benchmark there are a varying number of shapes with just one client subs

Latency and peak memory use rises linearly. Average memory use is flat.

<a id="_7-write-throughput-with-optimized-where-clauses"></a>
#### 7. Write throughput with optimised where clauses

<figure>
Expand All @@ -202,10 +203,10 @@ is using an optimised where clause, specifically `field = constant`.
> so that we can evaluate millions of where clauses at once, providing the where clauses follow various patterns, which we call optimised where clauses.
> `field = constant` is one of the patterns we optimise, we can evaluate millions of these where clauses at once by indexing the shapes based on the constant
> value for each shape. This index is internal to Electric, and nothing to do with Postgres indexes. It's a hashmap if you're interested.
> `field = const AND another_condition` is another pattern we optimise. We aim to optimise a large subset of Postgres where clauses in the future.
> `field = const AND another_condition` is another pattern we optimise, and some indexable `OR` combinations are now optimised too. This benchmark still measures the specific `field = constant` case shown above.
> Optimised where clauses mean that we can process writes in a quarter of a millisecond, regardless of how many shapes there are.
>
> For more information on optimised where clauses, see the [shape API](/docs/guides/shapes#optimised-where-clauses).
> For more information on optimised where clauses, see the [shape API](/docs/guides/shapes#optimized-where-clauses).

The top graph shows throughput for Postgres 14, the bottom graph for Postgres 15.

Expand Down
Loading