Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion site/docs/concepts/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
---
sidebar_position: 0
description: Learn key Estuary concepts to build your Data Flows, including captures, collections, materializations, connectors, schemas, derivations, and catalogs.
pagination_next: concepts/captures
---

# Concepts
# Core Concepts

Estuary helps you define data pipelines, known as **Data Flows**, that connect multiple data systems, APIs, and storage, and optionally transform data along the way.
Data Flows are defined by a set of **Data Flow specifications** and deployed using either the web application or the command-line interface.
Expand Down
4 changes: 4 additions & 0 deletions site/docs/concepts/advanced/evolutions.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Understand Estuary schema evolutions and how Data Flow specs are updated after collection changes, backfills, incompatible schema changes, and new bindings.
---

# Schema Evolution

**Schema evolutions** are a feature you can use to update your whole Data Flow to reflect edits to a collection, preventing your Data Flow from failing due to mismatched components.
Expand Down
4 changes: 4 additions & 0 deletions site/docs/concepts/advanced/journals.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Learn how Estuary journals store collection data as real-time logs, organized as fragment files in cloud storage. Understand physical partitions, hive layouts, and managing journals through spec files.
---

# Journals

:::tip
Expand Down
4 changes: 4 additions & 0 deletions site/docs/concepts/advanced/logs-stats.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Learn how Estuary stores task logs and stats in ops collections. Use these special collections to debug workflows, track throughput, and review metrics.
---

# Logs and Statistics

Estuary collects logs and statistics of catalog tasks to aid in debugging and refinement of your workflows.
Expand Down
4 changes: 4 additions & 0 deletions site/docs/concepts/advanced/projections.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Learn how to use Estuary projections to map field locations in JSON documents and logically partition collections.
---

# Projections

:::tip
Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/advanced/reduction-strategies/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Estuary's default reduction behaviors and available strategies to customize them
description: Learn how Estuary reduction strategies combine documents with shared keys using append, merge, set, sum, minimize, maximize, and first/last-write-wins behavior.
slug: /reference/reduction-strategies/
---

Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/advanced/reduction-strategies/append.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the append reduction strategy
description: Learn about Estuary's append reduction strategy to reduce arrays. Covers append reduction behavior and test examples.
slug: /reference/reduction-strategies/append/
sidebar_position: 1
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
description: Using conditionals statements to fine-tune reductions
description: Fine-tune reduction strategies with conditional statements. Use specific reduction strategies based on document shape or field values to customize behavior.
slug: /reference/reduction-strategies/composing-with-conditionals/
sidebar_position: 30
---

# Composing with conditionals
# Composing Reductions with Conditionals

Reduction strategies are JSON Schema [annotations](https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.7.7). As such, their applicability at a given document location can be controlled through the use of [conditional](https://json-schema.org/understanding-json-schema/reference/conditionals.html) keywords within the schema, like `oneOf` or `if/then/else`. This means Estuary’s built-in strategies can be combined with schema conditionals to construct a wider variety of custom reduction behaviors.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the firstWriteWins and lastWriteWins reduction strategies
description: Learn about Estuary's firstWriteWins and lastWriteWins reduction strategies to reduce document fields to their first or last values.
slug: /reference/reduction-strategies/firstwritewins-and-lastwritewins/
sidebar_position: 2
---
Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/advanced/reduction-strategies/merge.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the merge reduction strategy
description: Learn about Estuary's merge reduction strategy to reduce objects or arrays. Covers merge reduction behavior, merge keys, and test examples.
slug: /reference/reduction-strategies/merge/
sidebar_position: 3
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the minimize and maximize reduction strategies
description: Learn about Estuary's minimize and maximize reduction strategies to reduce document fields to their smallest or largest values.
slug: /reference/reduction-strategies/minimize-and-maximize/
sidebar_position: 4
---
Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/advanced/reduction-strategies/set.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the set reduction strategy
description: Learn about Estuary's set reduction strategy to interpret document locations as set updates. Covers set reduction behavior, reduction keys, and test examples.
slug: /reference/reduction-strategies/set/
sidebar_position: 5
---
Expand Down
2 changes: 1 addition & 1 deletion site/docs/concepts/advanced/reduction-strategies/sum.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Using the sum reduction strategy
description: Use Estuary's sum reduction strategy to reduce number fields by adding their values.
slug: /reference/reduction-strategies/sum/
sidebar_position: 6
---
Expand Down
4 changes: 4 additions & 0 deletions site/docs/concepts/advanced/shards.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Understand Estuary task shards for captures, derivations, and materializations, including scaling, transactions, and recovery logs.
---

# Task Shards

Catalog [tasks](../README.md#tasks) — captures, derivations, and materializations —
Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/captures.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 1
description: Learn how Estuary captures ingest source data into collections using discovery, auto-updates, and schema evolution. See an example connector specification to define the endpoint and bindings.
---

# Captures

A **capture** is how Estuary ingests data from an external source.
Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/catalogs.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 2
description: Learn how Estuary catalogs organize captures, collections, materializations, derivations, schemas, and tests, using namespaces.
---

# Catalog

The **catalog** is the set of published entities that comprise all **Data Flows**: captures, materializations, derivations, collections, schemas, tests, and more.
Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/collections.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 3
description: Learn how Estuary collections store real-time data lakes of JSON documents. See a specification example to define keys, schemas, projections, and derivations.
---

# Collections

The documents of your Data Flows are stored in **collections**:
Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/connectors.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 4
description: Learn how Estuary connectors work for captures and materializations, including configuration, Docker images, flowctl, SSH tunneling, and open architecture.
---

import Tabs from '@theme/Tabs';
Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/derivations.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 5
description: Create continuous transformations in Estuary using derivations. Supports SQL, TypeScript, and Python. Join collections or modify data in-flight.
---

import Mermaid from '@theme/Mermaid';
Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/flowctl.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 6
description: Understand Estuary’s flowctl CLI to manage data flows programmatically. Commands let you work with catalogs, specification files and drafts, logs, and preview functionality.
---
import Mermaid from '@theme/Mermaid';

Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/import.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 7
description: Learn how imports resolve Estuary Data Flow specs across files, fetch resources locally, and are used with tranformation resources.
---

# Imports

When you work on a draft Data Flow [using `flowctl draft`](../guides/flowctl/edit-draft-from-webapp.md),
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Understand Estuary’s materialization protocol for transactions, implementation patterns that support exactly-once semantics, and sequence phases.
slug: /reference/Connectors/materialization-protocol/
---

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Use Estuary materialization triggers to fire webhooks after a materialization commits new data. Configure webhook URLs, headers, payloads, and retries to integrate with Slack, dbt, and GitHub.
slug: /concepts/materialization-triggers/
---

Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/materialization/materialization.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Learn how Estuary's materializations push data collections to destinations using merge or delta updates, projected fields, and schema naming rules. See an example connector specification to define the endpoint and bindings.
slug: /concepts/materialization/
---

Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/schemas.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 9
description: Learn how Estuary schemas define document structure and provide validation. Automatically infer schemas and customize behavior with reductions and other annotations.
---

# Schemas

Estuary's documents and [collections](collections.md) always have an associated schema
Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/storage-mappings.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 10
description: Learn how Estuary storage mappings route collection data and recovery logs to cloud buckets, with retention policy options and the ability to configure by prefix.
---

# Storage Mappings
Expand Down
2 changes: 2 additions & 0 deletions site/docs/concepts/tests.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
sidebar_position: 11
description: Learn how to use tests to validate Data Flow behavior in Estuary with ingest and verify steps. Handle partition selectors, test reductions, and reuse common fixtures.
---

# Tests

Estuary's web application automatically performs basic tests to validate the configurations of captures and materializations.
Expand Down
1 change: 1 addition & 0 deletions site/docs/concepts/web-app.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 12
description: Learn how to use the Estuary dashboard to create, manage, and monitor captures, collections, materializations, users, and billing details.
---

# Web Application
Expand Down
1 change: 1 addition & 0 deletions site/docs/features/configuring-task-shards.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 2
description: Tune Estuary task shard behavior by configuring log levels, minimum and maximum transaction durations, and disabling processing.
slug: /reference/Configuring-task-shards/
---

Expand Down
1 change: 1 addition & 0 deletions site/docs/features/mcp-integration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 1
description: Connect Estuary docs to your IDE or AI tool with MCP setup guides for Cursor, VS Code, Claude, ChatGPT Desktop, and other clients.
---

# MCP Integration
Expand Down
1 change: 1 addition & 0 deletions site/docs/features/openmetrics-api.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Use Estuary’s OpenMetrics API to track capture, derivation, and materialization metrics. Integrate with observability platforms like Prometheus and Datadog.
slug: /reference/openmetrics-api/
---

Expand Down
3 changes: 3 additions & 0 deletions site/docs/features/redaction.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
description: Use Estuary redaction to block or hash sensitive capture fields at write time. Guide covers dashboard setup, spec config using schema annotations, and custom salts.
---

# Redaction

Expand Down
1 change: 1 addition & 0 deletions site/docs/features/time-travel.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 1
description: Restrict Estuary materializations to a date range with notBefore and notAfter filters in the dashboard or Data Flow specification file.
slug: /reference/time-travel/
---

Expand Down
1 change: 1 addition & 0 deletions site/docs/features/using-dekaf.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Read Estuary collections using Kafka consumers with Dekaf. Includes topic emulation, schema registry support, connection details, and examples.
slug: /guides/dekaf_reading_collections_from_kafka/
---

Expand Down
1 change: 1 addition & 0 deletions site/docs/features/working-logs-stats.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 3
description: Access Estuary logs and stats from the dashboard or using flowctl. Debug tasks, inspect shards, and monitor bytes and docs.
slug: /reference/working-logs-stats/
---

Expand Down
2 changes: 1 addition & 1 deletion site/docs/getting-started/comparisons.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
sidebar_position: 4
description: High level explanations of Estuary in terms of the systems you already know
description: High level explanations of Estuary in terms of the systems you already know. Compare Estuary with batch ELT systems, streaming solutions, Kafka, Spark, and more.
---

# Comparisons
Expand Down
1 change: 1 addition & 0 deletions site/docs/getting-started/deployment-options.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: deployment-options
title: Deployment Options
description: Compare Estuary public, private, and BYOC deployments for SaaS setups to dedicated infrastructure for compliance and data residency needs.
sidebar_position: 3
---

Expand Down
2 changes: 1 addition & 1 deletion site/docs/getting-started/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
sidebar_position: 1
description: Get to know Estuary and learn how to get started.
description: Learn what Estuary is and how to use it to build real-time and batch data pipelines with CDC, connectors, and transformations. Tutorials and deployment guides cover all aspects of the platform.
pagination_next: getting-started/who-should-use-flow
slug: /
---
Expand Down
2 changes: 1 addition & 1 deletion site/docs/getting-started/installation.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Configure Cloud Storage
description: Learn how to configure Estuary to use your cloud storage
description: Configure Estuary to use your own cloud storage with storage mappings for GCS, Amazon S3, or Azure Blob storage. Manage data planes, set access policies, and run connection tests.
slug: installation
sidebar_position: 5
---
Expand Down
3 changes: 3 additions & 0 deletions site/docs/getting-started/pricing.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
description: Understand Estuary pricing for data volume and connector task hours, how private or BYOC deployments affect pricing, PAYG billing vs. annual plans, and free tiers and trials.
---

# Estuary Pricing

Expand Down
1 change: 1 addition & 0 deletions site/docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 2
description: Build your first Estuary streaming CDC pipeline from PostgreSQL to Snowflake with guided setup steps for your capture and materialization.
---

# Quickstart for Estuary
Expand Down
5 changes: 4 additions & 1 deletion site/docs/getting-started/sso-setup.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
---
description: Set up Estuary SSO login on your tenant for a specific company domain. Guide includes Estuary's SAML 2.0 URLs and adding new users to your tenant.
---

# Log into Estuary Using SSO
# Set Up SSO Login

You can register or log into the [Estuary dashboard](https://dashboard.estuary.dev/) with one of several methods. Sign in with either:
* A Google account
Expand Down
3 changes: 3 additions & 0 deletions site/docs/getting-started/support.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
---
description: Learn how to get Estuary support through Slack and email. Report issues to GitHub, subscribe to status updates, and temporarily share data access for troubleshooting.
---

# Support

Expand Down
4 changes: 4 additions & 0 deletions site/docs/getting-started/tutorials/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Explore hands-on Estuary tutorials for practical Data Flows with sample data. Connect specific systems, like MongoDB, S3, and Snowflake, and transform data with derivations.
---

# Estuary Tutorials

Estuary's tutorials are complete learning experiences that help you get to know Estuary using sample data.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
sidebar_position: 4
description: Create a continuously updated PostgreSQL materialized view with Estuary derivations. Demo covers using flowctl, SQL transformations, and publishing steps.
---

import Tabs from '@theme/Tabs';
Expand Down
4 changes: 3 additions & 1 deletion site/docs/getting-started/tutorials/dataflow-s3-snowflake.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
---
sidebar_position: 3
description: Build your first Estuary Data Flow from Amazon S3 to Snowflake. Guide covers capture setup, collections, materialization, and publishing.
---
# Create Your First Dataflow with Amazon S3 and Snowflake

# Amazon S3 to Snowflake Data Flow

In this tutorial, you'll create your first complete **Data Flow** with Estuary using publicly available data.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: derivations_acmebank
title: Implementing Derivations for AcmeBank
description: Learn how to transform data with Estuary derivations. This tutorial covers filtering data, delayed reads, recursive data flows, and reductions using SQL and TypeScript.
---

<head>
Expand Down
1 change: 1 addition & 0 deletions site/docs/getting-started/tutorials/firestore-to-dwh.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
description: Build a real-time Firestore to Snowflake Data Flow with Estuary, including service account setup, capture, collections, materialization, and publishing.
slug: /guides/system-specific-dataflows/firestore-to-dwh/
---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
id: postgresql_cdc_to_snowflake
title: PostgreSQL CDC Streaming to Snowflake
sidebar_position: 1
description: Build your first Estuary Data Flow from PostgreSQL to Snowflake using CDC. Guide covers capture setup, collections, materialization, and publishing.
---

<head>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
id: real_time_cdc_with_mongodb
title: Real-Time CDC with MongoDB
sidebar_position: 2
description: Set up your first Estuary capture to read data from MongoDB in real-time. Guide covers CDC using change streams, MongoDB setup, and Estuary capture configuration.
---

import ReactPlayer from "react-player"
Expand Down
2 changes: 1 addition & 1 deletion site/docs/getting-started/who-should-use-flow.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
sidebar_position: 1
description: Common pain points you might have, and how Estuary addresses them.
description: See how Estuary helps data teams unify batch and streaming pipelines, reduce warehouse costs, transform data, improve integrity, scale workflows, and address other common pain points.
---

# Who Should Use Estuary?
Expand Down
Loading
Loading