diff --git a/site/docs/features/agent-skills.md b/site/docs/features/agent-skills.md new file mode 100644 index 00000000000..8f4c99bd314 --- /dev/null +++ b/site/docs/features/agent-skills.md @@ -0,0 +1,130 @@ +--- +sidebar_position: 3 +--- + +# Agent Skills + +Set up and operate Estuary pipelines through your AI assistant. Agent skills are step-by-step playbooks that give your assistant everything it needs to run the right commands, build the right specs, and explain the gotchas — no docs to stitch together, no flowctl commands to memorize. + +Ask your assistant to "capture my Postgres into Snowflake" or "why is this materialization failing?", and the relevant skill guides it through the work. + +Skills are distributed from the open-source [`estuary/agent-skills`](https://github.com/estuary/agent-skills) repository and work with [Claude Code, GitHub Copilot, Cursor, OpenAI Codex, Gemini CLI, and 30+ other tools](https://agentskills.io) via the open [SKILL.md](https://agentskills.io) standard. + +:::tip +Pair skills with the [MCP integration](./mcp-integration.md) so your assistant's explanations stay grounded in current Estuary documentation while the skills drive the work. See [Using coding agents with Estuary](./using-coding-agents.md) for the full setup. +::: + +## What's included + +### Captures (sources) + +Capture data from databases, APIs, and webhooks into Estuary collections. + +| Skill | Description | +|-------|-------------| +| `capture-postgres-create` | PostgreSQL CDC (vanilla, RDS, Aurora, Cloud SQL, Supabase, Neon) | +| `capture-mysql-create` | MySQL CDC via binlog replication (RDS, Aurora, Cloud SQL, Azure) | +| `capture-mongodb-create` | MongoDB CDC (Atlas, DocumentDB, self-hosted) | +| `capture-sqlserver-create` | SQL Server CDC (RDS, Azure SQL, Cloud SQL) | +| `capture-http-ingest-create` | HTTP webhook capture (GitHub, Shopify, Stripe, or any JSON source) | +| `capture-generic-create` | Any of 148+ source connectors via dynamic schema discovery | + +### Materializations (destinations) + +Stream Estuary collections into downstream databases and warehouses. + +| Skill | Description | +|-------|-------------| +| `materialize-postgres-create` | PostgreSQL destination | +| `materialize-snowflake-create` | Snowflake destination (JWT auth) | +| `materialize-bigquery-create` | BigQuery destination (GCS staging) | +| `materialize-redshift-create` | Amazon Redshift destination (S3 staging) | +| `materialize-databricks-create` | Databricks destination (Unity Catalog) | +| `materialize-generic-create` | Any destination connector via dynamic schema discovery | + +### Operations + +Manage and troubleshoot running pipelines. + +| Skill | Description | +|-------|-------------| +| `estuary-flowctl-setup` | Install, authenticate, and update the flowctl CLI | +| `estuary-task-health` | End-to-end health check for a task: status, data flow, errors, and history | +| `estuary-catalog-status` | Check control-plane status of a task with `flowctl catalog status` | +| `estuary-task-stats` | Inspect data volume, document counts, and hourly throughput for a task | +| `estuary-catalog-history` | Review publication history and recent spec changes | +| `estuary-logs` | Search and analyze task logs with jq filtering | +| `estuary-connector-restart` | Pause and restart connectors via shard management | +| `estuary-ssh-tunnels` | Diagnose and fix SSH tunnel connection issues | + +## Prerequisites + +- An [Estuary account](https://dashboard.estuary.dev/register) +- The [flowctl](../guides/get-started-with-flowctl.md) CLI — the `estuary-flowctl-setup` skill walks you through installation and authentication + +## Installation + +### Skills CLI + +Install all skills at once, in any supported tool: + +```bash +npx skills add estuary/agent-skills +``` + +### Claude Code + +Add the Estuary marketplace: + +```bash +/plugin marketplace add estuary/agent-skills +``` + +Then install by group: + +```bash +/plugin install estuary-captures@estuary +/plugin install estuary-materializations@estuary +/plugin install estuary-operations@estuary +``` + +Or run `/plugin` to browse from the Discover tab. Installed skills auto-update when the marketplace refreshes. + +### Manual installation + +Clone the repository and copy the skill folders into your AI tool's skill directory: + +```bash +git clone https://github.com/estuary/agent-skills.git +cp -r agent-skills/skills/* your-project/.claude/skills/ +``` + +Common paths by tool: + +| Tool | Path | +|------|------| +| Claude Code | `.claude/skills/` | +| Cursor | `.cursor/skills/` | +| GitHub Copilot / VS Code | `.github/skills/` | +| OpenCode | `.opencode/skills/` | +| Codex | `.codex/skills/` | + +## Usage + +Once installed, ask your AI assistant in plain English: + +> "Capture my Postgres database into Estuary." +> +> "Materialize my collections into Snowflake." +> +> "Capture from MySQL and materialize into Redshift." +> +> "Why is my materialization failing?" + +Your assistant picks the matching skill and walks through the commands, spec, and common pitfalls. + +## Related pages + +- [Using coding agents with Estuary](./using-coding-agents.md) — end-to-end setup for flowctl, MCP, and skills +- [MCP integration](./mcp-integration.md) — connect Estuary's documentation to your assistant +- [flowctl setup](../guides/get-started-with-flowctl.md) — install and authenticate the CLI diff --git a/site/docs/features/configuring-task-shards.md b/site/docs/features/configuring-task-shards.md index 76dab3b4bc3..3b5b1cdec0f 100644 --- a/site/docs/features/configuring-task-shards.md +++ b/site/docs/features/configuring-task-shards.md @@ -1,5 +1,5 @@ --- -sidebar_position: 2 +sidebar_position: 5 slug: /reference/Configuring-task-shards/ --- diff --git a/site/docs/features/mcp-integration.md b/site/docs/features/mcp-integration.md index d2b1269b4b5..522fe9a4bc7 100644 --- a/site/docs/features/mcp-integration.md +++ b/site/docs/features/mcp-integration.md @@ -1,5 +1,5 @@ --- -sidebar_position: 1 +sidebar_position: 2 --- # MCP Integration @@ -10,6 +10,10 @@ Connect to Estuary's documentation directly from your IDE or AI tool using the M The Model Context Protocol (MCP) allows AI assistants in your development environment to access Estuary's documentation contextually. This means you can ask questions about Estuary directly in your IDE and get accurate, documentation-backed answers without leaving your workflow. +:::tip +MCP keeps your assistant's answers accurate. To have it also build and operate pipelines for you, pair it with [agent skills](./agent-skills.md). See [Using coding agents with Estuary](./using-coding-agents.md) for the full setup. +::: + ## Setup **Server URL:** `https://estuary.mcp.kapa.ai` @@ -152,3 +156,9 @@ Once connected, you can ask context-aware questions about Estuary from within yo The AI assistant will use Estuary's documentation to provide accurate, up-to-date answers while you code. +## Related pages + +- [Using coding agents with Estuary](./using-coding-agents.md) — end-to-end setup for flowctl, MCP, and skills +- [Agent skills](./agent-skills.md) — playbooks that let your assistant build and operate pipelines +- [flowctl setup](../guides/get-started-with-flowctl.md) — install and authenticate the CLI + diff --git a/site/docs/features/time-travel.md b/site/docs/features/time-travel.md index 26aa36f21a5..bf01f6eb51d 100644 --- a/site/docs/features/time-travel.md +++ b/site/docs/features/time-travel.md @@ -1,5 +1,5 @@ --- -sidebar_position: 1 +sidebar_position: 4 slug: /reference/time-travel/ --- diff --git a/site/docs/features/using-coding-agents.md b/site/docs/features/using-coding-agents.md new file mode 100644 index 00000000000..d238fbd8e0d --- /dev/null +++ b/site/docs/features/using-coding-agents.md @@ -0,0 +1,82 @@ +--- +sidebar_position: 1 +--- + +# Using Coding Agents with Estuary + +Build and operate Estuary pipelines from your AI coding assistant — Claude Code, Cursor, GitHub Copilot, OpenAI Codex, Gemini CLI, and others. Instead of stitching together docs and remembering flowctl commands, you can ask in plain English: "capture my Postgres into Snowflake" or "why is this materialization failing?" + +Three pieces work together: + +1. **[flowctl](../guides/get-started-with-flowctl.md)** — Estuary's CLI. This is what your agent actually runs to discover sources, build specs, publish, and inspect tasks. +2. **[MCP integration](./mcp-integration.md)** — connects your assistant to Estuary's documentation so its answers are backed by current docs, without leaving your editor. +3. **[Agent skills](./agent-skills.md)** — step-by-step playbooks that tell your assistant the exact commands, spec shapes, and gotchas for each connector and operation. + +You can use any one of these on its own, but they're most useful together: skills drive flowctl to do the work, and MCP keeps the assistant's explanations accurate. + +## Setup + +### 1. Install and authenticate flowctl + +flowctl is the CLI your agent drives. Install it: + +```bash +# Mac (Homebrew) +brew tap estuary/flowctl +brew install flowctl +``` + +For Linux, direct download, or Windows (WSL), see the [flowctl setup guide](../guides/get-started-with-flowctl.md). + +Then authenticate: + +```bash +flowctl auth login +``` + +This opens the Estuary dashboard's CLI-API tab. Copy the access token and paste it back into the terminal. For CI/CD and other non-interactive environments, use a refresh token via `FLOW_AUTH_TOKEN` — see the [flowctl setup guide](../guides/get-started-with-flowctl.md). + +### 2. Connect the MCP server + +Point your assistant at Estuary's MCP server so it can answer documentation questions in context: + +```bash +# Claude Code +claude mcp add --transport http estuary https://estuary.mcp.kapa.ai +``` + +Setup differs per tool (Cursor, VS Code, ChatGPT Desktop, Claude Desktop, and others). See [MCP integration](./mcp-integration.md) for the full instructions and authentication steps. + +### 3. Install the agent skills + +Add the [Estuary agent skills](./agent-skills.md) so your assistant knows the right commands and spec shapes for each connector: + +```bash +# Skills CLI (works across tools) +npx skills add estuary/agent-skills +``` + +```bash +# Claude Code (plugin marketplace) +/plugin marketplace add estuary/agent-skills +``` + +See [Agent skills](./agent-skills.md) for the Claude Code plugin groups, manual installation, and per-tool skill directories. + +## What you can do + +Once set up, ask your assistant in plain English. For example: + +- "Capture my PostgreSQL database into Estuary." +- "Materialize my collections into Snowflake." +- "Capture from MySQL and materialize into Redshift." +- "Why is my materialization failing?" +- "What's the difference between captures and materializations?" + +The skills tell your assistant which flowctl commands to run and how to build the specs; the MCP integration keeps its explanations grounded in Estuary's documentation. + +## Next steps + +- [flowctl setup](../guides/get-started-with-flowctl.md) — install, authenticate, and update the CLI +- [MCP integration](./mcp-integration.md) — connect Estuary's docs to your assistant +- [Agent skills](./agent-skills.md) — the full catalog of capture, materialization, and operations skills diff --git a/site/docs/features/working-logs-stats.md b/site/docs/features/working-logs-stats.md index 6e7fd94b0f3..4197454c621 100644 --- a/site/docs/features/working-logs-stats.md +++ b/site/docs/features/working-logs-stats.md @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 6 slug: /reference/working-logs-stats/ --- diff --git a/site/docs/guides/get-started-with-flowctl.md b/site/docs/guides/get-started-with-flowctl.md index 7c92cc63e55..012a0ba4782 100644 --- a/site/docs/guides/get-started-with-flowctl.md +++ b/site/docs/guides/get-started-with-flowctl.md @@ -64,7 +64,17 @@ Once you've installed flowctl and are ready to begin working, authenticate your The token will expire after a predetermined duration. Repeat this process to re-authenticate. +## Drive flowctl from an AI coding agent + +You don't have to run flowctl commands by hand. AI assistants like Claude Code, Cursor, and GitHub Copilot can build and operate your pipelines for you: + +- **[Agent skills](../features/agent-skills.md)** give your assistant step-by-step playbooks for each connector and operation, so it knows the exact flowctl commands and spec shapes to use. Ask it to "capture my Postgres into Snowflake" and it does the work. +- **[MCP integration](../features/mcp-integration.md)** connects your assistant to Estuary's documentation, so its answers stay accurate while you code. + +See [Using coding agents with Estuary](../features/using-coding-agents.md) for the full setup. + # Next steps 1. [flowctl concepts](../concepts/flowctl.md): Learn more about using flowctl. 2. [User guides](../guides/flowctl/README.md): Check out some of the detailed user guides to see flowctl in action. +3. [Using coding agents with Estuary](../features/using-coding-agents.md): Let an AI assistant build and operate your pipelines.