diff --git a/skills/README.md b/skills/README.md index 091389cbd..051531110 100644 --- a/skills/README.md +++ b/skills/README.md @@ -11,6 +11,10 @@ This is the GreptimeDB documentation for AI agents. - `greptimedb-flow`: For creating greptimedb flow, continuous aggregation tasks - `greptimedb-trigger`: For creating greptimedb trigger - `self-monitoring-export`: For GreptimeDB Cluster incidents — infer the required log export time range from the user's abnormal-behavior description, then export complete cluster self-monitoring logs and relevant monitoring metrics for engineering investigation +- `greptimedb-performance-diagnosis`: For diagnosing a performance problem — find the bottleneck behind a slow query, lagging/stalled ingestion, or high CPU/memory, then hand off to a fix skill +- `greptimedb-performance-tuning`: For tuning server config and table options — cache sizing, write buffer, WAL, and ingestion throughput +- `greptimedb-table-design`: For designing a table schema for performance, or improving an existing one — primary key, indexes, append-only/merge mode, and partitioning +- `greptimedb-cluster-health-check`: For verifying a deployment is healthy after a deploy/restart — pod readiness, object-store smoke test, error logs, and key metrics The `greptimedb-quickstart` skill is also hosted at (and ), so any AI coding agent can load it with a single instruction: @@ -41,3 +45,19 @@ Using `skills` cli tool to install the skill to your coding agents. [![asciicast](https://asciinema.org/a/N59fXqonAiRYMk4L.svg)](https://asciinema.org/a/N59fXqonAiRYMk4L) `npx skills add https://github.com/GreptimeTeam/docs/tree/main/skills/self-monitoring-export` + +### `greptimedb-performance-diagnosis` + +`npx skills add https://github.com/GreptimeTeam/docs/tree/main/skills/greptimedb-performance-diagnosis` + +### `greptimedb-performance-tuning` + +`npx skills add https://github.com/GreptimeTeam/docs/tree/main/skills/greptimedb-performance-tuning` + +### `greptimedb-table-design` + +`npx skills add https://github.com/GreptimeTeam/docs/tree/main/skills/greptimedb-table-design` + +### `greptimedb-cluster-health-check` + +`npx skills add https://github.com/GreptimeTeam/docs/tree/main/skills/greptimedb-cluster-health-check` diff --git a/skills/greptimedb-cluster-health-check/SKILL.md b/skills/greptimedb-cluster-health-check/SKILL.md new file mode 100644 index 000000000..ded0f7821 --- /dev/null +++ b/skills/greptimedb-cluster-health-check/SKILL.md @@ -0,0 +1,164 @@ +--- +name: greptimedb-cluster-health-check +description: Guide for verifying a GreptimeDB deployment is healthy after deploying or restarting — checks pod/deployment readiness, object-store access via a functional write/read smoke test, error logs, key metrics, and that ingestion recovered to the expected rate. Use right after a fresh deploy, upgrade, restart, or scale event when the user wants confirmation the cluster is healthy. Triggers on phrases like "is my cluster healthy", "集群健康检查", "after deploy/restart", "部署/重启后检查", "verify cluster", "health check", "smoke test", "fresh cluster", "新部署的集群", "check pods", "object store permission". If a check fails, hand off to greptimedb-performance-diagnosis. +--- + +# GreptimeDB Cluster Health Check + +Verify a GreptimeDB deployment is healthy after a deploy, upgrade, restart, or scale event. +This is a **positive verification checklist** — confirm things work — not a root-cause hunt. +If a check fails or a metric looks off, hand off to `greptimedb-performance-diagnosis`. + +Ask up front: **GreptimeDB version**, **deployment mode** (standalone or cluster), and +**how it's deployed** (Kubernetes/operator, Docker, binary). Older versions may lack some +metrics, and older Grafana dashboards may lack some panels — supplement from the +[grafana dashboards directory](https://github.com/GreptimeTeam/greptimedb/tree/main/grafana/dashboards), +where each panel's PromQL is the canonical source for the metric names used below. + +**Running SQL and viewing metrics/logs.** To run the SQL in this guide, use any of: the +`greptimedb-mcp-server` `execute_sql` tool if available; a **MySQL** or **PostgreSQL** client; +or the built-in +[GreptimeDB Dashboard](https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/). +For metrics and logs, use Grafana/self-monitoring (below). **GreptimeDB Enterprise** users can +use the [Management Console](https://docs.greptime.com/enterprise/console-ui/), which provides +SQL, metrics, and logs in one place. + +## The workflow + +### Phase 1. Deployment & infra readiness (especially a fresh cluster) + +**On Kubernetes**, confirm the cluster actually came up: + +```bash +kubectl get pods -n +kubectl describe pod -n # for any pod not Running/Ready +``` + +Every enabled/deployed GreptimeDB role should be `Running`/`Ready` with no `CrashLoopBackOff` +and no unexpected restarts. Core cluster roles are metasrv, datanode, and frontend; also check +flownode when it is enabled/deployed. Check the GreptimeDBCluster CR / operator status too. For +Docker or binary deployments, confirm each configured process is up and its HTTP endpoint +responds. + +**Object store permission (fresh cluster)** — a common first-deploy failure. A process can +start fine yet be unable to read/write the configured object store (wrong credentials, bucket, +region, or endpoint). Don't assume it works because the pod is `Running`; verify it with the +smoke test below. + +### Phase 2. Functional smoke test (first-time deployment) + +Only needed when the cluster is deployed for the **first time** — to prove the configured +object store is actually reachable and writable. On a restart, upgrade, or scale of a cluster +that was already serving writes, object-store access is already proven, so skip this and rely +on Phase 4 (confirm ingestion recovered). Run it anyway if you have any doubt the write path +works. + +Run an end-to-end write/read to exercise the real write path (object store + WAL), not just +process liveness (see access methods above). Use a unique table name for each run, replace it +consistently in every statement, and only drop the table created by this smoke test. For +example, append the current timestamp: `_health_check_YYYYMMDD_HHMMSS`. + +```sql +CREATE TABLE _health_check_20260706_123000 ( + host STRING, + val DOUBLE, + ts TIMESTAMP TIME INDEX, + PRIMARY KEY (host) +); + +INSERT INTO _health_check_20260706_123000 (host, val, ts) +VALUES ('node-1', 1.0, now()), ('node-2', 2.0, now()); + +SELECT * FROM _health_check_20260706_123000; + +-- Force a flush so the object-store write path is exercised, then confirm: +ADMIN flush_table("_health_check_20260706_123000"); +SELECT count(*) FROM _health_check_20260706_123000; + +DROP TABLE _health_check_20260706_123000; +``` + +If the insert or flush fails (e.g. object-store auth/permission error), stop here and treat it +as a deployment problem — check Phase 3 logs for the exact error. + +### Phase 3. Check logs for errors + +Inspect datanode, frontend, and metasrv logs for `ERROR` (and notable `WARN`) lines that +indicate an unhealthy state: + +- Object store authentication/permission failures +- Region open failures +- Flush or compaction failures +- Repeated restarts / panics + +On Kubernetes: `kubectl logs -n ` (add `--previous` for a crashed container). +For self-monitoring deployments, use the **`self-monitoring-export`** skill to pull logs from +the self-monitoring instance. GreptimeDB Enterprise users can also browse logs in the +[Management Console](https://docs.greptime.com/enterprise/console-ui/). + +### Phase 4. Metrics checklist + +Check these against the Grafana dashboard groups (or run the PromQL directly if no dashboard). +Standalone and cluster dashboards share these panels (cluster panels filter by instance). + +**Overview** + +- Uptime, and **Recent Restarts**: `changes(process_start_time_seconds[$__range])` — should be + flat after the restart settles. +- Query rate and HTTP/gRPC P99 latency (`greptime_servers_http_requests_elapsed_bucket`, + `greptime_servers_grpc_requests_elapsed_bucket`). +- User-facing error rate: `sum(rate(greptime_servers_error[$__rate_interval]))`. + +**Ingestion** + +- `sum(rate(greptime_table_operator_ingest_rows[$__rate_interval]))` — **compare against the + rate before the restart/deploy** and confirm it recovered to the expected level (not merely + that it is non-zero). Break down by protocol if needed. + +**Health / errors** + +- `greptime_servers_error`, `greptime_datanode_region_request_fail_count` +- `greptime_mito_write_reject_total`, `greptime_mito_write_stall_total`, + `greptime_mito_flush_failure_total`, `greptime_mito_compaction_failure_total` +- `greptime_pending_rows_flush_dropped_rows` (buffered ingestion loss) + +**Resources** + +- CPU: `rate(process_cpu_seconds_total[$__rate_interval]) * 1000` vs + `greptime_cpu_limit_in_millicores` (both in millicores) +- Memory: `process_resident_memory_bytes` vs `greptime_memory_limit_in_bytes` + +### Phase 5. Interpret + +- All green (pods ready, smoke test passes on a first deploy, no error logs, ingestion + recovered, errors/stalls flat, resources within limits) → the cluster is healthy. +- Anything off → hand off to `greptimedb-performance-diagnosis` with the specific signal. + +If a needed panel is missing because the dashboard is old, give the user the PromQL above so +they can add it, or upgrade the dashboard from the grafana directory. + +## Escalation + +If a check fails and the cause is unclear, collect context (failing `kubectl describe`/logs, +the smoke-test error, relevant metrics, version and deployment mode) and open an issue or +discussion at https://github.com/GreptimeTeam/greptimedb. For self-monitoring users, use the +**`self-monitoring-export`** skill to export the incident logs/metrics. + +## Reference + +1. Self-monitoring deployment: + https://docs.greptime.com/user-guide/deployments-administration/monitoring/cluster-monitoring-deployment/ +2. Grafana dashboards (canonical source for metric names / PromQL): + https://github.com/GreptimeTeam/greptimedb/tree/main/grafana/dashboards +3. Performance tuning tips (ingestion/health metric meanings): + https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/performance-tuning-tips/ +4. REGION_STATISTICS: + https://docs.greptime.com/reference/sql/information-schema/region-statistics/ +5. CREATE TABLE: + https://docs.greptime.com/reference/sql/create/ +6. ADMIN functions (flush_table, compact_table): + https://docs.greptime.com/reference/sql/admin/ +7. GreptimeDB Dashboard (run SQL from a web UI): + https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/ +8. GreptimeDB Enterprise Management Console (SQL, metrics, logs): + https://docs.greptime.com/enterprise/console-ui/ diff --git a/skills/greptimedb-performance-diagnosis/SKILL.md b/skills/greptimedb-performance-diagnosis/SKILL.md new file mode 100644 index 000000000..637aa79b7 --- /dev/null +++ b/skills/greptimedb-performance-diagnosis/SKILL.md @@ -0,0 +1,203 @@ +--- +name: greptimedb-performance-diagnosis +description: Guide for diagnosing GreptimeDB performance problems — find the bottleneck behind a slow query, lagging or stalled ingestion, or high CPU/memory, then hand off to the right fix. Use when the user reports something is slow or resource-hungry and wants to know why. Triggers on phrases like "query is slow", "查询慢", "ingestion is slow", "写入慢", "write stall", "写入卡顿/被拒绝", "high CPU", "CPU 高", "high memory", "内存高/OOM", "EXPLAIN ANALYZE", "why is GreptimeDB slow", "性能排查/诊断". This skill diagnoses; apply fixes with greptimedb-performance-tuning and greptimedb-table-design. +--- + +# GreptimeDB Performance Diagnosis + +Help the user find the **bottleneck** behind a GreptimeDB performance problem, then point +them at the right fix. This skill is the triage hub: it observes and concludes, it does not +apply tuning itself. Once the cause is clear, hand off to: + +- `greptimedb-performance-tuning` — server config and table-option changes (caches, write + buffer, WAL, batching). +- `greptimedb-table-design` — schema changes (primary key, indexes, append-only, + partitioning). + +Before going deep, ask for the **GreptimeDB version** and deployment mode (standalone or +cluster). Older versions may not expose some of the metrics below, and older Grafana +dashboards may lack some panels — you can supplement them from the +[grafana dashboards directory](https://github.com/GreptimeTeam/greptimedb/tree/main/grafana/dashboards), +where each panel's PromQL is the canonical source for the metric names used below. + +**Running SQL and viewing metrics/logs.** To run the SQL in this guide (`EXPLAIN ANALYZE`, +`SELECT`, `SHOW CREATE TABLE`), use any of: the `greptimedb-mcp-server` `execute_sql` tool if +available; a **MySQL** or **PostgreSQL** client; or the built-in +[GreptimeDB Dashboard](https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/). +**GreptimeDB Enterprise** users can use the +[Management Console](https://docs.greptime.com/enterprise/console-ui/), which also surfaces +metrics and logs. + +## The workflow + +### Phase 1. Make signals observable (only as needed) + +Metrics dashboards are **not** a prerequisite. Many diagnoses need nothing but SQL: + +- A **slow query** can be investigated immediately with `EXPLAIN ANALYZE VERBOSE` (Phase 3). +- Write distribution can be checked with `REGION_STATISTICS` (Phase 4). + +Recommend setting up observability only when the symptom needs time-series metrics (write +pressure trends, cache hit ratios, CPU/memory over time): + +1. Self-monitoring setup: + https://docs.greptime.com/user-guide/deployments-administration/monitoring/cluster-monitoring-deployment/ +2. Import the Grafana dashboards (standalone or cluster): + https://github.com/GreptimeTeam/greptimedb/tree/main/grafana/dashboards + +If the dashboard version is old and missing a panel, give the user the raw PromQL from the +phases below so they can add it. + +### Phase 2. Classify the symptom + +Route to the matching phase based on what the user reports: + +| Symptom | Go to | +|---|---| +| A specific query is slow | Phase 3 | +| Ingestion is slow / latency high / writes stall or get rejected | Phase 4 | +| A node has high CPU or memory (cause unclear) | Phase 5 | +| Cache seems ineffective / lots of object-store reads | Phase 6 | + +If the user is unsure, start from the broadest available signal (ingestion rate, request +latency, CPU/memory) and narrow down. + +### Phase 3. Diagnose a slow query + +Run the query with `EXPLAIN ANALYZE VERBOSE` (see access methods above): + +```sql +EXPLAIN ANALYZE VERBOSE ; +``` + +`EXPLAIN ANALYZE` executes the query and reports per-stage runtime metrics; `VERBOSE` adds +scan-level metrics. When comparing repeated runs, remember the second run may be served from +cache — shift the time range slightly to keep the shape equivalent but avoid a fully cached +repeat. + +Map common findings to causes and the fix skill: + +| Finding in output | Likely cause | Fix | +|---|---|---| +| High `num_file_ranges`, many files < 1 row group | Many small SST files (backfill across windows, compaction pressure) | Tuning: `compaction.twcs.time_window`; Table-design: write/backfill by time window | +| High `scan_cost` **and** high `rows_precise_filtered` | Reads many rows then filters them out — missing a selective index | Table-design: add an index on the selective column | +| High `fetch_metrics.cache_miss` / `pages_to_fetch_store` / `store_fetch_elapsed` | Reading pages from object store instead of cache | Tuning: increase `write_cache_size` / `page_cache_size` | +| High `build_parts_cost` / `build_reader_cost` | Time spent building scan ranges/readers | Correlate with `num_file_ranges`, metadata cache misses | +| High `metadata_cache_metrics.metadata_load_cost` / `cache_miss` | SST metadata cache too small | Tuning: increase `sst_meta_cache_size` (check `greptime_mito_cache_bytes{type="sst_meta"}`) | + +If the cause is still unclear, capture the **full** `EXPLAIN ANALYZE VERBOSE` output plus +`SHOW CREATE TABLE` and escalate (see [Escalation](#escalation)). + +### Phase 4. Diagnose ingestion & write pressure + +First check whether the write buffer is under pressure: + +- `greptime_mito_write_buffer_bytes` +- `greptime_mito_write_stall_total` +- `greptime_mito_write_stalling_count` +- `greptime_mito_write_reject_total` + +Stalls mean GreptimeDB is applying backpressure (global write buffer near +`global_write_buffer_size`, or a region temporarily not writable). A client error like +`Engine write buffer is full, rejecting write requests` means the reject threshold +(`global_write_buffer_reject_size`) was reached. + +**Don't jump to raising the write buffer.** First find out *why* the buffer fills: + +1. **Flush performance** — `greptime_mito_flush_elapsed`, plus + `greptime_mito_flush_requests_total`, `greptime_mito_flush_bytes_total`, + `greptime_mito_flush_file_total`. Datanode logs also help: search for + `Successfully flush memtables` and divide `total_rows`/`total_bytes` by `cost` to estimate + throughput. Slow flushes (e.g. ≥30s) or flush throughput below ingestion → the fix is + partitioning/region distribution, not a bigger buffer. +2. **Write distribution** — `greptime_mito_write_rows_total` per datanode and + `greptime_mito_handle_request_elapsed_bucket`; query + [`REGION_STATISTICS`](https://docs.greptime.com/reference/sql/information-schema/region-statistics/) + for `written_bytes_since_open` and `memtable_size`. If one region/datanode takes most of + the load → repartition (Table-design). + +Conclusion → fix: + +- Flush slow or unbalanced load → **Table-design** (partitioning / region distribution). +- Flush fast but frequent under steady load → **Tuning** (increase `global_write_buffer_size`). + +Also compare frontend protocol latency (`greptime_servers_http_requests_elapsed_bucket`, +`greptime_servers_grpc_requests_elapsed_bucket`, filter by `path`/`method`/`code`) against +datanode-side `greptime_mito_handle_request_elapsed_bucket` and +`greptime_mito_write_stage_elapsed_bucket` to locate whether the bottleneck is before or +inside the storage engine. + +### Phase 5. Diagnose high CPU / memory + +When a node has high CPU or memory and the cause is unclear, collect a profile from that +node and provide it to developers. Profiling is node-local — use the HTTP address of the +affected node (replace `127.0.0.1:4000`). + +CPU flamegraph (collect while CPU is high): + +```bash +curl -X POST -s 'http://127.0.0.1:4000/debug/prof/cpu?seconds=10&output=flamegraph' > greptime-cpu.svg +``` + +Memory flamegraph (collect while memory is high or growing): + +```bash +curl -X POST -s 'http://127.0.0.1:4000/debug/prof/mem?output=flamegraph' > greptime-mem.svg +``` + +Notes: + +- Memory profiling is Linux-only (not macOS/Windows) and requires heap profiling enabled. + Official Docker images enable it by default; for self-built deployments start with + `MALLOC_CONF=prof:true ./greptime ...`. +- Generating a flamegraph uses extra memory (parsing + symbolization in process) — be careful + on a node near its memory limit. +- To find growing allocations, collect two `output=proto` profiles at different times and + compare them. + +See https://docs.greptime.com/reference/http-endpoints/#profiling-tools for the reference. +Attach the flamegraph/profile when escalating. + +### Phase 6. Diagnose cache pressure + +Use `greptime_mito_cache_bytes{type="..."}` as the primary signal: if a cache sits near its +configured capacity, it is under pressure. Supporting signals are `greptime_mito_cache_hit`, +`greptime_mito_cache_miss`, and `greptime_mito_cache_eviction` (all labeled by `type`). Cache +types include `file`, `index`, `sst_meta`, `page`, `vector`, `selector_result`, +`range_result`, `manifest`, and index-related caches. + +A low hit ratio with high miss/eviction on a near-full cache → hand off to **Tuning** to +resize that specific cache. + +## Escalation + +When you cannot reach a confident conclusion, recommend the user collect context and open an +issue or discussion at https://github.com/GreptimeTeam/greptimedb. Gather, as relevant: + +- Full `EXPLAIN ANALYZE VERBOSE` output (for query issues). +- Table schema (`SHOW CREATE TABLE`). +- GreptimeDB version and deployment mode. +- Relevant metrics (PromQL results or dashboard screenshots) and logs. +- CPU/memory flamegraphs for resource issues, plus the approximate time range and workload. + +If the user runs GreptimeDB self-monitoring, use the **`self-monitoring-export`** skill to +export the logs/metrics for the incident window. + +## Reference + +1. Performance tuning tips (metrics tables, EXPLAIN findings): + https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/performance-tuning-tips/ +2. EXPLAIN reference: + https://docs.greptime.com/reference/sql/explain/ +3. Self-monitoring deployment: + https://docs.greptime.com/user-guide/deployments-administration/monitoring/cluster-monitoring-deployment/ +4. Grafana dashboards (canonical source for metric names / PromQL): + https://github.com/GreptimeTeam/greptimedb/tree/main/grafana/dashboards +5. HTTP profiling endpoints: + https://docs.greptime.com/reference/http-endpoints/#profiling-tools +6. REGION_STATISTICS: + https://docs.greptime.com/reference/sql/information-schema/region-statistics/ +7. GreptimeDB Dashboard (run SQL from a web UI): + https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/ +8. GreptimeDB Enterprise Management Console (SQL, metrics, logs): + https://docs.greptime.com/enterprise/console-ui/ diff --git a/skills/greptimedb-performance-tuning/SKILL.md b/skills/greptimedb-performance-tuning/SKILL.md new file mode 100644 index 000000000..19aadacaa --- /dev/null +++ b/skills/greptimedb-performance-tuning/SKILL.md @@ -0,0 +1,135 @@ +--- +name: greptimedb-performance-tuning +description: Guide for tuning GreptimeDB server config and table options for better performance — cache sizing, write buffer, WAL, and ingestion throughput. Use when the user wants to speed up queries or writes, reduce object-store reads, fix write stalls/rejects, or right-size caches and the write buffer. Triggers on phrases like "tune performance", "性能调优", "increase cache", "调大缓存", "page cache / write cache", "write buffer", "写缓冲", "global_write_buffer_size", "disable WAL", "关闭 WAL", "improve write throughput", "提升写入吞吐", "batch writes". For schema/table changes (primary key, indexes, partitioning) use greptimedb-table-design; to first find the bottleneck use greptimedb-performance-diagnosis. +--- + +# GreptimeDB Performance Tuning + +Apply **server config and table-option** changes to improve GreptimeDB performance. This +skill covers the live-tunable / deployment-level knobs. Schema-level decisions (primary key, +indexes, append-only, partitioning) live in `greptimedb-table-design`. If you don't yet know +*what* to tune, run `greptimedb-performance-diagnosis` first to find the bottleneck. + +Read the source guide first when you need detail: +https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/performance-tuning-tips/ + +## The workflow + +### Phase 1. Identify the goal and version + +Confirm what to optimize for (query latency, write throughput, or memory/disk footprint) and +ask for the **GreptimeDB version** — defaults and available options differ across versions. + +To apply `ALTER` / table-option changes, run SQL via the `greptimedb-mcp-server` `execute_sql` +tool if available, a **MySQL**/**PostgreSQL** client, the built-in +[GreptimeDB Dashboard](https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/), +or (Enterprise) the [Management Console](https://docs.greptime.com/enterprise/console-ui/). +Config-file (`toml`) changes require a restart. See defaults in +https://docs.greptime.com/user-guide/deployments-administration/configuration/. + +### Phase 2. Cache sizing + +Use `greptime_mito_cache_bytes{type="..."}` as the primary signal: if a cache sits near its +configured capacity, grow it; if it stays far below, shrink it and give the resource to a +cache under pressure. Use hit/miss/eviction as supporting signals. + +```toml +[[region_engine]] +[region_engine.mito] +# Write cache (data files); `type` label is `file`. +write_cache_size = "10G" +# Percentage of write cache for index (puffin) files; `type` label `index`. +index_cache_percent = 20 +# `type` label `sst_meta`. +sst_meta_cache_size = "128MB" +# `type` label `vector`. +vector_cache_size = "512MB" +# Pages of SST row groups; `type` label `page`. +page_cache_size = "512MB" +# Time series selector results (e.g. last_value()); `type` label `selector_result`. +selector_result_cache_size = "512MB" +# Flat range scan results; `type` label `range_result`. +range_result_cache_size = "512MB" +# Manifest files; `type` label `manifest`. +manifest_cache_size = "256MB" +``` + +Decision rules: + +- **Check available memory and disk first, and leave headroom.** Size in-memory caches + (`page`, `vector`, `*_result`, `sst_meta`, `manifest`) against free RAM and the node's + memory limit so they don't trigger OOM; size the `write_cache_size` (disk) against free disk + so it doesn't fill the volume. Grow incrementally and re-check resource usage rather than + maxing caches out. +- Allocate **at least 1/10 of disk** to the write cache; prefer a large write cache with + object storage. +- Allocate **at least 1/4 of memory** to `page_cache_size` if memory usage is under 20%. +- **Double** a cache if its hit ratio is below 50%. +- If `greptime_mito_cache_bytes{type="index"}` is full but `{type="file"}` is not, increase + `index_cache_percent`. +- Only tune `index.staging_size` for index-search workloads where + `greptime_mito_cache_bytes{type="index_staging"}` approaches capacity — not for every + deployment. + +### Phase 3. Ingestion throughput + +- **Batching** — send multiple rows per request; ~1000 rows/batch is a good start, enlarge if + latency and resources allow. +- **Write by time window** — out-of-order writes across time windows hurt performance. For + backfilling a long time range, create the table in advance and set + `compaction.twcs.time_window`. +- **Write buffer** — only after confirming (via diagnosis) that flush performance and write + distribution are healthy, increase `region_engine.mito.global_write_buffer_size` + (default: auto, ~1/8 OS memory, max 1GB). Increase gradually (2x–4x) and watch datanode + memory. Leave `global_write_buffer_reject_size` unset to use the default 2x; set it manually + (commonly 1.5x–2x, and it must exceed `global_write_buffer_size`) only if you want writes to + fail earlier under memory pressure. + +```toml +[[region_engine]] +[region_engine.mito] +global_write_buffer_size = "2GB" +# Optional. Leave unset unless you need a custom reject margin. +global_write_buffer_reject_size = "3GB" +``` + +If flush is slow or one region/datanode takes most writes, repartition instead — see +`greptimedb-table-design`. + +### Phase 4. WAL and table options + +- **Disable WAL** for replayable sources (e.g. Kafka) to boost write throughput. Unflushed + data is not recoverable from GreptimeDB and must be replayed from the source. + + ```sql + CREATE TABLE logs ( + message STRING, + ts TIMESTAMP TIME INDEX + ) WITH (skip_wal = 'true'); + ``` + +- **`append_mode`** and **`sst_format`** — append-only tables and the `flat` SST format + improve scan performance. `flat` is the default; the only reason to set `sst_format='flat'` + manually is upgrading from an old version whose default was the legacy `primary_key` format. + These are schema-level concerns — see `greptimedb-table-design`. + +## Escalation + +If tuning doesn't help and the cause is unclear, collect context (relevant metrics including +`greptime_mito_cache_bytes`/hit/miss, `EXPLAIN ANALYZE VERBOSE` for query cases, +`SHOW CREATE TABLE`, version, logs) and open an issue or discussion at +https://github.com/GreptimeTeam/greptimedb. For self-monitoring users, use the +**`self-monitoring-export`** skill to export the incident data. + +## Reference + +1. Performance tuning tips: + https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/performance-tuning-tips/ +2. Configuration (default cache and buffer sizes): + https://docs.greptime.com/user-guide/deployments-administration/configuration/ +3. CREATE TABLE (table options, `skip_wal`, `append_mode`, `sst_format`, compaction options): + https://docs.greptime.com/reference/sql/create/ +4. GreptimeDB Dashboard (run SQL from a web UI): + https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/ +5. GreptimeDB Enterprise Management Console (SQL, metrics, logs): + https://docs.greptime.com/enterprise/console-ui/ diff --git a/skills/greptimedb-quickstart/SKILL.md b/skills/greptimedb-quickstart/SKILL.md index e591adb3f..3a981359d 100644 --- a/skills/greptimedb-quickstart/SKILL.md +++ b/skills/greptimedb-quickstart/SKILL.md @@ -19,6 +19,10 @@ skills (each hosted as a fetchable markdown file at the URL below): - [`greptimedb-pipeline`](https://docs.greptime.com/skills/greptimedb-pipeline/SKILL.md) — parse, transform, and route logs. - [`greptimedb-flow`](https://docs.greptime.com/skills/greptimedb-flow/SKILL.md) — continuous aggregation / materialized views. - [`greptimedb-trigger`](https://docs.greptime.com/skills/greptimedb-trigger/SKILL.md) — alerting rules (Enterprise). +- [`greptimedb-performance-diagnosis`](https://docs.greptime.com/skills/greptimedb-performance-diagnosis/SKILL.md) — find the bottleneck behind slow queries, stalled ingestion, or high resource usage. +- [`greptimedb-performance-tuning`](https://docs.greptime.com/skills/greptimedb-performance-tuning/SKILL.md) — tune server config and table options for performance. +- [`greptimedb-table-design`](https://docs.greptime.com/skills/greptimedb-table-design/SKILL.md) — design or improve table schemas, primary keys, indexes, and partitioning. +- [`greptimedb-cluster-health-check`](https://docs.greptime.com/skills/greptimedb-cluster-health-check/SKILL.md) — verify a deployment is healthy after deploy, restart, upgrade, or scale. To load a sister skill at runtime, fetch its `SKILL.md` URL and follow the instructions inside. To install it persistently into the user's agent config, @@ -322,6 +326,18 @@ should enable it. See [`greptimedb-trigger`](https://docs.greptime.com/skills/greptimedb-trigger/SKILL.md) (Enterprise only; for open source, point the user at Prometheus Alertmanager with GreptimeDB as the PromQL backend). +- **Slow query, slow ingestion, write stall, high CPU, or high memory** — hand + off to + [`greptimedb-performance-diagnosis`](https://docs.greptime.com/skills/greptimedb-performance-diagnosis/SKILL.md) + first to identify the bottleneck. +- **Known server config or table-option tuning** — cache sizing, write buffer, + WAL, or ingestion throughput — hand off to + [`greptimedb-performance-tuning`](https://docs.greptime.com/skills/greptimedb-performance-tuning/SKILL.md). +- **Schema, primary key, index, partitioning, append-only, or merge-mode + decisions** — hand off to + [`greptimedb-table-design`](https://docs.greptime.com/skills/greptimedb-table-design/SKILL.md). +- **Post-deploy, restart, upgrade, or scale health verification** — hand off to + [`greptimedb-cluster-health-check`](https://docs.greptime.com/skills/greptimedb-cluster-health-check/SKILL.md). ## Phase 5. Use llms.txt for deeper detail diff --git a/skills/greptimedb-table-design/SKILL.md b/skills/greptimedb-table-design/SKILL.md new file mode 100644 index 000000000..55111e2e3 --- /dev/null +++ b/skills/greptimedb-table-design/SKILL.md @@ -0,0 +1,171 @@ +--- +name: greptimedb-table-design +description: Guide for designing a GreptimeDB table schema for performance, and for improving an existing table. Covers primary key, append-only vs deduplication + merge mode, indexes (inverted/skipping/fulltext), wide vs multiple tables, and partitioning. Use when the user asks how to model a table, choose a primary key, pick an index, partition for scale, or improve a schema that already exists. Triggers on phrases like "design table", "表设计", "schema design", "建表", "primary key", "主键", "which index", "选索引", "partition table", "分区/分表", "append-only", "merge mode", "improve my schema". For server config tuning use greptimedb-performance-tuning; to find a bottleneck first use greptimedb-performance-diagnosis. +--- + +# GreptimeDB Table Design + +Help the user design a table schema for performance, or improve an existing one. The schema +is the most impactful performance decision in GreptimeDB. This skill has two entry points — +**a new table** (every lever is open) and **an existing table** (most key decisions are baked +in at creation; only some things can be altered). + +For server-level config (caches, write buffer, WAL) see `greptimedb-performance-tuning`. To +diagnose why a query/write is slow before redesigning, see `greptimedb-performance-diagnosis`. + +Ask up front: **is this a new table or an existing one?** and the **GreptimeDB version**. + +To run the SQL in this guide (`CREATE TABLE`, `ALTER TABLE`, `SHOW CREATE TABLE`, +`INSERT ... SELECT`), use any of: the `greptimedb-mcp-server` `execute_sql` tool if available; +a **MySQL** or **PostgreSQL** client; the built-in +[GreptimeDB Dashboard](https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/); +or (Enterprise) the [Management Console](https://docs.greptime.com/enterprise/console-ui/). + +## The workflow + +### Phase 1. Understand how GreptimeDB stores and reads data + +Read the design guide; the rest of this skill builds on these ideas: +https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/design-table/ + +Key facts: + +- Rows are stored sorted by `(primary key, timestamp)`. Rows sharing a primary key form one + time series, stored contiguously — making single-series scans cheap and improving + compression. +- A scan prunes in stages: **time range** → **row-group min/max stats** → **index** → read & + filter. The leading primary-key column gets the most benefit from row-group pruning; there + is no global primary-key index, so deep-key or high-cardinality point lookups need an index. +- Non-append-only tables **merge and deduplicate** by `(primary key, timestamp)` at scan time; + append-only tables skip this and are faster. + +### Phase 2. New table — make the decisions + +Understand the data (cardinality of each column), the common queries (filters, GROUP BY, +ORDER BY), and whether updates/deletes are needed. Then decide: + +**Primary key (tags).** + +- Optional. An append-only table without a primary key (sorted by time) is a fine baseline. +- Set one when most queries benefit from its ordering, or you need to deduplicate/delete by + it. +- Lead with the **most frequently filtered, most selective low-cardinality** column + (`namespace`, `cluster`, `region`, `application`). Keep it to **≤5 columns**; a long key + slows inserts and grows memory. +- Use short strings/integers; avoid `FLOAT`/`DOUBLE`/`TIMESTAMP` as tags. +- For high-cardinality columns (`trace_id`, `span_id`, `user_id`), prefer a **skipping index** + over adding them to the primary key — add them to the key only when you must order or + deduplicate by them. + +**Append-only vs deduplication.** + +- `append_mode = 'true'` — keep every row, no dedup/delete. Fastest; ideal for logs/traces. +- Default (deduplicating) — one row per `(primary key, timestamp)`, with `merge_mode`: + `last_row` (default, faster) or `last_non_null` (update individual fields, keeps latest + non-null). Make sure the primary key uniquely identifies a series so distinct rows aren't + merged. + +**Indexes** (optional; add when a filter is common and not already fast enough): + +| Index | Best for | +|---|---| +| Inverted (`INVERTED INDEX`) | Filtering low-cardinality columns; supports `=`, `<`, `<=`, `>`, `>=`, `IN`, `BETWEEN`, `~` | +| Skipping (`SKIPPING INDEX`) | Precise filtering of high-cardinality columns; equality only; low storage/memory overhead | +| Full-text (`FULLTEXT INDEX`) | Tokenized text search in unstructured message bodies | + +Primary key and index are complementary: the primary key gives ordering/locality and enables +dedup; an index targets selective filters ordering can't cover. A query can use both. + +**Wide vs multiple tables.** Put metrics collected together in one wide table (better +throughput and compression). Use separate tables for logically distinct data (different +schema, columns, or TTL). For Prometheus remote-write, the metric engine handles this +automatically. + +**Partitioning** — only when one node can no longer serve the table (disk, CPU, memory, or +ingestion limits). Choose partition columns that are evenly distributed, stable, and appear in +common query filters. For non-append-only tables, **partition columns must be a subset of the +primary key** (dedup happens within a partition). + +Example append-only logs table: + +```sql +CREATE TABLE http_logs ( + access_time TIMESTAMP TIME INDEX, + application STRING, + remote_addr STRING, + http_status STRING, + http_method STRING INVERTED INDEX, + http_refer STRING, + user_agent STRING, + request_id STRING SKIPPING INDEX, + request STRING, + PRIMARY KEY (application) +) WITH ('append_mode' = 'true'); +``` + +### Phase 3. Existing table — what you can and cannot change + +Be explicit about the constraints. The highest-impact choices are fixed at `CREATE`: + +**Not alterable** (require recreating the table): + +- Primary key +- `merge_mode` + +**Alterable on a live table:** + +- Add/drop **indexes** via `ALTER TABLE` — applies to **newly flushed data**, not existing + SST files (see the data-index guide). +- `append_mode` can change from `false` to `true`, but not from `true` back to `false`: + + ```sql + ALTER TABLE my_table SET 'append_mode' = 'true'; + ``` + +- Existing unpartitioned tables can be repartitioned with + `ALTER TABLE ... PARTITION ON COLUMNS`, and partitioned tables can be adjusted with + `SPLIT PARTITION` / `MERGE PARTITION`. Repartitioning requires a distributed cluster, + shared object storage, and GC enabled. +- `sst_format` → `flat` (only needed when upgrading from an old `primary_key`-format version): + + ```sql + ALTER TABLE my_table SET 'sst_format' = 'flat'; + ``` + +- `compaction.twcs.time_window`, `TTL`, and other table options via `ALTER TABLE ... SET`. + +**When a baked-in choice is wrong** (e.g. wrong primary key, wrong `merge_mode`, or need to +switch from append-only back to deduplicating mode), the path is to **create a new table with the +right schema and migrate**: + +```sql +CREATE TABLE my_table_v2 ( ... corrected schema ... ); +INSERT INTO my_table_v2 SELECT ... FROM my_table; +-- verify, then switch writes/reads to my_table_v2 and drop the old table +``` + +Confirm the time range and row volume before migrating large tables, and consider migrating in +time-window chunks. + +## Escalation + +If you're unsure whether a schema change will help, recommend benchmarking on the user's own +dataset (a baseline table without extra indexes is a good control). When still unclear, collect +context (`SHOW CREATE TABLE`, representative queries with `EXPLAIN ANALYZE VERBOSE`, cardinality +estimates, version) and open an issue or discussion at +https://github.com/GreptimeTeam/greptimedb. + +## Reference + +1. Design your table schema: + https://docs.greptime.com/user-guide/deployments-administration/performance-tuning/design-table/ +2. Data index (types, ALTER to add/drop indexes): + https://docs.greptime.com/user-guide/manage-data/data-index/ +3. CREATE TABLE (append-only, merge mode, SST format, compaction options): + https://docs.greptime.com/reference/sql/create/ +4. Table sharding / partitioning: + https://docs.greptime.com/user-guide/deployments-administration/manage-data/table-sharding/ +5. GreptimeDB Dashboard (run SQL from a web UI): + https://docs.greptime.com/getting-started/installation/greptimedb-dashboard/ +6. GreptimeDB Enterprise Management Console (SQL, metrics, logs): + https://docs.greptime.com/enterprise/console-ui/