Skip to content

Fix prod DB table dump workflow#1017

Merged
arielr-lt merged 7 commits intomasterfrom
feature/prod-db-table-dump-fixes
Mar 31, 2026
Merged

Fix prod DB table dump workflow#1017
arielr-lt merged 7 commits intomasterfrom
feature/prod-db-table-dump-fixes

Conversation

@arielr-lt
Copy link
Copy Markdown
Collaborator

Summary

  • Mount main-app-config configmap in the K8s Job so POSTGRESQL_DATABASE and POSTGRESQL_USERNAME are available
  • Switch from pg_dump to psql COPY — streams directly to S3 (no /tmp disk usage), supports arbitrary WHERE conditions, outputs gzipped CSV
  • Add optional where_clause input to the workflow for row filtering (e.g. created_at > '2026-01-01')
  • Bump activeDeadlineSeconds to 4h to handle large tables
  • Use postgres:17-alpine to match the RDS server version

Ariel Rolfo and others added 5 commits March 30, 2026 11:06
Introduces a manually-triggered GitHub Actions workflow that spins up a
K8s Job in credreg-prod to pg_dump a specified table, uploads the result
to a dedicated private S3 bucket, and notifies Slack with a 1-hour
presigned download URL.

- New terraform module: db_dumps_s3 (private bucket, AES-256, 7-day expiry)
- IRSA application policy extended with PutObject on cer-db-dumps-prod
- github-oidc-widget gets s3:GetObject for presigned URL generation
- K8s Job template: postgres:16-alpine, uses existing app-secrets for DB creds
- GH Actions workflow: validates table name, creates job, waits, presigns, notifies Slack
@excelsior excelsior self-requested a review March 31, 2026 22:39
@arielr-lt arielr-lt merged commit 9fcccc8 into master Mar 31, 2026
11 of 12 checks passed
@arielr-lt arielr-lt deleted the feature/prod-db-table-dump-fixes branch March 31, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants