diff --git a/ci/vale/dictionary.txt b/ci/vale/dictionary.txt index 10ee9a02a4b..1c86447fc11 100644 --- a/ci/vale/dictionary.txt +++ b/ci/vale/dictionary.txt @@ -1468,6 +1468,8 @@ Matthes Matz Mavan mawk +max_replication_slots +max_wal_senders maxconn maxdepth maxretry @@ -2832,6 +2834,7 @@ vvv vz vznetaddbr wafs +wal_level walkthrough walkthroughs WAN diff --git a/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/akamai-cloud-manager-database-host.png b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/akamai-cloud-manager-database-host.png new file mode 100644 index 00000000000..4355aa637a5 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/akamai-cloud-manager-database-host.png differ diff --git a/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/index.md b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/index.md new file mode 100644 index 00000000000..0b304b6038d --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/index.md @@ -0,0 +1,492 @@ +--- +slug: logical-replication-to-a-linode-managed-postgresql-database +title: "Logical Replication to a Linode Managed PostgreSQL Database" +description: "Learn how to configure PostgreSQL logical replication from a cloud-hosted source database to a Linode Managed Database." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-10-24 +keywords: ['logical replication', 'postgresql logical replication', 'linode managed database', 'akamai cloud', 'database migration', 'low downtime migration', 'postgresql subscription', 'postgresql publication', 'replication slot', 'write-ahead log', 'pg_stat_subscription', 'pg_current_wal_lsn', 'pg_dump schema only'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +--- + +Logical replication with PostgreSQL allows you to stream changes from one database (the publisher) to another (the subscriber) at the table level. This makes it possible to replicate data across environments in near real-time, without disrupting the source database by taking it offline. + +[Logical replication](https://www.postgresql.org/docs/current/logical-replication.html) is an alternative approach to the "dump-and-restore" method often used for data replication or migration. Logical replication is useful if you want: + +- **Zero Downtime Migration**: Keep the source database online while synchronizing data to the destination. +- **Continuous Synchronization**: Maintain an ongoing sync between source and destination databases. +- **Selective Table Replication**: Only replicate specific tables, rather than the entire database. + +This guide walks through the process of setting up logical replication from an existing PostgreSQL database to a [Linode Managed Database](https://www.linode.com/products/databases/) running PostgreSQL. + +## Things to Know Before Getting Started + +Before configuring logical replication, review the following key concepts and requirements. Understanding these can help you configure your databases correctly and avoid common issues during replication. + +### Key Terminology + +Logical replication in PostgreSQL follows a *publisher/subscriber* (pub/sub) model. Here, the source database publishes table-level changes and one or more subscribers receive and apply them. + +- **Publisher**: The source PostgreSQL database that exposes data changes through one or more publications. +- **Subscriber**: The destination PostgreSQL database that connects to a publisher and applies changes locally. +- **Publication**: A named set of tables on the publisher whose changes are made available to subscribers. +- **Subscription**: A replication object on the subscriber that connects to a publisher and retrieves changes from its publications. +- **Write-Ahead Log (WAL)**: A sequential log of all database changes. Logical replication reads table-level changes decoded from this log. +- **Replication Slot**: Tracks the progress of each subscriber and retains required WAL segments until they’ve been consumed. +- **WAL Sender (`walsender`)**: A background process on the publisher that streams WAL-derived changes to the subscriber. +- **Apply Worker**: A background process on the subscriber that replays incoming changes in transactional order. + +The diagram below illustrates the logical replication flow between the publisher and subscriber: + +(refer to diagram below) Logical Replication Flow + +![Diagram showing PostgreSQL logical replication flow from publisher to subscriber.](postgresql-logical-replication-flow.png) + +- The publisher defines what to replicate via a publication. +- Changes are captured from the Write-Ahead Log (WAL) and managed through a replication slot. +- A WAL sender streams these changes to the subscriber, which uses an apply worker to write them to local tables. + +### PostgreSQL Version Compatibility + +Logical replication works across major versions of PostgreSQL, but some features may have compatibility limitations. Confirm that your source and destination PostgreSQL versions both support logical replication and are compatible with each other. + +### Server Settings + +To enable logical replication, the source database server must be configured with the following parameters: + +- `wal_level`: Must be set to `logical` to enable logical decoding. +- `max_replication_slots`: At least one slot is required per subscription. +- `max_wal_senders`: Should be equal to or greater than `max_replication_slots`. + +### Roles and Permissions + +The source database should have a dedicated replication user. This user must have the `REPLICATION` attribute and `SELECT` privileges on any tables to be replicated. This is a security best practice, as it avoids using superuser or administrator accounts for replication purposes. + +### Network Access + +Replication requires that the destination database can connect to the source over the network. This means ensuring the source database accepts connections from the Linode Managed Database host. + +## Before You Begin + +1. Follow our [Get started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide to create an Akamai Cloud account if you do not already have one. + +1. Follow the steps in the Akamai Cloud documentation to [create a new database cluster](https://techdocs.akamai.com/cloud-computing/docs/aiven-manage-database#create-a-new-database-cluster). + +1. [Set up access control](https://techdocs.akamai.com/cloud-computing/docs/aiven-manage-database#access-control) so that you can connect to the database from your local machine. + +1. Install the [Linode CLI](https://techdocs.akamai.com/cloud-computing/docs/install-and-configure-the-cli) on your local machine. + +1. Install the `psql` client on your local machine: + + ```command + sudo apt install postgresql-client + ``` + +1. Ensure you have administrative access to your source PostgreSQL database. + +### Placeholders and Examples + +The following placeholders and example values are used in commands throughout this guide: + +| Parameter | Placeholder | Example Value | +|------------|--------------|----------------| +| Destination Hostname or IP Address | {{< placeholder "DEST_HOST" >}} | `a334568-akamai-prod-183144-default.g2a.akamaidb.net` | +| Destination Port Number | {{< placeholder "DEST_PORT" >}} | `10033` | +| Destination Database Name | {{< placeholder "DEST_DB" >}} | `defaultdb` | +| Destination Username | {{< placeholder "DEST_USER" >}} | `akmadmin` | +| Destination Password | {{< placeholder "DEST_PASSWORD" >}} | `thisismydestinationpassword` | +| Source Hostname or IP Address | {{< placeholder "SOURCE_HOST" >}} | `psql.managed-db-services.example.com` | +| Source Port Number | {{< placeholder "SOURCE_PORT" >}} | `5432` | +| Source Database Name | {{< placeholder "SOURCE_DB" >}} | `postgres` | +| Replication Username | {{< placeholder "REPL_USER" >}} | `linode_replicator` | +| Replication Password | {{< placeholder "REPL_PASSWORD" >}} | `thisismyreplicatorpassword` | +| Publication Name | {{< placeholder "PUBLICATION_NAME" >}} | `my_publication` | +| Subscription Name | {{< placeholder "SUBSCRIPTION_NAME" >}} | `my_subscription` | + +Replace these placeholders with your own connection details when running commands in your environment. + +Additionally, the examples used in this guide assume the source database contains three tables (`customers`, `products`, and `orders`) that you want to replicate to a Linode Managed Database. + +### Example Scenario + +The examples used in this guide assume you have an existing PostgreSQL database hosted with a managed service from a cloud provider. This source database contains three tables (`customers`, `products`, and `orders`) that you want to replicate to a Linode Managed Database for PostgreSQL. + + +## Configure the Source Database + +The source PostgreSQL database must be properly configured to support logical replication and accept incoming connections from your Linode Managed Database. + +### Obtain the IP Address of the Destination Database Host + +To configure access control on your source database, you first need the IP address of your Linode Managed Database. Begin by identifying your destination host ({{< placeholder "DEST_HOST" >}}). + +{{< tabs >}} +{{< tab "Akamai Cloud Manager" >}} +1. In the [Akamai Cloud Manager](https://cloud.linode.com/), navigate to **Databases**, then select your database cluster and copy the **Host** string: + + ![Screenshot of Akamai Cloud Manager showing Linode Managed Database host information.](akamai-cloud-manager-database-host.png) + + In the example above, the {{< placeholder "DEST_HOST" >}} value is `a334568-akamai-prod-183144-default.g2a.akamaidb.net` +{{< /tab >}} +{{< tab "Linode CLI" >}} +1. To find the host using the Linode CLI, run the following command to list your databases: + + ```command + linode databases list --json + ``` + + Locate the `primary` host listed for your destination database: + + ```output + [ + { + "allow_list": [ + "2606:4700:3035::ac43:bc7a/128", + ], + "engine": "postgresql", + "hosts": { + "primary": "a334568-akamai-prod-183144-default.g2a.akamaidb.net" + }, + "id": 334568, + "label": "destination-db", + "port": 10033, + "region": "us-sea", + "status": "active", + "total_disk_size_gb": 9, + "type": "g6-nanode-1", + "version": "16.9", + ... + }, + ... + ] + ``` + + In the example above, the {{< placeholder "DEST_HOST" >}} value is `a334568-akamai-prod-183144-default.g2a.akamaidb.net` +{{< /tab >}} +{{< /tabs >}} + +2. Use an online service or `nslookup` to determine the IP address of the database host. Replace {{< placeholder "DEST_HOST" >}} with your actual value (e.g., `a334568-akamai-prod-183144-default.g2a.akamaidb.net`): + + ```command + nslookup {{< placeholder "DEST_HOST" >}} + ``` + + ```output + Non-authoritative answer: + Name: a334568-akamai-prod-183144-default.g2a.akamaidb.net + Address: 172.232.188.122 + Name: a334568-akamai-prod-183144-default.g2a.akamaidb.net + Address: 2600:3c0a::2000:fbff:fe65:756d + ``` + +### Prepare the Source Database for Logical Replication + +With your Linode Managed Database host’s IP address located, follow the corresponding guide to prepare your source database for logical replication: + +- [Preparing your AWS RDS PostgreSQL Database for Logical Replication to Linode Managed Database](/docs/guides/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/) +- [Preparing your Azure PostgreSQL Database for Logical Replication to Linode Managed Database](/docs/guides/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/) +- [Preparing your Google Cloud SQL PostgreSQL Database for Logical Replication to Linode Managed Database](/docs/guides/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/) + +After completing the steps in one of the above guides, gather the following information for your *source database*: + +| Parameter | Placeholder | Example Value | +|------------|--------------|----------------| +| Source Hostname or IP Address | {{< placeholder "SOURCE_HOST" >}} | `psql.managed-db-services.example.com` | +| Source Port Number | {{< placeholder "SOURCE_PORT" >}} | `5432` | +| Source Database Name | {{< placeholder "SOURCE_DB" >}} | `postgres` | +| Replication Username | {{< placeholder "REPL_USER" >}} | `linode_replicator` | +| Replication Password | {{< placeholder "REPL_PASSWORD" >}} | `thisismyreplicatorpassword` | +| Publication Name | {{< placeholder "PUBLICATION_NAME" >}} | `my_publication` | + +The remainder of this guide uses the example values shown above. + +## Configure the Destination Database + +Configure your Linode Managed PostgreSQL database as the replication target. This involves two main tasks: + +- Recreating the schema +- Subscribing to the publication you defined on the source + +### Create the Database Schema + +Logical replication in PostgreSQL does not copy table definitions, it only replicates the data. This means you must manually create the destination tables to match the source database schema exactly. Use the same column names, types, constraints, and indexes to avoid replication errors. + +1. At the command line, use `pg_dump` with the `--schema-only` flag to connect to your source database and write the resulting SQL statements to a file. Replace {{< placeholder "SOURCE_HOST" >}} (e.g. `psql.managed-db-services.example.com`), {{< placeholder "REPL_USER" >}} (e.g. `linode_replicator`), and {{< placeholder "SOURCE_PORT" >}} (e.g. `5432`) with your own values: + + ```command + pg_dump \ + -h {{< placeholder "SOURCE_HOST" >}} \ + -U {{< placeholder "REPL_USER" >}} \ + -p {{< placeholder "SOURCE_PORT" >}} \ + --schema-only \ + defaultdb > schema.sql + ``` + + {{< note >}} + If you only want to create the schema for specific tables rather than the entire database, use the `--table` flag. For example, the following command only dumps the schema for the `customers` and `orders` tables: + + ```command + pg_dump \ + -h {{< placeholder "SOURCE_HOST" >}} \ + -U {{< placeholder "REPL_USER" >}} \ + -p {{< placeholder "SOURCE_PORT" >}} \ + --schema-only \ + --table customers \ + --table orders \ + defaultdb > schema.sql + ``` + {{< /note >}} + +1. Inspect the generated SQL statements to confirm that all expected tables and constraints are present: + + ```command + cat schema.sql + ``` + + ```output + ... + CREATE TABLE public.customers ( + id integer NOT NULL, + name text NOT NULL, + email text NOT NULL, + created_at timestamp without time zone DEFAULT CURRENT_TIMESTAMP + ); + + + ALTER TABLE public.customers OWNER TO demouser; + + CREATE SEQUENCE public.customers_id_seq + AS integer + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1; + + + ALTER SEQUENCE public.customers_id_seq OWNER TO demouser; + + ALTER SEQUENCE public.customers_id_seq OWNED BY public.customers.id; + ... + ``` + +1. In the resulting file, remove or comment out any `ALTER` lines that change the `OWNER`. One way to accomplish this is with `sed`: + + ```command + sed -i '/^ALTER .* OWNER TO /d' schema.sql + ``` + + {{< note >}} + On macOS, use `-i ''` instead of the above command: + + ```command + sed -i '' '/^ALTER .* OWNER TO /d' schema.sql + ``` + {{< /note >}} + +1. Run the schema file on your Linode Managed Database to create the tables. Replace {{< placeholder "DEST_HOST" >}} (e.g, `a334568-akamai-prod-183144-default.g2a.akamaidb.net`), {{< placeholder "DEST_USER" >}} (e.g., `akmadmin`), {{< placeholder "DEST_PORT" >}} (e.g, `10033`), and {{< placeholder "DEST_DB" >}} (e.g., `defaultdb`) with your own values: + + ```command + psql \ + -h {{< placeholder "DEST_HOST" >}} \ + -U {{< placeholder "DEST_USER" >}} \ + -p {{< placeholder "DEST_PORT" >}} \ + -d {{< placeholder "DEST_DB" >}} \ + -f schema.sql + ``` + +### Create a Subscription + +Once the schema is in place, create a subscription on the destination database. This step connects your Linode Managed Database (the subscriber) to your source database (the publisher) and begins streaming changes. + +1. Use the `psql` client to connect to your Linode Managed Database and enter the interactive prompt: + + ```command + psql \ + -h {{< placeholder "DEST_HOST" >}} \ + -U {{< placeholder "DEST_USER" >}} \ + -p {{< placeholder "DEST_PORT" >}} \ + -d {{< placeholder "DEST_DB" >}} + ``` + +1. In the `psql` prompt, run the [`CREATE SUBSCRIPTION`](https://www.postgresql.org/docs/current/sql-createsubscription.html) command to create a subscription (e.g., `my_subscription`) on the destination database. Remember to substitute placeholders for your own source database values: + + ```command {title="Destination psql Prompt"} + CREATE SUBSCRIPTION {{< placeholder "SUBSCRIPTION_NAME" >}} + CONNECTION 'host={{< placeholder "SOURCE_HOST" >}} port={{< placeholder "SOURCE_PORT" >}} user={{< placeholder "REPL_USER" >}} password={{< placeholder "REPL_PASSWORD" >}} dbname={{< placeholder "SOURCE_DB" >}} sslmode=require' + PUBLICATION {{< placeholder "PUBLICATION_NAME" >}}; + ``` + + ```output + NOTICE: created replication slot "my_subscription" on publisher + CREATE SUBSCRIPTION + ``` + + Once the subscription is created, PostgreSQL begins replicating data from the source to the destination. + +1. When done, type `\q` and press Enter to exit the destination `psql` shell. + +## Monitor and Verify Replication + +Once replication is active, monitor its status and verify that data is syncing correctly. + +### Check Subscription Status + +1. Use `psql` to establish a connection to the destination (subscriber) database and enter the interactive prompt: + + ```command + psql \ + -h {{< placeholder "DEST_HOST" >}} \ + -U {{< placeholder "DEST_USER" >}} \ + -p {{< placeholder "DEST_PORT" >}} \ + -d {{< placeholder "DEST_DB" >}} + ``` + +1. On the destination database, run the following SQL command to retrieve information about the subscription: + + ```command {title="Destination psql Prompt"} + SELECT * FROM pg_catalog.pg_stat_subscription; + ``` + + ```output + -[ RECORD 1 ]---------+------------------------------ + subid | 16492 + subname | my_subscription + pid | 463099 + leader_pid | + relid | + received_lsn | 0/D90038A0 + last_msg_send_time | 2025-07-24 17:38:44.539465+00 + last_msg_receipt_time | 2025-07-24 17:38:44.575312+00 + latest_end_lsn | 0/D90038A0 + latest_end_time | 2025-07-24 17:38:44.539465+00 + ``` + + Note the following important fields in the output: + + - `subname`: The name of the subscription + - `received_lsn`: The last WAL location received + - `latest_end_lsn`: The last WAL location applied + - `last_msg_send_time` and `last_msg_receipt_time`: Timestamps for the most recent activity + +1. When done, type `\q` and press Enter to exit the destination `psql` shell. + +### Compare WAL Positions + +To verify how far along the subscriber is, compare the `received_lsn` (or `latest_end_lsn`) from the destination with the current WAL position on the source. + +1. Use `psql` to establish a connection to the source (publisher) database: + + ```command + psql \ + -h {{< placeholder "SOURCE_HOST" >}} \ + -U {{< placeholder "REPL_USER" >}} \ + -p {{< placeholder "SOURCE_PORT" >}} \ + -d {{< placeholder "SOURCE_DB" >}} + ``` + +1. On the source database, run the following SQL command to retrieve the current WAL log sequence number: + + ```command {title="Source psql Prompt"} + SELECT pg_current_wal_lsn(); + ``` + + ```output + pg_current_wal_lsn + -------------------- + 0/DA0009D8 + (1 row) + ``` + + Compare this value with the `latest_end_lsn` shown on the destination: + + - If they match or are close, replication is current and lag-free. + - A large gap may indicate replication delay or a connectivity issue. + + {{< note >}} + Small differences are normal on active databases as new WAL entries are generated continuously. + {{< /note >}} + +1. When done, type `\q` and press Enter to exit the source `psql` shell. + +### Validate Replicated Data + +Verify that the expected data has appeared in the destination (subscriber) database. + +1. Use `psql` to reconnect to your Linode Managed Database and enter the interactive prompt: + + ```command + psql \ + -h {{< placeholder "DEST_HOST" >}} \ + -U {{< placeholder "DEST_USER" >}} \ + -p {{< placeholder "DEST_PORT" >}} \ + -d {{< placeholder "DEST_DB" >}} + ``` + +1. Run validation queries and other table checks to ensure that all rows were copied and continue to stay in sync for example: + + ```command + SELECT COUNT(*) FROM customers + ``` + +1. When done, type `\q` and press Enter to exit the source `psql` shell. + +You may want to repeat this validation periodically until the cutover is complete. + +## Finalize the Cutover + +Once replication is stable and the destination database is fully caught up, complete the migration by redirecting application traffic and decommissioning the replication setup. + +### Redirect Application Traffic + +Update your applications to point to the Linode Managed Database instead of the old source database. This involves changing the database host, port, username, and password in each application's connection settings. + +Test application connectivity to the new database in a staging environment or during a maintenance window before routing production traffic to the new database. + +### Remove the Subscription + +After all applications are using the Linode Managed database as the primary data store, remove the subscription. This stops replication and turns the destination into an independent, writable database. + +1. Use `psql` to reconnect to your Linode Managed Database and enter the interactive prompt: + + ```command + psql \ + -h {{< placeholder "DEST_HOST" >}} \ + -U {{< placeholder "DEST_USER" >}} \ + -p {{< placeholder "DEST_PORT" >}} \ + -d {{< placeholder "DEST_DB" >}} + ``` + +1. Run the following SQL command to remove the subscription from destination database: + + ```command {title="Destination psql Prompt"} + DROP SUBSCRIPTION {{< placeholder "SUBSCRIPTION_NAME" >}}; + ``` + + ```output + NOTICE: dropped replication slot "my_subscription" on publisher + DROP SUBSCRIPTION + ``` + + This command also removes the replication slot from the source database, preventing unnecessary WAL retention. + +1. When done, type `\q` and press Enter to exit the `psql` shell. + +### Retire the Source Database + +Once replication has stopped and all applications have switched to the destination database, you can safely decommission the source. Before doing so, make sure: + +- All applications connect exclusively to the new Linode Managed Database. +- Replicated data has been verified as complete and accurate. +- Backups exist for both the source and destination databases. + +After confirming these items, follow your cloud provider's procedures to delete or archive the source database. + +## Considerations and Potential Challenges + +Logical replication provides flexibility, but a few common challenges can arise: + +- **Schema Drift**: Changes to table structures on the source are not automatically replicated. Keep schemas aligned manually to avoid replication errors. +- **Conflicting Writes**: The subscriber should remain read-only during replication. Avoid making manual changes that could conflict with incoming data. +- **Network Reliability**: Logical replication requires a stable network connection between the destination and source databases. Temporary disconnects can cause lag or stalled replication until connectivity is restored. +- **Replication Slot Management**: Orphaned replication slots on the source can lead to WAL buildup. Always drop the subscription to clean up replication slots after cutover. \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/postgresql-logical-replication-flow.png b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/postgresql-logical-replication-flow.png new file mode 100644 index 00000000000..c36686c36fe Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/logical-replication-to-a-linode-managed-postgresql-database/postgresql-logical-replication-flow.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image1.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image1.png new file mode 100644 index 00000000000..844f964b7ed Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image1.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png new file mode 100644 index 00000000000..34ccb3c41f8 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png new file mode 100644 index 00000000000..3b9ec8d7381 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png new file mode 100644 index 00000000000..2b84befb7e6 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png new file mode 100644 index 00000000000..f2bd32acfa0 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image7.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image7.png new file mode 100644 index 00000000000..5fabe5fd374 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image7.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image8.png b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image8.png new file mode 100644 index 00000000000..7f98f8af3b9 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/image8.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/index.md b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/index.md new file mode 100644 index 00000000000..0123438dc12 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database/index.md @@ -0,0 +1,347 @@ +--- +slug: preparing-your-aws-rds-postgresql-database-for-logical-replication-to-linode-managed-database +title: "Preparing Your AWS RDS PostgreSQL Database for Logical Replication to Linode Managed Database" +description: "Two to three sentences describing your guide." +og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-10-10 +keywords: ['list','of','keywords','and key phrases'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +external_resources: +- '[Link Title 1](http://www.example.com)' +- '[Link Title 2](http://www.example.net)' +--- + +This guide explains how to prepare an AWS RDS PostgreSQL database for logical replication to a Linode Managed Database. If you're following the "Logical Replication to a Linode Managed PostgreSQL Database" guide, use this document to complete the AWS-side preparation before returning to create the subscription on Linode. + +Following the steps in this guide will help you to: + +1. Configure your RDS instance to support logical replication. +1. Ensure secure network access from Linode. +1. Create a dedicated replication user. +1. Set up a publication for the tables you wish to replicate. + +After completing these steps, return to the main replication guide to configure the subscriber and finalize the setup. + +## Prerequisites + +You will need administrative access to your AWS account, including permissions to modify RDS instance settings and security groups. You’ll also need the AWS CLI installed and configured—with a user or role that has the necessary privileges. + +In addition, you should have the IP address or CIDR range for the host machine of your Linode Managed Database so you can configure inbound access on the RDS instance. + +## Modify the RDS Parameter Group + +To support logical replication, you must configure your RDS instance with the correct parameter settings. Follow these steps: + +### Step 1: Locate Your RDS Instance’s Parameter Group + +In the AWS Management Console, go to **RDS > Databases**. Select your instance and look under the **Configuration** tab to find the associated DB instance parameter group (not to be confused with the "option group"). Click on the parameter group. + +![](image5.png) + +When using the `aws` CLI, obtain the name of the parameter group with the following command: + +```command {title="Retrieve name of parameter group for RDS instance"} +aws rds describe-db-instances \ + --db-instance-identifier {{< placeholder "DB_INSTANCE_ID" >}} \ + --region {{< placeholder "AWS-REGION" >}} \ + --query "DBInstances[0].DBParameterGroups[*].DBParameterGroupName" +``` + +```output +[ + "default.postgres16" +] +``` + +### Step 2: Review Current Parameter Settings + +In the list of parameters, filter the parameters to find the values for `rds.logical_replication`, `max_replication_slots`, and `max_wal_senders`. For example: + +![](image???) + +The values should be: + +- `rds.logical_replication`: 1 (which will subsequently set the PostgreSQL setting of `wal_level` to `logical`). +- `max_replication_slots`: Greater than or equal to 1 +- `max_wal_senders`: Greater than or equal to `max_replication_slots`, depending on expected replication concurrency + +To determine the parameter values for a given parameter group, use the `describe-db-parameters` subcommand. For example: + +```command {title="Obtain parameter values for a given parameter group"} +aws rds describe-db-parameters \ + --region {{< placeholder "AWS_REGION" >}} \ + --db-parameter-group-name {{< placeholder "PARAMETER_GROUP_NAME" >}} \ + --query "Parameters[?ParameterName=='rds.logical_replication' || ParameterName=='max_replication_slots' || ParameterName=='max_wal_senders'].[ParameterName, ParameterValue, ApplyType, Description]" +``` + +```output +[ + [ + "max_replication_slots", + "20", + "static", + "Sets the maximum number of replication slots that the server can support." + ], + [ + "max_wal_senders", + "35", + "static", + "Sets the maximum number of simultaneously running WAL sender processes." + ], + [ + "rds.logical_replication", + "0", + "static", + "Enables logical decoding." + ] +] +``` + +If these are already set correctly, you can skip ahead in this guide. Otherwise, you will need to modify the parameter group. + +### Step 3: Determine If the Parameter Group Is Modifiable + +Your RDS instance may be using the default parameter group. You can determine this under the **Details** for this parameter group to see if it is "Default" or "Custom". + +![](image2.png) + +The default parameter groups cannot be modified. If your RDS instance is using the default parameter group, you will need to create a custom parameter group to make the necessary modifications. + +Create a new custom parameter group by navigating to **Parameter groups**. Click **Create parameter group**. Specify a name and description for the parameter group. Choose "PostgreSQL" for the engine type, then choose the appropriate parameter group family based on the version of PostgreSQL used by your RDS instance. Click **Create**. + +![](image6.png) + +This creates a parameter group that begins with values that match the default parameter group. + +You can also create the parameter group with the CLI. For example: + +```command {title="Create a new parameter group based on the postgres16 parameter group family"} +aws rds create-db-parameter-group \ + --region {{< placeholder "AWS_REGION" >}} \ + --db-parameter-group-name with-logical-replication-on \ + --db-parameter-group-family postgres16 \ + --description "sets wal_level to logical" +``` + +```output +{ + "DBParameterGroup": { + "DBParameterGroupName": "with-logical-replication-on", + "DBParameterGroupFamily": "postgres16", + "Description": "sets wal_level to logical", + ... + } +} +``` + +On the details page for the newly created parameter group, click **Edit**. Find each of the three parameters that may need modification. Set the values as required. Click **Save Changes**. + +![](image8.png) + +With the CLI, you can set parameter values as follows: + +```command {title="Modify parameter values for a custom parameter group"} +aws rds modify-db-parameter-group \ + --region {{< placeholder "AWS_REGION" >}} \ + --db-parameter-group-name with-logical-replication-on \ + --parameters \ +"ParameterName=rds.logical_replication,ParameterValue=1,ApplyMethod=pending-reboot" \ +"ParameterName=max_replication_slots,ParameterValue=10,ApplyMethod=pending-reboot" \ +"ParameterName=max_wal_senders,ParameterValue=10,ApplyMethod=pending-reboot" +``` + +### Step 4: Associate the Parameter Group with Your RDS Instance + +Navigate to **Databases**, select your instance, and choose **Modify**. Under **Additional configuration**, set the DB parameter group to your modified (or newly created) parameter group. + +![](image1.png) + +Click **Continue**. For the modification schedule, select **Apply immediately**, then click **Modify DB instance**. + +With the AWS CLI, run the following command to associate your RDS instance with the parameter group: + +```command {title="Modify parameter group for an RDS instance"} +aws rds modify-db-instance \ + --region {{< placeholder "AWS_REGION" >}} \ + --db-instance-identifier {{< placeholder "DB-INSTANCE-ID" >}} \ + --db-parameter-group-name {{< placeholder "PARAMETER_GROUP_NAME" >}} \ + --apply-immediately +``` + +### Step 5: Reboot the RDS Instance + +After applying the parameter group, reboot the database to apply the changes. For example + +```command {title="Reboot RDS instance"} +aws rds reboot-db-instance \ + --region {{< placeholder "AWS_REGION" >}} \ + --db-instance-identifier {{< placeholder "DB_INSTANCE_ID" >}} +``` + +## Configure Network Access + +Before the Linode Managed Database can connect to your RDS instance, you must ensure that the RDS instance allows network access from the Linode database host. + +Under **Connectivity & security** for your database instance, find and click on the security group associated with the instance. + +To get the name of the security group with the AWS CLI, run the following command: + +```command {title="Retrieve name of security group for RDS instance"} +aws rds describe-db-instances \ + --db-instance-identifier {{< placeholder "DB_INSTANCE_ID" >}} \ + --region {{< placeholder "AWS_REGION" >}} \ + --query "DBInstances[0].{SecurityGroups:VpcSecurityGroups[*].VpcSecurityGroupId}" +``` + +```output +{ + "SecurityGroups": [ + "sg-78944f70" + ] +} +``` + +Examine the inbound rules for the security group. With the CLI, use the following command: + +```command {title="Retrieve inbound rules for a security group"} +aws ec2 describe-security-groups \ + --group-ids sg-78944f70 \ + --region {{< placeholder "AWS_REGION" >}} \ + --query "SecurityGroups[0].IpPermissions" +``` + +```output +[ + { + "FromPort": 5432, + "IpProtocol": "tcp", + "IpRanges": [ + { + "CidrIp": "..." + }, + { + "CidrIp": "..." + } + ], + "Ipv6Ranges": [], + "PrefixListIds": [], + "ToPort": 5432, + "UserIdGroupPairs": [] + }, + ... +] +``` + +If an existing rule already allows access from `0.0.0.0/0`, then your Linode Managed Database likely already has access. However, it’s considered a security best practice to scope access more narrowly when possible. + +If no rule exists that would allow access by our Linode Managed Database, click **Edit inbound rules**. Then, click **Add rule**. Set the rule type to "PostgreSQL" (TCP protocol to port `5432`). For source, add the CIDR block of your Linode Managed Database host. + +![](image7.png) + +Click **Save rules**. + +Using the AWS CLI, you would run the following command to add the inbound rule: + +```command {title="Add a CIDR block to allow inbound PostgreSQL traffic"} +aws ec2 authorize-security-group-ingress \ + --group-id sg-78944f70 \ + --region {{< placeholder "AWS_REGION" >}} \ + --protocol tcp \ + --port 5432 \ + --cidr 172.232.188.122/32 +``` + +With network access configured, your Linode Managed Database should be able to reach the RDS instance during the subscription creation step in the main guide. + +## Create a Replication User + +While logical replication can technically be performed using the master user, it's strongly recommended to create a dedicated user with the `REPLICATION` privilege and appropriate access to the tables being published. Follow these steps to create this limited-privileges user on your RDS instance. + +Connect to your RDS PostgreSQL instance using the `psql` client. The default database name for RDS PostgreSQL instances is `postgres`. + +```command +psql -h {{< placeholder "RDS_ENDPOINT" >}} -p {{< placeholder "RDS_PORT" >}} -U {{< placeholder "MASTER_USERNAME" >}} -d postgres +``` + +Create a user, then grant it replication privileges. In RDS, this means attaching the `rds_replication` role. Then, grant SELECT privileges for the tables you plan to replicate. For example: + +```command {title="Create user and grant privileges"} +CREATE ROLE linode_replicator LOGIN PASSWORD 'thisismyreplicatorpassword'; +GRANT rds_replication TO linode_replicator; +GRANT SELECT ON customers, products, orders TO linode_replicator; +``` + +```output +CREATE ROLE +GRANT ROLE +GRANT +``` + +Replace the table names with your actual schema as needed. For simplicity, this example assumes a public schema and three tables typically found in an ecommerce database. Alternatively, you can grant privileges on all tables with the following command: + +```command {title="Grant select privileges on all tables"} +GRANT SELECT ON ALL TABLES in SCHEMA public to linode_replicator; +``` + +The newly created user (for example: `linode_replicator`) will be referenced by the Linode Managed Database when creating the subscription. + +## Create a Publication + +A publication defines which tables and changes (inserts, updates, deletes) should be streamed to the subscriber. You'll need at least one publication for logical replication. + +While still connected via the `psql` client, create a publication for specific tables. For example: + +```command {title="PostgreSQL command to create a publication for specific tables"} +CREATE PUBLICATION my_publication FOR TABLE customers, products, orders; +``` + +Alternatively, you can create a publication for all current and future tables in the database: + +```command {title="PostgreSQL command to create a publication for all tables"} +CREATE PUBLICATION my_publication FOR ALL TABLES; +``` + +The subscriber must have matching tables with compatible schemas for replication to succeed. + +To view any created publications, run the following command: + +```command {title="PostgreSQL command to see existing publications"} +SELECT * FROM pg_publication_tables; +``` + +```output +-[ RECORD 1 ]----------------------------------------------- +pubname | my_publication +schemaname | public +tablename | customers +attnames | {id,name,email,created_at} +rowfilter | +-[ RECORD 2 ]----------------------------------------------- +pubname | my_publication +schemaname | public +tablename | products +attnames | {id,name,price,in_stock} +rowfilter | +-[ RECORD 3 ]----------------------------------------------- +pubname | my_publication +schemaname | public +tablename | orders +attnames | {id,customer_id,product_id,quantity,order_date} +rowfilter | +``` + +Your source database is now ready for logical replication. Return to the main guide to configure the Linode Managed Database and create the subscription. + +## Additional Resources + +The resources below are provided to help you become familiar with logical replication with a PostgreSQL database when working with AWS RDS. + +- AWS: + - [Setting up PostgreSQL logical replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_MultiAZDBCluster_LogicalRepl.html#multi-az-db-clusters-logical-replication) + - [Parameter groups for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) + - [CLI documentation for AWS RDS](https://docs.aws.amazon.com/cli/latest/reference/rds/) +- PostgreSQL: + - [Logical replication](https://www.postgresql.org/docs/current/logical-replication.html) + - [CREATE SUBSCRIPTION](https://www.postgresql.org/docs/current/sql-createsubscription.html) \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-firewall-rule-linode-managed-database.png b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-firewall-rule-linode-managed-database.png new file mode 100644 index 00000000000..a64214d6982 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-firewall-rule-linode-managed-database.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-navigation-server-parameters.png b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-navigation-server-parameters.png new file mode 100644 index 00000000000..203d6a8c7e0 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-navigation-server-parameters.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-networking-public-access.png b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-networking-public-access.png new file mode 100644 index 00000000000..6e0b3de102c Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-networking-public-access.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameter-wal-level-default.png b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameter-wal-level-default.png new file mode 100644 index 00000000000..d6a29891513 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameter-wal-level-default.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameters-save-logical-replication.png b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameters-save-logical-replication.png new file mode 100644 index 00000000000..276ec316b12 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/azure-flexible-server-parameters-save-logical-replication.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/index.md b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/index.md new file mode 100644 index 00000000000..1bd1a5e0d10 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database/index.md @@ -0,0 +1,279 @@ +--- +slug: preparing-your-azure-postgresql-database-for-logical-replication-to-linode-managed-database +title: "Preparing Your Azure PostgreSQL Database for Logical Replication to Linode Managed Database" +description: "Prepare your Azure Database for PostgreSQL for logical replication to a Linode Managed Database. Configure server parameters, network access, and a replication user." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-10-10 +keywords: ['azure postgresql logical replication','linode managed database','flexible server','pg_publication','azure replication setup'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +external_resources: +- '[Logical replication and logical decoding in Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical)' +- '[Server parameters in Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-server-parameters)' +- '[Azure CLI Documentation for `az postgres`](https://learn.microsoft.com/en-us/cli/azure/postgres?view=azure-cli-latest)' +--- + +[Logical replication](https://www.postgresql.org/docs/current/logical-replication.html) continuously synchronizes database tables, allowing you to prepare the destination database in advance. This approach minimizes downtime when you switch application traffic and retire the source database. + +This guide explains how to prepare an Azure Database for PostgreSQL for logical replication to a [Linode Managed Database](https://www.linode.com/products/databases/). Follow this guide before returning to the [Logical Replication to a Linode Managed PostgreSQL Database](/docs/guides/logical-replication-to-a-linode-managed-postgresql-database/) guide to [create the subscription](https://www.postgresql.org/docs/current/sql-createsubscription.html) on Akamai Cloud. + +The steps presented in this guide cover: + +- Configuring your Azure Database instance to support logical replication. +- Ensuring secure network access from Linode. +- Creating a dedicated replication user. +- Setting up a publication for the tables you wish to replicate. + +After completing these steps, return to the main replication guide to configure the subscriber and finalize the setup. + +## Before You Begin + +1. Follow the [Logical Replication to a Linode Managed PostgreSQL Database](/docs/guides/logical-replication-to-a-linode-managed-postgresql-database/) guide up to the **Prepare the Source Database for Logical Replication** section to obtain the public IP address or CIDR range of your Linode Managed Database host. + +1. Ensure your Azure account has permissions to modify PostgreSQL server parameters, networking settings, and firewall rules. + +1. Install and authenticate the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) on your local machine: + + ```command + az login + az account set --subscription {{< placeholder "YOUR_AZURE_SUBSCRIPTION_ID" >}} + ``` + +{{< note >}} +Ensure you have an *Azure Database for PostgreSQL – Flexible Server* instance. Logical replication requires Flexible Server, as Single Server was retired in March 2025. +{{< /note >}} + +### Placeholders and Examples + +The following placeholders and example values are used in commands throughout this guide: + +| Parameter | Placeholder | Example Value | +|------------|--------------|----------------| +| Azure Server Name | {{< placeholder "AZURE_SERVER_NAME" >}} | `source-database` | +| Azure Resource Group | {{< placeholder "AZURE_RESOURCE_GROUP" >}} | `pg-repl-rg` | +| Source Hostname or IP Address | {{< placeholder "SOURCE_HOST" >}} | `source-database.postgres.database.azure.com` | +| Source Port Number | {{< placeholder "SOURCE_PORT" >}} | `5432` | +| Source Database Name | {{< placeholder "SOURCE_DB" >}} | `postgres` | +| Source Username | {{< placeholder "SOURCE_USER" >}} | `azureadmin` | +| Source Password | {{< placeholder "SOURCE_PASSWORD" >}} | `thisismysourcepassword` | +| Replication Username | {{< placeholder "REPL_USER" >}} | `linode_replicator` | +| Replication Password | {{< placeholder "REPL_PASSWORD" >}} | `thisismyreplicatorpassword` | +| Publication Name | {{< placeholder "PUBLICATION_NAME" >}} | `my_publication` | + +Replace these placeholders with your own connection details when running commands in your environment. + +Additionally, the examples used in this guide assume the source database contains three tables (`customers`, `products`, and `orders`) that you want to replicate to a Linode Managed Database. + +## Configure Server Parameters + +To support logical replication, you’ll need to adjust a few parameters on your Azure Database for PostgreSQL instance. + +{{< tabs >}} +{{< tab "Azure Portal" >}} +1. In the Azure Portal, locate your database resource, then navigate to **Settings > Server parameters**: + + ![Azure Database for PostgreSQL Flexible Server - Navigation menu with Server parameters selected.](azure-flexible-server-navigation-server-parameters.png) + +1. In the list of server parameters, use the search filter to find the values for `wal_level`, `max_replication_slots`, and `max_wal_senders`: + + ![Azure portal Server parameters page showing the wal_level parameter set to REPLICA by default.](azure-flexible-server-parameter-wal-level-default.png) + + The values should be: + + - `wal_level`: `LOGICAL` + - `max_replication_slots`: Greater than or equal to 1 + - `max_wal_senders`: Greater than or equal to `max_replication_slots`, depending on expected replication concurrency + +1. Adjust the values as needed, then click **Save**: + + ![Azure portal Server parameters page showing wal_level set to LOGICAL, max_wal_senders and max_replication_slots set to 5, with the Save button highlighted.](azure-flexible-server-parameters-save-logical-replication.png) + +1. When Azure notifies you that you need to restart the server for changes to take effect, click **Save and Restart**. + +{{< /tab >}} +{{< tab "Azure CLI" >}} +1. Run the following `az` CLI command to list the relevant server parameters for the instance: + + ```command + az postgres flexible-server parameter list \ + --server-name {{< placeholder "AZURE_DATABASE_SERVER" >}} \ + --resource-group {{< placeholder "AZURE_RESOURCE_GROUP" >}} \ + --output json \ + --query "[?name=='wal_level' || name=='max_replication_slots' || name=='max_wal_senders'].{name:name, description:description, dataType:dataType, value:value}" + ``` + + ```output + [ + { + "dataType": "Integer", + "description": "Specifies the maximum number of replication slots that the server can support.", + "name": "max_replication_slots", + "value": "10" + }, + { + "dataType": "Integer", + "description": "Sets the maximum number of simultaneously running WAL sender processes.", + "name": "max_wal_senders", + "value": "10" + }, + { + "dataType": "Enumeration", + "description": "It determines how much information is written to the WAL.", + "name": "wal_level", + "value": "REPLICA" + } + ] + ``` + + The values should be: + + - `wal_level`: `LOGICAL` + - `max_replication_slots`: Greater than or equal to 1 + - `max_wal_senders`: Greater than or equal to `max_replication_slots`, depending on expected replication concurrency + + To adjust the values with the Azure CLI, you need to run the `parameter set` command for each parameter. + +1. Use the following command to adjust the value of `wal_level` to `logical`: + + ```command {title="Modify server parameters to support logical replication"} + az postgres flexible-server parameter set \ + --server-name {{< placeholder "AZURE_DATABASE_SERVER" >}} \ + --resource-group {{< placeholder "AZURE_RESOURCE_GROUP" >}} \ + --name wal_level \ + --value logical + ``` + +1. Use the following command to adjust the value of `max_replication_slots` to `5`: + + ```command + az postgres flexible-server parameter set \ + --server-name {{< placeholder "AZURE_DATABASE_SERVER" >}} \ + --resource-group {{< placeholder "AZURE_RESOURCE_GROUP" >}} \ + --name max_replication_slots \ + --value 5 + ``` + +1. Use the following command to adjust the value of `max_wal_senders` to `5`: + + ```command + az postgres flexible-server parameter set \ + --server-name {{< placeholder "AZURE_DATABASE_SERVER" >}} \ + --resource-group {{< placeholder "AZURE_RESOURCE_GROUP" >}} \ + --name max_wal_senders \ + --value 5 + ``` + +1. After modifying these parameters, restart the database instance: + + ```command + az postgres flexible-server restart \ + --name {{< placeholder "AZURE_DATABASE_SERVER" >}} \ + --resource-group {{< placeholder "AZURE_RESOURCE_GROUP" >}} + ``` +{{< /tab >}} +{{< /tabs >}} + +## Configure Network Access + +Before the Linode Managed Database can connect to your Azure Database instance, ensure that the instance allows network access from the Linode database host. + +1. Navigate to the **Settings > Networking** page for the instance. Make sure that the **Public access** option is checked. + + ![Azure Database for PostgreSQL Networking page showing the Public access option enabled.](azure-flexible-server-networking-public-access.png) + +1. In the list of firewall rules, add a rule to allow access to your Linode Managed Database. Specify a name for the firewall rule. Enter the IP address of your Linode Managed Database host as both the **Start IP address** and the **End IP address**: + + ![Azure PostgreSQL Networking page showing a firewall rule named allow-linode-managed-database with the Linode host IP entered for Start and End addresses.](azure-flexible-server-firewall-rule-linode-managed-database.png) + +1. Click **Save** at the top of the page. + +With network access configured, your Linode Managed Database can reach the Azure Database instance during the subscription creation step in the main guide. + +## Create a Replication User + +The source database should have a dedicated replication user with the `REPLICATION` attribute and `SELECT` privileges to the tables to be replicated. While logical replication can be performed as administrator, it's a security best practice to create a dedicated user. + +Follow the steps below to create this scope-limited user on your Azure Database instance. + +1. Connect to your instance using the `psql` client and the connection string information found on the **Settings > Connect** page, replacing any placeholders with your own values: + + ```command + psql "host={{< placeholder "AZURE_SERVER_NAME" >}}.postgres.database.azure.com \ + port=5432 \ + dbname={{< placeholder "SOURCE_DB" >}} \ + user={{< placeholder "SOURCE_USER" >}} \ + password={{< placeholder "SOURCE_PASSWORD" >}} \ + sslmode=require" + ``` + +1. After connecting successfully, create a user with replication privileges (e.g., `linode_replicator`), provide a password (e.g., `thisismyreplicatorpassword`), then grant SELECT privileges for the tables you plan to replicate. For simplicity, this example assumes a public schema and three tables typically found in an ecommerce database (e.g., `customers`, `products`, and `orders`). Replace the table names with your actual schema as needed: + + ```command {title="Source psql Prompt"} + CREATE ROLE linode_replicator + WITH REPLICATION + LOGIN PASSWORD 'thisismyreplicatorpassword'; + GRANT SELECT ON customers, products, orders TO linode_replicator; + ``` + + ```output + CREATE ROLE + GRANT + ``` + + {{< note >}} + Alternatively, you can grant privileges on all tables with the following command: + + ```command {title="Source psql Prompt"} + GRANT SELECT ON ALL TABLES in SCHEMA public to linode_replicator; + ``` + {{< /note >}} + +The newly created user is referenced by the Linode Managed Database when creating the subscription. + +## Create a Publication + +A publication defines which tables and changes (inserts, updates, deletes) should be streamed to the subscriber. You need at least one publication for logical replication. + +1. While still connected via the `psql` client, create a publication (e.g., `my_publication`) for specific tables (e.g., `customers` , `products`, and `orders`): + + ```command {title="Source psql Prompt"} + CREATE PUBLICATION {{< placeholder "PUBLICATION_NAME" >}} FOR TABLE customers, products, orders; + ``` + + {{< note >}} + The subscriber must have matching tables with compatible schemas for replication to succeed. Alternatively, you can create a publication for all current and future tables in the database: + + ```command {title="Source psql Prompt"} + CREATE PUBLICATION {{< placeholder "PUBLICATION_NAME" >}} FOR ALL TABLES; + ``` + {{< /note >}} + + +1. Run the following command to view any created publications: + + ```command {title="Source psql Prompt"} + SELECT * FROM pg_publication_tables; + ``` + + ```output + -[ RECORD 1 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | customers + attnames | {id,name,email,created_at} + rowfilter | + -[ RECORD 2 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | products + attnames | {id,name,price,in_stock} + rowfilter | + -[ RECORD 3 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | orders + attnames | {id,customer_id,product_id,quantity,order_date} + rowfilter | + ``` + +Your Azure source database is now ready for logical replication. Return to the main guide to configure the Linode Managed Database and create the subscription. \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png new file mode 100644 index 00000000000..f9af56f2455 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image2.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png new file mode 100644 index 00000000000..9a9861b722f Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image3.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image4.png b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image4.png new file mode 100644 index 00000000000..6d40ff8a884 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image4.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png new file mode 100644 index 00000000000..818e1547323 Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image5.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png new file mode 100644 index 00000000000..276db86580d Binary files /dev/null and b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/image6.png differ diff --git a/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/index.md b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/index.md new file mode 100644 index 00000000000..92162336a29 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database/index.md @@ -0,0 +1,225 @@ +--- +slug: preparing-your-google-cloud-sql-postgresql-database-for-logical-replication-to-linode-managed-database +title: "Preparing Your Google Cloud SQL PostgreSQL Database for Logical Replication to Linode Managed Database" +description: "Two to three sentences describing your guide." +og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-10-10 +keywords: ['list','of','keywords','and key phrases'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +external_resources: +- '[Link Title 1](http://www.example.com)' +- '[Link Title 2](http://www.example.net)' +--- + +This guide explains how to prepare a PostgreSQL instance hosted on Google Cloud SQL (publisher) for logical replication to a Linode Managed Database (subscriber). If you're following the "Logical Replication to a Linode Managed PostgreSQL Database" guide, use this document to complete the GCP-side preparation before returning to create the subscription on Linode. + +Following the steps in this guide will help you to: + +1. Configure your Cloud SQL instance to support logical replication. +1. Ensure secure network access from Linode. +1. Create a dedicated replication user. +1. Set up a publication for the tables you wish to replicate. + +After completing these steps, return to the main replication guide to configure the subscriber and finalize the setup. + +## Prerequisites + +This guide applies to **Cloud SQL for PostgreSQL**, Google's managed PostgreSQL offering. + +In order to follow the steps in this guide, you need: + +- Administrative access to your GCP project, including permissions to modify Cloud SQL instance flags and authorized networks. +- The Google Cloud CLI (`gcloud`) installed and authenticated. +- The IP address or CIDR range for the Linode Managed Database's host, so you can configure inbound access on the Cloud SQL instance. + +## Configure Database Flags + +To support logical replication, you’ll need to adjust a few flags on your Cloud SQL PostgreSQL instance. These changes can be made from the Google Cloud Console or with `gcloud`. + +Logical replication requires enabling specific PostgreSQL flags on your Cloud SQL for PostgreSQL instance. You can set these from the Google Cloud Console or using the gcloud CLI. + +{{< tabs >}} +{{< tab "Google Cloud Console" >}} +1. In the Google Cloud Console, navigate to **SQL** and select your PostgreSQL instance. + + ![](image5.png) + +1. On the instance page, click **Edit**. + +1. Locate the **Flags and parameters** section, then click **Add a database flag**. + +1. Add the following flags: + + - `cloudsql.logical_decoding`: `On` (sets `wal_level` to `logical`) + - `max_replication_slots`: `10` or higher + - `max_wal_senders`: Greater than or equal to `max_replication_slots`, depending on expected replication concurrency + + ![](image6.png) + +1. Click **Save** at the bottom of the page. + +1. When prompted, click **Save and restart** to restart and apply the changes. + + ![](image3.png) + +{{< /tab >}} +{{< tab "`gcloud` CLI" >}} +To set CloudSQL instance database flags from the CLI, run the following command: + +```command +gcloud sql instances patch {{< placeholder "INSTANCE_NAME" >}} \ + --database-flags=cloudsql.logical_decoding=on,max_replication_slots=4,max_wal_senders=5 +``` + +```output +The following message will be used for the patch API method. + +{ + "name": "source-database", + "settings": { + "databaseFlags": [ + {"name": "cloudsql.logical_decoding", "value": "off"}, + {"name": "max_replication_slots", "value": "10"}, + {"name": "max_wal_senders", "value": "10"} + ] + } +} + +WARNING: This patch modifies database flag values, which may require your +instance to be restarted. Check the list of supported flags - +https://cloud.google.com/sql/docs/postgres/flags - to see if your +instance will be restarted when this patch is submitted. + +Do you want to continue (Y/n)? +``` + +Confirm the request to restart the instance. +{{< /tab >}} +{{< /tabs >}} + +## Configure Network Access + +Before the Linode Managed Database can connect to your Cloud SQL instance, you must ensure that the instance allows network access from the Linode database host. + +{{< tabs >}} +{{< tab "Google Cloud Console" >}} +1. In the Google Cloud Console, open your Cloud SQL instance. + +1. Navigate to the **Connections** page, then select the **Networking** tab. + +1. Ensure that the **Public IP** option is checked: + + ![](image4.png) + +1. In the list of Authorized networks, add the CIDR range of your Linode Managed Database host: + + ![](image2.png) + +1. Click **Save** at the bottom of the page. +{{< /tab >}} +{{< tab "`gcloud` CLI" >}} +You can also add authorized networks with the `gcloud` CLI, however, you are only able to set the CIDR range values (as a comma-separated list) and cannot specify a name for each network. For example: + +```command +gcloud sql instances patch {{< placeholder "INSTANCE_NAME" >}} \ + --authorized-networks="172.232.188.122/32" +``` +{{< /tab >}} +{{< /tabs >}} + +With network access configured, your Linode Managed Database should be able to reach the Cloud SQL instance during the subscription creation step in the main guide. + +## Create a Replication User + +While logical replication can technically be performed using the primary database user, it's best practice to create a dedicated replication user. This user should have the `REPLICATION` privilege and `SELECT` access to the tables being published. + +Follow the steps below to create this limited-privileges user on your Cloud SQL instance. + +1. Connect to your instance using the `psql` client and the connection details shown under **Connections > Summary** in the Google Cloud Console. + +1. Create a replication user and grant `SELECT` privileges for the tables you plan to replicate. Replace the table names with your actual schema as needed. For simplicity, this example assumes a public schema and three tables typically found in an ecommerce database: + + ```command + CREATE ROLE linode_replicator + WITH REPLICATION + LOGIN PASSWORD 'thisismyreplicatorpassword'; + GRANT SELECT ON customers, products, orders TO linode_replicator; + ``` + + ```output + CREATE ROLE + GRANT + ``` + +The newly created user (e.g., `linode_replicator`) is referenced by the Linode Managed Database when creating the subscription in the main replication guide. + +{{< note >}} +Alternatively, you can grant privileges on all tables with the following command: + +```command +GRANT SELECT ON ALL TABLES in SCHEMA public to linode_replicator; +``` +{{< /note >}} + +## Create a Publication + +A publication defines which tables and changes (e.g., `INSERT`, `UPDATE`, and `DELETE`) should be streamed to the subscriber. At least one publication is required for logical replication. + +1. While still connected to your Cloud SQL instance via `psql`, create a publication for the specific tables you want to replicate. For example: + + ```command + CREATE PUBLICATION my_publication FOR TABLE customers, products, orders; + ``` + + {{< note >}} + Alternatively, you can create a publication for all current and future tables in the database: + + ```command + CREATE PUBLICATION my_publication FOR ALL TABLES; + ``` + {{< /note >}} + +1. The subscriber database already contains matching tables and compatible schemas. Replication will fail if table definitions differ between the publisher and subscriber. + +1. Run the following command to view all existing publications: + + ```command + SELECT * FROM pg_publication_tables; + ``` + + ```output + -[ RECORD 1 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | customers + attnames | {id,name,email,created_at} + rowfilter | + -[ RECORD 2 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | products + attnames | {id,name,price,in_stock} + rowfilter | + -[ RECORD 3 ]----------------------------------------------- + pubname | my_publication + schemaname | public + tablename | orders + attnames | {id,customer_id,product_id,quantity,order_date} + rowfilter | + ``` + +Your source database is now ready for logical replication. Return to the main guide to configure the Linode Managed Database and create the subscription. + +## Additional Resources + +The resources below are provided to help you become familiar with logical replication with a PostgreSQL database when working with Google Cloud SQL PostgreSQL. + +- Google Cloud SQL PostgreSQL: + - [Setting up PostgreSQL logical replication](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication) + - [Documentation](https://cloud.google.com/sql/docs/postgres) + - [CLI documentation for gcloud sql](https://cloud.google.com/sdk/gcloud/reference/sql) +- PostgreSQL: + - [Logical replication](https://www.postgresql.org/docs/current/logical-replication.html) + - [CREATE SUBSCRIPTION](https://www.postgresql.org/docs/current/sql-createsubscription.html) \ No newline at end of file