feat: Yugabyte CQL DDL issues fixes, curio set to log debug, e2e-test-pdp-caching-layers for cache testing

[redpanda-f]

One-line summary

• Adds an e2e GitHub Actions workflow to test PDP caching and fixes CQL/Curio issues.

What changed (short)

• Added: .github/workflows/e2e-test-pdp-caching-layers.yml — E2E workflow that builds + runs a devnet, runs a Synapse storage e2e, and asserts PDP caching behavior in YCQL.
• Updated: src/commands/start/yugabyte/mod.rs — removed --advertise_address=0.0.0.0 to fix gocql discovery/reconnect issues.
• Updated: src/commands/start/curio/{db_setup.rs,daemon.rs} — inject GOLOG_LOG_LEVEL=pdp=debug into Curio containers for better PDP debugging.

Why

• Fixes a connector/discovery bug that prevented CQL DDL and subsequent queries from working reliably.
• Adds automated verification that PDP SaveCache/SaveLayer flows populate pdp_cache_layer and clear needs_save_cache.

Quick test notes

• Run the workflow on a capable self-hosted runner or locally build/start the devnet and run the workflow steps.
• Check Curio logs for PDPv0_SaveCache and that curio.pdp_cache_layer contains rows.

Notes for reviewers

• The workflow downloads large proof params and expects a powerful self-hosted runner.
• Removing --advertise_address relies on Docker network auto-detection — verify this is acceptable for tests.

[redpanda-f]

@rjan90 you may need to tag this properly, this is 4.2

[Copilot]

Pull request overview

This PR addresses Yugabyte CQL connectivity issues with Curio's gocql driver, enables debug logging for PDP operations, and introduces a comprehensive end-to-end test workflow for PDP caching functionality. The changes fix a critical networking issue where gocql would fail to connect to YugabyteDB due to an incorrect advertise_address configuration.

Changes:

• Fixed Yugabyte networking by removing --advertise_address=0.0.0.0 to allow gocql to properly discover and connect to YugabyteDB instances
• Added debug logging (GOLOG_LOG_LEVEL=pdp=debug) to Curio containers for better observability of PDP operations
• Introduced a new GitHub Actions workflow for end-to-end testing of PDP caching layers, including storage operations, cache validation, and database state verification

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
src/commands/start/yugabyte/mod.rs	Removed --advertise_address=0.0.0.0 flag and added detailed explanation of why this fixes gocql connection issues
src/commands/start/curio/db_setup.rs	Added debug log level environment variable for PDP logging in database setup containers; introduced unused import
src/commands/start/curio/daemon.rs	Added debug log level environment variable for PDP logging in Curio daemon containers
.github/workflows/e2e-test-pdp-caching-layers.yml	New comprehensive workflow testing PDP caching functionality through multiple phases including cluster startup, Synapse SDK e2e tests, and cache validation

[Copilot]

The import use crate::commands::build::docker; is added but never used in this file. This will cause a compiler warning about an unused import. Please remove this import.

Suggested change

use crate::commands::build::docker;

[Copilot]

According to the coding guidelines, magic strings should be defined as constants. The string "GOLOG_LOG_LEVEL=pdp=debug" is duplicated in three locations (here, line 319, and daemon.rs:252). This should be defined as a constant in the constants module (e.g., pub const CURIO_PDP_DEBUG_LOG_LEVEL: &str = "GOLOG_LOG_LEVEL=pdp=debug";) and referenced from there.

[Copilot]

According to the coding guidelines, magic strings should be defined as constants. The string "GOLOG_LOG_LEVEL=pdp=debug" is duplicated in three locations (line 206, here, and daemon.rs:252). This should be defined as a constant in the constants module (e.g., pub const CURIO_PDP_DEBUG_LOG_LEVEL: &str = "GOLOG_LOG_LEVEL=pdp=debug";) and referenced from there.

[Copilot]

According to the coding guidelines, magic strings should be defined as constants. The string "GOLOG_LOG_LEVEL=pdp=debug" is duplicated in three locations (db_setup.rs lines 206 and 319, and here). This should be defined as a constant in the constants module (e.g., pub const CURIO_PDP_DEBUG_LOG_LEVEL: &str = "GOLOG_LOG_LEVEL=pdp=debug";) and referenced from there.

[rvagg]

Up to phase 4 this workflow is nearly identical to ci.yml, but without caching. I think we would benefit from having this logic repeated only once where possible, otherwise this is going to be a maintenance headache.

You could make a composite action so it's all reusable, e.g. .github/actions/devnet-setup/action.yml and then uses: ./.github/actions/devnet-setup within both workflows that need those common components. You can even parameterise it (with:) if you need subtle differences. I think the main catches with composite actions are needing to take care with getting caching right (do it within the composite, don't rely on external), and it won't inherit shell, so shell: bash inside it. Otherwise it's going to save the pain of having two almost the same workflows and one of them just having additional steps.

There's an alternative that may just be simpler: add workflow_dispatch to ci.yml with a boolean input defaulting to true (e.g. pdp_caching_test), then add the additional steps in there but if: github.event_name == 'workflow_dispatch' && inputs.pdp_caching_test. It would need the component override logic from here duplicated in but gated by github.event_name == 'workflow_dispatch', and there's a --notest diff between the two, but again, could be gated if that's necessary.

[rvagg]

Seems fine to me, the Copilot notes are worth dealing with, and the huge semi-copypasta into the new workflow is a maintenance risk that we could minimise with a bit of crafting.