sink: Add schema/table routing support for mysql and redo sinks

[a-cong]

What problem does this PR solve?

Issue Number: close #3700

What is changed and how it works?

Porting changes from pingcap/tiflow#12435 to new ticdc architecture.

The dispatchers config is the same as in tiflow -

[[sink.dispatchers]]
matcher = ["uds_000.*"]
schema = "tidb_failover_test"
table = "{table}"

[[sink.dispatchers]]
matcher = ["oyster_production.*"]
schema = "oyster_production"
table = "{table}"

We also add the TargetSchema and TargetTable fields to the TableName struct. Since there is now a shared TableInfo object that can be accessed by multiple changefeeds and a new concept of Dispatcher objects, we clone the TableInfo in each dispatcher with the routing information for that table in handleHandshakeEvent (which is executed when the dispatcher starts), and then store that cloned/routed TableInfo for the dispatcher. Subsequent DMLs then look up the stored info. Subsequent DDLs also explicitly update routing in handleSingleDataEvents.

Another difference between ticdc and tiflow is that DDLs are now handled per-sink instead of in one single entry point. Because of this, we need to apply routing in both the mysql and redo sinks separately - the shared rewriting functionality is in pkg/sink/util/ddl_routing.go.

We also copied FetchDDLTables and RenameDDLTable (and the corresponding tests) from https://github.com/pingcap/tiflow/blob/master/dm/pkg/parser/common.go so we could use them to rewrite DDLs.

Check List

Tests

• Unit test
• Integration test (will add, in progress)
• Manual test - deployed to internal dev cluster and bootstrapped a migration shadow with schema mapping, verified changefeed routes DDLs/DMLs correctly

Questions

Do you need to update user documentation, design documentation or monitoring documentation?

Yes - will do so in separate PR

Release note

Support schema and table routing for mysql-compatible sinks by extending the dispatchers config

[ti-chi-bot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign flowbehappy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

Hi @a-cong. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

Welcome @a-cong!

It looks like this is your first PR to pingcap/ticdc 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/ticdc. 😃

[pingcap-cla-assistant]

All committers have signed the CLA.

[gemini-code-assist]

Summary of Changes

Hello @a-cong, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive schema and table routing capabilities for MySQL and Redo sinks. It allows users to configure rules to map source database and table names to different target names in the downstream, providing greater flexibility for data migration and synchronization scenarios. The changes involve extending configuration options, integrating a new routing engine into the event processing pipeline, and ensuring that both DDL and DML operations correctly apply these routing rules while maintaining data integrity and avoiding race conditions.

Highlights

• Schema and Table Routing Configuration: Introduced new SchemaRule and TableRule fields within the dispatchers configuration. These fields allow users to define expressions for mapping source schema and table names to different target names in the downstream, supporting placeholders like {schema} and {table}.
• Integrated Routing for DDL Operations: Implemented a new Router utility and integrated it into both MySQL and Redo sinks. DDL queries are now parsed and rewritten based on configured routing rules before being executed by the MySQL sink or written to the redo log. This ensures DDLs create/modify tables with the correct target names.
• Dynamic TableInfo Routing for DMLs: Modified the dispatcher logic to apply routing to TableInfo objects. When a dispatcher starts or receives a DDL event, it clones the TableInfo and sets TargetSchema and TargetTable fields based on routing rules. Subsequent DML SQL generation then uses these routed names, ensuring DMLs are written to the correct target tables.
• Shared DDL Parsing and Rewriting Utilities: Copied and adapted DDL parsing (FetchDDLTables) and rewriting (RenameDDLTable) utilities from the TiFlow project. These utilities leverage TiDB's parser to safely extract and modify table references within DDL statements, forming the foundation for DDL routing.
• Enhanced Data Safety and Consistency: Introduced cloning mechanisms for DDLEvent and TableInfo objects when routing is applied. This prevents data races and ensures that different changefeeds or components (like the event dispatcher and redo dispatcher) can independently apply their specific routing rules without inadvertently mutating shared event data.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces schema and table routing for MySQL and redo sinks, a significant and valuable feature. The implementation is well-structured, with routing logic centralized in a new Router utility and consistently applied to both DDL and DML paths. The approach of rewriting DDLs in the sink and applying routing to TableInfo for DMLs is sound. The addition of extensive tests covering various routing scenarios is commendable.

I've identified a critical race condition concerning BatchDMLEvent processing when redo logging is enabled. Similar to how DDLEvent is cloned to prevent data races between sinks, BatchDMLEvent also needs to be cloned before its in-place modification. I have provided detailed comments and suggestions to address this issue. Once this is fixed, the PR should be in good shape.