Skip to content

feat: implement filter in cdc and incremental#756

Merged
hash-data merged 77 commits intostagingfrom
feat/filter-in-cdc&incremental
Mar 23, 2026
Merged

feat: implement filter in cdc and incremental#756
hash-data merged 77 commits intostagingfrom
feat/filter-in-cdc&incremental

Conversation

@vikaxsh
Copy link
Copy Markdown
Collaborator

@vikaxsh vikaxsh commented Jan 14, 2026

Description

This PR adds filtering capability for CDC and incremental sync modes, allowing users to selectively sync records based on configurable filter conditions.

Fixes # (issue)

Type of change

  • Multiple Conditions: Support for multiple filter conditions with AND/OR logical operators
  • Operators: Supports =, !=, <, >, <=, >=
  • Type-Aware Filtering: Automatic type conversion and validation based on schema
  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Comprehensive test coverage in destination/filter_test.go including:

  • Legacy filter support

  • Multiple operators and data types

  • AND/OR logic

  • Null value handling

  • Type coercion

  • Edge cases

  • Scenario A

  • Scenario B

Screenshots or Recordings

Documentation

Related PR's (If Any):

New Filter Format

Filters can be specified in the stream configuration using the new FilterInput structure:

{
  "selected_streams": {
    "namespace": [
      {
        "stream_name": "table_1",
        "filter_input": {
          "logical_operator": "AND",
          "conditions": [
            {
              "column": "id",
              "operator": ">",
              "value": 100
            },
            {
              "column": "status",
              "operator": "=",
              "value": "active"
            }
          ]
        }
      }
    ]
  }
}

Legacy Filter Format (Still Supported)

The legacy string-based filter format continues to work:

{
  "filter": "id > 100 AND status = \"active\""
}

@vikaxsh vikaxsh force-pushed the feat/filter-in-cdc&incremental branch from b75a508 to ab0848f Compare January 20, 2026 10:26
@vikaxsh vikaxsh marked this pull request as ready for review January 21, 2026 08:21
Comment thread constants/constants.go Outdated
Comment thread destination/filter.go Outdated
Comment thread destination/parquet/parquet.go Outdated
Comment thread types/catalog.go Outdated
Comment thread types/stream_configured.go
Comment thread types/stream_configured.go Outdated
Comment thread drivers/mongodb/internal/backfill.go
Comment thread drivers/mongodb/internal/mon_test.go Outdated
Comment thread drivers/mongodb/internal/mon_test_util.go Outdated
Comment thread pkg/jdbc/jdbc.go Outdated
Comment thread pkg/jdbc/jdbc.go Outdated
Comment thread pkg/jdbc/jdbc.go Outdated
Comment thread pkg/jdbc/jdbc.go Outdated
Comment thread pkg/jdbc/jdbc.go Outdated
Comment thread drivers/oracle/internal/oracle_test.go
Comment thread destination/writers.go Outdated
Comment thread protocol/sync.go Outdated
Comment thread destination/writers.go
Comment thread drivers/mysql/internal/mysql_test.go Outdated
@vikaxsh vikaxsh had a problem deploying to integration_tests March 12, 2026 11:28 — with GitHub Actions Failure
@vikaxsh vikaxsh temporarily deployed to integration_tests March 12, 2026 12:04 — with GitHub Actions Inactive
Comment thread utils/typeutils/reformat.go Outdated
@vikaxsh vikaxsh temporarily deployed to integration_tests March 16, 2026 05:53 — with GitHub Actions Inactive
vishalm0509
vishalm0509 previously approved these changes Mar 16, 2026
Comment thread protocol/sync.go
Comment thread utils/typeutils/reformat.go Outdated
@vikaxsh vikaxsh had a problem deploying to integration_tests March 16, 2026 12:29 — with GitHub Actions Failure
@vikaxsh vikaxsh had a problem deploying to integration_tests March 16, 2026 12:48 — with GitHub Actions Failure
@vikaxsh vikaxsh temporarily deployed to integration_tests March 16, 2026 13:13 — with GitHub Actions Inactive
Comment thread types/stream_configured.go
Comment thread types/stream_configured.go
@shubham19may shubham19may temporarily deployed to integration_tests March 18, 2026 17:12 — with GitHub Actions Inactive
@hash-data hash-data temporarily deployed to integration_tests March 20, 2026 12:08 — with GitHub Actions Inactive
@hash-data hash-data merged commit 8208b6b into staging Mar 23, 2026
10 checks passed
@hash-data hash-data deleted the feat/filter-in-cdc&incremental branch March 23, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants