Skip to content

consumer: ignore fallback dml event #3864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ti-chi-bot merged 3 commits into from
Jan 5, 2026
+18 −5
Merged

consumer: ignore fallback dml event #3864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8cb71ac
init
wk989898 Dec 31, 2025
93736d8
update
wk989898 Dec 31, 2025
859a20d
chore
wk989898 Jan 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions cmd/kafka-consumer/writer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,14 @@ func (w *writer) appendRow2Group(dml *commonEvent.DMLEvent, progress *partitionP
group = util.NewEventsGroup(progress.partition, tableID)
progress.eventsGroup[tableID] = group
}
if commitTs < progress.watermark {
log.Warn("DML Event fallback row, since less than the partition watermark, ignore it",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Any("offset", offset),
zap.Uint64("watermark", progress.watermark), zap.Any("watermarkOffset", progress.watermarkOffset),
zap.String("schema", schema), zap.String("table", table))
Comment on lines +528 to +532
Copy link

@gemini-code-assist gemini-code-assist bot Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The log message is a bit verbose. The reason for ignoring the event ("since less than the partition watermark") is already evident from the logged commitTs and watermark fields. A more concise message would improve log readability and maintainability.

Suggested change
log.Warn("DML Event fallback row, since less than the partition watermark, ignore it",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Any("offset", offset),
zap.Uint64("watermark", progress.watermark), zap.Any("watermarkOffset", progress.watermarkOffset),
zap.String("schema", schema), zap.String("table", table))
log.Warn("Ignore fallback DML event",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Any("offset", offset),
zap.Uint64("watermark", progress.watermark), zap.Any("watermarkOffset", progress.watermarkOffset),
zap.String("schema", schema), zap.String("table", table))

return
}
if commitTs >= group.HighWatermark {
group.Append(dml, false)
log.Info("DML event append to the group",
Expand Down
7 changes: 7 additions & 0 deletions cmd/pulsar-consumer/writer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,13 @@ func (w *writer) appendRow2Group(dml *commonEvent.DMLEvent, progress *partitionP
group = util.NewEventsGroup(progress.partition, tableID)
progress.eventsGroup[tableID] = group
}
if commitTs < progress.watermark {
log.Warn("DML Event fallback row, since less than the partition watermark, ignore it",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Uint64("watermark", progress.watermark),
zap.String("schema", schema), zap.String("table", table))
Comment on lines +438 to +441
Copy link

@gemini-code-assist gemini-code-assist bot Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The log message is a bit verbose. The reason for ignoring the event ("since less than the partition watermark") is already evident from the logged commitTs and watermark fields. A more concise message would improve log readability and maintainability.

Suggested change
log.Warn("DML Event fallback row, since less than the partition watermark, ignore it",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Uint64("watermark", progress.watermark),
zap.String("schema", schema), zap.String("table", table))
log.Warn("Ignore fallback DML event",
zap.Int64("tableID", tableID), zap.Int32("partition", group.Partition),
zap.Uint64("commitTs", commitTs), zap.Uint64("watermark", progress.watermark),
zap.String("schema", schema), zap.String("table", table))

return
}
if commitTs >= group.HighWatermark {
group.Append(dml, false)
log.Info("DML event append to the group",
Expand Down
4 changes: 1 addition & 3 deletions pkg/sink/codec/common/ddl.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,6 @@ func GetBlockedTables(
tableName = ddl.TableName
action = timodel.ActionType(ddl.Type)
)
blockedTableIDs := accessor.GetBlockedTables(schemaName, tableName)
if action == timodel.ActionRenameTable {
stmt, err := parser.New().ParseOneStmt(ddl.Query, "", "")
if err != nil {
Expand All @@ -175,9 +174,8 @@ func GetBlockedTables(

ddl.ExtraSchemaName = schemaName
ddl.ExtraTableName = tableName
extraTableIDs := accessor.GetBlockedTables(schemaName, tableName)
blockedTableIDs = append(blockedTableIDs, extraTableIDs...)
}
blockedTableIDs := accessor.GetBlockedTables(schemaName, tableName)

if action == timodel.ActionExchangeTablePartition {
stmt, err := parser.New().ParseOneStmt(ddl.Query, "", "")
Expand Down
4 changes: 2 additions & 2 deletions pkg/sink/codec/debezium/helper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,11 +117,11 @@ func parseBit(s string, n int) string {
}

func getValueFromDefault(defaultVal any, tp *types.FieldType) any {
// defaultValue shoul be string
// defaultValue should be string
// see https://github.com/pingcap/tidb/blob/72b1b7c564c301de33a4bd335a05770c78528db4/pkg/ddl/add_column.go#L791
val, ok := defaultVal.(string)
if !ok {
log.Debug("default value is not string", zap.Any("defaultValue", defaultVal))
log.Warn("default value is not string", zap.Any("defaultValue", defaultVal))
Copy link
Collaborator

@3AceShowHand 3AceShowHand Jan 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is only used in the testing code, set it to panic.

return defaultVal
}
// TODO: more data types need be consider
Expand Down