[SPARK-58082][SQL] Add DSv2 relation and row-level command tree patterns

[yyanyy]

What changes were proposed in this pull request?

Add Catalyst tree-pattern identity bits for the DSv2 relation leaves and the
row-level command nodes, and use them to prune the optimizer/analyzer rules
that match those nodes.

• Add DATA_SOURCE_V2_RELATION and DATA_SOURCE_V2_SCAN_RELATION to
  TreePattern, and attach them to DataSourceV2Relation and
  DataSourceV2ScanRelation respectively. DATA_SOURCE_V2_SCAN_RELATION is used
  as a pruning guard (see below); DATA_SOURCE_V2_RELATION is attached to the
  relation leaf but has no consumer rule in this PR (reserved for future pruning).
• Add DELETE_FROM_TABLE, UPDATE_TABLE, MERGE_INTO_TABLE, REPLACE_DATA
  and WRITE_DELTA. Since every command already carries the shared COMMAND
  bit (declared final on the Command trait), Command now exposes a
  nodePatternsInternal() hook, mirroring how PlanExpression combines
  PLAN_EXPRESSION with a per-subquery bit, so a command node contributes its
  own identity bit while COMMAND stays guaranteed-present. DeleteFromTable,
  UpdateTable, MergeIntoTable, ReplaceData and WriteDelta override it.
• Guard the previously-unguarded rules that key off these nodes with
  containsPattern / containsAnyPattern:
  RewriteDeleteFromTable, RewriteUpdateTable, RewriteMergeIntoTable,
  V2ScanPartitioningAndOrdering, GroupBasedRowLevelOperationScanPlanning,
  OptimizeMetadataOnlyDeleteFromTable and
  RowLevelOperationRuntimeGroupFiltering.
• Add tree-pattern contract tests.

Why are the changes needed?

A tree-pattern bit is an O(1) bitset lookup on a plan's cached treePatternBits.
Rules that today run an unconditional transform/resolveOperators walk over
every node, on every invocation, even for plans that contain none of the target
nodes, can instead cheaply skip whole subtrees that provably don't contain the
node they rewrite. The DSv2 scan/relation leaves and the row-level command nodes
previously had no identity pattern (the command nodes carried only the coarse
COMMAND bit shared by all commands), so rules matching them either ran
unguarded or could only prune on the broad COMMAND bit. This adds the missing
per-node bits and applies them as pruning guards, matching the existing idiom
used throughout the optimizer (for example ReplaceExceptWithFilter on EXCEPT,
ResolveRowLevelCommandAssignments on COMMAND).

Does this PR introduce any user-facing change?

No. This only adds internal Catalyst tree-pattern metadata and rule pruning
guards. Query results and plans are unchanged; the guarded rules fire on exactly
the same nodes as before, just skipping trees that cannot contain them.

How was this patch tested?

• New unit tests pinning the pattern contract:
  DataSourceV2RelationSuite (asserts the relation/scan-relation identity bits)
  and a new V2CommandTreePatternSuite (asserts each of the five row-level
  command nodes -- DeleteFromTable, UpdateTable, MergeIntoTable,
  ReplaceData and WriteDelta -- carries both COMMAND and its own bit).
• Regression suites covering the guarded rules, confirming the pruning guards do
  not drop any rule firing:
  • Row-level operations (Group/Delta-based Delete/Update/Merge): 347 tests, all
    passing.
  • Scan pushdown / partitioning (DataSourceV2Suite,
    KeyGroupedPartitioningSuite, ProjectedOrderingAndPartitioningSuite,
    DataSourceV2TableSampleSuite): all passing, including TABLESAMPLE SYSTEM
    validation over DSv1 relations.
  • V2 write analysis (V2AppendData*, V2Overwrite* ANSI/Strict suites):
    388 tests, all passing, confirming the Seq(COMMAND) ++ nodePatternsInternal()
    composition does not disturb existing COMMAND-pruned analysis rules.
• ./build/sbt catalyst/compile sql/compile and git diff --check.

Was this patch authored or co-authored using generative AI tooling?

yes, opus 4.8

[gengliangwang]

0 blocking, 1 non-blocking, 0 nits.
Clean, well-scoped performance change with a sound design that mirrors the PlanExpression precedent; the only note is a test/description completeness gap on the two untested command nodes.

Suggestions (1)

• V2CommandTreePatternSuite.scala:29: only 3 of 5 command nodes tested — ReplaceData/WriteDelta (the bits most guards rely on) are untested, and the PR description overstates coverage — see inline

Verification

Traced all 8 pruning guards for equivalence to the unguarded rules they replace. treePatternBits propagates every node's bits up to the root (TreeNode.getDefaultTreePatternBits), and each rule's matched nodes all declare the guarded bit — commands via nodePatternsInternal(), relations via override val nodePatterns — so a containsPattern guard descends into exactly the subtrees that can match. V2ScanRelationPushDown's top-level early-return is sound because ScanBuilderHolder (which every sub-rule matches) is constructed only from a DataSourceV2Relation, so no relation in the input means no holder and no sub-rule firing. No input produces a different plan.

[gengliangwang]

This suite pins the COMMAND + identity-bit contract for 3 of the 5 command nodes the PR touches — ReplaceData and WriteDelta are missing. Their REPLACE_DATA/WRITE_DELTA bits are the most load-bearing in the PR: GroupBasedRowLevelOperationScanPlanning, OptimizeMetadataOnlyDeleteFromTable, and RowLevelOperationRuntimeGroupFiltering all prune exclusively on them, so a wrong bit there would silently disable those rules with no unit-level signal. Worth adding two cases here; the nodePatternsInternal() overrides return compile-time constants, so a minimal instance pins the bit without needing resolution.

(Relatedly, the PR description's "asserts each command node carries both COMMAND and its own bit" reads as all five — worth adjusting to match the actual coverage.)