[SPARK-58069][SQL] approx_top_k returns incorrect/garbage sort-key bytes instead of actual values for collated strings

[jiwen624]

What changes were proposed in this pull request?

For a non-UTF8_BINARY collated STRING column, approx_top_k (and approx_top_k_accumulate / _combine / _estimate) stored the collation sort key in the sketch and returned it verbatim as the item. This makes the sketch store CollatedString instead: it dedupes by the collation key (so collation-equal values still count as one) but retains an actual input value to return, the way mode() does. The serde writes only the original value and recomputes the key on read, so the UTF8_BINARY format is unchanged.

Note: while working on the fix, two new issues are found in the approx_top_k_ expressions. The following issues are raised to address them separately - so this PR will only focus on the issue mentioned in SPARK-57177:

• https://issues.apache.org/jira/browse/SPARK-58096
• https://issues.apache.org/jira/browse/SPARK-58095

Why are the changes needed?

The returned items were wrong: raw ICU sort-key bytes (invalid UTF-8) for ICU collations, and the lower-cased form for UTF8_LCASE (a value that may never have appeared). Counts were correct, but the item labels were garbage. Silent wrong result, no error.

Before the fix:

scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
     |   SELECT CAST(col AS STRING COLLATE UNICODE_CI) AS c
     |   FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world') AS t(col)
     | )""").show(100, false)

+--------------------------------+
|approx_top_k(c, 5, 100)         |
+--------------------------------+
|[{93AAG\t, 4}, {WGMA1\t, 1}]|
+--------------------------------+

scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
     |   SELECT CAST(col AS STRING COLLATE UNICODE) AS c
     |   FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world') AS t(col)
     | )""").show(100, false)
+------------------------------------------------------------+
|approx_top_k(c, 5, 100)                                     |
+------------------------------------------------------------+
|[{93AAG\t�����, 3}, {93AAG\t\t, 1}, {WGMA1\t\t, 1}]|
+------------------------------------------------------------+

scala> spark.sql("""SELECT approx_top_k(c, 5, 100) FROM (
     |   SELECT CAST(col AS STRING COLLATE UTF8_LCASE) AS c
     |   FROM VALUES ('HELLO'), ('HELLO'), ('HELLO'), ('hello'), ('world') AS t(col)
     | )""").show(100, false)
+------------------------+
|approx_top_k(c, 5, 100) |
+------------------------+
|[{hello, 4}, {world, 1}]|
+------------------------+

Does this PR introduce any user-facing change?

Yes. approx_top_k over a collated string column now returns an actual input value instead of the internal collation key. Sketches persisted by approx_top_k_accumulate over a non-binary collated column before this change are not readable after it (they contained wrong data); the UTF8_BINARY format is unchanged.

How was this patch tested?

Added test cases.

Was this patch authored or co-authored using generative AI tooling?

Yes.