[SPARK-57555][FOLLOWUP][SQL] Add TimeType support to MySQLDialect#57198
Open
shrirangmhalgi wants to merge 3 commits into
Open
[SPARK-57555][FOLLOWUP][SQL] Add TimeType support to MySQLDialect#57198shrirangmhalgi wants to merge 3 commits into
shrirangmhalgi wants to merge 3 commits into
Conversation
shrirangmhalgi
changed the title
Add TimeType support to MySQLDialect: - getCatalystType: maps Types.TIME to TimeType(precision) when timeType.enabled=true and legacyJdbcTimeMappingEnabled=false. Precision is derived from JDBC metadata (scale parameter). - getJDBCType: maps TimeType(p) to TIME(p) DDL for write path. - Integration tests: read round-trip, write round-trip, and legacy escape hatch verification.
…clamping, proactive fixes
- getCatalystType: use md.build().getLong("scale") for fractional-second
precision (getScale reports DECIMAL_DIGITS, not display width). Accept
scale=0 for bare TIME (TIME(0) is valid).
- getJDBCType: use TIME(p) for p<=6, bare TIME for out-of-range
precisions (e.g. nanosecond TimeType), matching PostgreSQL pattern.
- Fix read test: bare TIME column reports scale=0, truncates fractional
seconds (value 13:31:24 not 13:31:24.123).
- Add precision preservation round-trip test (TimeType(3) -> TIME(3))
- Add nanosecond TimeType(9) write test (MySQL truncates to micros)
- Add comment explaining precision source from JDBC metadata
…SION MySQL Connector/J Bug #84308: getPrecision()=8 and getScale()=0 for all TIME columns regardless of declared precision. Override updateExtraColumnMeta to query INFORMATION_SCHEMA.COLUMNS.DATETIME_PRECISION and inject the real precision into the metadata, which getCatalystType then reads.
shrirangmhalgi
force-pushed
the
SPARK-57555-mysql-time-dialect
branch
from
499448d to
75cd5c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add
TimeTypesupport toMySQLDialectand precision-preserving DDL for writes:getCatalystType: mapTypes.TIMEtoTimeType(scale)whenspark.sql.timeType.enabledis true, gated bylegacyJdbcTimeMappingEnabledgetJDBCType: addTimeType => TIME(p)for p in [0,6], bare TIME for out-of-range precisions (e.g. nanosecond TimeType)Why are the changes needed?
This is a followup to #56653 which added core JDBC TIME support in
JdbcUtils. That PR handled scalar read/write correctly via the generic path, but:getJDBCTypein the base JdbcUtils emitsTIME(${t.precision})which produces TIME(9) for nanosecond TimeTypes - MySQL rejects precisions > 6getCatalystTyperuns before the genericJdbcUtilsmapper (confirmed in [SPARK-57555][FOLLOWUP][SQL] Add escape-hatch conf to keep legacy JDBC TIME-to-timestamp mapping #56884), so MySQL needs its own TIME handling gated by the escape-hatch configDoes this PR introduce any user-facing change?
Yes. When
spark.sql.timeType.enabledis true, MySQLTIMEandTIME(p)columns now correctly read asTimeTypeand write with precision-preservingTIME(p)DDL. Non-TIME columns are unaffected.How was this patch tested?
Added integration tests in
MySQLIntegrationSuite:TIMEcolumn is read asTimeType(0)andTIME(3)asTimeType(3)with correct valuesTimeType(0)andTimeType(6)values and reads them backTimeType(3)->TIME(3)-> read back asTimeType(3)TimeType(9)-> bareTIME-> read back asTimeType(6)with microsecond truncationTIMEreads as non-TimeType whenlegacyJdbcTimeMappingEnabled=trueUnit tests: JDBCSuite (127 tests, all passing).
Was this patch authored or co-authored using generative AI tooling?
CoAuthored using Claude Opus 4.6