Skip to content

[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183

Open
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:split-func-types-sql
Open

[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:split-func-types-sql

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This documents, for every built-in function argument, which input types it accepts, in the SQL
@ExpressionDescription arguments blocks (rendered on the SQL functions reference page), using a
numeric umbrella (a numeric / a numeric or interval) where numeric subtypes cast in, and
timestamp for TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only
the SQL surface.

Why are the changes needed?

The SQL function reference previously had no per-argument type information for most built-in
functions. This fills that gap.

Does this PR introduce any user-facing change?

Yes, documentation only. No behavioral or API change.

How was this patch tested?

Existing ExpressionInfoSuite and the SPARK-32870 argument-block-format tests pass. Types were
derived from and verified against the analyzer's accept/reject behavior and each expression's
inputTypes / checkInputDataTypes.

Co-authored-by: Isaac

…unctions in the SQL function reference

### What changes were proposed in this pull request?

This documents, for every built-in function argument, which input types it accepts, in the SQL
`@ExpressionDescription` `arguments` blocks (rendered on the SQL functions reference page), using a
numeric umbrella (`a numeric` / `a numeric or interval`) where numeric subtypes cast in, and
`timestamp` for TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only
the SQL surface.

### Why are the changes needed?

The SQL function reference previously had no per-argument type information for most built-in
functions. This fills that gap.

### Does this PR introduce _any_ user-facing change?

Yes, documentation only. No behavioral or API change.

### How was this patch tested?

Existing `ExpressionInfoSuite` and the SPARK-32870 argument-block-format tests pass. Types were
derived from and verified against the analyzer's accept/reject behavior and each expression's
`inputTypes` / `checkInputDataTypes`.

Co-authored-by: Isaac

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR seems to break the CI at describe function unit test.

2026-07-10T10:04:14.2381680Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m- describe function *** FAILED *** (115 milliseconds)�[0m�[0m
2026-07-10T10:04:14.2390933Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Results do not match for query:�[0m�[0m
2026-07-10T10:04:14.2436987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=311,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]�[0m�[0m
2026-07-10T10:04:14.2449058Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Timezone Env: �[0m�[0m
2026-07-10T10:04:14.2450581Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2451468Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Parsed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2452454Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  'DescribeFunction true�[0m�[0m
2026-07-10T10:04:14.2453548Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  +- 'UnresolvedFunctionName [^], DESCRIBE FUNCTION�[0m�[0m
2026-07-10T10:04:14.2454477Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2455282Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Analyzed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2456358Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  function_desc: string�[0m�[0m
2026-07-10T10:04:14.2458012Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2459987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2460987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Optimized Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2463845Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  CommandResult [function_desc#1140035], Execute DescribeFunctionCommand, [[Function: ^], [Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor], [Usage: expr1 ^ expr2 - Returns the result of bitwise exclusive OR of `expr1` and `expr2`.], [Extended Usage:�[0m�[0m
2026-07-10T10:04:14.2466350Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Arguments:�[0m�[0m
2026-07-10T10:04:14.2467543Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        * expr1 - The first operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2468867Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m          An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2470447Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        * expr2 - The second operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2471784Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m          An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2472804Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m    �[0m
2026-07-10T10:04:14.2473892Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Examples:�[0m�[0m
2026-07-10T10:04:14.2474855Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        > SELECT 3 ^ 5;�[0m�[0m
2026-07-10T10:04:14.2475761Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m         6�[0m�[0m
2026-07-10T10:04:14.2476546Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m    �[0m
2026-07-10T10:04:14.2477361Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Since: 1.4.0�[0m�[0m
2026-07-10T10:04:14.2478191Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  ]]�[0m�[0m
2026-07-10T10:04:14.2479891Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m     +- DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2481555Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants