[SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions by HyukjinKwon · Pull Request #57079 · apache/spark

HyukjinKwon · 2026-07-07T10:28:10Z

What changes were proposed in this pull request?

This PR documents, for every built-in function argument, which input types it accepts, and for
the DataFrame APIs, what type the function returns — across the three public documentation
surfaces:

SQL @ExpressionDescription arguments blocks (the SQL functions reference page),
Scala functions.scala Scaladoc (@param / @return),
PySpark builtin.py docstrings (Parameters / Returns).

Each argument names its accepted type, e.g. SQL An expression that evaluates to an integer. /
DataFrame A column that evaluates to an integer.. The wording adapts to the argument kind:

Value arguments name their accepted type(s), e.g. a numeric, a string or binary,
a numeric, interval, date, timestamp, or time.
Constant-only arguments are marked Must be a constant. (e.g. ceil/floor's scale).
Lambda arguments of higher-order functions are described as lambdas, e.g. A lambda function..

Return types are documented on the DataFrame surfaces as, e.g., Returns a column that evaluates to a double., Returns a column of the same type as the input. (e.g. abs, coalesce, greatest),
or Returns a column of the element type of the input array, or the value type of the input map.
for collection accessors (e.g. element_at, explode).

Conventions used for the documented types:

Numeric umbrella. Where numeric subtypes are accepted via implicit cast (e.g. the statistical
aggregates corr/covar_*/stddev_*/var_*/regr_*, or divide), the argument is documented
as a numeric (or a numeric or interval) rather than enumerating double/decimal/long,
matching how mainstream SQL references document argument types. Genuinely narrow sets are kept
(e.g. an integral, an integral, decimal, or interval for div).
TIMESTAMP_NTZ is documented as timestamp in type phrases (the distinction is kept in
function names and examples, not in user-facing type descriptions).

This PR contains only documentation text. It has no functional or API change.

Why are the changes needed?

Users of the SQL reference, the Scala functions API, and the PySpark functions API previously
had no per-argument type information for most built-in functions, and no documented return type on
the DataFrame APIs. This PR fills that gap consistently across all three surfaces.

How was this patch generated and verified?

The documented types were derived from the analyzer's actual behavior, then adjusted to the
conventions above and cross-checked against each expression's inputTypes / checkInputDataTypes
/ dataType and the ExpressionsSchemaSuite output schema:

A probe ran every function/argument position through the analyzer, recording which types are
accepted (natively or via cast), rejected, or constant-only, and measured each function's output
type (including functions whose return type follows their input or an input's element/value type).
The measured types were rendered into the three surfaces, applying the numeric-umbrella and
TIMESTAMP_NTZ conventions.
Accepted/return types were then verified against Catalyst source, correcting cases the raw probe
got wrong — e.g. to_char/to_varchar accept numeric, date, timestamp, time, or binary;
divide dividend is numeric or interval and divisor numeric; array element-args
(array_contains/append/prepend/position/remove/insert) take the array's element type;
element_at/explode document the map case; unix_nanos returns a decimal.

ExpressionInfoSuite, the SPARK-32870 argument-format tests, the docs build, Scala style/format
(scalastyle + scalafmt), and Python lint (flake8 + black) all pass.

Does this PR introduce any user-facing change?

Yes — documentation only. The SQL built-in function reference, Scala functions Scaladoc, and
PySpark functions docstrings now state each argument's accepted input type and each DataFrame
function's return type. There is no behavioral or API change.

How was this patch tested?

Existing documentation and expression tests (ExpressionInfoSuite, docs build) pass. The
documentation text was cross-checked against the analyzer's measured accept/reject behavior and
Catalyst source for every function and argument position, as described above.

…ypes for built-in functions ### What changes were proposed in this pull request? Documents, for every built-in function argument, which input types it accepts, and for the DataFrame APIs, what type the function returns, across the three public surfaces: SQL `@ExpressionDescription` arguments blocks, Scala `functions.scala` Scaladoc, and PySpark docstrings. Each argument names its canonical/expected type (e.g. "An expression that evaluates to an integer." for SQL, "A column that evaluates to an integer." for the DataFrame APIs), return types are documented on the DataFrame surfaces (e.g. "Returns a column that evaluates to a double." / "Returns a column of the same type as the input."), and constant-only arguments are marked "Must be a constant.". Documentation only; no functional or API change. ### Why are the changes needed? The SQL reference and the Scala/PySpark `functions` APIs previously had no per-argument type information for most built-in functions and no documented return type on the DataFrame APIs. ### Does this PR introduce any user-facing change? Yes, documentation only. The built-in function reference and the Scala/PySpark `functions` docstrings now state the accepted input type of each argument and the return type of each DataFrame function. ### How was this patch tested? The documented types were derived from and cross-checked against the analyzer's measured accept/reject behavior for every function and argument position (see the PR description for the generation-and-verification process). Existing documentation and expression tests (`ExpressionInfoSuite`, docs build) pass.

@return

…run-ons - approx_top_k/approx_top_k_accumulate: expr is gated by isDataTypeSupported (atomic allowlist), not any type - decode: bin/charset are binary/string, not any type - comparison ops (=, <=>, <, <=, >, >=), equal_null, nullif, nullifzero, max_by/min_by ord: gated by checkForOrderingExpr -> "any orderable type" - functions.scala: add missing period before appended "Returns a column..." sentence in 33 @return blocks (run-on Scaladoc) Co-authored-by: Isaac

…cala The earlier sweep matched "Returns a column" on a single line, missing 16 blocks where the appended sentence wraps as "...`sinh` Returns a" / "column..." across two lines. Add the missing period before "Returns" in sinh, tanh, add_months, date_format, the date-part extractors (year/quarter/month/ dayofweek/dayofmonth/dayofyear and siblings), last_day, and 3 more. Co-authored-by: Isaac

@return

…wording - to_timestamp @return: the 17th run-on block missed by the prior sweep (prose line ended without a period, fusing with the next "Returns a column..." line). Add the terminal period. - any_value: "The expression to return some value of for a group of rows." read broken ("of for"); match the first/last peers: "...some value of." Co-authored-by: Isaac

@return

…x review findings Consistency: many functions.scala functions documented @param but had no @return (or only one overload did), while the PR documents return types elsewhere. Add "Returns a column..." @return to every function/overload that was missing one (355 functions), leaving functions where a value return type is not meaningful (sort builders asc/desc, column constructors col/column, partition transforms bucket/days/months/years, UDF/dispatch udf/udaf/callUDF/ reflect, generators posexplode) without one. Return types derived from the ExpressionsSchemaSuite golden file (sql-expression-schema.md) and Catalyst dataType; polymorphic functions (max/min/first/last/coalesce/greatest/...) use "of the same type as the input" rather than the misleading example type. Review findings (pullrequestreview-4667871389): - to_time/try_to_time @return said "string" but return TimeType — corrected on both Scala @return and PySpark Returns docstrings. - bit_or: replaced content-free "The expr1 argument." arg docs with wording matching the BitwiseAnd/BitwiseXor peers ("The first/second operand ..."). Co-authored-by: Isaac

@return

…rate Audited every accepted-input and return type phrase across the three doc surfaces (functions.scala Scaladoc, builtin.py docstrings, SQL ExpressionDescription blocks) against each expression's inputTypes / dataType. - Fold TIMESTAMP_NTZ into "timestamp" in all type phrases (e.g. "string, date, timestamp, or timestamp_ntz" -> "string, date, or timestamp"), keeping the distinction only in function names / SQL literals / code, not user-facing type descriptions. - Numeric umbrella: where numeric subtypes cast in, say "a numeric" (or "a numeric or interval") rather than enumerating double/decimal/long — e.g. divide operands, ceil/floor, and the statistical aggregates (corr, covar_*, stddev_*, var_*, skewness, kurtosis, regr_*) which accept any numeric via implicit cast to double. Kept genuinely-narrow sets ("an integral" for kll_sketch_agg_bigint, "an integral, decimal, or interval" for div). - Fix incorrect accepted types: to_char/to_varchar accept numeric, date, timestamp, time, or binary (date was missing); divide dividend is "numeric or interval", divisor "numeric"; array element-args (array_contains/append/prepend/position/remove/insert) take the array's element type, not an integer; element_at index is an integer for an array or the key type for a map; nullifzero takes a numeric. - Fix incorrect return types: unix_nanos returns a decimal (DECIMAL(21,0)), not a long; hll_union_agg usage said "estimated number of unique values" but returns the merged sketch's binary representation. - element_at/try_element_at/explode/explode_outer @return now document the map case (value type / key and value columns), not only the array case. Verified against the ExpressionsSchemaSuite golden file and Catalyst source via an audit pass with adversarial verification; no run-on/orphan @return regressions. Co-authored-by: Isaac

HyukjinKwon · 2026-07-10T05:45:30Z

I think it's quite good now. I am triggering the last multi-agent orchastrated review

HyukjinKwon · 2026-07-10T05:46:51Z

cc @cloud-fan too

dongjoon-hyun · 2026-07-10T06:47:42Z

Could you check the linter failures?

Scalastyle checks failed at following occurrences:
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:2967: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:2996: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3864: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3899: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3934: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4190: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4212: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4937: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4958: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4980: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5001: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5047: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5093: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8757: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8884: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8935: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:9355: File line length exceeds 100 characters
[error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:9408: File line length exceeds 100 characters

dongjoon-hyun · 2026-07-10T06:51:47Z

If you don't mind, shall we split this into more smaller reviewable PRs? For example, at least two; one for SQL, one for Python, @HyukjinKwon ? Although we live in agentic era, the PR should be human-reviewable as much as possible.

@return

…udit A multi-agent verification pass over every multi-argument function's type phrases (all three surfaces, checked against Catalyst inputTypes/dataType) surfaced these: - array_insert: the `pos` arg (1-based integer index) was mislabeled "the same type as the array elements" on the PySpark surface — only `value` is the element type. `pos` is coerced to IntegerType. (review finding) - max_by/min_by (3-arg, with `k`): route through MaxMinByK whose dataType is ArrayType(valueType), so @return is "an array", not "the same type as the input". - listagg/string_agg: ListAgg.dataType = child.dataType (string or binary), so the return is polymorphic — "of the same type as the input", not "a string". - overlay: Overlay.dataType = input.dataType (string or binary) — polymorphic. - shiftleft/shiftright/shiftrightunsigned: dataType = left.dataType (integer or long) — "of the same type as the input", not "an integer". - try_divide: Divide.dataType mirrors the input (double or decimal) — matches the try_add/try_subtract/try_multiply peers' "of the same type as the input". - reduce: `merge`/`finish` are lambda args; removed the stray column type-phrase lines so they match `aggregate` and the other higher-order functions (whose lambda params carry no "A column ..." type line on the PySpark surface). Co-authored-by: Isaac

HyukjinKwon · 2026-07-10T07:05:46Z

Yeah will do

@return

- Wrap 20 Scaladoc @return/@param lines that exceeded 100 characters after the type sentences were appended (the scalastyle failures dongjoon-hyun flagged), preserving the standard 3-space payload indentation. - schema_of_json(json, options): @return wrongly said "evaluates to an integer"; SchemaOfJson returns a string (DDL schema). Corrected to "a string". Co-authored-by: Isaac

HyukjinKwon · 2026-07-10T07:49:16Z

Superseded by smaller, per-surface PRs for reviewability (per @dongjoon-hyun's suggestion). SPARK-57999 is now an umbrella, with one sub-task and PR per documentation surface:

SQL (@ExpressionDescription reference): SPARK-58085 — [SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference #57183
Scala (functions.scala): SPARK-58086 — [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API #57184
PySpark (builtin.py): SPARK-58087 — [SPARK-58087][PYTHON][DOCS] Document accepted input and return types in the PySpark functions API #57185

Each carries the exact reviewed content from this PR for its surface (scalastyle line-length fixes and the schema_of_json return-type fix included). Closing this in favor of the three above.

dongjoon-hyun · 2026-07-10T16:59:35Z

Thank you for the decision, @HyukjinKwon !

…unctions in the SQL function reference ### What changes were proposed in this pull request? This documents, for every built-in function argument, which input types it accepts, in the SQL `@ExpressionDescription` `arguments` blocks (rendered on the SQL functions reference page), using a numeric umbrella (`a numeric` / `a numeric or interval`) where numeric subtypes cast in, and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the SQL surface. ### Why are the changes needed? The SQL function reference previously had no per-argument type information for most built-in functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Existing `ExpressionInfoSuite` and the SPARK-32870 argument-block-format tests pass. Types were derived from and verified against the analyzer's accept/reject behavior and each expression's `inputTypes` / `checkInputDataTypes`. Co-authored-by: Isaac

…the Scala functions API ### What changes were proposed in this pull request? This documents, for every built-in function in the Scala `functions` API (`functions.scala`), the accepted input type of each argument (`@param`) and the return type of each DataFrame function (`@return`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the Scala `functions.scala` surface. ### Why are the changes needed? The Scala `functions` Scaladoc previously had no per-argument type information and no documented return type for most functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Scala style/format (scalastyle + scalafmt) and the docs build pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac

…in the PySpark functions API ### What changes were proposed in this pull request? This documents, for every built-in function in the PySpark `functions` API (`builtin.py`), the accepted input type of each argument (`Parameters`) and the return type of each function (`Returns`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the PySpark `builtin.py` surface. ### Why are the changes needed? The PySpark `functions` docstrings previously had no per-argument type information for most functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Python lint (flake8 + black) and the docstring doctests pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac

HyukjinKwon marked this pull request as draft July 7, 2026 10:29

HyukjinKwon changed the title ~~[SPARK-57999][SQL][TESTS] Add generated table of accepted input types for built-in scalar functions~~ [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input types for built-in functions, backed by a generated test Jul 7, 2026

HyukjinKwon changed the title ~~[SPARK-57999][SQL][PYTHON][DOCS] Document accepted input types for built-in functions, backed by a generated test~~ [DO-NOT-MERGE][SQL][PYTHON][DOCS] Document accepted input types for built-in functions, backed by a generated test Jul 7, 2026

HyukjinKwon force-pushed the spark-func-accepted-types branch 3 times, most recently from 263bea6 to f2d7147 Compare July 9, 2026 04:02

HyukjinKwon changed the title ~~[DO-NOT-MERGE][SQL][PYTHON][DOCS] Document accepted input types for built-in functions, backed by a generated test~~ [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions Jul 9, 2026

HyukjinKwon marked this pull request as ready for review July 9, 2026 04:03

dongjoon-hyun reviewed Jul 9, 2026

View reviewed changes

Comment thread ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala Outdated

dongjoon-hyun reviewed Jul 9, 2026

View reviewed changes

Comment thread python/pyspark/sql/functions/builtin.py Outdated

dongjoon-hyun reviewed Jul 9, 2026

View reviewed changes

Comment thread python/pyspark/sql/functions/builtin.py Outdated

HyukjinKwon marked this pull request as draft July 9, 2026 06:04

HyukjinKwon force-pushed the spark-func-accepted-types branch from f2d7147 to 5afa847 Compare July 9, 2026 07:31