[FEATURE]: Extend profiler with additional metrics

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

The currently list of metrics is limited to basics such as min, max, stdev etc.

Proposed Solution

Include additional statistics beyond minimum and maximum values, such as the 10th percentile, along with the percentage of occurrences within each bucket or range.

Additional Context

No response

[passionworkeer]

I’d keep this incremental: add percentile support first (p10/p25/p50/p75/p90) at the profiler aggregation layer, then expose bucket percentages as a second step so the output contract stays easy to review. For implementation, it helps if metric calculators return a structured dict keyed by statistic name instead of appending ad-hoc columns. Tests should cover null-heavy columns, identical values, and skewed distributions so percentile behavior is stable before wiring it into report rendering.

[mwojtyczka]

Sounds good. We have percentile support for p25, p50, p75 already. Just need to be extended for p10/p90

[passionworkeer]

A practical way to add this without making the profiler heavy: keep the current aggregates, then add configurable percentiles (p10/p25/p50/p75/p90/p95) and an optional histogram summary. If this runs on Spark data, percentile_approx is usually good enough and much cheaper than exact quantiles. I’d also expose bucket edges as config so teams can tune ranges per metric instead of hardcoding one distribution.

[mwojtyczka]

yes percentile_approx should be ok

[passionworkeer]

Got it — I can take the implementation from here. I’ll keep it scoped to extending the existing percentile support with p10/p90 first, reusing the current aggregation path instead of changing the output shape more broadly. If that looks good, I’ll work from that direction and keep bucket/histogram-style percentages as a follow-up step.

[passionworkeer]

I've submitted a PR to add percentile support: #1071

The implementation adds p10/p90 (and p25/p50/p75) to the numeric column profiler using �pprox_percentile() for efficient computation. Percentiles are included in DQProfile.parameters when outlier removal is enabled, maintaining backward compatibility.

[mwojtyczka]

@passionworkeer we recently refactored the profiler and it will also be much easier to extend it now. If you still plan to complete this please let us know. For now we removed your assignment to free up the task