[SPARK-57891][CORE] Add CredentialProvider SPI and ServiceLoader discovery

[yadavay-amzn]

What changes were proposed in this pull request?

This is task 2 of the OIDC Credential Propagation SPIP (SPARK-57703), building on the core types added in SPARK-57890. It introduces the pluggable provider SPI in org.apache.spark.security (spark-core):

• CredentialProvider - a @DeveloperApi interface that providers (AWS STS, Azure, GCP, etc.) implement to exchange a user identity for a short-lived ServiceCredential
• CredentialResolutionException - a @DeveloperApi checked exception thrown by resolve.
• CredentialProviderLoader - discovers implementations via ServiceLoader and selects a provider per scheme. Scheme keys are normalized to lowercase. Selection follows an explicit-configuration policy: spark.security.credentials.provider.<scheme> names the fully-qualified provider class to use for that scheme. If exactly one discovered provider supports a scheme, no configuration is needed; if multiple support it and the conf is unset, a clear error is raised listing the candidates. Each selected provider is initialized exactly once.

A FakeCredentialProvider (and a second one sharing a scheme) plus a META-INF/services registration are added under core/src/test to exercise discovery and selection.

Why are the changes needed?

There is currently no pluggable SPI for exchanging an identity token for short-lived service credentials. This is the extension point the credential-propagation framework builds on. See the SPIP design document, Appendix A.

Does this PR introduce any user-facing change?

No behavior change. It adds new @DeveloperApi types only; nothing uses them until the follow-up (SPARK-57892 / the manager task).

How was this patch tested?

New JUnit CredentialProviderLoaderSuite covering the acceptance criteria, ServiceLoader discovery, provider selection/ambiguity, error cases, and null-input guards.

Design notes (feedback/suggestions welcome)

Two choices worth considering:

• Binding policy: explicit selection via spark.security.credentials.provider.<scheme> (single fully-qualified class name). This can be extended to an ordered-priority list later without breaking the conf if preferred.
• init lifecycle: each provider is initialized once, on first selection, with that call's configuration (first-conf-wins), inside the loader's lock. This sets the provider lifecycle the manager task will consume.

Was this patch authored or co-authored using generative AI tooling?

No.