[FLINK-37683][Connectors/AWS] Enable SSO credentials by soisyourface · Pull Request #197 · apache/flink-connector-aws

soisyourface · 2025-04-09T15:56:14Z

There is a conversation in slack that has some additional information.

Purpose of the change

SSO Credentials do not work without including the sso and ssooidc dependencies in the shaded jar.

Verifying this change

I've tested this locally by doing mvn clean install and then using 5.1-SNAPSHOT in my flink job. I can add further tests if this PR is wanted.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

Dependencies have been added or upgraded
Public API has been changed (Public API is any class annotated with @Public(Evolving))
Serializers have been changed
New feature has been introduced
- If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

add sso and ssooidc dependencies to the base pom, similar to sts

leekeiabstraction

Thank you Ryan for raising this PR!

Can we add test case? An example would be using ProfileTokenProviderBuilder to load a test implementation of profile file returns a token provider of class SsoProfileCredentialsProvider?

… config

…t the SSO stuff... but maybe this is valuable?

soisyourface · 2025-04-17T17:32:38Z

added this test
i'm not confident that it is doing much, so let me know if this is way off from what you were thinking.

i imagine the only "real" way to test the shadedness is to use an E2E test, yeah?

leekeiabstraction · 2025-04-23T13:26:28Z

+    assertThatThrownBy(() -> factory.create(context))
+      .isInstanceOf(java.io.UncheckedIOException.class)
+      .hasRootCauseInstanceOf(java.nio.file.NoSuchFileException.class)
+      .hasMessageContaining(".aws/sso/cache/");
+  }


Can you elaborate why we're asserting that an exception is thrown here? What would the full exception message look like?

here is the full exception message:

[ERROR] org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile Time elapsed: 0.925 s <<< ERROR! java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:58) at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.resolveToken(SsoAccessTokenProvider.java:51) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:107) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:88) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory.create(SsoProfileCredentialsProviderFactory.java:68) at org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile(AWSGeneralUtilTest.java:991) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.lambda$execute$1(JUnitPlatformProvider.java:199) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:193) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) Caused by: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) at java.base/java.nio.file.Files.newByteChannel(Files.java:380) at java.base/java.nio.file.Files.newByteChannel(Files.java:432) at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) at java.base/java.nio.file.Files.newInputStream(Files.java:160) at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:55) ... 75 more

in this case the exception is being thrown because there is no cache with the credentials in it. we can always just remove the check but this essentially means that if it makes it this far then the sso/ssooidc class loader stuff worked because the dependencies were found.

at least that was my initial thinking. but i really don't know if these tests are valuable, as mentioned in a previous comment.

leekeiabstraction · 2025-04-23T13:28:10Z

    }
+
+  @Test
+  void testGetCredentialsProviderReturnsSsoProviderForSsoProfile() throws IOException {


The test case name indicates that we're expecting SsoProvider to be returned but actual assertion is on UncheckedIOException being thrown. Can you elaborate?

likely a leftover name from a previous iteration of writing the tests.
renamed.

the bigger question is: does this test even do anything?

soisyourface and others added 2 commits April 7, 2025 17:11

add sso and ssooidc dependencies to the base pom, similar to sts

2aad3fb

Merge pull request #1 from soisyourface/fix_kinesis_sso_credentials

6a3fc51

add sso and ssooidc dependencies to the base pom, similar to sts

soisyourface changed the title ~~[FLINK-XXXX][Connectors/AWS] Enable SSO credentials~~ [FLINK-37683][Connectors/AWS] Enable SSO credentials Apr 15, 2025

leekeiabstraction suggested changes Apr 16, 2025

View reviewed changes

soisyourface added 2 commits April 16, 2025 17:57

fix: Remove unsupported 'allowUndeclaredRTE' property from Checkstyle…

65ac913

… config

add a test... but i think an e2e test is really the only thing to tes…

60d4bfc

…t the SSO stuff... but maybe this is valuable?

boring-cyborg bot added the component=BuildSystem label Apr 17, 2025

soisyourface and others added 4 commits April 17, 2025 10:15

Merge branch 'apache:main' into main

f19f84a

whoops, removed this to try and fix an error

e516173

Merge branch 'main' of github.com:soisyourface/flink-connector-aws

6d1ab91

grr, IDEA added this plugin...

c987345

leekeiabstraction reviewed Apr 23, 2025

View reviewed changes

rename the test

f9c8ed4

soisyourface requested a review from leekeiabstraction April 25, 2025 18:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-37683][Connectors/AWS] Enable SSO credentials#197

[FLINK-37683][Connectors/AWS] Enable SSO credentials#197
soisyourface wants to merge 9 commits intoapache:mainfrom
soisyourface:main

soisyourface commented Apr 9, 2025 •

edited

Loading

Uh oh!

leekeiabstraction left a comment

Uh oh!

soisyourface commented Apr 17, 2025

Uh oh!

Uh oh!

leekeiabstraction Apr 23, 2025

Uh oh!

soisyourface Apr 25, 2025

Uh oh!

leekeiabstraction Apr 23, 2025

Uh oh!

soisyourface Apr 25, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

soisyourface commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Verifying this change

Significant changes

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

soisyourface commented Apr 17, 2025

Uh oh!

Uh oh!

leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

soisyourface Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

soisyourface Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

soisyourface commented Apr 9, 2025 •

edited

Loading

soisyourface Apr 25, 2025 •

edited

Loading