403 on new install #118
Description
Did a fresh helm install on EKS
=========================================================================
CHT Containers Collector Environment
CHT_API_TOKEN: ****
CHT_CLUSTER_NAME: dev-us-east-2
JAVA_OPTS:
-XX:+ExitOnOutOfMemoryError -Xms10M -Xmx891M
CHT_INTERVAL: 900
=========================================================================
2024-04-24T22:41:32.215Z [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
CHT Containers Collector : version 5f1b5a2 starting
D, [2024-04-24T22:41:46.848458 #10] DEBUG -- : Ensuring cache directory is present: /tmp/cache
D, [2024-04-24T22:41:46.862460 #10] DEBUG -- : Ensuring metrics cache directory is present: /tmp/metrics_cache
D, [2024-04-24T22:41:46.935558 #10] DEBUG -- : Fetching collector agent version
D, [2024-04-24T22:41:46.943248 #10] DEBUG -- : Kubernetes collector agent version found : 1398
D, [2024-04-24T22:41:47.028912 #10] DEBUG -- : loaded K8S config from with master @ https://172.22.0.1:443/ with ca certificate /var/run/secrets/kubernetes.io/serviceaccount/ca.crt with client_cert_file with client key file with trust_certs false with trust store file with proxy username
D, [2024-04-24T22:41:47.635460 #10] DEBUG -- : Fetching Cluster UID
D, [2024-04-24T22:41:47.640977 #10] DEBUG -- : Connecting to URL: https://172.22.0.1:443/api/v1/namespaces/kube-system
E, [2024-04-24T22:41:48.137155 #10] ERROR -- : [Java::JavaIo::IOException]: Server returned HTTP response code: 403 for URL: https://172.22.0.1:443/api/v1/namespaces/kube-system
E, [2024-04-24T22:41:48.138684 #10] ERROR -- : Backtrace:
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:77)
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstanceWithCaller(java/lang/reflect/Constructor.java:499)
java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:480)
sun.net.www.protocol.http.HttpURLConnection$10.run(sun/net/www/protocol/http/HttpURLConnection.java:2057)
sun.net.www.protocol.http.HttpURLConnection$10.run(sun/net/www/protocol/http/HttpURLConnection.java:2052)
java.security.AccessController.doPrivileged(java/security/AccessController.java:569)
sun.net.www.protocol.http.HttpURLConnection.getChainedException(sun/net/www/protocol/http/HttpURLConnection.java:2051)
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(sun/net/www/protocol/http/HttpURLConnection.java:1609)
sun.net.www.protocol.http.HttpURLConnection.getInputStream(sun/net/www/protocol/http/HttpURLConnection.java:1589)
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(sun/net/www/protocol/https/HttpsURLConnectionImpl.java:224)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:329)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:193)
RUBY.get_input_stream(uri:classloader:/mesos-collector/lib/common/fetcher.rb:37)
RUBY.fetch_uid(uri:classloader:/mesos-collector/lib/common/fetcher.rb:50)
RUBY.do_fetch_cluster_uid(uri:classloader:/mesos-collector/lib/common/collection_cycle.rb:104)
RUBY.do_upload(uri:classloader:/mesos-collector/lib/common/collection_cycle.rb:24)
RUBY.upload_k8s_state_v4(uri:classloader:/mesos-collector/bin/container-collector.rb:88)
org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:522)
org.jruby.RubyBasicObject.send(org/jruby/RubyBasicObject.java:1687)
org.jruby.RubyKernel.send(org/jruby/RubyKernel.java:2315)
org.jruby.RubyKernel$INVOKER$s$send.call(org/jruby/RubyKernel$INVOKER$s$send.gen)
RUBY.delegate_runner_methods(uri:classloader:/gems/boson-1.3.0/lib/boson/runner_library.rb:12)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:378)
org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:144)
org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)
RUBY.redefine_command_block(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:87)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:333)
org.jruby.RubyProc$INVOKER$i$call.call(org/jruby/RubyProc$INVOKER$i$call.gen)
RUBY.call_original_command(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:123)
RUBY.during_analyze(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:115)
RUBY.analyze(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:107)
RUBY.redefine_command_block(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:86)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:378)
org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:522)
org.jruby.RubyBasicObject.send(org/jruby/RubyBasicObject.java:1687)
org.jruby.RubyKernel.send(org/jruby/RubyKernel.java:2315)
org.jruby.RubyKernel$INVOKER$s$send.call(org/jruby/RubyKernel$INVOKER$s$send.gen)
RUBY.full_invoke(uri:classloader:/gems/boson-1.3.0/lib/boson.rb:79)
RUBY.execute_command(uri:classloader:/gems/boson-1.3.0/lib/boson/bare_runner.rb:42)
RUBY.execute_command(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:35)
RUBY.execute(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:31)
RUBY.start(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:26)
RUBY.<main>(uri:classloader:/mesos-collector/bin/container-collector.rb:120)
org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1290)
org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2964)
org.jruby.RubyKernel.loadCommon(org/jruby/RubyKernel.java:1121)
org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1091)
org.jruby.RubyKernel$INVOKER$s$load.call(org/jruby/RubyKernel$INVOKER$s$load.gen)
RUBY.<main>(uri:classloader:/META-INF/main.rb:1)
org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1290)
org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2964)
org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:1064)
org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1057)
org.jruby.RubyKernel$INVOKER$s$1$0$require.call(org/jruby/RubyKernel$INVOKER$s$1$0$require.gen)
RUBY.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85)
RUBY.<main>(<script>:3)
org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1312)
org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1316)
org.jruby.embed.internal.EmbedEvalUnitImpl.run(org/jruby/embed/internal/EmbedEvalUnitImpl.java:119)
org.jruby.embed.ScriptingContainer.runUnit(org/jruby/embed/ScriptingContainer.java:1296)
org.jruby.embed.ScriptingContainer.runScriptlet(org/jruby/embed/ScriptingContainer.java:1289)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)
JarMain.invokeMethod(JarMain.java:263)
JarMain.invokeMethod(JarMain.java:256)
JarMain.launchJRuby(JarMain.java:141)
JarMain.start(JarMain.java:158)
JarMain.doStart(JarMain.java:233)
JarMain.main(JarMain.java:227)
E, [2024-04-24T22:41:48.139484 #10] ERROR -- : Could not open an input stream to cluster's Kubernetes API: Server returned HTTP response code: 403 for URL: https://172.22.0.1:443/api/v1/namespaces/kube-system
I've validated that the service account and bindings were made as expected in the chart