[WIP] debug InstanceProfileCredentialsProvider usage

[agebhar1]

Description

Currently only for debugging broken authentication with kube2iam (InstanceProfileCredentialsProvider).

Test Steps

Build

mvn clean package -Drevision=$(git describe --tags --always)

and deploy artifact to Kafka Connect.

Checklist:

• I have tested my changes. No regression in existing tests.
• I have modified and/or added unit-tests to cover the code changes in this Pull Request.

Related Issue

#396

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[agebhar1]

Running the code local yields to this error:

connect-1  | org.apache.kafka.connect.errors.ConnectException: software.amazon.awssdk.core.exception.SdkClientException: Failed to load credentials from IMDS.
connect-1  |    at software.amazon.event.kafkaconnector.EventBridgeWriter.<init>(EventBridgeWriter.java:141)
connect-1  |    at software.amazon.event.kafkaconnector.EventBridgeSinkTask.start(EventBridgeSinkTask.java:43)
connect-1  |    at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:323)
connect-1  |    at org.apache.kafka.connect.runtime.WorkerTask.doStart(WorkerTask.java:175)
connect-1  |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:224)
connect-1  |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280)
connect-1  |    at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
connect-1  |    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
connect-1  |    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
connect-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
connect-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
connect-1  |    at java.base/java.lang.Thread.run(Unknown Source)
connect-1  | Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to load credentials from IMDS.
connect-1  |    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
connect-1  |    at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
connect-1  |    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:167)
connect-1  |    at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
connect-1  |    at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448)
connect-1  |    at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
connect-1  |    at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
connect-1  |    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:149)
connect-1  |    at software.amazon.event.kafkaconnector.EventBridgeWriter.<init>(EventBridgeWriter.java:139)
connect-1  |    ... 11 more
connect-1  | Caused by: java.lang.NullPointerException: Cannot invoke "java.nio.file.Path.getFileSystem()" because "<parameter1>" is null
connect-1  |    at java.base/java.nio.file.Files.provider(Unknown Source)
connect-1  |    at java.base/java.nio.file.Files.newInputStream(Unknown Source)
connect-1  |    at software.amazon.awssdk.profiles.ProfileFile$BuilderImpl.lambda$build$0(ProfileFile.java:314)
connect-1  |    at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
connect-1  |    at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
connect-1  |    at software.amazon.awssdk.profiles.ProfileFile$BuilderImpl.build(ProfileFile.java:314)
connect-1  |    at software.amazon.awssdk.profiles.ProfileFileSupplier.lambda$defaultSupplier$2(ProfileFileSupplier.java:58)
connect-1  |    at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.resolveProfileFile(Ec2MetadataConfigProvider.java:129)
connect-1  |    at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.resolveProfile(Ec2MetadataConfigProvider.java:121)
connect-1  |    at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.configFileEndpointOverride(Ec2MetadataConfigProvider.java:117)
connect-1  |    at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.getEndpointOverride(Ec2MetadataConfigProvider.java:102)
connect-1  |    at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.getEndpoint(Ec2MetadataConfigProvider.java:70)
connect-1  |    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getImdsEndpoint(InstanceProfileCredentialsProvider.java:224)
connect-1  |    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:210)
connect-1  |    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:158)

I guess this will be the same on your side @sugarcrm-jgminder. /cc @embano1

[embano1]

@agebhar1 thx! I guess the error is due to the fact that you don't run in an EKS IMDS environment?

[embano1]

@sugarcrm-jgminder You can download the build for this test PR here: https://github.com/awslabs/eventbridge-kafka-connector/actions/runs/11048078087?pr=397

[agebhar1]

@agebhar1 thx! I guess the error is due to the fact that you don't run in an EKS IMDS environment?

The error is more a result of the missing profile file, introduced by the change of v1.3.1

[embano1]

@agebhar1 got it! @maschnetwork this comparison of the change between the versions from Andi was helpful.

[agebhar1]

Yes, the difference is in the behavior of ProfileFileSupplier.defaultSupplier() (>= v1.3.1) and ProfileFile.defaultProfileFile() (< v1.3.1) which is used implicit.

My understanding is they behave the same in absence of ~/.aws/credentials and/or ~/.aws/config, but they don't:

package software.amazon.event.kafkaconnector;

import software.amazon.awssdk.profiles.ProfileFile;
import software.amazon.awssdk.profiles.ProfileFileSupplier;

public class Test {

    public static void main(String[] args) {
        try {
            var supplier = ProfileFileSupplier.defaultSupplier();
            var profileFile = supplier.get();
            System.out.println(profileFile);
        } catch (Exception e) {
            System.out.println(e); // java.lang.NullPointerException
        }

        var profileFile = ProfileFile.defaultProfileFile();
        System.out.println(profileFile); // ProfileFile(sections=[])
    }

}

IDK if one can call it a bug?