Unable to load AWS credentials from any provider in the chain

[Malladi-Satish]

Hi,

we triggered our streaming jobs in emr on eks with fargate and jobs running good but some times we are facing incosistent issue i.e at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: org.apache.spark.sql.kinesis.shaded.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: Unable to execute HTTP request: sts.us-east-1.amazonaws.com, org.apache.spark.sql.kinesis.shaded.amazonaws.auth.profile.ProfileCredentialsProvider@2d65bc9d: profile file cannot be null,

could you please someone face this issue previously please let us know how did you resolve it

[debora-ito]

Hi @Malladi-Satish

The error Unable to load AWS credentials from any provider in the chain indicates that the SDK tried to find aws credentials in each of the providers of the default credential provider chain but could not find one.

By the stacktrace I can see that Apache Spark has shaded the Java SDK modules, so you'll need to reach out to their support team if you have further questions.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.