Cross Account Buckets get region calls fail with 403

[HarshitGupta11]

Describe the bug

We have a cross region bukcet setting. Both the accounts have all the permissions on the bucket but after performing an sdk upgrade to 2.25.3 our getBucketRegion calls fail with 403.
After some investigation we found that in the earlier versio(2.21.) we were sending the call to the global endpoint whereas for the newer version(2.25.) we see its being sent to the local endpoint.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The bucket region call should work.

Current Behavior

Described above.

Reproduction Steps

Create two buckets in different accounts and create trust relationships between them. Then make a get bucket location call from both the accounts to both the buckets.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.25.53

JDK version used

1.8

Operating System and version

Redhat 8

[bhoradc]

Hi @HarshitGupta11,

Thank you for reporting the issue. We are looking into it and will get back to you soon. Meanwhile, are you able to share the error stack trace for the same?

-update
I see you might be affected due to PR #4849 - where we disabled "GlobalEndpoints" while doing Cross region enabled calls. This change was introduced in SDK version 2.23.13. Refer ChangeLog.

After some investigation we found that in the earlier versio(2.23.) we were sending the call to the global endpoint whereas for the newer version(2.25.) we see its being sent to the local endpoint.

Can you please confirm if your earlier SDK version was older than 2.23.13?

Regards,
Chaitanya

[bhoradc]

Hi @HarshitGupta11,

Kindly share below details to be able to determine the problem you are facing.

1. How does your S3Client builder looks like? Are you using the .region(AWS_GLOBAL) on the client config? Kindly provide a self-contained code sample that we can use to reproduce the 403 error.
2. Debug SDK logs - instructions to enable debug-level SDK logging can be found here
3. Last Java SDK working version (including minor version).
4. Does the issue persists when using latest SDK version - 2.28.11?

Regards,
Chaitanya

[HarshitGupta11]

Hi @bhoradc, no we are using US-EAST-1 as the default region, like:

this.s3Client = S3Client.builder().
            credentialsProvider(context.getAwsCredentialProvider()).region(Region.US_EAST_1)
              .crossRegionAccessEnabled(true).build();

7.2.18 raz logs_SDK_2.21.46.txt
7.3.1 RAZ logs_SDK_2.25.53.txt
3. Minor version is 2.21.46, I have updated the description about it.
4. We have not tried with the newest version, but can you confirm that none of the interceptors have changed since the upgrade?

[bhoradc]

Hi @HarshitGupta11,

Thanks for providing above details. Below are the steps I took to reproduce the 403 error for cross-accounts, cross-region bucket access.

1. Created bucket x in account B in us-west-2 region.
2. Configured below bucket policy for bucket x. This allows the IAM user role in account A to access bucket X in account B.

{
    "Version": "2012-10-17",
    "Statement": [
 {
    "Sid": "statement1",
    "Effect": "Allow",
    "Principal": {
    "AWS": "arn:aws:iam::<<accountA>>:user/<<testuser>>"
 },
    "Action": "*",
    "Resource": "arn:aws:s3:::x/*"
                }
                ]
 }

3. Sample code as below.

        String bucketName ="x"; 
        S3Client s3 = S3Client.builder()
                .crossRegionAccessEnabled(true)
                .region(Region.US_EAST_1)
                .build();

        GetBucketLocationRequest request = GetBucketLocationRequest.builder()
                .bucket(bucketName)
                .build();

        GetBucketLocationResponse response = s3.getBucketLocation(request);

Using above reproduction steps, I receive 403 error irrespective of the Java SDK version I use. Tested with SDK v2.20.111 (support for cross-Region access became available) thru to latest v2.28.11.

Therefore, I do not see this as a regression, but there seems an issue with the getBucket calls for cross-accounts and cross-region buckets.

For cross-region bucket access within same account, the getBucket calls works fine. I only see this not working for cross-account access. Let me know your thoughts.

I will work with the Java SDK team to further investigate on this issue. But for now, as a workaround you can try doing headBucket operation before the getBucket call and SDK should be able to utilize the x-amz-bucket-region to retry using the target bucket region.

        s3.headBucket(h->h.bucket(bucketName));
        GetBucketLocationResponse response = s3.getBucketLocation(request);

Regards,
Chaitanya

[HarshitGupta11]

Hi @bhoradc, thanks for the response and resolution of the issue, we reverted back the sdk for the time being as we were already late for the release but I will keep this method of getting the location in mind when we are updating next.

[debora-ito]

@HarshitGupta11 we discussed this issue further and I want to give some additional details.

To make the crossRegionAccess work, the SDK sends the request and then parsers the error response, expecting a 3xx code and looking for the response header that indicates the expected region. For example, with HeadBucket, S3 will first return a 301 Moved Permanently error with a x-amz-bucket-region: <region> header. The SDK will automatically pick this region and send a new request to the expected regional endpoint.

With GetBucketLocation though, even with the cross-account access configured correctly, the request call will fail with 403 Forbidden (not 3xx) and no expected region is sent in the response. There's no way for the SDK to work around this. It is very unlikely S3 will change this behavior, the API documentation calls out that the GetBucketLocation API is only supported for backwards compatibility, and HeadBucket is recommended.

So our recommendation here is the same as S3's: use HeadBucket, it will work for cross-account cross-region use cases.

Let us know if you have any questions or comments.