Add a functionality to exclude headers during custom signing.

[mukund-thakur]

Describe the feature

In V1 SDK we are overriding this https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWS4Signer.java#L537 custom method to achieve this functionality. But there is no similar option in V2 SDK.

Use Case

We use this feature internally in some of our authorization use cases which won't work without this/similar method in V2 SDK.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Java SDK version used

2.19.12

JDK version used

java version "1.8.0_161"

Operating System and version

Mac OS

[debora-ito]

The method is not present in v2, but the same headers are excluded in a Helper class here -
https://github.com/aws/aws-sdk-java-v2/blob/21bdc9566bcebf816429c031dddeec48b6eaf30f/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/util/HeaderTransformsHelper.java

You can write a custom ExecutionInterceptor and use one of the interceptor hooks to remove the headers you need -
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/interceptor/ExecutionInterceptor.html

Can you use ExecutionInterceptors in your use case?

[mukund-thakur]

Thanks for your response. Yes, the default headers are getting excluded using this helper method but we have a use case to exclude some other custom headers during signing.
Just taking a quick look I don't think we can use ExecutionInterceptors as that seems to help with updating the actual request to S3. But in our we just want to remove some extra headers only during the signing process but those headers will still be present during actual execution of HTTP request. Also the whole signing of the request just happens in the SDK.

[debora-ito]

Are you sending the requests to a third-party endpoint or to S3?
If you are sending to S3, can you show an example with actual headers?

I'm trying to understand how your use case works... if you are removing headers that are not signed, why do you need to remove them before the signature calculation and add them later?

[mukund-thakur]

No, the request is going to S3. We are at Cloudera where the Hadoop S3A connector is used.

Adding one example request below. As you can see SignedHeaders doesn't contain all the headers being sent for example referrer.

23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "GET /?list-type=2&delimiter=%2F&max-keys=2&prefix=mukund%2F&fetch-owner=false HTTP/1.1[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Host: harshit-s3-uswest-2.s3.amazonaws.com[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 03d175d6-ab85-8167-4887-d703a5c2e05f[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-request: ttl=20230814T171105Z;attempt=1;max=21[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=/20230814/us-west-2/s3/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Referer: https://audit.example.org/hadoop/1/op_get_file_status/51572c79-3905-4b7f-bfda-07c87b8ae682-00000006/?op=op_get_file_status&p1=mukund&pr=csso_mthakur@HARSHIT.XCU2-8Y8X.DEV.CLDR.WORK&ps=36491d99-9595-471e-a7c2-181cee10f020&cm=FsShell&id=51572c79-3905-4b7f-bfda-07c87b8ae682-00000006&t0=1&fs=51572c79-3905-4b7f-bfda-07c87b8ae682&t1=1&ts=1692032865033[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "User-Agent: user:csso_mthakur,Hadoop 3.1.1.7.2.17.0-330, aws-sdk-java/1.12.316 Linux/3.10.0-1160.90.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 groovy/3.0.7 vendor/Red_Hat,_Inc. cfg/retry-mode/legacy[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20230814T170745Z[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Security-Token: *******[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: 4v9/xxxx/xx/3+oBneYeic=[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 34BJYWGS5G6V6P6A[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "Date: Mon, 14 Aug 2023 17:07:46 GMT[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-west-2[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
23/08/14 17:07:45 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"

[debora-ito]

A couple of questions more:

1. What happens if Referer is added to the signature when using Java SDK v2? Does S3 return an error?
2. In the example above, why it's important that Referer is sent as a header to S3? How you're going to use it?

[steveloughran]

we attach audit information about the calling operation which is then preserved in the s3 server logs for fielding support calls or other analysis (which jobs trigger throttling, who is doing the most HEAD requests, in a job what are the ranges of a file being read...)

https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/auditing.html

[mukund-thakur]

Referer is one example I gave. We don't want to create signatures for each request but rather cache and re-use it. For example, not adding range header in the signature for multiple get requests on the same file, reuse the same signature for multiple get requests. Does this answer your question? Let me know if you have any other. Thanks

[mukund-thakur]

Hi @debora-ito Any update on this? Or do you have any other questions?
CC @ahmarsuhail

[debora-ito]

@mukund-thakur Sorry for the long silence. We are still not entirely sure that excluding headers for signing is a feature we'd like to support. I'm going to work on proof of concept using ExecutionInterceptors to see if it satisfy your use case.

[steveloughran]

We are still not entirely sure that excluding headers for signing is a feature we'd like to support

well, this marks a regression from v1. 😞

[debora-ito]

Quick update: we will work on this feature.

[shorea]

I recently ran into this as well. For anyone who would like a quick workaround you can do something like this for the headers you'd wish to exclude. Note that if you are using S3 there is a different base Signer (AbstractAwsS3V4Signer) used but the concept would be the same.

public class ExcludeHeadersSigner extends BaseAws4Signer {

    public static final List<String> EXCLUDED_HEADERS = List.of("headerOne", "headerTwo");

    @Override
    public SdkHttpFullRequest sign(SdkHttpFullRequest request, ExecutionAttributes executionAttributes) {
        Map<String, List<String>> excludedHeaderOrig = new HashMap<>();
        EXCLUDED_HEADERS.forEach(header -> {
            List<String> values = request.headers().get(header);
            if (!CollectionUtils.isNullOrEmpty(values)) {
                excludedHeaderOrig.put(header, values);
            }
        });
        // Remove excluded headers before signing
        SdkHttpFullRequest.Builder builder = request.toBuilder();
        EXCLUDED_HEADERS.forEach(builder::removeHeader);

        // Sign the request
        Aws4SignerParams signingParams = extractSignerParams(Aws4SignerParams.builder(), executionAttributes)
            .build();
        SdkHttpFullRequest.Builder afterSign = sign(builder.build(), signingParams).toBuilder();

        // Add excluded headers back after signing
        excludedHeaderOrig.forEach(afterSign::putHeader);
        return afterSign.build();
    }
}

You can register this custom signer during client creation like so

        return SomeClient.builder()
                              .overrideConfiguration(o -> o.putAdvancedOption(SdkAdvancedClientOption.SIGNER,
                                                                              new ExcludeHeadersSigner()))
                              .build();

Note that you can re-write this as an interface decorator rather than extending the Base signer. This would be more flexible and less coupled to SDK internals.

[debora-ito]

Thank you for the sample, Andrew! 👋🏻 @shorea

Note to all: this path is for advanced use cases, you must implement your own logic to exclude the headers from signing, there's no explicit API in the SDK that provides this feature.

Closing the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.