CRT HTTP Client: HTTP Connections are not properly released

[i-volkov]

Describe the bug

We are using AwsCrtAsyncHttpClient for DynamoDB API calls.
Everything was working fine, however after we upgraded to the new version of SDK we noticed, that over the time LeasedConcurrency metric value is growing steadily and when it reaches maximum capacity all API requests starting to fail with timeout exception apparently because client is not able to acquire new connections from the pool.
Another observation is that the speed of increase in LeasedConcurrency is correlated to the RetryCount, so it might be that connections from the failed requests are not properly released.

Expected Behavior

All connections are properly released and connection pool always has available connections

Current Behavior

Some connections are not properly released which leads to the lack of available connections

Reproduction Steps

var clientBuilder = DynamoDbAsyncClient.builder()
clientBuilder.overrideConfiguration(ClientOverrideConfiguration.builder()
   .apiCallAttemptTimeout(Duration.ofMillis(dynamoDbConfiguration.getApiCallAttemptTimeout().toMillis()))
   .apiCallTimeout(Duration.ofMillis(dynamoDbConfiguration.getApiCallTimeout().toMillis()))
   .retryPolicy(RetryPolicy.builder()
      .numRetries(dynamoDbConfiguration.getNumRetries())
      .retryCondition(RetryCondition.defaultRetryCondition())
      .build())
   .addMetricPublisher(new DynamoDbDataDogMetricPublisher())
   .build());

var client = clientBuilder.httpClient(AwsCrtAsyncHttpClient.builder()
   .connectionMaxIdleTime(httpClientConfiguration.getConnectionMaxIdleTime())
   .connectionTimeout(httpClientConfiguration.getConnectionTimeout())
   .maxConcurrency(httpClientConfiguration.getMaxConcurrency())
   .build());

Possible Solution

No response

Additional Information/Context

The last working SDK version is 2.18.34, so theoretically the change which caused the regression should be here:
2.18.34...2.18.35
The issue is reproducible on the latest SDK version 2.22.10.

AWS Java SDK version used

2.18.35

JDK version used

openjdk version "17.0.7" 2023-04-18 LTS OpenJDK Runtime Environment Zulu17.42+19-CA (build 17.0.7+7-LTS)

Operating System and version

Linux 6.1.61-85.141.amzn2023.x86_64

[debora-ito]

Hi @i-volkov thank you for reaching out.

• Was there any change in code or in the environment, besides upgrading the SDK version?
• Which DynamoDB API are you using?
• Do you have the logs with the retry errors that correlates to the increased LeasedConcurrency? They might help in reproducing the issue on our side

[i-volkov]

Hi @i-volkov thank you for reaching out.

• Was there any change in code or in the environment, besides upgrading the SDK version?
• Which DynamoDB API are you using?
• Do you have the logs with the retry errors that correlates to the increased LeasedConcurrency? They might help in reproducing the issue on our side

Thank you for the resposne.
Answering your questions:

• Only SDK version was updated
• GetItem and PutItem
• By errors I mean exceptions caused by the exceeded api attempt timeout. With 50 ms limit for attempt there are more retires than with 150 ms and LeasedConcurrency is growing more slowly. We don't have logs for that, because these exceptions are logged by AWS SDK using DEBUG logging level as I understand

[debora-ito]

@i-volkov I think I 'm able to reproduce, still need to confirm if it's the same problem you are running into.

If I set a low apiCallAtemptTimeout like 20ms, some requests will succeed, but some will fail and will be retried. When using version 2.18.34 (and aws-crt-client 2.18.34-PREVIEW) and calling GetItem in a loop, LeasedConcurrency is 2 at max. Running the same code with version 2.18.35 (and 2.18.35-PREVIEW for aws-crt-client), the LeasedConcurrency jumps to 50.

We still need to investigate the root cause.

[i-volkov]

@debora-ito
Yes, this is pretty much the same behaviour we are observing, the only difference is that in our case it eventually leads to lack of available connections in the pool and all requests start to time out.

With 50 ms limit for attempt there are more retires than with 150 ms and LeasedConcurrency is growing more slowly

The above sentence from my previous post is incorrect, the right way of phrasing is: the more retries we have the faster LeasedConcurrency metric grows. Sorry for the confusion.

[joviegas]

The issue was occurring because of PR-3603 added in 2.18.35. Prior to this fix in 2.18.34, since we did not return when Future.isCompletedExceptionally complted, the connection used to get closed in successResponseHandler.onStream(publisher), which is right below it.
However, the latest fix PR-4825 introduced a new ResponseHandlerHelper that releases the connection in these cases. I tried with Version 2.23.2 and no longer saw the issue.

The fix is available in version 2.23.2. I would recommend using the latest version.

[i-volkov]

The issue was occurring because of PR-3603 added in 2.18.35. Prior to this fix in 2.18.34, since we did not return when Future.isCompletedExceptionally complted, the connection used to get closed in successResponseHandler.onStream(publisher), which is right below it. However, the latest fix PR-4825 introduced a new ResponseHandlerHelper that releases the connection in these cases. I tried with Version 2.23.2 and no longer saw the issue.

The fix is available in version 2.23.2. I would recommend using the latest version.

Thank you! We did a quick test using version 2.23.2 and the issue is gone

[debora-ito]

@i-volkov thank you for confirming the fix is working. Closing this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.