Add support for spring.config.import with S3 backend (#849)

* initial commit.

* Refactor S3 Import integration

* Add enableImport flag

* Fix failing test

* Refactor

* Fix review comments

* Refactor s3 import integration

* Polish

* Polish

* Polish

* Align to other integrations

* delete unused pom.xml

* remove AwsS3ClientCustomizer

* polish docs

* polish docs

* delete S3ManagerClientCustomizer

---------

Co-authored-by: matejnedic <matejnedic1@gmail.com>
Co-authored-by: Maciej Walkowiak <walkowiak.maciej@yahoo.com>

docs/src/main/asciidoc/_configprops.adoc

@@ -46,14 +46,23 @@
 |spring.cloud.aws.s3.accelerate-mode-enabled |  | Option to enable using the accelerate endpoint when accessing S3. Accelerate endpoints allow faster transfer of objects by using Amazon CloudFront's globally distributed edge locations.
 |spring.cloud.aws.s3.checksum-validation-enabled |  | Option to disable doing a validation of the checksum of an object stored in S3.
 |spring.cloud.aws.s3.chunked-encoding-enabled |  | Option to enable using chunked encoding when signing the request payload for {@link software.amazon.awssdk.services.s3.model.PutObjectRequest} and {@link software.amazon.awssdk.services.s3.model.UploadPartRequest}.
+|spring.cloud.aws.s3.config.enabled | `+++true+++` | Enables S3 Config File import integration.
+|spring.cloud.aws.s3.config.reload.max-wait-for-restart | `+++2s+++` | If {@link ReloadStrategy#RESTART_CONTEXT} is configured, maximum waiting time for server restart.
+|spring.cloud.aws.s3.config.reload.period | `+++1m+++` | Refresh period for {@link PollingAwsPropertySourceChangeDetector}.
+|spring.cloud.aws.s3.config.reload.strategy |  | Reload strategy to run when properties change.
 |spring.cloud.aws.s3.cross-region-enabled |  | Enables cross-region bucket access.
 |spring.cloud.aws.s3.crt.initial-read-buffer-size-in-bytes |  | Configure the starting buffer size the client will use to buffer the parts downloaded from S3. Maintain a larger window to keep up a high download throughput; parts cannot download in parallel unless the window is large enough to hold multiple parts. Maintain a smaller window to limit the amount of data buffered in memory.
 |spring.cloud.aws.s3.crt.max-concurrency |  | Specifies the maximum number of S3 connections that should be established during transfer.
 |spring.cloud.aws.s3.crt.minimum-part-size-in-bytes |  | Sets the minimum part size for transfer parts. Decreasing the minimum part size causes multipart transfer to be split into a larger number of smaller parts. Setting this value too low has a negative effect on transfer speeds, causing extra latency and network communication for each part.
 |spring.cloud.aws.s3.crt.target-throughput-in-gbps |  | The target throughput for transfer requests. Higher value means more S3 connections will be opened. Whether the transfer manager can achieve the configured target throughput depends on various factors such as the network bandwidth of the environment and the configured `max-concurrency`.
 |spring.cloud.aws.s3.enabled | `+++true+++` | Enables S3 integration.
+|spring.cloud.aws.s3.encryption.enable-delayed-authentication-mode | `+++false+++` | 
+|spring.cloud.aws.s3.encryption.enable-legacy-unauthenticated-modes | `+++false+++` | 
+|spring.cloud.aws.s3.encryption.enable-multipart-put-object | `+++false+++` | 
+|spring.cloud.aws.s3.encryption.key-id |  | 
 |spring.cloud.aws.s3.endpoint |  | Overrides the default endpoint.
 |spring.cloud.aws.s3.path-style-access-enabled |  | Option to enable using path style access for accessing S3 objects instead of DNS style access. DNS style access is preferred as it will result in better load balancing when accessing S3.
+|spring.cloud.aws.s3.plugin.enable-fallback | `+++false+++` | If set to false if Access Grants does not find/return permissions, S3Client won't try to determine if policies grant access If set to true fallback policies S3/IAM will be evaluated.
 |spring.cloud.aws.s3.region |  | Overrides the default region.
 |spring.cloud.aws.s3.transfer-manager.follow-symbolic-links |  | Specifies whether to follow symbolic links when traversing the file tree in `S3TransferManager#uploadDirectory` operation.
 |spring.cloud.aws.s3.transfer-manager.max-depth |  | Specifies the maximum number of levels of directories to visit in `S3TransferManager#uploadDirectory` operation.

docs/src/main/asciidoc/s3.adoc

@@ -378,6 +378,231 @@ There can be multiple `S3ClientCustomizer` beans present in single application c
 
 Note that `S3ClientCustomizer` beans are applied **after** `AwsSyncClientCustomizer` beans and therefore can overwrite previously set configurations.
 
+=== Loading External Configuration
+
+Just like Spring Boot supports configuring application through `application.properties` stored in the file system, Spring Cloud AWS S3 integration extends this capability with fetching application configuration the S3 bucket through `spring.config.import` property.
+
+For example, assuming that there is a file `config.properties` in a bucket named `bucket-name`, to include it as Spring Boot configuration, add a following property to `application.properties` or `application.yml`:
+
+[source,properties]
+----
+spring.config.import=aws-s3:/bucket-name/config.properties
+----
+
+If a file with given name does not exist in S3, application will fail to start. If file configuration is not required for the application, and it should continue to startup even when file configuration is missing, add `optional` before prefix:
+
+[source,properties]
+----
+spring.config.import=optional:aws-s3:/bucket-name/config.properties
+----
+
+To load multiple files, separate their names with `;`:
+
+[source,properties]
+----
+spring.config.import=aws-s3:/bucket-name/config.properties;/another-name/config.yml
+----
+
+If some files are required, and other ones are optional, list them as separate entries in `spring.config.import` property:
+
+[source,properties]
+----
+spring.config.import[0]=optional:bucket-name/config.properties
+spring.config.import[1]=aws-s3=/another-name/config.yml
+----
+
+Fetched files configuration can be referenced with `@Value`, bound to `@ConfigurationProperties` classes, or referenced in `application.properties` file.
+
+`JSON`, Java Properties and `YAML` configuration file formats are supported.
+
+File resolved with `spring.config.import` can be also referenced in `application.properties`.
+For example, with a file `config.json` containing following JSON structure:
+
+[source,json]
+----
+{
+      "url": "someUrl"
+}
+----
+
+
+`spring.config.import` entry is added to `application.properties`:
+
+[source, properties]
+----
+spring.config.import=aws-s3:/bucket-name/config.json
+----
+
+File configuration values can be referenced by JSON key names:
+
+[source, java]
+----
+@Value("${url}"
+private String url;
+----
+
+=== Customizing S3Client
+
+To use custom `S3Client` in `spring.config.import`, provide an implementation of `BootstrapRegistryInitializer`. For example:
+
+[source,java]
+----
+package com.app;
+
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.s3.S3Client;
+
+import org.springframework.boot.BootstrapRegistry;
+import org.springframework.boot.BootstrapRegistryInitializer;
+
+public class S3ClientBootstrapConfiguration implements BootstrapRegistryInitializer {
+
+    @Override
+    public void initialize(BootstrapRegistry registry) {
+        registry.register(S3Client.class, context -> {
+            AwsCredentialsProvider awsCredentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create("yourAccessKey", "yourSecretKey"));
+            return S3Client.builder().credentialsProvider(awsCredentialsProvider).region(Region.EU_WEST_2).build();
+        });
+    }
+}
+----
+
+Note that this class must be listed under `org.springframework.boot.BootstrapRegistryInitializer` key in `META-INF/spring.factories`:
+
+[source, properties]
+----
+org.springframework.boot.BootstrapRegistryInitializer=com.app.S3ClientBootstrapConfiguration
+----
+
+If you want to use autoconfigured `S3Client` but change underlying SDKClient or `ClientOverrideConfiguration` you will need to register bean of type `S3ClientCustomizer`:
+Autoconfiguration will configure `S3Client` Bean with provided values after that, for example:
+
+[source,java]
+----
+package com.app;
+
+import io.awspring.cloud.autoconfigure.s3.S3ClientCustomizer;
+import java.time.Duration;
+import org.springframework.boot.BootstrapRegistry;
+import org.springframework.boot.BootstrapRegistryInitializer;
+import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
+import software.amazon.awssdk.http.SdkHttpClient;
+import software.amazon.awssdk.http.apache.ApacheHttpClient;
+import software.amazon.awssdk.services.s3.S3Client;
+
+class S3ClientBootstrapConfiguration implements BootstrapRegistryInitializer {
+
+	@Override
+	public void initialize(BootstrapRegistry registry) {
+		registry.register(S3ClientCustomizer.class, context -> (builder -> {
+			builder.overrideConfiguration(builder.overrideConfiguration().copy(c -> {
+				c.apiCallTimeout(Duration.ofMillis(2001));
+			}));
+		}));
+	}
+}
+----
+
+=== `PropertySource` Reload
+
+Some applications may need to detect changes on external property sources and update their internal status to reflect the new configuration.
+The reload feature of Spring Cloud AWS S3 config import integration is able to trigger an application reload when a related file value changes.
+
+By default, this feature is disabled. You can enable it by using the `spring.cloud.aws.s3.config.reload.strategy` configuration property (for example, in the `application.properties` file) and adding following dependencies.
+
+[source,xml]
+----
+<dependency>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-actuator</artifactId>
+</dependency>
+<dependency>
+    <groupId>org.springframework.cloud</groupId>
+    <artifactId>spring-cloud-commons</artifactId>
+</dependency>
+<dependency>
+    <groupId>org.springframework.cloud</groupId>
+    <artifactId>spring-cloud-context</artifactId>
+</dependency>
+----
+
+The following levels of reload are supported (by setting the `spring.cloud.aws.s3.config.reload.strategy` property):
+
+* `refresh` (default): Only configuration beans annotated with `@ConfigurationProperties` or `@RefreshScope` are reloaded.
+This reload level leverages the refresh feature of Spring Cloud Context.
+
+* `restart_context`: the whole Spring `ApplicationContext` is gracefully restarted. Beans are recreated with the new configuration.
+In order for the restart context functionality to work properly you must enable and expose the restart actuator endpoint
+[source,yaml]
+====
+----
+management:
+  endpoint:
+    restart:
+      enabled: true
+  endpoints:
+    web:
+      exposure:
+        include: restart
+----
+====
+
+Assuming that the reload feature is enabled with default settings (`refresh` mode), the following bean is refreshed when the file changes:
+
+====
+[java, source]
+----
+@Configuration
+@ConfigurationProperties(prefix = "bean")
+public class MyConfig {
+
+    private String message = "a message that can be changed live";
+
+    // getter and setters
+
+}
+----
+====
+
+To see that changes effectively happen, you can create another bean that prints the message periodically, as follows
+
+====
+[source,java]
+----
+@Component
+public class MyBean {
+
+    @Autowired
+    private MyConfig config;
+
+    @Scheduled(fixedDelay = 5000)
+    public void hello() {
+        System.out.println("The message is: " + config.getMessage());
+    }
+}
+----
+====
+
+The reload feature periodically re-creates the configuration from S3 file to see if it has changed.
+You can configure the polling period by using the `spring.cloud.aws.s3.config.reload.period` (default value is 1 minute).
+
+=== Configuration
+
+The Spring Boot Starter for S3 provides the following configuration options:
+
+[cols="2,3,1,1"]
+|===
+| Name | Description | Required | Default value
+| `spring.cloud.aws.s3.config.enabled` | Enables the S3 config import integration. | No | `true`
+| `spring.cloud.aws.s3.config.reload.strategy` | `Enum` | `refresh` | The strategy to use when firing a reload (`refresh`, `restart_context`)
+| `spring.cloud.aws.s3.config.reload.period` | `Duration`| `15s` | The period for verifying changes
+| `spring.cloud.aws.s3.config.reload.max-wait-time-for-restart` | `Duration`| `2s` | The maximum time between the detection of changes in property source and the application context restart when `restart_context` strategy is used.
+|===
+
+
 === IAM Permissions
 
 Following IAM permissions are required by Spring Cloud AWS:

pom.xml

@@ -58,7 +58,7 @@
 		<module>spring-cloud-aws-test</module>
 		<module>spring-cloud-aws-modulith</module>
 		<module>docs</module>
-	</modules>
+    </modules>
 
 	<dependencyManagement>
 		<dependencies>

spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/config/s3/S3ConfigDataLoader.java

@@ -0,0 +1,72 @@
+/*
+ * Copyright 2013-2023 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.awspring.cloud.autoconfigure.config.s3;
+
+import io.awspring.cloud.autoconfigure.config.BootstrapLoggingHelper;
+import io.awspring.cloud.s3.config.S3PropertySource;
+import java.util.Collections;
+import java.util.Map;
+import org.springframework.boot.context.config.ConfigData;
+import org.springframework.boot.context.config.ConfigDataLoader;
+import org.springframework.boot.context.config.ConfigDataLoaderContext;
+import org.springframework.boot.context.config.ConfigDataResourceNotFoundException;
+import org.springframework.boot.logging.DeferredLogFactory;
+import org.springframework.core.env.MapPropertySource;
+import org.springframework.lang.Nullable;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * Loads config data from AWS S3.
+ *
+ * @author Kunal Varpe
+ * @since 3.3.0
+ */
+public class S3ConfigDataLoader implements ConfigDataLoader<S3ConfigDataResource> {
+
+	public S3ConfigDataLoader(DeferredLogFactory logFactory) {
+		BootstrapLoggingHelper.reconfigureLoggers(logFactory, "io.awspring.cloud.s3.config.S3PropertySource",
+				"io.awspring.cloud.autoconfigure.config.s3.S3PropertySources");
+	}
+
+	@Override
+	@Nullable
+	public ConfigData load(ConfigDataLoaderContext context, S3ConfigDataResource resource) {
+		try {
+			// resource is disabled if s3 integration is disabled via
+			// spring.cloud.aws.s3.config.enabled=false
+			if (resource.isEnabled()) {
+				S3Client s3Client = context.getBootstrapContext().get(S3Client.class);
+				S3PropertySource propertySource = resource.getPropertySources()
+						.createPropertySource(resource.getContext(), resource.isOptional(), s3Client);
+				if (propertySource != null) {
+					return new ConfigData(Collections.singletonList(propertySource));
+				}
+				else {
+					return null;
+				}
+			}
+			else {
+				// create dummy empty config data
+				return new ConfigData(Collections.singletonList(new MapPropertySource("aws-s3:" + context, Map.of())));
+			}
+		}
+		catch (Exception e) {
+			throw new ConfigDataResourceNotFoundException(resource, e);
+		}
+
+	}
+
+}