Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Replace the use of aws-auth configmap with EKS cluster access entry #2858

Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
51a1625
feat: Replace `resolve_conflicts` with `resolve_conflicts_on_create`/…
bryantbiggs Oct 9, 2023
57378ee
fix: Replace dynamic DNS suffix for `sts:AssumeRole` API calls for st…
bryantbiggs Nov 7, 2023
d238c00
feat: Add module tag
bryantbiggs Nov 7, 2023
ff1447c
feat: Align Karpenter permissions with Karpenter v1beta1/v0.32 permis…
bryantbiggs Nov 9, 2023
f57bcb9
refactor: Move `aws-auth` ConfigMap functionality to its own sub-module
bryantbiggs Nov 11, 2023
f8fc282
chore: Update examples
bryantbiggs Nov 14, 2023
a497463
feat: Add state `moved` block for Karpenter Pod Identity role re-name
bryantbiggs Nov 15, 2023
5eda1f3
Merge branch 'master' of github.com:terraform-aws-modules/terraform-a…
bryantbiggs Dec 19, 2023
e4c6a71
fix: Correct variable `create` description
bryantbiggs Dec 19, 2023
4f3be36
feat: Add support for cluster access entries
bryantbiggs Jan 19, 2024
005885b
chore: Bump MSV of Terraform to `1.3`
bryantbiggs Jan 26, 2024
9f1227d
fix: Replace defunct kubectl provider with an updated forked equivalent
bryantbiggs Jan 26, 2024
8cee96d
chore: Update and validate examples for access entry; clean up provid…
bryantbiggs Jan 26, 2024
570065e
docs: Correct double redundant variable descriptions
bryantbiggs Jan 26, 2024
ee48891
feat: Add support for Cloudwatch log group class argument
bryantbiggs Jan 26, 2024
685e9a1
fix: Update usage tag placement, fix Karpenter event spelling, add up…
bryantbiggs Jan 29, 2024
ef2bef9
feat: Update Karpenter module to generalize naming used and align pol…
bryantbiggs Jan 29, 2024
35685c1
feat: Add native support for Windows based managed nodegroups similar…
bryantbiggs Jan 29, 2024
aeeedd4
feat: Update self-managed nodegroup module to use latest features of ASG
bryantbiggs Jan 29, 2024
09ff91d
docs: Update and simplify docs
bryantbiggs Jan 29, 2024
2e94601
fix: Correct variable description for AMI types
bryantbiggs Jan 29, 2024
79a71dc
fix: Update upgrade guide with changes; rename Karpenter controller r…
bryantbiggs Jan 29, 2024
9cb886e
docs: Complete upgrade guide docs for migration and changes applied
bryantbiggs Jan 30, 2024
42cb888
Update examples/karpenter/README.md
bryantbiggs Feb 2, 2024
22514f8
Update examples/outposts/README.md
bryantbiggs Feb 2, 2024
0db5ca4
Update modules/karpenter/README.md
bryantbiggs Feb 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.83.5
rev: v1.86.0
hooks:
- id: terraform_fmt
- id: terraform_validate
Expand Down
163 changes: 17 additions & 146 deletions README.md

Large diffs are not rendered by default.

105 changes: 105 additions & 0 deletions docs/UPGRADE-20.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Upgrade from v19.x to v20.x

Please consult the `examples` directory for reference example configurations. If you find a bug, please open an issue with supporting configuration to reproduce.

## List of backwards incompatible changes

- Minium supported AWS provider version increased to `v5.34`
- Minimum supported Terraform version increased to `v1.3` to support Terraform state `moved` blocks as well as other advanced features
- The `resolve_conflicts` argument within the `cluster_addons` configuration has been replaced with `resolve_conflicts_on_create` and `resolve_conflicts_on_delete` now that `resolve_conflicts` is deprecated
- The default/fallback value for the `preserve` argument of `cluster_addons`is now set to `true`. This has shown to be useful for users deprovisioning clusters while avoiding the situation where the CNI is deleted too early and causes resources to be left orphaned resulting in conflicts.
- The Karpenter sub-module's use of the `irsa` naming convention has been removed, along with an update to the Karpenter controller IAM policy to align with Karpenter's `v1beta1`/`v0.32` changes. Instead of referring to the role as `irsa` or `pod_identity`, its simply just an IAM role used by the Karpenter controller and there is support for use with either IRSA and/or Pod Identity (default) at this time
- The `aws-auth` ConfigMap resources have been moved to a standalone sub-module. This removes the Kubernetes provider requirement from the main module and allows for the `aws-auth` ConfigMap to be managed independently of the main module. This sub-module will be removed entirely in the next major release of the module.
- Support for cluster access management has been added with the default authentication mode set as `API_AND_CONFIG_MAP`. This is a one way change if applied; if you wish to use `CONFIG_MAP`, you will need to set `authentication_mode = "CONFIG_MAP"` explicitly when upgrading.

### ⚠️ Upcoming Changes Planned in v21.0 ⚠️

To give users advanced notice and provide some future direction for this module, these are the following changes we will be looking to make in the next major release of this module:

1. The `aws-auth` sub-module will be removed entirely from the project. Since this sub-module is captured in the v20.x releases, users can continue using it even after the module moves forward with the next major version. The long term strategy and direction is cluster access entry and to rely only on the AWS Terraform provider.
2. The default value for `authentication_mode` will change to `API`. Aligning with point 1 above, this is a one way change, but users are free to specify the value of their choosing in place of this default (when the change is made). This module will proceed with an EKS API first strategy.

## Additional changes

### Added

- A module tag has been added to the cluster control plane
- Support for cluster access entries. The `bootstrap_cluster_creator_admin_permissions` setting on the control plane has been hardcoded to `false` since this operation is a one time operation only at cluster creation per the EKS API. Instead, users can enable/disable `enable_cluster_creator_admin_permissions` at any time to achieve the same functionality. This takes the identity that Terraform is using to make API calls and maps it into a cluster admin via an access entry. For users on existing clusters, you will need to remove the default cluster administrator that was created by EKS prior to the cluster access entry APIs - see the section [`Removing the default cluster administrator`](https://aws.amazon.com/blogs/containers/a-deep-dive-into-simplified-amazon-eks-access-management-controls/) for more details.
- Added support for specifying the CloudWatch log group class (standard or infrequent access)

### Modified

- For `sts:AssumeRole` permissions by services, the use of dynamically looking up the DNS suffix has been replaced with the static value of `amazonaws.com`. This does not appear to change by partition and instead requires users to set this manually for non-commercial regions.
- The default value for `kms_key_enable_default_policy` has changed from `false` to `true` to align with the default behavior of the `aws_kms_key` resource
- The Karpenter default value for `create_instance_profile` has changed from `true` to `false` to align with the changes in Karpenter v0.32

### Removed

-

### Variable and output changes

1. Removed variables:

- `cluster_iam_role_dns_suffix` - replaced with a static string of `amazonaws.com`
- Karpenter
- `irsa_tag_key`
- `irsa_tag_values`
- `irsa_subnet_account_id`
- `enable_karpenter_instance_profile_creation`

2. Renamed variables:

- Karpenter
- `create_irsa` -> `create_iam_role`
- `irsa_name` -> `iam_role_name`
- `irsa_use_name_prefix` -> `iam_role_name_prefix`
- `irsa_path` -> `iam_role_path`
- `irsa_description` -> `iam_role_description`
- `irsa_max_session_duration` -> `iam_role_max_session_duration`
- `irsa_permissions_boundary_arn` -> `iam_role_permissions_boundary_arn`
- `irsa_tags` -> `iam_role_tags`
- `policies` -> `iam_role_policies`
- `irsa_policy_name` -> `iam_policy_name`
- `irsa_ssm_parameter_arns` -> `ami_id_ssm_parameter_arns`

3. Added variables:

- `enable_cluster_creator_admin_permissions`
- `access_entries`
- `cloudwatch_log_group_class`

- Karpenter
- `iam_policy_use_name_prefix`
- `iam_policy_description`
- `enable_irsa`

4. Removed outputs:

- `aws_auth_configmap_yaml`

5. Renamed outputs:

- Karpenter
- `irsa_name` -> `iam_role_name`
- `irsa_arn` -> `iam_role_arn`
- `irsa_unique_id` -> `iam_role_unique_id`

6. Added outputs:

- `access_entries`

## Upgrade Migrations

### Diff of Before (v18.x) vs After (v19.x)

```diff
module "eks" {
source = "terraform-aws-modules/eks/aws"
- version = "~> 19.17.1"
+ version = "~> 20.0"

}
```

## Terraform State Moves
8 changes: 0 additions & 8 deletions docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,6 @@ If you require a public endpoint, setting up both (public and private) and restr

The module is configured to ignore this value. Unfortunately, Terraform does not support variables within the `lifecycle` block. The setting is ignored to allow autoscaling via controllers such as cluster autoscaler or Karpenter to work properly and without interference by Terraform. Changing the desired count must be handled outside of Terraform once the node group is created.

### How can I deploy Windows based nodes?

To enable Windows support for your EKS cluster, you will need to apply some configuration manually. See the [Enabling Windows Support (Windows/MacOS/Linux)](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html#enable-windows-support).

In addition, Windows based nodes require an additional cluster RBAC role (`eks:kube-proxy-windows`).

Note: Windows based node support is limited to a default user data template that is provided due to the lack of Windows support and manual steps required to provision Windows based EKS nodes.

### How do I access compute resource attributes?

Examples of accessing the attributes of the compute resource(s) created by the root module are shown below. Note - the assumption is that your cluster module definition is named `eks` as in `module "eks" { ... }`:
Expand Down
84 changes: 0 additions & 84 deletions docs/irsa_integration.md

This file was deleted.

1 change: 1 addition & 0 deletions docs/user_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Users can see the various methods of using and providing user data through the [
- By default, any supplied user data is pre-pended to the user data supplied by the EKS Managed Node Group service
- If users supply an `ami_id`, the service no longers supplies user data to bootstrap nodes; users can enable `enable_bootstrap_user_data` and use the module provided user data template, or provide their own user data template
- `bottlerocket` platform user data must be in TOML format
- `windows` platform user data must be in powershell/PS1 script format
- Self Managed Node Groups
- `linux` platform (default) -> the user data template (bash/shell script) provided by the module is used as the default; users are able to provide their own user data template
- `bottlerocket` platform -> the user data template (TOML file) provided by the module is used as the default; users are able to provide their own user data template
Expand Down
107 changes: 0 additions & 107 deletions examples/complete/README.md

This file was deleted.

Loading
Loading