diff --git a/charts/metrics-server/README.md b/charts/metrics-server/README.md index 187fbb3636..bbbf93679f 100644 --- a/charts/metrics-server/README.md +++ b/charts/metrics-server/README.md @@ -86,3 +86,133 @@ The following table lists the configurable parameters of the _Metrics Server_ ch | `topologySpreadConstraints` | Pod Topology Spread Constraints. | `[]` | | `deploymentAnnotations` | Annotations to add to the deployment. | `{}` | | `schedulerName` | scheduler to set to the deployment. | `""` | +| `tls.type` | TLS option to use. Either use `metrics-server` for self-signed certificates, `secret`, `existingSecret`, `cert-manager` or `helm` | `"metrics-server"` | +| `tls.certManager.clusterDomain` | Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate | `"cluster.local"` | +| `tls.certManager.addInjectorAnnotations` | Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource. | `true` | +| `tls.certManager.existingIssuer.enabled` | Use an existing cert-manager issuer | `false` | +| `tls.certManager.existingIssuer.kind` | Kind of the existing cert-manager issuer | `"Issuer"` | +| `tls.certManager.existingIssuer.name` | Name of the existing cert-manager issuer | `"my-issuer"` | +| `tls.certManager.duration` | Set the requested duration (i.e. lifetime) of the Certificate. | `""` | +| `tls.certManager.renewBefore` | How long before the currently issued certificate’s expiry cert-manager should renew the certificate. | `""` | +| `tls.certManager.annotations` | Add extra annotations to the Certificate resource | `{}` | +| `tls.certManager.labels` | Add extra labels to the Certificate resource | `{}` | +| `tls.existingSecret.name` | Name of the existing Secret to use for TLS | `""` | +| `tls.secret.annotations` | Add extra annotations to the Secret resource | `{}` | +| `tls.secret.labels` | Add extra labels to the Secret resource | `{}` | +| `tls.secret.crt` | The server certificate to use for metrics-server. Use PEM format | `""` | +| `tls.secret.key` | The private key of the certificate to use for metrics-server. Use PEM format. | `""` | +| `extraObjects` | List of extra manifests to deploy. Will be passed through `tpl` to support templating | `[]` | + +## Hardening metrics-server + +By default, metrics-server is using a self-signed certificate which is generated during startup. The APIservice is registered with `.spec.insecureSkipTLSVerify` set to `true` as you can see here: + +```yaml +apiVersion: apiregistration.k8s.io/v1 +kind: APIService +metadata: + name: v1beta1.metrics.k8s.io +spec: + #.. + insecureSkipTLSVerify: true # <-- see here + service: + name: metrics-server + #.. +``` + +To harden metrics-server, you have these 4 options described in the following section. + +### Option 1: Let helm generate a self-signed certificate + +This option is probably the easiest solution for you. We delegate helm the generation process for self-signed certificates. +As helm generates them during deploy time, helm can also inject the `apiService.caBundle` for you. +**The only disadvantage of using this method is that it is not GitOps friendly** (e.g. Argo CD). If you are using one of these +GitOps tools with drift detection, it will always detect changes. However if you are deploying the helm chart via Terraform +for example (or maybe even Flux), this method is perfectly fine. + +To use this method, please setup your values file like this: + +```yaml +apiService: + insecureSkipTLSVerify: false +tls: + type: helm +``` + +### Option 2: Use cert-manager + +> **Requirement:** cert-manager needs to be installed before you install metrics-server + +To use this method, please setup your values file like this: + +```yaml +apiService: + insecureSkipTLSVerify: false +tls: + type: cert-manager +``` + +There are other optional parameters, if you want to customize the behavior of the certificate even more. + +### Option 3: Provide certificate data + +You can use an arbitrary PKI solution and generate a certificate for metrics-server. You need the following data in PEM format: + +- The server certificate which is issued with 3 Subject Alt Names: + - `metrics-server.` + - `metrics-server..svc` + - `metrics-server..svc.cluster.local` +- The server key file +- The CA certificate *(optional, you can also provide the server certificate to `apiService.caBundle`)* + +To use this method, please setup your values file like this: + +```yaml +apiService: + insecureSkipTLSVerify: false + caBundle: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + + +tls: + type: secret + secret: + crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + + key: | + -----BEGIN RSA PRIVATE KEY----- + ... + -----END RSA PRIVATE KEY----- +``` + +### Option 4: Use existing Secret + +This option allows you to reuse an existing Secret. This Secrets can have an arbitrary origin, e.g. + +- Created via kubectl / Terraform / etc. +- Synced from a secrets management solution like AWS SecretsManager, HashiCorp Vault, etc. + +You still need to pass the CA certificate to ensure proper configuration of the `APIservice` resource, +but the sensitive information (the private key) can be read via Secret. Same as in the previous option "2", +`apiService.caBundle` can also be the server certificate. + +To use this method, please setup your values file like this: + +```yaml +apiService: + insecureSkipTLSVerify: false + caBundle: + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + +tls: + type: existingSecret + existingSecret: + name: metrics-server-existing +``` diff --git a/charts/metrics-server/ci/tls-certManager-values.yaml b/charts/metrics-server/ci/tls-certManager-values.yaml new file mode 100644 index 0000000000..3dcfd0ea6f --- /dev/null +++ b/charts/metrics-server/ci/tls-certManager-values.yaml @@ -0,0 +1,8 @@ +args: + - --kubelet-insecure-tls + +apiService: + insecureSkipTLSVerify: false + +tls: + type: cert-manager diff --git a/charts/metrics-server/ci/tls-helm-values.yaml b/charts/metrics-server/ci/tls-helm-values.yaml new file mode 100644 index 0000000000..dc8342865c --- /dev/null +++ b/charts/metrics-server/ci/tls-helm-values.yaml @@ -0,0 +1,8 @@ +args: + - --kubelet-insecure-tls + +apiService: + insecureSkipTLSVerify: false + +tls: + type: helm diff --git a/charts/metrics-server/templates/apiservice.yaml b/charts/metrics-server/templates/apiservice.yaml index f58931d9e9..392d59a7da 100644 --- a/charts/metrics-server/templates/apiservice.yaml +++ b/charts/metrics-server/templates/apiservice.yaml @@ -1,17 +1,56 @@ -{{- if .Values.apiService.create -}} +{{ $commonName := include "metrics-server.fullname" . }} +{{ $ns := .Release.Namespace }} +{{- $altNames := list -}} +{{- $altNames = append $altNames (printf "%s.%s" $commonName $ns) -}} +{{- $altNames = append $altNames (printf "%s.%s.svc" $commonName $ns) -}} +{{- $altNames = append $altNames (printf "%s.%s.svc.%s" $commonName $ns .Values.tls.certManager.clusterDomain) -}} +{{- $certs := genSelfSignedCert $commonName nil $altNames 36500 -}} +{{- if or (eq .Values.tls.type "secret") (eq .Values.tls.type "helm") }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "metrics-server.fullname" . }} + {{- with .Values.tls.secret.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} + labels: + {{- include "metrics-server.labels" . | nindent 4 }} + {{- with .Values.tls.secret.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +type: Opaque +data: + {{- if eq .Values.tls.type "secret" }} + tls.crt: {{ .Values.tls.secret.crt | b64enc | quote }} + tls.key: {{ .Values.tls.secret.key | b64enc | quote }} + {{- else }} + tls.crt: {{ $certs.Cert| b64enc | quote }} + tls.key: {{ $certs.Key | b64enc | quote }} + {{- end }} +{{- end }} +--- +{{- if .Values.apiService.create }} apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.metrics.k8s.io labels: {{- include "metrics-server.labels" . | nindent 4 }} - {{- with .Values.apiService.annotations }} + {{- if or .Values.apiService.annotations .Values.tls.certManager.addInjectorAnnotations }} annotations: - {{- toYaml . | nindent 4 }} + {{- if and (eq .Values.tls.type "cert-manager") .Values.tls.certManager.addInjectorAnnotations }} + cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ include "metrics-server.fullname" . }} + {{- end }} + {{- with .Values.apiService.annotations }} + {{- toYaml . | nindent 4 }} + {{- end }} {{- end }} spec: - {{- with .Values.apiService.caBundle }} - caBundle: {{ b64enc . }} + {{- if eq .Values.tls.type "helm" }} + caBundle: {{ $certs.Cert | b64enc }} + {{- else if and .Values.apiService.caBundle (ne .Values.tls.type "cert-manager") }} + caBundle: {{ .Values.apiService.caBundle | b64enc }} {{- end }} group: metrics.k8s.io groupPriorityMinimum: 100 @@ -22,4 +61,4 @@ spec: port: {{ .Values.service.port }} version: v1beta1 versionPriority: 100 -{{- end -}} +{{- end }} diff --git a/charts/metrics-server/templates/certificate.yaml b/charts/metrics-server/templates/certificate.yaml new file mode 100644 index 0000000000..74ec6a3f98 --- /dev/null +++ b/charts/metrics-server/templates/certificate.yaml @@ -0,0 +1,47 @@ +{{- if eq .Values.tls.type "cert-manager" }} + {{- if not .Values.tls.certManager.existingIssuer.enabled }} +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + annotations: + {{- toYaml .Values.additionalAnnotations | nindent 4 }} + name: {{ include "metrics-server.fullname" . }}-issuer + namespace: {{ .Release.Namespace }} +spec: + selfSigned: {} + {{- end }} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: {{ include "metrics-server.fullname" . }} + namespace: {{ .Release.Namespace }} +spec: + commonName: {{ include "metrics-server.fullname" . }} + dnsNames: + - {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }} + - {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc + - {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.tls.certManager.clusterDomain }} + secretName: {{ include "metrics-server.fullname" . }} + usages: + - server auth + - client auth + privateKey: + algorithm: RSA + size: 2048 + {{- with .Values.tls.certManager.duration }} + duration: {{ . }} + {{- end }} + {{- with .Values.tls.certManager.renewBefore }} + renewBefore: {{ . }} + {{- end }} + issuerRef: + {{- if .Values.tls.certManager.existingIssuer.enabled }} + name: {{ .Values.tls.certManager.existingIssuer.name }} + kind: {{ .Values.tls.certManager.existingIssuer.kind }} + {{- else }} + name: {{ include "metrics-server.fullname" . }}-issuer + kind: Issuer + {{- end }} + group: cert-manager.io +{{- end }} diff --git a/charts/metrics-server/templates/deployment.yaml b/charts/metrics-server/templates/deployment.yaml index 9f44be441d..f031a83b1b 100644 --- a/charts/metrics-server/templates/deployment.yaml +++ b/charts/metrics-server/templates/deployment.yaml @@ -62,6 +62,10 @@ spec: {{- if .Values.metrics.enabled }} - --authorization-always-allow-paths=/metrics {{- end }} + {{- if ne .Values.tls.type "metrics-server" }} + - --tls-cert-file=/tmp/tls-certs/tls.crt + - --tls-private-key-file=/tmp/tls-certs/tls.key + {{- end }} {{- range .Values.args }} - {{ . }} {{- end }} @@ -80,6 +84,11 @@ spec: volumeMounts: - name: tmp mountPath: /tmp + {{- if ne .Values.tls.type "metrics-server" }} + - mountPath: /tmp/tls-certs + name: certs + readOnly: true + {{- end }} {{- with .Values.extraVolumeMounts }} {{- toYaml . | nindent 12 }} {{- end }} @@ -126,6 +135,15 @@ spec: configMap: name: {{ include "metrics-server.addonResizer.configMap" . }} {{- end }} + {{- if ne .Values.tls.type "metrics-server" }} + - name: certs + secret: + {{- if and (eq .Values.tls.type "existingSecret") .Values.tls.existingSecret.name }} + secretName: {{ .Values.tls.existingSecret.name }} + {{- else }} + secretName: {{ include "metrics-server.fullname" . }} + {{- end }} + {{- end }} {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} diff --git a/charts/metrics-server/templates/extra-manifests.yaml b/charts/metrics-server/templates/extra-manifests.yaml new file mode 100644 index 0000000000..a9bb3b6ba8 --- /dev/null +++ b/charts/metrics-server/templates/extra-manifests.yaml @@ -0,0 +1,4 @@ +{{ range .Values.extraObjects }} +--- +{{ tpl (toYaml .) $ }} +{{ end }} diff --git a/charts/metrics-server/values.yaml b/charts/metrics-server/values.yaml index 7520a947bc..3ed6ee638a 100644 --- a/charts/metrics-server/values.yaml +++ b/charts/metrics-server/values.yaml @@ -174,3 +174,84 @@ topologySpreadConstraints: [] deploymentAnnotations: {} schedulerName: "" + +tls: + # Either use `metrics-server`, `secret`, `existingSecret`, `cert-manager` or `helm`. Details: + # `metrics-server` : Metrics-server will generate self-signed certs + # `secret` : Create a new secret with user-provided key and certificate + # `existingSecret` : Reuse an existing secret. No new secret will be created + # `cert-manager` : Use cert-manager.io to create and maintain the certificate for you + # `helm` : Helm will generate the certificate + type: "metrics-server" + + certManager: + # Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate + clusterDomain: cluster.local + # Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource. + # See https://cert-manager.io/docs/concepts/ca-injector + addInjectorAnnotations: true + existingIssuer: + # Use an existing cert-manager issuer + enabled: false + # Kind of the existing cert-manager issuer + kind: "Issuer" + # Name of the existing cert-manager issuer + name: "my-issuer" + # Set the requested duration (i.e. lifetime) of the Certificate. + # See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec + duration: "" + # How long before the currently issued certificate’s expiry cert-manager should renew the certificate. + # See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec + renewBefore: "" + # Add extra annotations to the Certificate resource + annotations: {} + # Add extra labels to the Certificate resource + labels: {} + + existingSecret: + # Name of the existing Secret to use for TLS + name: "" + + secret: + # Add extra annotations to the Secret resource + annotations: {} + # Add extra labels to the Secret resource + labels: {} + # The server certificate to use for metrics-server. Use PEM format + crt: "" + # The private key of the certificate to use for metrics-server. Use PEM format. + key: "" + +# List of extra manifests to deploy. Will be passed through `tpl` to support templating +extraObjects: [] + # - apiVersion: external-secrets.io/v1beta1 + # kind: ExternalSecret + # metadata: + # name: my-external-secret + # namespace: '{{ .Release.Namespace }}' + # spec: + # secretStoreRef: + # kind: ClusterSecretStore + # name: my-secret-store + # target: + # name: my-kubernetes-secret + # data: + # - secretKey: secretKey + # remoteRef: + # key: /path/to/my-secret + + # - apiVersion: secrets.hashicorp.com/v1beta1 + # kind: VaultStaticSecret + # metadata: + # name: my-external-vault-secret + # namespace: '{{ .Release.Namespace }}' + # spec: + # type: kv-v2 + # mount: kvv2 + # path: webapp/config + # destination: + # name: secretkv + # create: true + # refreshAfter: 30s + # refreshAfter: 30s + # vaultAuthRef: static-auth