Merge pull request #39 from fultonj/k1

Restructure for kustomize
openstack-k8s-operators · Jan 8, 2024 · d1bd4f7 · d1bd4f7
2 parents f064d3c + ca0eaef
commit d1bd4f7
Show file tree

Hide file tree

Showing 85 changed files with 2,090 additions and 1,072 deletions.
diff --git a/README.md b/README.md
@@ -1,6 +1,11 @@
 # OpenStack K8S Operators Architectures
 
-This repo stores validated OpenStack K8S Operators architectures [VAs](va), represented as CRs for the various operators.
+This repository may be used with to create validated
+architectures ([VAs](examples/va)), represented as CRs for
+[openstack-k8s-operators](https://github.com/openstack-k8s-operators).
+It may also be used to create deployed topologies
+([DTs](examples/dt)) which should only be used for testing.
 
-It also stores deployment topologies [DTs](dt) which should only be
-used for testing.
+The following VAs are available.
+
+- [Hyperconverged OpenStack and Ceph](examples/va/hci/)
diff --git a/docs/contributing/documentation.md b/docs/contributing/documentation.md
@@ -19,3 +19,27 @@ mkdocs serve
 
 Click the link it outputs. As you save changes to files modified in your editor,
 the browser will automatically show the new content.
+
+## Structure and Content
+
+The `MkDocs` output generates nice looking HTML pages that link to the
+content genereated by github.com.
+
+This is because the authors believe it's more valuable to have
+[github.com/openstack-k8s-operators/architecture](https://github.com/openstack-k8s-operators/architecture)
+be navigable relative to the github pages which contain the CRs,
+than have all of the documentation isolated in the `docs` directory.
+Thus, there are non-relative links in the `MkDocs` content to the
+pages hosted on github.
+
+Though it's
+[possible](https://www.mkdocs.org/user-guide/writing-your-docs/#linking-to-pages)
+to create symbolic links to README files or link to a directory above
+the `docs` directory, the resulting HTML will contain invalid links
+unless all READMEs are moved out of the directories that they
+describe. However, this would make reading the CRs more complicated as
+they wouldn't have a corresponding README.
+
+Thus, if you add a new VA or DT, then please just link it in the
+`mkdocs.yml` file, similar to the way the HCI VA is linked, in to keep
+the `MkDocs` output up to date.
diff --git a/docs/dt.md b/docs/dt.md
@@ -0,0 +1 @@
+../examples/dt/README.md
diff --git a/docs/index.md b/docs/index.md
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,9 @@
+# OpenStack K8S Operators Architectures
+
+The
+[Architectures](https://github.com/openstack-k8s-operators/architecture/)
+repository may be used with to create validated architectures (VAs),
+represented as custom resources (CRs) for
+[openstack-k8s-operators](https://github.com/openstack-k8s-operators).
+It may also be used to create deployed topologies
+([DTs](dt.md)) which should only be used for testing.
diff --git a/docs/stage1.md b/docs/stage1.md
diff --git a/docs/stage2.md b/docs/stage2.md
diff --git a/docs/stage3.md b/docs/stage3.md
diff --git a/docs/stage4.md b/docs/stage4.md
diff --git a/docs/stage5.md b/docs/stage5.md
diff --git a/docs/stage6.md b/docs/stage6.md
diff --git a/va/hci/stage1/README.md → examples/common/README.md b/va/hci/stage1/README.md → examples/common/README.md
@@ -1,25 +1,28 @@
-# Stage 1
+# Common CRs
 
-Install the dependencies for Red Hat OpenStack Services on OpenShift operators
+All VAs and DTs assume that these CRs have been created.
 
-## Notes
+## OLM
 
-Requires an OpenShift 4.12+ cluster with 3 master/worker combo-nodes
+The [olm](olm) directory contains a kustomization which will generate
+Namespace, OperatorGroup, and Subscription CRs. Creating these CRs
+will install the OpenStack K8S operators and their dependencies.
 
-## Steps
-
-1. Create Namespaces
-```bash
-oc apply -f cert_manager_namespace.yaml -f metallb_namespace.yaml -f nmstate_namespace.yaml
+Observe CRs which will be generated.
 ```
-2. Create OperatorGroups
-```bash
-oc apply -f cert_manager_operatorgroup.yaml -f metallb_operatorgroup.yaml -f nmstate_operatorgroup.yaml
+kustomize build examples/common/olm/
+```
+Create the CRs.
+```
+oc apply -k examples/common/olm/
+```
+Watch the OpenStack operator pods start.
+```
+oc get pods -w -n openstack-operators
+```
+The following commands can be used to confirm that each step of this
+procedure is complete.
 ```
-3. Create Subscriptions
-```bash
-oc apply -f cert_manager_subscription.yaml -f metallb_subscription.yaml -f nmstate_subscription.yaml
-
 while ! (oc get pod --no-headers=true -l name=cert-manager-operator -n cert-manager-operator| grep "cert-manager-operator"); do sleep 10; done
 oc wait pod -n cert-manager-operator --for condition=Ready -l name=cert-manager-operator --timeout=300s
 while ! (oc get pod --no-headers=true -l app=cainjector -n cert-manager | grep "cert-manager-cainjector"); do sleep 10; done
@@ -37,17 +40,3 @@ oc wait pod -n metallb-system --for condition=Ready -l component=webhook-server
 timeout 300 bash -c "while ! (oc get deployments/nmstate-operator -n openshift-nmstate); do sleep 10; done"
 oc wait deployments/nmstate-operator -n openshift-nmstate --for condition=Available --timeout=300s
 ```
-4. Create Deploys
-```bash
-# MetalLB
-oc apply -f metallb_deploy.yaml
-timeout 300 bash -c "while ! (oc get pod --no-headers=true -l component=speaker -n metallb-system | grep speaker); do sleep 10; done"
-oc wait pod -n metallb-system -l component=speaker --for condition=Ready --timeout=300s
-
-# NMState
-oc apply -f nmstate_deploy.yaml
-timeout 300 bash -c "while ! (oc get pod --no-headers=true -l component=kubernetes-nmstate-handler -n openshift-nmstate| grep nmstate-handler); do sleep 10; done"
-oc wait pod -n openshift-nmstate -l component=kubernetes-nmstate-handler --for condition=Ready --timeout=300s
-timeout 300 bash -c "while ! (oc get deployments/nmstate-webhook -n openshift-nmstate); do sleep 10; done"
-oc wait deployments/nmstate-webhook -n openshift-nmstate --for condition=Available --timeout=300s
-```
diff --git a/examples/common/olm/kustomization.yaml b/examples/common/olm/kustomization.yaml
@@ -0,0 +1,3 @@
+resources:
+  - ../../../lib/olm-deps
+  - ../../../lib/olm-openstack
diff --git a/dt/README.md → examples/dt/README.md b/dt/README.md → examples/dt/README.md
@@ -1,6 +1,6 @@
-# Deployment Topologies
+# Deployed Topologies
 
-All validated architectures ([VAs](../va/)) are deployment topologies
+All validated architectures (VAs) are deployed topologies
 (DTs), but not all DTs are VAs.
 
 DTs represent CI optimizations. We design them to test lots of things

diff --git a/examples/dt/one/edpm/README.md b/examples/dt/one/edpm/README.md
@@ -0,0 +1 @@
+The EDPM directory of the first DT will go here
diff --git a/examples/dt/one/kustomization.yaml b/examples/dt/one/kustomization.yaml
diff --git a/examples/dt/one/values.yaml b/examples/dt/one/values.yaml
diff --git a/examples/va/hci/README.md b/examples/va/hci/README.md
@@ -0,0 +1,39 @@
+# Hyperconverged OpenStack and Ceph
+
+**Based on OpenStack K8S operators from the "main" branch of the [OpenStack Operator repo](https://github.com/openstack-k8s-operators/openstack-operator/commit/aa63bf3931f74722dd48af8a0914233b2b384330) on Dec 19th, 2023**
+
+This is a collection of CR templates that represent a validated Red Hat OpenStack Services on OpenShift deployment that has the following characteristics:
+
+- 3 master/worker combo-node OpenShift cluster
+- 3-replica Galera database
+- RabbitMQ
+- OVN networking
+- Network isolation over a single NIC
+- 3 compute nodes
+- CephHCI installed on compute nodes and used by various OSP services
+    - Cinder Volume using RBD for backend
+    - Cinder Backup using RBD for backend
+    - Glance using RBD for backend
+    - Nova using RBD for ephemeral storage
+    - Manila using CephFS for backend
+
+## Considerations
+
+1. These CRs are validated for the overall functionality of the OSP cloud deployed, but they nonetheless require customization for the particular environment in which they are utilized.  In this sense they are _templates_ meant to be consumed and tweaked to fit the specific constraints of the hardware available.
+
+2. The CRs are applied against an OpenShift cluster in _stages_.  That is, there is an ordering in which each grouping of CRs is fed to the cluster.  It is _not_ a case of simply taking all CRs from all stages and applying them all at once.
+
+3. In stages 1 and 2 [kustomize](https://kustomize.io/) is used to genereate the CRs dynamically. The `values.yaml` file(s) must be updated to fit your environment. kustomize version 5 or newer required.
+
+4. In stages 3 and 4 YAML comments are placed throughout the CRs to aid in the process of customizing the CRs.  Fields that _must_ (or most likely need to be) changed are commented with "# CHANGEME" either on the field itself or somewhere nearby.  Other comments are added to explain fields that can be changed and, sometimes, to explain additions that can be made.
+
+5. Between stages 3 and 4, _it is assumed that the user installs Ceph on the 3 OSP compute nodes._  OpenStack K8S CRDs do not provide a way to install Ceph via any sort of combination of CRs.
+
+## Stages
+
+All stages must be executed in the order listed below. Everything is required unless otherwise indicated.
+
+1. [Install the OpenStack K8S operators and their dependencies](../../common/)
+2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
+3. [Configure and deploy the initial data plane to prepare for Ceph installation](edpm-pre-ceph)
+4. [Update the control plane and finish deploying the data plane after Ceph has been installed](edpm-post-ceph)
diff --git a/examples/va/hci/control-plane.md b/examples/va/hci/control-plane.md
@@ -0,0 +1,66 @@
+# Configuring networking and deploy the OpenStack control plane
+
+## Assumptions
+
+- A storage class called `local-storage` should already exist.
+
+## Initialize
+
+Switch to the "openstack" namespace
+```
+oc project openstack
+```
+Change to the hci directory
+```
+cd architecture/examples/va/hci
+```
+Edit the [values.yaml](values.yaml) file to suit your environment.
+```
+vi values.yaml
+```
+Alternatively use your own copy of `values.yaml` and edit 
+[kustomization.yaml](kustomization.yaml) to use that copy.
+```
+resources:
+  - values-ci-framework.yaml
+```
+
+Generate the control-plane and networking CRs.
+```
+kustomize build > control-plane.yaml
+```
+
+## Create CRs
+```
+oc apply -f control-plane.yaml
+```
+
+Wait for NNCPs to be available
+```
+oc wait nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=300s
+```
+
+Wait for control plane to be available
+```
+oc wait osctlplane controlplane --for condition=Ready --timeout=600s
+```
+
+## Workaround
+
+The `control-plane.yaml` file contains CRs for both `NMState` and
+`NodeNetworkConfigurationPolicy` (NNCP). When `oc apply -f` is
+passed this file, OpenShift might try to create the NNCPs while
+`NMState` CRDs are still installing and produce the following message.
+
+```
+nmstate.nmstate.io/nmstate created
+[resource mapping not found for name:
+"ostest-master-0" namespace: "openstack" from "control-plane.yaml":
+no matches for kind "NodeNetworkConfigurationPolicy" in version "nmstate.io/v1"
+ensure CRDs are installed first,
+resource mapping not found for name: "ostest-master-1" namespace: "openstack"
+from "control-plane.yaml": no matches for kind "NodeNetworkConfigurationPolicy"
+in version "nmstate.io/v1"
+```
+Retrying `oc apply -f contol-plane.yaml` a few seconds later should
+resolve the problem however.
diff --git a/examples/va/hci/edpm-post-ceph/README.md b/examples/va/hci/edpm-post-ceph/README.md
@@ -0,0 +1,54 @@
+# EDPM Post Ceph
+
+Finish deploying the data plane after Ceph is available
+
+## Notes
+
+Assumes that a Ceph cluster is available.
+
+At this stage the CRs in this directory need to be edited to match the
+values in your environment. Look for `#CHANGEME` comments in each CR
+and update them accordingly. In the future this will not be necessary
+when this `edpm-pre-ceph` and `edpm-post-ceph` directories are updated
+to use `kustomize`.
+
+During this stage the OpenStackDataPlaneNodeSet
+(openstackdataplanenodeset.yaml) and OpenStackControlPlane
+(openstackcontrolplane.yaml) are updated. New instances of these
+services are not created. Thus, if you have modified these CRs beyond
+after [creating the OpenStackControlPlane](../control-plane.md) or after
+[creating the OpenStackDataPlaneNodeSet](../edpm-pre-ceph/README.md),
+then your personal changes could be lost when you run `oc apply` as
+described below. To avoid this, `diff` the files and apply the changes
+using `oc edit` or `oc patch` during steps 2 and 3 below.
+
+```bash
+cd architecture/examples/va/hci
+diff -u control-plane.yaml edpm-post-ceph/openstackcontrolplane.yaml
+diff -u edpm-pre-ceph/openstackdataplanenodeset.yaml edpm-post-ceph/openstackdataplanenodeset.yaml
+```
+
+## Steps
+
+1. Create Secrets
+```bash
+oc apply -f ceph_secret.yaml -f nova_ceph.yaml -f nova_migration_ssh_key.yaml
+```
+2. Update OpenStackControlPlane and wait for it to finish
+```bash
+oc apply -f openstackcontrolplane.yaml
+oc wait osctlplane controlplane --for condition=Ready --timeout=300s
+```
+3. Update OpenStackDataPlaneNodeSet
+```bash
+oc apply -f openstackdataplanenodeset.yaml
+```
+4. Create a post-Ceph OpenStackDataPlaneDeployment and wait for it to finish
+```bash
+oc apply -f openstackdataplanedeployment.yaml
+oc wait osdpd deployment-post-ceph --for condition=Ready --timeout=720s
+```
+5. Ask Nova to discover all compute hosts
+```bash
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
+```
diff --git a/va/hci/stage6/ceph_secret.yaml → ...es/va/hci/edpm-post-ceph/ceph_secret.yaml b/va/hci/stage6/ceph_secret.yaml → ...es/va/hci/edpm-post-ceph/ceph_secret.yaml
diff --git a/va/hci/stage6/nova_ceph.yaml → ...ples/va/hci/edpm-post-ceph/nova_ceph.yaml b/va/hci/stage6/nova_ceph.yaml → ...ples/va/hci/edpm-post-ceph/nova_ceph.yaml
diff --git a/va/hci/stage6/nova_migration_ssh_key.yaml → ...dpm-post-ceph/nova_migration_ssh_key.yaml b/va/hci/stage6/nova_migration_ssh_key.yaml → ...dpm-post-ceph/nova_migration_ssh_key.yaml
diff --git a/va/hci/stage6/openstackcontrolplane.yaml → ...edpm-post-ceph/openstackcontrolplane.yaml b/va/hci/stage6/openstackcontrolplane.yaml → ...edpm-post-ceph/openstackcontrolplane.yaml
@@ -11,7 +11,7 @@
 apiVersion: core.openstack.org/v1beta1
 kind: OpenStackControlPlane
 metadata:
-  name: openstack-galera-network-isolation-3replicas
+  name: controlplane
   namespace: openstack
 spec:
   ceilometer:

diff --git a/.../stage6/openstackdataplanedeployment.yaml → ...st-ceph/openstackdataplanedeployment.yaml b/.../stage6/openstackdataplanedeployment.yaml → ...st-ceph/openstackdataplanedeployment.yaml
diff --git a/va/hci/stage6/openstackdataplanenodeset.yaml → ...-post-ceph/openstackdataplanenodeset.yaml b/va/hci/stage6/openstackdataplanenodeset.yaml → ...-post-ceph/openstackdataplanenodeset.yaml
diff --git a/va/hci/stage5/README.md → examples/va/hci/edpm-pre-ceph/README.md b/va/hci/stage5/README.md → examples/va/hci/edpm-pre-ceph/README.md
@@ -1,9 +1,15 @@
-# Stage 5
+# EDPM Pre Ceph
 
 Deploy the initial data plane to prepare for CephHCI installation
 
 ## Notes
 
+At this stage the CRs in this directory need to be edited to match the
+values in your environment. Look for `#CHANGEME` comments in each CR
+and update them accordingly. In the future this will not be necessary
+when this `edpm-pre-ceph` and `edpm-post-ceph` directories are updated
+to use `kustomize`.
+
 Requires 3 pre-provisioned compute nodes that are accessible via the control plane IPs
 enumerated in step 2's `OpenStackDataPlaneNodeSet` using the SSH credentials provided
 in step 1's `Secret`.

diff --git a/va/hci/stage5/dataplanesshsecret.yaml → ...hci/edpm-pre-ceph/dataplanesshsecret.yaml b/va/hci/stage5/dataplanesshsecret.yaml → ...hci/edpm-pre-ceph/dataplanesshsecret.yaml
diff --git a/.../stage5/openstackdataplanedeployment.yaml → ...re-ceph/openstackdataplanedeployment.yaml b/.../stage5/openstackdataplanedeployment.yaml → ...re-ceph/openstackdataplanedeployment.yaml
diff --git a/va/hci/stage5/openstackdataplanenodeset.yaml → ...m-pre-ceph/openstackdataplanenodeset.yaml b/va/hci/stage5/openstackdataplanenodeset.yaml → ...m-pre-ceph/openstackdataplanenodeset.yaml
diff --git a/examples/va/hci/kustomization.yaml b/examples/va/hci/kustomization.yaml
@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+components:
+  - ../../../va/hci/
+  # - https://github.com/openstack-k8s-operators/architecture/va/hci?ref=main
+  ## It's possible to replace ../../../va/hci/ with a git checkout URL as per:
+  ## https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md
+
+resources:
+  - values.yaml
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		The EDPM directory of the first DT will go here