Merge pull request #62 from abays/va2_sriov

SRIOV VA

examples/va/hci/kustomization.yaml

@@ -7,5 +7,54 @@ components:
   ## It's possible to replace ../../../va/hci/ with a git checkout URL as per:
   ## https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md
 
+# Add StorageMgmt network template, as it is needed for CephHCI
+patches:
+- target:
+    version: v1beta1
+    kind: NetConfig
+    name: netconfig
+  patch: |-
+    - op: add
+      path: /spec/networks/-
+      value:
+        dnsDomain: _replaced_
+        name: StorageMgmt
+        subnets:
+          - _replaced_
+        mtu: 1500
+
 resources:
   - values.yaml
+
+# Add StorageMgmt network replacements
+replacements:
+# NetConfig dnsDomain specific to this VA
+- source:
+    kind: ConfigMap
+    name: network-values
+    fieldPath: data.storagemgmt.dnsDomain
+  targets:
+  - select:
+      kind: NetConfig
+    fieldPaths:
+    - spec.networks.[name=StorageMgmt].dnsDomain
+# NetConfig MTU specific to this VA
+- source:
+    kind: ConfigMap
+    name: network-values
+    fieldPath: data.storagemgmt.mtu
+  targets:
+  - select:
+      kind: NetConfig
+    fieldPaths:
+    - spec.networks.[name=StorageMgmt].mtu
+# NetConfig subnets specific to this VA
+- source:
+    kind: ConfigMap
+    name: network-values
+    fieldPath: data.storagemgmt.subnets
+  targets:
+  - select:
+      kind: NetConfig
+    fieldPaths:
+    - spec.networks.[name=StorageMgmt].subnets

examples/va/nfv/sriov/.gitignore

@@ -0,0 +1 @@
+control-plane.yaml

examples/va/nfv/sriov/README.md

@@ -0,0 +1,30 @@
+# OpenStack SRIOV
+
+**Based on OpenStack K8S operators from the "main" branch of the [OpenStack Operator repo](https://github.com/openstack-k8s-operators/openstack-operator/tree/12c57baeca4ae33dd30a7707d330eb094309b4cd) on Dec 6th, 2023**
+
+This is a collection of CR templates that represent a validated Red Hat OpenStack Services on OpenShift deployment that has the following characteristics:
+
+- 3 master/worker combo-node OpenShift cluster
+- 3-replica Galera database
+- RabbitMQ
+- OVN networking
+- Network isolation over a single NIC
+- 2 compute nodes with an additional SRIOV-enabled NIC
+
+## Considerations
+
+1. These CRs are validated for the overall functionality of the OSP cloud deployed, but they nonetheless require customization for the particular environment in which they are utilized.  In this sense they are _templates_ meant to be consumed and tweaked to fit the specific constraints of the hardware available.
+
+2. The CRs are applied against an OpenShift cluster in _stages_.  That is, there is an ordering in which each grouping of CRs is fed to the cluster.  It is _not_ a case of simply taking all CRs from all stages and applying them all at once.
+
+3. In stages 1 and 2 [kustomize](https://kustomize.io/) is used to genereate the CRs dynamically. The `*-values.yaml` file(s) must be updated to fit your environment. kustomize version 5 or newer required.
+
+4. In stage 3 YAML comments are placed throughout the CRs to aid in the process of customizing the CRs.  Fields that _must_ (or most likely need to be) changed are commented with "# CHANGEME" either on the field itself or somewhere nearby.  Other comments are added to explain fields that can be changed and, sometimes, to explain additions that can be made.
+
+## Stages
+
+All stages must be executed in the order listed below. Everything is required unless otherwise indicated.
+
+1. [Install the OpenStack K8S operators and their dependencies](../../../common/)
+2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
+3. [Configure and deploy the data plane](edpm)

examples/va/nfv/sriov/control-plane.md

@@ -0,0 +1,70 @@
+# Configuring networking and deploy the OpenStack control plane
+
+## Assumptions
+
+- A storage class called `local-storage` should already exist.
+
+## Initialize
+
+Switch to the "openstack" namespace
+```
+oc project openstack
+```
+Change to the nfv/sriov directory
+```
+cd architecture/examples/va/nfv/sriov
+```
+Edit the [values.yaml](values.yaml) and
+[service-values.yaml](service-values.yaml) files to suit 
+your environment.
+```
+vi values.yaml
+vi service-values.yaml
+```
+Alternatively use your own copies of those files and edit
+[kustomization.yaml](kustomization.yaml) to use those copies.
+```
+resources:
+  - values-ci-framework.yaml
+  - service-values-ci-framework.yaml
+```
+
+Generate the control-plane and networking CRs.
+```
+kustomize build > control-plane.yaml
+```
+
+## Create CRs
+```
+oc apply -f control-plane.yaml
+```
+
+Wait for NNCPs to be available
+```
+oc wait nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=300s
+```
+
+Wait for control plane to be available
+```
+oc wait osctlplane controlplane --for condition=Ready --timeout=600s
+```
+
+## Workaround
+
+The `control-plane.yaml` file contains CRs for both `NMState` and
+`NodeNetworkConfigurationPolicy` (NNCP). When `oc apply -f` is
+passed this file, OpenShift might try to create the NNCPs while
+`NMState` CRDs are still installing and produce the following message.
+
+```
+nmstate.nmstate.io/nmstate created
+[resource mapping not found for name:
+"ostest-master-0" namespace: "openstack" from "control-plane.yaml":
+no matches for kind "NodeNetworkConfigurationPolicy" in version "nmstate.io/v1"
+ensure CRDs are installed first,
+resource mapping not found for name: "ostest-master-1" namespace: "openstack"
+from "control-plane.yaml": no matches for kind "NodeNetworkConfigurationPolicy"
+in version "nmstate.io/v1"
+```
+Retrying `oc apply -f contol-plane.yaml` a few seconds later should
+resolve the problem however.

examples/va/nfv/sriov/edpm/README.md

@@ -0,0 +1,30 @@
+# SRIOV EDPM
+
+Deploy the SRIOV-enabled data plane
+
+## Notes
+
+At this stage the CRs in this directory need to be edited to match the
+values in your environment. Look for `#CHANGEME` comments in each CR
+and update them accordingly. In the future this will not be necessary
+when this is updated to use `kustomize`.
+
+## Steps
+
+1. Create Secrets
+```bash
+oc apply -f dataplanesshsecret.yaml -f nova_migration_ssh_key.yaml
+```
+2. Create SRIOV ConfigMaps and associated OpenStackDataPlaneService
+```bash
+oc apply -f nova_sriov.yaml
+```
+3. Create OpenStackDataPlaneNodeSet
+```bash
+oc apply -f openstackdataplanenodeset.yaml
+```
+4. Create OpenStackDataPlaneDeployment and wait for it to finish
+```bash
+oc apply -f openstackdataplanedeployment.yaml
+oc wait osdpd edpm-sriov-deployment --for condition=Ready --timeout=720s
+```

examples/va/nfv/sriov/edpm/dataplanesshsecret.yaml

@@ -0,0 +1,24 @@
+#
+# CHANGEME:
+#
+# - Change "data.authorized_keys" as indicated below
+# - Change "data.ssh-privatekey" as indicated below
+# - Change "data.ssh-publickey" as indicated below
+#
+
+apiVersion: v1
+data:
+  # authorized_keys are the public keys that are allowed to SSH to the compute nodes.  It can be set like so:
+  # KEYDATA=$(cat <pub key file> | base64 -w 0) yq -i '(.data["authorized_keys"])=("${KEYDATA}" | envsubst)' dataplanesshsecret.yaml
+  authorized_keys: # CHANGEME
+  # ssh-privatekey is the private key that can SSH to compute nodes.  It can be set like so:
+  # KEYDATA=$(cat <private key file> | base64 -w 0) yq -i '(.data["ssh-privatekey"])=("${KEYDATA}" | envsubst)' dataplanesshsecret.yaml
+  ssh-privatekey: # CHANGEME
+  # ssh-publickey is the public key that can SSH to compute nodes. It can be set like so:
+  # KEYDATA=$(cat <pub key file> | base64 -w 0) yq -i '(.data["ssh-publickey"])=("${KEYDATA}" | envsubst)' dataplanesshsecret.yaml
+  ssh-publickey: # CHANGEME
+kind: Secret
+metadata:
+  name: dataplane-ansible-ssh-private-key-secret
+  namespace: openstack
+type: Opaque

examples/va/nfv/sriov/edpm/nova_migration_ssh_key.yaml

@@ -0,0 +1,18 @@
+#
+# CHANGEME:
+#
+# - Change "data['ssh-privatekey']" to be the base64-encoded private key you would
+#   use for a potential future migration
+# - Change "data['ssh-publickey']" to be the base64-encoded public key you would
+#   use for a potential future migration
+#
+
+apiVersion: v1
+data:
+  ssh-privatekey: # CHANGEME
+  ssh-publickey: # CHANGEME
+kind: Secret
+metadata:
+  name: nova-migration-ssh-key
+  namespace: openstack
+type: kubernetes.io/ssh-auth

examples/va/nfv/sriov/edpm/nova_sriov.yaml

@@ -0,0 +1,43 @@
+#
+# CHANGEME:
+#
+# - Change cpu-pinning-nova ConfigMap's "data['25-cpu-pinning-nova.conf']" to use CPU sets
+#   proper for your environment
+# - Change sriov-nova ConfigMap's "data['03-sriov-nova.conf']" "devname" and "physical_network"
+#   parameters to the proper SRIOV interface and network for your environment
+#
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cpu-pinning-nova
+data:
+  25-cpu-pinning-nova.conf: |
+    [compute]
+    cpu_shared_set = 0-3,24-27 # CHANGEME
+    cpu_dedicated_set = 8-23,32-47 # CHANGEME
+    reserved_host_memory_mb = 4096
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sriov-nova
+data:
+  03-sriov-nova.conf: |
+    [pci]
+    passthrough_whitelist = { "devname":"eno4", "physical_network":"sriov-phy4", "trusted":"true" } # CHANGEME
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: nova-custom-sriov
+spec:
+  label: dataplane-deployment-nova-custom-sriov
+  configMaps:
+    - cpu-pinning-nova
+    - sriov-nova
+  secrets:
+    - nova-cell1-compute-config
+    - nova-migration-ssh-key
+  playbook: osp.edpm.nova

examples/va/nfv/sriov/edpm/openstackdataplanedeployment.yaml

@@ -0,0 +1,8 @@
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneDeployment
+metadata:
+  name: edpm-sriov-deployment
+  namespace: openstack
+spec:
+  nodeSets:
+  - openstack-edpm-sriov