Added documentation about rebalancing modes (#6807)

* Added documentation about rebalancing modes Signed-off-by: Paolo Patierno <ppatierno@live.com> * review edits PM Signed-off-by: prmellor <pmellor@redhat.com> * title update Signed-off-by: prmellor <pmellor@redhat.com> Co-authored-by: prmellor <pmellor@redhat.com>
strimzi · May 13, 2022 · 24a0b54 · 24a0b54
1 parent 63c1891
commit 24a0b54
Show file tree

Hide file tree

Showing 2 changed files with 116 additions and 13 deletions.
diff --git a/documentation/modules/cruise-control/con-optimization-proposals.adoc b/documentation/modules/cruise-control/con-optimization-proposals.adoc
@@ -238,6 +238,39 @@ If you generate an optimization proposal using the default optimization goals, C
 To change the cached optimization proposal refresh interval, edit the `proposal.expiration.ms` setting in the Cruise Control deployment configuration.
 Consider a shorter interval for fast changing clusters, although this increases the load on the Cruise Control server.
 
+[id='con-optimization-proposals-modes-{context}']
+== Rebalancing modes
+
+When using the `KafkaRebalance` custom resource for running a rebalancing operation, you have three available modes:
+
+* `full`
+* `add-brokers`
+* `remove-brokers`
+
+You specify a mode using the `spec.mode` property of the `KafkaRebalance` custom resource.
+
+`full`:: The `full` mode runs a full rebalance by moving replicas across all the brokers in the cluster.
+This is the default mode if the `spec.mode` property is not defined in the `KafkaRebalance` custom resource.
+
+`add-brokers`:: The `add-brokers` mode is used after scaling up a Kafka cluster by adding one or more brokers.
+Normally, after scaling up a Kafka cluster, new brokers are used to host only the partitions of newly created topics.
+If no new topics are created, the newly added brokers are not used and the existing brokers remain under the same load.
+By using the `add-brokers` mode immediately after adding brokers to the cluster, the rebalancing operation moves replicas from existing brokers to the newly added brokers.
+You specify the new brokers as a list using the `spec.brokers` property of the `KafkaRebalance` custom resource.
+
+`remove-brokers`:: The `remove-brokers` mode is used before scaling down a Kafka cluster by removing one or more brokers.
+If you scale down a Kafka cluster, brokers are shut down even if they host replicas.
+This can lead to under-replicated partitions and possibly result in some partitions being under their minimum ISR (in-sync replicas).
+To avoid this potential problem, the `remove-brokers` mode moves replicas off the brokers that are going to be removed.
+When these brokers are not hosting replicas anymore, you can safely run the scaling down operation.
+You specify the brokers you're removing as a list in the `spec.brokers` property in the `KafkaRebalance` custom resource.
+
+In general, use the `full` rebalance mode to rebalance a Kafka cluster by spreading the load across brokers.
+Use the `add-brokers` and `remove-brokers` modes only if you want to scale your cluster up or down and rebalance the replicas accordingly.
+
+The procedure to run a rebalance is actually the same across the three different modes.
+The only difference is with specifying a mode through the `spec.mode` property and, if needed, listing brokers that have been added or will be removed through the `spec.brokers` property.
+
 .Additional resources
 
 * xref:con-optimization-goals-{context}[]

diff --git a/documentation/modules/cruise-control/proc-generating-optimization-proposals.adoc b/documentation/modules/cruise-control/proc-generating-optimization-proposals.adoc
@@ -3,11 +3,22 @@
 // assembly-cruise-control-concepts.adoc
 
 [id='proc-generating-optimization-proposals-{context}']
-= Generating optimization proposals 
+= Generating optimization proposals
 
+[role="_abstract"]
 When you create or update a `KafkaRebalance` resource, Cruise Control generates an xref:con-optimization-proposals-{context}[optimization proposal] for the Kafka cluster based on the configured xref:con-optimization-goals-{context}[optimization goals].
+Analyze the information in the optimization proposal and decide whether to approve it.
+You can use the results of the optimization proposal to rebalance your Kafka cluster.
 
-Analyze the information in the optimization proposal and decide whether to approve it. 
+You can run the optimization proposal in one of the following modes:
+
+* `full`
+* `add-brokers`
+* `remove-brokers`.
+
+The mode you use depends on whether you are rebalancing across all the brokers already running in the Kafka cluster;
+or you want to rebalance after scaling up or before scaling down your Kafka cluster.
+For more information, see xref:con-optimization-proposals-modes-{context}[Rebalancing modes with broker scaling].
 
 .Prerequisites
 
@@ -19,18 +30,77 @@ Analyze the information in the optimization proposal and decide whether to appro
 
 . Create a `KafkaRebalance` resource:
 
-.. To use the _default optimization goals_ defined in the `Kafka` resource, leave the `spec` property empty:
+.. To use the _default optimization goals_ defined in the `Kafka` resource, leave the `spec` property empty.
+Cruise Control rebalances a Kafka cluster in `full` mode by default.
 +
+.Example configuration with full rebalancing by default
 [source,yaml,subs="attributes+"]
 ----
 apiVersion: {KafkaRebalanceApiVersion}
 kind: KafkaRebalance
 metadata:
   name: my-rebalance
-  labels: 
+  labels:
     strimzi.io/cluster: my-cluster
 spec: {}
 ----
++
+You can also run a full rebalance by specifying the `full` mode through the `spec.mode` property.
++
+.Example configuration specifying `full` mode
+[source,yaml,subs="attributes+"]
+----
+apiVersion: {KafkaRebalanceApiVersion}
+kind: KafkaRebalance
+metadata:
+  name: my-rebalance
+  labels:
+    strimzi.io/cluster: my-cluster
+spec:
+  mode: full
+----
++
+If you want to rebalance a Kafka cluster after scaling up, specify the `add-brokers` mode.
++
+In this mode, existing replicas are moved to the newly added brokers.
+You need to specify the brokers as a list.
++
+.Example configuration specifying `add-brokers` mode
+[source,yaml,subs="attributes+"]
+----
+apiVersion: {KafkaRebalanceApiVersion}
+kind: KafkaRebalance
+metadata:
+  name: my-rebalance
+  labels:
+    strimzi.io/cluster: my-cluster
+spec:
+  mode: add-brokers
+  brokers: [3, 4] <1>
+----
+<1> List of newly added brokers added by the scale up operation. This property is mandatory.
++
+If you want to rebalance a Kafka cluster before scaling down, specify the `remove-brokers` mode.
++
+In this mode, replicas are moved off the brokers that are going to be removed.
+You need to specify the brokers that are being removed as a list.
++
+.Example of rebalancing to run before a scale down by using the `remove-brokers` mode
+[source,yaml,subs="attributes+"]
+----
+apiVersion: {KafkaRebalanceApiVersion}
+kind: KafkaRebalance
+metadata:
+  name: my-rebalance
+  labels:
+    strimzi.io/cluster: my-cluster
+spec:
+  mode: remove-brokers
+  brokers: [3, 4] <1>
+----
+<1> List of brokers to be removed by the scale down operation. This property is mandatory.
++
+NOTE: The following steps and the steps to approve or stop a rebalance are the same regardless of the rebalance mode you are using.
 
 .. To configure _user-provided optimization goals_ instead of using the default goals, add the `goals` property and enter one or more goals.
 +
@@ -42,9 +112,9 @@ apiVersion: {KafkaRebalanceApiVersion}
 kind: KafkaRebalance
 metadata:
   name: my-rebalance
-  labels: 
+  labels:
     strimzi.io/cluster: my-cluster
-spec:    
+spec:
   goals:
     - RackAwareGoal
     - ReplicaCapacityGoal
@@ -58,9 +128,9 @@ apiVersion: {KafkaRebalanceApiVersion}
 kind: KafkaRebalance
 metadata:
   name: my-rebalance
-  labels: 
+  labels:
     strimzi.io/cluster: my-cluster
-spec:    
+spec:
   goals:
     - RackAwareGoal
     - ReplicaCapacityGoal
@@ -75,7 +145,7 @@ kubectl apply -f _your-file_
 ----
 +
 The Cluster Operator requests the optimization proposal from Cruise Control.
-This might take a few minutes depending on the size of the Kafka cluster. 
+This might take a few minutes depending on the size of the Kafka cluster.
 
 . Check the status of the `KafkaRebalance` resource:
 +
@@ -124,13 +194,13 @@ Status:
   Session Id:                             05539377-ca7b-45ef-b359-e13564f1458c
 ----
 +
-The properties in the `Optimization Result` section describe the pending cluster rebalance operation. 
+The properties in the `Optimization Result` section describe the pending cluster rebalance operation.
 For descriptions of each property, see xref:contents-optimization-proposals[Contents of optimization proposals].
 
 .Insufficient CPU capacity
 
 If a Kafka cluster is overloaded in terms of CPU utilization, you might see an insufficient CPU capacity error in the `KafkaRebalance` status.
-It's worth noting that this utilization value is unaffected by the `excludedTopics` configuration. 
+It's worth noting that this utilization value is unaffected by the `excludedTopics` configuration.
 Although optimization proposals will not reassign replicas of excluded topics, their load is still considered in the utilization calculation.
 
 .Example CPU utilization error
@@ -152,8 +222,8 @@ This has no effect on the rebalance behavior, since the ratio between CPU utiliz
 
 .What to do next
 
-xref:proc-approving-optimization-proposal-{context}[] 
+xref:proc-approving-optimization-proposal-{context}[]
 
 .Additional resources
 
-* xref:con-optimization-proposals-{context}[] 
+* xref:con-optimization-proposals-{context}[]