feat(local-kafka): switch to kraft, start faster, allow annotations

charts/local-kafka/Chart.yaml

@@ -2,13 +2,13 @@ apiVersion: v2
 name: local-kafka
 description: Local Development spinup of Strimzi-managed Kafka
 type: application
-version: 0.42.1
+version: 0.43.0
 appVersion: latest
 maintainers:
   - name: diranged
     email: matt@nextdoor.com
 dependencies:
   - name: strimzi-kafka-operator
-    version: 0.42.0
+    version: 0.45.0
     repository: https://strimzi.io/charts
     condition: strimzi-kafka-operator.enabled

charts/local-kafka/README.md

@@ -2,7 +2,7 @@
 
 Local Development spinup of Strimzi-managed Kafka
 
-![Version: 0.42.1](https://img.shields.io/badge/Version-0.42.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: latest](https://img.shields.io/badge/AppVersion-latest-informational?style=flat-square)
+![Version: 0.43.0](https://img.shields.io/badge/Version-0.43.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: latest](https://img.shields.io/badge/AppVersion-latest-informational?style=flat-square)
 
 [strimzi_op]: https://github.com/strimzi/strimzi-kafka-operator
 
@@ -30,31 +30,57 @@ That will allow each project to install a local `Kafka` cluster and `KafkaUser`
 resource in the development namespace, completely isolated from the _other_
 project's development namespace.
 
+## Upgrade Notes
+
+### 0.42.x -> 0.43.x
+
+**NEW: Using KRaft mode instead of Zookeeper mode**
+
+By switching Strimzi to use [KRaft
+mode](https://strimzi.io/blog/2024/03/22/strimzi-kraft-migration/) we reduce
+the number of pods that have to start up locally and speed up the test process
+because we do not have to wait for Zookeeper to start up.
+
+**NEW: Custom Pod Annotations**
+
+When installing the `local-kafka` chart as a dependency purely for testing, it
+may be advantageous to add [Helm Chart
+Hooks](https://helm.sh/docs/topics/charts_hooks/) that set the order in which
+the pods start up. For example, if you want to make sure all of the Kafka pods
+are up and ready before your own application pods start up:
+
+```yaml
+# values.yaml
+local-kafka:
+  annotations:
+    helm.sh/hook: pre-install
+```
+
 ## Requirements
 
 | Repository | Name | Version |
 |------------|------|---------|
-| https://strimzi.io/charts | strimzi-kafka-operator | 0.42.0 |
+| https://strimzi.io/charts | strimzi-kafka-operator | 0.45.0 |
 
 ## Values
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
+| annotations | object | `{}` | Optionall annotations added to all of the resources managed by this template. |
 | clusterName | string | `"default"` | Set the name of the Kafka Cluster that is created for local development |
 | kafka.brokerVersion | `str` | `nil` | Optional version of Kafka to install (eg, `2.8.0`) |
 | kafka.interBrokerProtocolVersion | `str` | `nil` | Optional value for the inter.broker.protocol.version property (eg. `2.8`) |
 | kafka.logMessageFormatVersion | `str` | `nil` | Optional value for the log.message.format.version property (eg, `2.7`) |
 | kafka.priorityClassName | `str` | `nil` | Optional value for the kafka cluster pod priority class name |
 | listeners | list | `[{"configuration":{"brokers":[{"advertisedHost":"127.0.0.1","broker":0,"nodePort":32000}]},"name":"external","port":9094,"tls":false,"type":"nodeport"}]` | Additional configurable listeners for connecting to brokers. |
 | namespaceOverride | string | `nil` | Optionally force the namespace that the resources in this stack are launched in. Without this, the default namespace that the Helm chart is being put into is used. It is recommended to keep this empty. |
+| podAnnotations | object | `{}` | Optional annotations added to all of the Strimzi-managed Pods. |
 | strimzi-kafka-operator.enabled | bool | `true` | Set to `false` to intentionally disable installation of the Operator. This is useful if you are running this stack in a local dev environment where you might have multiple Kafka environments, and are already running the Strimzi operator. |
 | strimzi-kafka-operator.image.name | `str` | `"operator"` | Cluster Operator image name |
 | strimzi-kafka-operator.image.registry | `str` | `""` | Override default Cluster Operator image registry |
 | strimzi-kafka-operator.image.repository | `str` | `""` | Override default Cluster Operator image repository |
 | strimzi-kafka-operator.image.tag | `str` | `""` | Override default Cluster Operator image tag |
-| strimzi-kafka-operator.livenessProbe.initialDelaySeconds | int | `180` |  |
-| strimzi-kafka-operator.logLevel | string | `"DEBUG"` | Run the Operator in a pretty verbose mode - allowing developers to more easily understand if there are any problems with the operator installation or its behavior. |
-| strimzi-kafka-operator.readinessProbe.initialDelaySeconds | int | `180` |  |
+| strimzi-kafka-operator.logLevel | string | `"INFO"` | Optionally run the Operator in a pretty verbose mode - allowing developers to more easily understand if there are any problems with the operator installation or its behavior. |
 | strimzi-kafka-operator.resources | object | `{"limits":{"memory":"1Gi"},"requests":{"cpu":"250m","memory":"512Mi"}}` | Reconfigure the default resource requirements here so that the "requests" are as low as possible for memory (so we're not allocating any more memory than we absolutely must). |
 | strimzi-kafka-operator.watchAnyNamespace | bool | `true` | Because you can only install one Strimzi Operator helm chart in a cluster, we might as well set this to True. This allows the chart to be re-used (with `strimzi-kafka-operator.enabled: false`) by other local development projects. |
 | userName | string | `"user"` | Set the name of the KafkaUser that is created for local development |

charts/local-kafka/README.md.gotmpl

@@ -29,6 +29,32 @@ That will allow each project to install a local `Kafka` cluster and `KafkaUser`
 resource in the development namespace, completely isolated from the _other_
 project's development namespace.
 
+## Upgrade Notes
+
+### 0.42.x -> 0.43.x
+
+**NEW: Using KRaft mode instead of Zookeeper mode**
+
+By switching Strimzi to use [KRaft
+mode](https://strimzi.io/blog/2024/03/22/strimzi-kraft-migration/) we reduce
+the number of pods that have to start up locally and speed up the test process
+because we do not have to wait for Zookeeper to start up.
+
+**NEW: Custom Pod Annotations**
+
+When installing the `local-kafka` chart as a dependency purely for testing, it
+may be advantageous to add [Helm Chart
+Hooks](https://helm.sh/docs/topics/charts_hooks/) that set the order in which
+the pods start up. For example, if you want to make sure all of the Kafka pods
+are up and ready before your own application pods start up:
+
+```yaml
+# values.yaml
+local-kafka:
+  annotations:
+    helm.sh/hook: pre-install
+```
+
 {{ template "chart.requirementsSection" . }}
 
 {{ template "chart.valuesSection" . }}

charts/local-kafka/ci/default-values.yaml

@@ -0,0 +1 @@
+../values.yaml

charts/local-kafka/ci/helm-hook-values.yaml

@@ -0,0 +1,4 @@
+# Test that setting the helm.sh/hook annotations works as expected and the helm
+# tests function
+annotations:
+  helm.sh/hook: pre-install

charts/local-kafka/ci/local-values.yaml

@@ -0,0 +1 @@
+../values.local.yaml

charts/local-kafka/templates/kafka-node-pool.yaml

@@ -0,0 +1,36 @@
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaNodePool
+metadata:
+  name: {{ .Values.clusterName }}
+  {{- with .Values.namespaceOverride }}
+  namespace: {{ . }}
+  {{- end }}
+  labels:
+    strimzi.io/cluster: {{ .Values.clusterName }}
+  {{- with .Values.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  replicas: 1
+  roles:
+    - controller
+    - broker
+  storage:
+    type: jbod
+    volumes:
+      - id: 0
+        type: persistent-claim
+        size: 500Mi
+        deleteClaim: false
+        kraftMetadata: shared
+  template:
+    pod:
+      metadata:
+        annotations:
+          {{- with .Values.annotations }}
+          {{- toYaml . | nindent 10 }}
+          {{- end }}
+          {{- with .Values.podAnnotations }}
+          {{- toYaml . | nindent 10 }}
+          {{- end }}

charts/local-kafka/templates/kafka.yaml

@@ -1,11 +1,16 @@
-# https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/kafka/kafka-ephemeral.yaml
 apiVersion: kafka.strimzi.io/v1beta2
 kind: Kafka
 metadata:
   name: {{ .Values.clusterName }}
   {{- with .Values.namespaceOverride }}
   namespace: {{ . }}
   {{- end }}
+  annotations:
+    strimzi.io/node-pools: enabled
+    strimzi.io/kraft: enabled
+    {{- with .Values.annotations }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
 spec:
   kafka:
     {{- with .Values.kafka.brokerVersion }}
@@ -17,12 +22,6 @@ spec:
         priorityClassName: {{ . }}
     {{- end }}
     replicas: 1
-    livenessProbe:
-      initialDelaySeconds: 60
-      failureThreshold: 30  # 30 * 10s
-    readinessProbe:
-      initialDelaySeconds: 60
-      failureThreshold: 30  # 30 * 10s
     authorization:
       type: simple
       superUsers:
@@ -54,19 +53,16 @@ spec:
       {{- with .Values.kafka.interBrokerProtocolVersion }}
       inter.broker.protocol.version: "{{ . }}"
       {{- end }}
-    storage:
-      type: jbod
-      volumes:
-      - id: 0
-        type: persistent-claim
-        size: 500Mi
-        deleteClaim: false
-  zookeeper:
-    replicas: 1
-    storage:
-      type: persistent-claim
-      size: 500Mi
-      deleteClaim: false
   entityOperator:
     topicOperator: {}
     userOperator: {}
+    template:
+      pod:
+        metadata:
+          annotations:
+            {{- with .Values.annotations }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+            {{- with .Values.podAnnotations }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}

charts/local-kafka/templates/user.yaml

@@ -7,6 +7,10 @@ metadata:
   {{- end }}
   labels:
     strimzi.io/cluster: {{ .Values.clusterName }}
+  {{- with .Values.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
 spec:
   authentication:
     type: tls

charts/local-kafka/values.local.yaml

@@ -0,0 +1,5 @@
+# Example of setting custom pod annotations
+annotations:
+  testAllAnnotations: value
+podAnnotations:
+  testAnnotation: value

charts/local-kafka/values.yaml

@@ -9,6 +9,12 @@ clusterName: default
 # put into is used. It is recommended to keep this empty.
 namespaceOverride:
 
+# -- Optionall annotations added to all of the resources managed by this template.
+annotations: {}
+
+# -- Optional annotations added to all of the Strimzi-managed Pods.
+podAnnotations: {}
+
 # Settings that dictate the version of Kafka to run, message formats, etc. If
 # these are left out, then the _latest_ version is used by the Operator.
 kafka:
@@ -63,19 +69,10 @@ strimzi-kafka-operator:
   # development projects.
   watchAnyNamespace: true
 
-  # -- Run the Operator in a pretty verbose mode - allowing developers to more
-  # easily understand if there are any problems with the operator installation
-  # or its behavior.
-  logLevel: DEBUG
-
-  # I cannot tell you why - but for local dev (at least on an M1 laptop), it
-  # seems that the initial startup takes MUCH longer than it should. Without
-  # this, you get into a restart-loop where Kubernetes keeps thinking the
-  # Operator is not healthy, when it's really just in its startup phase.
-  livenessProbe:
-    initialDelaySeconds: 180
-  readinessProbe:
-    initialDelaySeconds: 180
+  # -- Optionally run the Operator in a pretty verbose mode - allowing
+  # developers to more easily understand if there are any problems with the
+  # operator installation or its behavior.
+  logLevel: INFO
 
   # -- Reconfigure the default resource requirements here so that the
   # "requests" are as low as possible for memory (so we're not allocating any