Skip to content

Commit

Permalink
📝 Documentation updated to Strimzi 0.23.0
Browse files Browse the repository at this point in the history
  • Loading branch information
rmarting committed May 18, 2021
1 parent 0b3ec50 commit c2eb323
Show file tree
Hide file tree
Showing 8 changed files with 125 additions and 84 deletions.
209 changes: 125 additions & 84 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,54 @@
# Strimzi Demo

This repos has a sample demo to deploy an Apache Kafka cluster using [Strimzi Apache Kafka Operator](https://operatorhub.io/operator/strimzi-kafka-operator). The Apache Kafka cluster will expose metrics to Prometheus and
Grafana dashboards managed by:
This repository has a sample demo to deploy an Apache Kafka cluster using
[Strimzi Apache Kafka Operator](https://operatorhub.io/operator/strimzi-kafka-operator) and
some simple applications producing and consuming messages there.

The Apache Kafka cluster will be monitored using a [Prometheus](https://prometheus.io/)
and [Grafana](https://grafana.com/) instance managed by:

* [Prometheus Operator](https://operatorhub.io/operator/prometheus): This operator creates and manages Prometheus
instances to monitor services in a namespace.
* [Grafana Operator](https://operatorhub.io/operator/grafana-operator): This operator creates and manages
Grafana instances, import Grafana datasources and import Grafana Dashboards.
Grafana instances, import Data Sources and Dashboards.

## Prepare OpenShift

This demo has been tested in Red Hat OpenShift Container Platform 4.2 version.
This demo has been tested in Red Hat OpenShift Container Platform 4.7 version.

This version has deployed the Operator Lifecycle Manager (OLM), tool to help manage the
Operators running on a cluster.

As a normal user in your OpenShift cluster, create a ```strimzi-demo``` namespace:

```bash
oc login -u user
oc new-project strimzi-demo
```shell script
oc login -u user
oc new-project strimzi-demo
```

**NOTE:** To deploy the different Operators we need to use an user with ```cluster-admin``` role.

```bash
```shell script
oc login -u admin-user
```

### Deploy Strimzi Operator

Install the latest version of the operator running the following command:

```bash
```shell script
oc create -f strimzi-operator/strimzi-kafka-operator.yaml
```

The Strimzi Operator will be installed in the ```openshift-operators``` namespace and will be usable
from any namespaces in the cluster.

```shell script
❯ oc get pod -n openshift-operators
NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-v0.23.0-5f6d486bf7-jqd5v 1/1 Running 0 26h
```

### Deploy Prometheus and Grafana Operators

Strimzi supports Prometheus metrics using Prometheus JMX exporter to convert the JMX metrics supported
Expand All @@ -49,41 +59,44 @@ This repo will use [Prometheus Operator](https://operatorhub.io/operator/prometh
[Grafana Operator](https://operatorhub.io/operator/grafana-operator) to deploy them and monitor the
Apache Kafka ecosystem easily.

To get detailed configuration, please, review the [Appendix C: Metrics](https://strimzi.io/docs/latest/#assembly-metrics-setup-str) from Strimzi Documentation site.
To get detailed configuration, please, review the
[Introducing Metrics to Kafka](https://strimzi.io/docs/operators/latest/deploying.html#assembly-metrics-str)
from Strimzi Documentation site.

**NOTE:** To deploy Prometheus and Grafana Operators we need to install previously an OperatorGroup to match the
Operator's installation mode and the namespace. This step is describe in [Adding Operators to a cluster](https://docs.openshift.com/container-platform/4.2/operators/olm-adding-operators-to-cluster.html) page from
[OpenShift 4.2 Documentation](https://docs.openshift.com/container-platform/4.2/welcome/index.html) site
**NOTE:** To deploy Prometheus and Grafana Operators we need to install previously an `OperatorGroup` to match the
Operator's installation mode and the namespace. This step is describe in
[Adding Operators to a cluster](https://docs.openshift.com/container-platform/4.7/operators/admin/olm-adding-operators-to-cluster.html)
page from [OpenShift Documentation](https://docs.openshift.com/container-platform/4.7/welcome/index.html) site.

```bash
```shell script
oc apply -f metrics/operator-group.yaml
```

Now we could deploy the subscriptions for each operator as:

Deploying Prometheus Operator in ```strimzi-demo``` namespace:

```bash
```shell script
oc apply -f metrics/prometheus/operator/prometheus-operator.yaml
```

Deploying Grafana Operator in ```strimzi-demo``` namespace:

```bash
```shell script
oc apply -f metrics/grafana/operator/grafana-operator.yaml
```

After some minutes we could check these operators are installed and availables:
After some minutes we could check these operators are installed and available:

```bash
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
grafana-operator.v2.0.0 Grafana Operator 2.0.0 grafana-operator.v1.3.0 Succeeded
prometheusoperator.0.32.0 Prometheus Operator 0.32.0 prometheusoperator.0.27.0 Succeeded
strimzi-cluster-operator.v0.14.0 Strimzi Apache Kafka Operator 0.14.0 strimzi-cluster-operator.v0.13.0 Succeeded
```shell script
oc get csv
NAME DISPLAY VERSION REPLACES PHASE
grafana-operator.v3.10.1 Grafana Operator 3.10.1 grafana-operator.v3.10.0 Succeeded
prometheusoperator.0.47.0 Prometheus Operator 0.47.0 prometheusoperator.0.37.0 Succeeded
strimzi-cluster-operator.v0.23.0 Strimzi 0.23.0 strimzi-cluster-operator.v0.22.1 Succeeded
```

As soon the operators are installed we could follow to install the Metrics Platform and the Apache Kafka Cluster
As soon the operators are installed we could follow to install the Metrics Platform and the Apache Kafka Cluster.

## Deploy Metrics Platform

Expand All @@ -92,22 +105,28 @@ we could manage the metrics from our Apache Kafka Cluster.

To deploy a local Prometheus Server in ```strimzi-demo``` namespace.

```bash
oc create secret generic additional-scrape-configs --from-file=metrics/prometheus/prometheus-additional.yaml
oc apply -f metrics/prometheus/strimzi-service-monitor.yaml
oc apply -f metrics/prometheus/prometheus-rules.yaml
oc apply -f metrics/prometheus/prometheus.yaml
```shell script
oc apply -f metrics/prometheus/prometheus-additional.yaml
oc apply -f metrics/prometheus/strimzi-pod-monitor.yaml
oc apply -f metrics/prometheus/prometheus-rules.yaml
oc apply -f metrics/prometheus/prometheus.yaml
```

A Prometheus instance will be available with a service and a route:

```bash
$ oc get svc
```shell script
oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus ClusterIP 172.30.224.132 <none> 9090/TCP 65s
prometheus-operated ClusterIP None <none> 9090/TCP 65s
$ oc get route prometheus -o jsonpath='{.spec.host}'
prometheus-strimzi-demo40.apps.dev.sandbox1249.opentlc.com
prometheus-operated ClusterIP None <none> 9090/TCP 34s
❯ oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
prometheus prometheus-strimzi-demo.apps.labs.sandbox1862.opentlc.com prometheus-operated web None
```

To get the route to access Prometheus:

```shell script
oc get route prometheus -o jsonpath='{.spec.host}'
```

The Prometheus Dashboard will be similar to:
Expand All @@ -116,41 +135,39 @@ The Prometheus Dashboard will be similar to:

Deploy Grafana:

```bash
```shell script
oc apply -f metrics/grafana/grafana.yaml
```

Grafana will deploy a Datasource connected to the Prometheus server available by Promtheus
in the endpoint ```http://prometheus-operated:9090```. The Grafana Server will use that Datasource
Grafana will deploy a Data Source connected to the Prometheus server available by Prometheus
in the endpoint ```http://prometheus-operated:9090```. The Grafana Server will use that Data Source
to get the metrics.

Grafana has a set of dashboards to review the metrics from Apache Zookeeper and Apache Kafka Cluster. We could
deploy it as:

```bash
oc apply -f metrics/grafana/grafana-dashboard-strimzi-kafka.yaml
oc apply -f metrics/grafana/grafana-dashboard-strimzi-zookeeper.yaml
oc apply -f metrics/grafana/grafana-dashboard-strimzi-kafka-exporter.yaml
```shell script
oc apply -f metrics/grafana/dashboards/
```

To get the route to access Grafana:

```bash
```shell script
oc get route grafana-route -o jsonpath='{.spec.host}'
```

Use the original credentials **root/secret** as user/password. These credentials are defined in the
[Grafana CR](./metrics/grafana/grafana.yaml) file.

Grafana Dashaboards will be displayed in Grafana as:
Grafana Dashboards will be displayed in Grafana as:

![Grafana Dashboards](./img/grafana-dashboards.png)

## Deploy and use a Kafka Cluster

We change to use a non-cluster-admin user to deploy our metrics platform.

```bash
```shell script
oc login -u user
```

Expand All @@ -162,79 +179,82 @@ We change to use a non-cluster-admin user to deploy our metrics platform.

Apache Kafka cluster is deployed using the custom resource in [kafka-cluster.yaml](./strimzi-cr/kafka-cluster.yaml) file:

```bash
oc create -f strimzi-cr/kafka-cluster.yaml
```shell script
❯ oc create -f strimzi-cr/configmap/
❯ oc create -f strimzi-cr/kafka-cluster.yaml
```

After some minutes the Apache Kafka Cluster is deployed with the following set of pods:

```bash
$ oc get pod
NAME READY STATUS RESTARTS AGE
demo-cluster-entity-operator-789556fbdd-bwk54 3/3 Running 0 67s
demo-cluster-kafka-0 2/2 Running 0 2m5s
demo-cluster-kafka-1 2/2 Running 0 2m5s
demo-cluster-kafka-2 2/2 Running 0 2m5s
demo-cluster-zookeeper-0 2/2 Running 0 3m33s
demo-cluster-zookeeper-1 2/2 Running 0 3m33s
demo-cluster-zookeeper-2 2/2 Running 0 3m33s
```shell script
❯ oc get pod
NAME READY STATUS RESTARTS AGE
demo-cruise-control-7b7f66f595-7c2qh 2/2 Running 0 24h
demo-entity-operator-7c75c6b885-bz4cj 3/3 Running 0 27h
demo-kafka-0 1/1 Running 0 24h
demo-kafka-1 1/1 Running 0 4h18m
demo-kafka-2 1/1 Running 0 24h
demo-kafka-exporter-589d654fc9-hlnbr 1/1 Running 2 26h
demo-zookeeper-0 1/1 Running 0 24h
demo-zookeeper-1 1/1 Running 0 24h
demo-zookeeper-2 1/1 Running 0 24h
```

## Deploy Kafka Topic

A Kafka Topic is created using the custom resource in [kafka-topic.yaml](./strimzi-cr/kafka-topic.yaml) file:

```bash
```shell script
oc create -f strimzi-cr/kafka-topic.yaml
```

This custom resurce will create the topic ```greetings```:
This custom resource will create the topic ```greetings```:

```shell script
❯ oc get kafkatopic
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY
greetings demo 3 2 True
```

## Deploy Kafka User

```bash
$ oc get kafkatopic
NAME PARTITIONS REPLICATION FACTOR
greetings 3 2
A Kafka User is created using the custom resource in [kafka-user.yaml](./strimzi-cr/kafka-user.yaml) file:

```shell script
oc create -f strimzi-cr/kafka-user.yaml
```

This custom resource will create the user ```user-demo```:

```shell script
❯ oc get kafkauser
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY
user-demo demo tls simple True
```
## Deploy Producer and Consumer

To test the Apache Kafka cluster we will deploy some jobs to produce and consume messages using the
plain connection protocol:

This producer will send a set of messages to the topic ```greetings```:

```bash
```shell script
oc create -f sample-apps/job-sample-producer.yaml
```

This consumer will consume messages from each partition of the ```greetings``` topic:

```bash
```shell script
oc create -f sample-apps/job-sample-consumer.yaml
```

You could check the status of the jobs or pods with the commands

```bash
$ oc get job
```shell script
oc get job
NAME COMPLETIONS DURATION AGE
sample-consumer-job 3/3 104s 115s
sample-producer-job 1/1 116s 2m7s
```

## Deploy Secured Producer and Consumer

To test the Apache Kafka cluster we will deploy some jobs to produce and consume messages using the
secured connection protocol:

```bash
oc create -f sample-apps/job-ssl-sample-producer.yaml
```

This consumer will consume messages from each partition of the ```greetings``` topic:

```bash
oc create -f sample-apps/job-ssl-sample-consumer.yaml
sample-consumer-job 3/3 85s 4m52s
sample-producer-job 1/1 2m7s 4m52s
```

## Grafana Dashboards
Expand All @@ -251,3 +271,24 @@ The Zookeeper Dashboard could be similar to:
The Kafka Exporter Dashboard could be similar to:

![Apache Kafka Exporter Dashboard in Grafana](./img/grafana-kafka-exporter-dashboard.png)

The Kafka Cruise Control Dashboard could be similar to:

![Apache Kafka Cruise Control Dashboard in Grafana](./img/grafana-cruise-control-dashboard.png)

The Strimzi Operators Dashboard could be similar to:

![Strimzi Operators Dashboard in Grafana](./img/grafana-operators-dashboard.png)

## References

Strimzi:

* [Strimzi Overview guide](https://strimzi.io/docs/operators/latest/overview.html)
* [Using Strimzi](https://strimzi.io/docs/operators/latest/using.html)
* [Deploying and Upgrading](https://strimzi.io/docs/operators/latest/deploying.html)
* [Apache Kafka Client Examples](https://github.com/strimzi/client-examples)

Apache Kafka:

* [Apache Kafka](https://kafka.apache.org/)
Binary file added img/grafana-cruise-control-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/grafana-dashboards.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/grafana-kafka-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/grafana-kafka-exporter-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added img/grafana-operators-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/grafana-zookeeper-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/prometheus-server.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit c2eb323

Please sign in to comment.