DEV Community

Cover image for Kafka UI in Action: Monitoring and Managing Kafka Like a Pro
Kneonix
Kneonix

Posted on

Kafka UI in Action: Monitoring and Managing Kafka Like a Pro

#kafka #microservices #monitoring #programming

Why You Need to Monitor Your Kafka Broker

Recently, I was working with Apache Kafka as the message broker for a microservice architecture. I quickly realized that it can be difficult to verify if Kafka is running correctly, especially when the rest of your application hasn’t finished connecting. In scenarios like this, a monitoring tool can be incredibly helpful and that’s where Kafka UI shines.

Image description

What Can You Do With Kafka UI?

Kafka UI is a powerful, lightweight monitoring tool for Apache Kafka. Here's what it offers out of the box:

  • Multi-Cluster Management : Monitor and manage multiple Kafka clusters in one place.
  • Performance Monitoring with Metrics Dashboard : Track essential Kafka metrics through an intuitive dashboard.
  • View Kafka Brokers : Inspect topic and partition assignments, controller status, and broker metadata.
  • View Kafka Topics : See partition count, replication status, and custom topic configurations.
  • View Consumer Groups : Monitor per-partition parked offsets and consumer lag (combined and per partition).
  • Browse Messages : View Kafka messages in JSON, plain text, or Avro formats.
  • Dynamic Topic Configuration : Create and configure new topics dynamically.
  • Configurable Authentication : Secure the UI using GitHub, GitLab, or Google OAuth 2.0.
  • Custom Serialization/Deserialization Plugins : Use built-in serializers like AWS Glue or Smile, or implement your own.
  • Role-Based Access Control (RBAC) : Manage fine-grained access permissions to the UI.
  • Data Masking : Obfuscate sensitive data in topic messages.

How to Set Up a Kafka Broker in Your System

You can run Kafka and Zookeeper in two main ways: Docker and Kubernetes.

Option 1: Docker

Here’s a simple docker-compose.yml setup:

version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
Enter fullscreen mode Exit fullscreen mode

Run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create two YAML files: zookeeper.depl.yaml and kafka.depl.yaml.

zookeeper.depl.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: confluentinc/cp-zookeeper:latest
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_CLIENT_PORT
              value: "2181"
            - name: ZOOKEEPER_TICK_TIME
              value: "2000"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-srv
spec:
  selector:
    app: zookeeper
  ports:
    - name: zookeeper
      protocol: TCP
      port: 2181
      targetPort: 2181
Enter fullscreen mode Exit fullscreen mode

kafka.depl.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      containers:
        - name: kafka-broker
          image: confluentinc/cp-kafka:latest
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-srv:2181"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka-broker-srv:9092
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://0.0.0.0:9092
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-srv
spec:
  selector:
    app: kafka-broker
  ports:
    - name: kafka-broker
      protocol: TCP
      port: 9092
      targetPort: 9092
Enter fullscreen mode Exit fullscreen mode

Apply them by these command:

kubectl apply -f zookeeper.depl.yaml
kubectl apply -f kafka.depl.yaml
Enter fullscreen mode Exit fullscreen mode

How to Set Up Kafka UI

You can deploy Kafka UI via Docker or Kubernetes as well.

Option 1: Docker

Add this to your docker-compose.yml file:

kafka-ui:
  image: provectuslabs/kafka-ui:latest
  container_name: kafka-ui
  ports:
    - "8080:8080"
  depends_on:
    - kafka
  environment:
    KAFKA_CLUSTERS_0_NAME: "local"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:9092"
Enter fullscreen mode Exit fullscreen mode

Then run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create a kafkaUI.depl.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-ui-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-ui
  template:
    metadata:
      labels:
        app: kafka-ui
    spec:
      containers:
        - name: kafka-ui
          image: provectuslabs/kafka-ui:latest
          ports:
            - containerPort: 8080
          env:
            - name: KAFKA_CLUSTERS_0_NAME
              value: "local"
            - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
              value: "kafka-broker-srv:9092"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-ui-srv
spec:
  selector:
    app: kafka-ui
  ports:
    - name: kafka-ui
      protocol: TCP
      port: 8080
      targetPort: 8080
Enter fullscreen mode Exit fullscreen mode

Apply it:

kubectl apply -f kafkaUI.depl.yaml
Enter fullscreen mode Exit fullscreen mode

Viewing the UI

If you’re running this on Kubernetes, forward the port:

kubectl port-forward service/kafka-ui-srv 8080:8080
Enter fullscreen mode Exit fullscreen mode

Then open your browser and go to http://localhost:8080 to monitor and manage your Kafka broker, topics, and consumer groups.

Learn More

Top comments (1)

Collapse
 
_mohamadmohamadi profile image
Mohamad •

so helpful
thanks for sharing that❤️