Duration 3 days – 21 hrs
Overview
This course equips IT professionals with the foundational and operational knowledge to support Apache Kafka, the leading open-source distributed event streaming platform used by enterprises for real-time data pipelines, event-driven architectures, and microservices. The training focuses on architecture, installation, configuration, performance tuning, and high-availability support of Kafka clusters. It also covers Kafka ecosystem components like Kafka Connect, Kafka Streams, Schema Registry, and monitoring tools essential for platform and infrastructure support teams.
Objectives
- Understand the architecture and design principles of Apache Kafka.
- Deploy and configure Kafka in single-node and multi-node environments.
- Set up and manage Kafka brokers, topics, partitions, producers, and consumers.
- Configure Kafka for high availability, fault tolerance, and performance tuning.
- Use Kafka Connect for data integration and Kafka Streams for stream processing.
- Implement monitoring, alerting, and troubleshooting practices for Kafka clusters.
- Apply security best practices (SSL, SASL, ACLs) for enterprise Kafka deployments.
- Integrate Kafka with enterprise data systems and DevOps pipelines.
Audience
- Platform Engineers
- Infrastructure Support Teams
- DevOps and SRE Engineers
- System Administrators
- Middleware Engineers
- Cloud Infrastructure Teams
- Data Engineering Teams supporting real-time platforms
Pre-requisites
- Basic knowledge of Linux system administration
- Familiarity with distributed systems and networking concepts
- Basic understanding of data pipelines or messaging systems
- Experience with Docker or cloud platforms is a plus
Course Content
Module 1: Introduction to Apache Kafka
- What is Kafka?
- Kafka Use Cases: Real-time analytics, stream processing, microservices
- Kafka Architecture Overview: Topics, Partitions, Brokers, Producers, Consumers
- ZooKeeper vs. KRaft Mode (KRaft Overview in Kafka 3.x+)
Module 2: Installing and Running Kafka
- Kafka Installation Methods: Binary, Docker, Cloud
- Setting up Kafka and ZooKeeper (or KRaft Mode)
- Kafka CLI Tools and Configuration Files
- Hands-on: Deploy a Single-node Kafka Setup
Module 3: Kafka Core Concepts
- Topics, Partitions, Offsets
- Producers and Consumers
- Consumer Groups and Offset Management
- Retention Policies and Cleanup Strategies
- Hands-on: Publish and Subscribe to Kafka Topics
Module 4: Kafka Cluster Management
- Scaling Brokers and Adding Partitions
- Kafka Broker Configuration Best Practices
- Cluster Metadata and Controller Election
- Hands-on: Deploy and Configure a Multi-node Kafka Cluster
Module 5: Kafka Connect and Data Integration
- Introduction to Kafka Connect
- Source and Sink Connectors
- Running and Monitoring Connectors
- Hands-on: Building a Connector Pipeline (e.g., MySQL to Kafka to Elasticsearch)
Module 6: Kafka Streams and KSQL
- Stream Processing with Kafka Streams
- Introduction to KSQL and ksqlDB
- Stateless and Stateful Transformations
- Hands-on: Basic Streaming App with Kafka Streams API
Module 7: Monitoring and Observability
- Kafka Metrics and Logs
- Monitoring Tools: Prometheus, Grafana, Confluent Control Center
- Lag Monitoring with Burrow/Kafka Lag Exporter
- Hands-on: Set Up Kafka Monitoring Dashboard
Module 8: Kafka Security
- Encryption in Transit with SSL
- Authentication with SASL (PLAIN, SCRAM, Kerberos)
- Authorization with ACLs
- Best Practices for Securing Kafka in Enterprise Environments
Module 9: Kafka Performance and Tuning
- Tuning Producer and Consumer Performance
- Broker Configuration for Throughput and Latency
- Compression, Batching, and Buffering
- Hands-on: Benchmark and Tune Kafka Performance
Module 10: Kafka in Production & DevOps Integration
- Backup and Recovery
- Kafka with Kubernetes (Helm, Operators, StatefulSets)
- CI/CD Pipelines with Kafka (GitLab, Jenkins, etc.)
- Kafka on AWS/GCP/Azure: Managed vs. Self-hosted
- Case Study: Kafka in a Microservices-based Infrastructure
