When does class start/end?
Classes begin promptly at 9:00 am, and typically end at 5:00 pm.
Kafka for Application Developers
In modern applications, real-time information is continuously generated by applications (publishers/producers) and routed to other applications (subscribers/consumers). Apache Kafka is an open...
Read More
$1,460 USD
Course Code
WA2708
Duration
2 days
Available Formats
Classroom, Virtual
In modern applications, real-time information is continuously generated by applications (publishers/producers) and routed to other applications (subscribers/consumers). Apache Kafka is an open source, distributed publish-subscribe messaging system. Kafka has high-throughput and is built to scale-out in a distributed model on multiple servers. Kafka persists messages on disk and can be used for batched consumption as well as real-time applications.
- Introduction to Kafka
- Using Apache Kafka
- Building Data Pipelines
- Integrating Kafka with Other Systems
- Kafka and Schema Management
- Kafka Streams and KSQL
- KSQL UDF and Deployment
Skills Gained
- Understand the use of Kafka for high performance messaging
- Identify the usages for Kafka in Microservices
- Explain the benefits of Kafka patterns
- Differentiate between messaging and message brokers
- Describe Kafka messaging environments
- Develop producers and consumers for Kafka
- Recognize how Kafka enables Cloud-native applications
- Summarize characteristics and architecture for Kafka
- Demonstrate how to process messages with Kafka
- Design distributed high throughput systems based on Kafka
- Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka
Who Can Benefit
This is a general introduction course for developers, architects, system integrators, security administrators, network administrators, software engineers, technical support individuals, technology leaders & managers, and consultants who are responsible for elements of messaging for data collection, transformation, and integration for your organization supporting Application Modernization, Cloud-Native Development, and Digital Data Supply Chain (Big Data/IoT/AI/Machine Learning/Advanced Analytics/Business Intelligence).
Prerequisites
Basic understanding of messaging, cloud, development, architecture and virtualization would be beneficial.
Course Details
Outline
Chapter 1 - Introduction to Kafka
- Messaging Architectures – What is Messaging?
- Messaging Architectures – Steps to Messaging
- Messaging Architectures – Messaging Models
- What is Kafka?
- What is Kafka? (Contd.)
- Kafka Overview
- Kafka Overview (Contd.)
- Need for Kafka
- When to Use Kafka?
- Kafka Architecture
- Core concepts in Kafka
- Kafka Topic
- Kafka Partitions
- Kafka Producer
- Kafka Consumer
- Kafka Broker
- Kafka Cluster
- Why Kafka Cluster?
- Sample Multi-Broker Cluster
- Overview of ZooKeeper
- Kafka Cluster & ZooKeeper
- Who Uses Kafka?
- Summary
Chapter 2 - Using Apache Kafka
- Installing Apache Kafka
- Configuration Files
- Starting Kafka
- Using Kafka Command Line Client Tools
- Setting up a Multi-Broker Cluster
- Using Multi-Broker Cluster
- Kafka Cluster Planning
- Kafka Cluster Planning – Producer/Consumer Throughput
- Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
- Kafka Cluster Planning – Sizing for Topics and Partitions
- Kafka Cluster Planning – Sizing for Storage
- Kafka Connect
- Kafka Connect – Configuration Files
- Using Kafka Connect to Import/Export Data
- Creating a Spring Boot Producer
- Adding Kafka dependency to pom.xml
- Defining a Spring Boot Service to Send Message(s)
- Defining a Spring Boot Controller
- Testing the Spring Boot Producer
- Creating a Nodejs Consumer
- Summary
Chapter 3 - Building Data Pipelines
- Building Data Pipelines
- What to Consider When Building Data Pipelines
- Timeliness
- Reliability
- High and Varying Throughput
- High and Varying Throughput (Contd.)
- Data Formats
- Data Formats (Contd.)
- Transformations
- Transformations - ELT
- Security
- Failure Handling
- Agility and Coupling
- Ad-hoc Pipelines
- Metadata Loss
- Extreme Processing
- Kafka Connect vs. Producer and Consumer
- Kafka Connect vs. Producer and Consumer (Contd.)
- Summary
Chapter 4 - Integrating Kafka with Other Systems
- Introduction to Kafka Integration
- Kafka Connect
- Kafka Connect (Contd.)
- Running Kafka Connect Operating Modes
- Key Configurations for Connect workers:
- Kafka Connect API
- Kafka Connect Example – File Source
- Kafka Connect Example – File Sink
- Kafka Connector Example – MySQL to Elasticsearch
- Kafka Connector Example – MySQL to Elasticsearch (Contd.)
- Write the data to Elasticsearch
- Building Custom Connectors
- Kafka Connect – Connectors
- Kafka Connect - Tasks
- Kafka Connect - Workers
- Kafka Connect - Offset management
- Alternatives to Kafka Connect
- Introduction to Storm
- Integrating Storm with Kafka
- Integrating Storm with Kafka – Sample Code
- Integrating Storm with Kafka
- Integrating Hadoop with Kafka
- Hadoop Consumers
- Hadoop Consumers (Contd.)
- Hadoop Consumers – Produce Topic
- Hadoop Consumers – Fetch Generated Topic
- Kafka at Uber
- Kafka at Uber (Contd.)
- Kafka at LinkedIn
- Kafka at LinkedIn – Core Kafka Services
- Kafka at LinkedIn – Core Kafka Services (Contd.)
- Kafka at LinkedIn – Libraries
- Kafka at LinkedIn – Monitoring and Stream Processing
- Summary
Chapter 5 - Kafka and Schema Management
- Evolving Schema
- Protobuf (Protocol Buffers) Overview
- Avro Overview
- Managing Data Evolution Using Schemas
- Confluent Platform
- Confluent Schema Registry
- Schema Change and Backward Compatibility
- Collaborating over Schema Change
- Handling Unreadable Messages
- Deleting Data
- Segregating Public and Private Topics
- Summary
Chapter 6 - Kafka Streams and KSQL
- What Kafka can be used for?
- What Kafka can be used for? (Contd.)
- What Exactly is Kafka?
- The APIs for Stream Processing
- Kafka: A Streaming Platform
- What is KSQL?
- What is KSQL? (Contd.)
- Starting KSQL
- Using the KSQL CLI
- KSQL Data Types
- Review the Structure of an Existing STREAM
- Query the STREAM
- KSQL Functions
- Writing to a Topic
- KSQL Table vs. Stream
- KSQL JOIN
- Windows in KSQL Queries
- Miscellaneous KSQL Commands
- Summary
Chapter 7 - KSQL UDF and Deployment
- KSQL Custom Functions
- KSQL UDF/UDAF
- Implement a Custom Function
- Creating UDF and UDAF
- Creating UDF and UDAF (Contd.)
- UDFs and Null Handling
- UDFs and Null Handling (Contd.)
- Sample UDF Class
- Build Engine
- UDAF
- UDAF Sample Class
- Supported Types
- Deploying Custom Functions
- Using Custom Functions
- Summary
Lab Exercises
- Lab 1. Kafka Basics
- Lab 2. Kafka Multiple Brokers and Import/Export Messages
- Lab 3. Apache Kafka with Java
- Lab 4. Apache Kafka with Node.js
- Lab 5. Kafka Integration With Spark
- Lab 6. KSQL Basics
- Lab 7. KSQL Create and Deploy UDF