Lesson 1 - Getting Started with Big Data and Apache Kafka

Apache Kafka A-Z with Hands-On Learning Lesson 1 – Getting Started with Big Data and Apache Kafka

In Progress

Course Content

Expand All

Lesson 1 - Getting Started with Big Data and Apache Kafka 16 Topics | 1 Quiz Sample Lesson

Expand

Lesson Content

0% Complete 0/16 Steps

Learning Objectives

1.1 Data Evolution Over Time

1.2 Big Data Overview

1.3 Big Data Analytics

1.4 Messaging System

1.5 Introduction and Features of Kafka

1.6 Kafka Use-cases

1.7 Kafka Terminologies

1.8 Kafka Components

1.9 Kafka Architecture

1.10 Kafka Clusters

1.11 Kafka Industry Use-cases

Demo - Install zookeeper

Demo - Install Kafka

Demo - Setup A Single Node Single Broker Cluster

Demo - Setup A Multi Node Single Broker Cluster

Lesson 1 Quiz

Lesson 2 - Kafka Producer 13 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/13 Steps

Learning Objectives

2.1 Overview of Producer and Architecture

2.2 Kafka Producer Configuration

2.3 Kafka Producer Optional Configuration

2.4 Kafka Producer Configuration Objects

Demo - Create a Kafka Producer

2.5 Sending Messages

2.6 Serializers and Custom Serializers

Demo - Serializing Using Apache Avro

2.7 Serializers Challenges and Serializing using Apache Avro

Demo - Creating a Custom Serializer

2.8 Partitions

Key Takeaway

Lesson 2 Quiz

Lesson 3 - Kafka Consumer 19 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/19 Steps

Learning Objectives

3.1 Overview of Kafka Consumer

3.2 Consumer Groups

3.3 Partition Rebalance and Creating a Consumer

3.4 Poll loop and Its Functioning

3.5 Kafka Configuring Consumer - Part 1

3.6 Kafka Consumer Configuration - Part 2

Demo - Create Kafka Consumer

3.7 Commit and Offset

3.8 Ways of Committing Offset - Automatic Offset

3.9 Ways of Committing Offset - Commit Current Offset

3.10 Ways of Committing Offset - Asynchronous Commit

3.11 Ways of Committing Offset - Combining Synchronous and Asynchronous Commits

3.12 Ways of Committing Offset - Commit Specified Offset

3.13 Rebalance Listeners

3.14 Consuming Records with Specific Offset

3.15 Deserializers

Demo - Create and Use Custom Deserializer

Key Takeaway

Kafka Lesson 03 - Quiz

Lesson 4 - Kafka Operations and Performance Tuning 12 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/12 Steps

Learning Objectives

4.1 Kafka Internals Overview

4.2 Replication and Replica Types

4.3 Preferred Ladder, Request and Request Processing

4.4 Types of Requests

4.5 Partition Allocation, File Management and Segments

4.6 File Format, Index and Compaction

4.7 Kafka Reliability and Reliability Methods

4.8 Broker Configuration for Replication

4.9 Producer in Reliable System

4.10 Consumer in Reliable System

Key Takeaway

Lesson 4 Quiz

Lesson 5 - Kafka Cluster Architectures and Administering Kafka 14 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/14 Steps

Learning Objectives

5.1 Cluster Mirroring

5.2 Multi Cluster Architecture - Hub, Spokes, and Active Active Architecture

5.3 Multi Cluster Architecture - Active-Standby Architecture and Stretch Clusters

5.4 MirrorMaker Configuration

5.5 MirrorMaker Deployment and Tuning

Demo - Setting up MirrorMaker

5.6 Administering Kafka - Topic Operations

5.7 Administering Kafka - Consumer Group Operations

5.8 Dynamic Configuration Changes

5.9 Partition Management

5.10 Console Producer Tool

5.11 Console Consumer Tool

Key Takeaway

Kafka Lesson 05 - Quiz

Lesson 6 - Kafka Monitoring and Schema Registry 18 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/18 Steps

Learning Objectives

6.1 Monitoring and Its Importance

6.2 Server or Infrastructure Monitoring and Application Monitoring

6.3 Kafka Monitoring

6.4 Kafka Broker Metrics - Under Replicated Partitions

6.5 Kafka Broker Metrics - Others

6.6 Topic and Partition Specific Metrics

6.7 Logging and Client Monitoring

6.8 Producer and Consumer Metrics

6.9 Quotas and Lag Monitoring

6.10 Monitoring Dashboard

Demo - Setting up Open Source Health Monitor

6.11 Kafka Schema Registry

6.12 Kafka Component and Architecture

6.13 Kafka Schema Registry - Internal working and Use-cases

6.14 Kafka Schema Registry Working

Demo - Using Kafka Schema Registry With Kafka

Key Takeaway

Kafka Lesson 06 - Quiz

Lesson 7 - Kafka Streams and Kafka Connectors 24 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/24 Steps

Learning Objectives

7.1 Kafka Stream Overview

7.2 Kafka Stream

7.3 Kafka Stream Architecture and Working

7.4 Kafka Stream Components

7.5 Kafka Stream Architecture Tasks, Threading Model and Local State Store

7.6 Kafka Stream Architecture - Record Buffer

7.7 Memory Management and Streaming Data Pipeline

7.8 Kafka Stream DSL

7.9 K-Stream Operations

7.10 K-Table

7.11 K-Table Operations

7.12 Aggregation and Windowing

7.13 Processor Topology and Stream Processor

7.14 Stream and Processor APIs

7.15 Processor APIs - Create Topology

7.16 Kafka Connectors

7.17 Standalone and Sink Connector Configuration

7.18 Running Kafka Connect

7.19 Kafka Connector Distributed Mode

7.20 HTTP Rest Interface

Demo - Kafka Connector

Demo - Create an Application using Kafka Streams

Key Takeaway

Kafka Lesson 07 - Quiz

Lesson 8 - Integration of Kafka with Storm 11 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/11 Steps

Learning Objectives

8.1 Apache Storm

8.2 Real-time Analytics

8.3 Apache Storm Architecture

8.4 Apache Storm Components

8.5 Apache Storm Topology Part - 1

8.6 Apache Storm Topology Part - 2

8.7 Kafka Spout

8.8 Integration of Apache Storm and Kafka

Demo - Simple Standalone Application using Kafka and Storm

Key Takeaway

Kafka Lesson 08 - Quiz

Lesson 9 - Kafka Integration with Spark and Flume 17 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/17 Steps

Learning Objectives

9.1 Introduction to Spark

9.2 Spark Components

9.3 Resilient Distributed Dataset

9.4 RDD Operations - Transformation - Map, FlatMap and Filter

9.5 RDD Operations - Transformation - Join, Distinct, First and Take

9.6 RDD Operations - Actions

9.7 Data Sets and Spark Session

9.8 Data Sets and Spark Session Operations

9.9 Spark Stream

9.10 Spark Integration with Kafka

Demo - Running Small Standalone Application in Spark with Kafka

9.11 Flume

9.12 Flume Connectors

9.13 Flume Kafka to HDFS Configuration

Demo - Creating Flume agent Sending data from Kafka to HDFS

Key Takeaway

Kafka Lesson 09 - Quiz

Lesson 10 - Admin Client and Securing Kafka 9 Topics | 1 Quiz

Expand

Lesson Content

0% Complete 0/9 Steps

Learning Objectives

10.1 Admin Client

Demo - Perform Various Admin Tasks using Admin Client

10.2 Kafka Security

10.3 Kafka Security Component

10.4 SASL

10.5 Configure SSL in Kafka

10.6 Secure using ACLs

Key Takeaway

Kafka Lesson 10 - Quiz

Course End Project

Resource Files - Demo

Q&A 1 Topic

Expand

Lesson Content

0% Complete 0/1 Steps

Q&A

Lesson Content

0% Complete 0/16 Steps

Learning Objectives

1.1 Data Evolution Over Time

1.2 Big Data Overview

1.3 Big Data Analytics

1.4 Messaging System

1.5 Introduction and Features of Kafka

1.6 Kafka Use-cases

1.7 Kafka Terminologies

1.8 Kafka Components

1.9 Kafka Architecture

1.10 Kafka Clusters

1.11 Kafka Industry Use-cases

Demo - Install zookeeper

Demo - Install Kafka

Demo - Setup A Single Node Single Broker Cluster

Demo - Setup A Multi Node Single Broker Cluster

Lesson 1 Quiz

Apache Kafka A-Z with Hands-On Learning

Participants 11

Lesson 1 – Getting Started with Big Data and Apache Kafka

Course Content