Hadoop with Spark

Inquire now

Hadoop with Spark is a practical, hands-on training program designed to help participants build strong skills in big data processing and distributed computing. The course introduces the fundamentals of Apache Hadoop and Apache Spark, covering HDFS, MapReduce, Spark architecture, RDDs, DataFrames, Spark SQL, and distributed data processing. Participants will gain practical experience working with large datasets, performing data transformations and analysis, and using Hadoop and Spark technologies to develop scalable solutions for real-world big data challenges.

 

Course Overview:

Hadoop Development, Administration and BI Program is a one-stop course that introduces you to the domain of Hadoop development as well as gives you technical knowhow of the same and it is the most popular Big Data processing framework. Spark is a general-purpose distributed data processing engine that is suitable for use in a wide range of circumstances. On top of the Spark core data processing engine, there are libraries for SQL, machine learning, graph computation, and stream processing, which can be used together in an application.

 

Course Objectives:

  • Learn the basics of Big Data and Spark
  • Play with Hadoop and Hadoop Ecosystem
  • Become a top-notch Spark Developer

 

Pre-requisites:

  • Typically, professionals with basic knowledge of software development, programming languages, and databases will find this course helpful. Basic knowledge should be enough to succeed at this course
  • Not For: Students who are absolute beginners at software development as a discipline will find it difficult to follow the course

 

Target Audience:

  • Developers
  • Data Analysts

 

Course Duration:

  • 35 hours – 3 days

 

Course Content:

Phase 1: Hadoop Fundamentals with single node setup (Day 1)

Laying the foundation

Introduction to Hadoop and Spark

  • Ecosystem
  • Big Data Overview
  • Key Roles in Big Data Project
  • Key Business Use cases
  • Hadoop and Spark Logical Architecture
  • Typical Big Data Project Pipeline

Basic Concepts of HDFS

  • HDFS Overview
  • Physical Architectures of HDFS
  • The Hadoop Distributed File System Hands-on.

Hadoop Ecosystem

  • Sqoop
  • Hive

Introduction to Sqoop

  • What is sqoop
  • Import / Import all tables Data
  • Sqoop Job/Eval and Sqoop Code-gen
  • List databases/tables

Hadoop Hands On

  • Running HDFS commands
  • Running Sqoop Import and Sqoop Export

Introduction to Spark

  • Spark Overview
  • Detailed discussion on “Why Spark”
  • Quick Recap of MapReduce
  • Spark vs MapReduce
  • Why Python for Spark
  • Just Enough Python for Spark
  • Understanding of CDH Spark and Apache Spark

Phase 2: Hadoop Development (Day 2)

Become a Pro developer with Spark and Hive Datawarehouse

Spark Core Framework and API

  • High level Spark Architecture
  • Role of Executor, Driver, SparkSession etc.
  • Resilient Distributed Datasets
  • Basic operations in Spark Core API i.e.
  • Actions and Transformations
  • Using the Spark REPL for performing interactive data analysis
  • Hands-on Exercises
  • Integrating with Hive

Delving Deeper into Spark API

  • Pair RDDs
  • Implementing Map Reduce Algorithms using Spark
  • Ways to create Pair RDDs JSON Processing Code Example on JSON Processing
  • XML Processing
  • Joins
  • Playing with Regular Expressions
  • Log File Processing using Regular Expressions
  • Hands-on Exercises

Executing a Spark Application

  • Writing Standalone Spark Application
  • Various commands to execute and configure
  • Spark Applications in various modes
  • Discussion on Application, Job, Stage,
  • Executor, Tasks
  • Interpreting RDD Metadata/Lineage/DAG
  • Controlling degree of Parallelism in Spark Job
  • Physical execution of a Spark application
  • Discussion on: How Spark is better than
  • MapReduce?
  • Hands-on Exercises

Phase 3: Hadoop with Spark Dataframes and park SQL (Day 3 and 4)

Spark SQL

  • Dataframes in Depth
  • Creating Dataframes
  • Discussion on Different file formatORC, Sequence, Avro, Parquet and sequence
  • Dataframe internals that makes it fast – Catalyst Optimizer and Tungsten
  • Load data into Spark from external data sources like relational databases
  • Saving dataframe to external sources like HDFS, RDBMS
  • SQL features of Data frame
  • Data formats – text format such csv, json, xml, binary formats such as parquet,orc
  • UDF in Spark Dataframe
  • When to use UDF of hive or not?
  • CDC use cases
  • Spark optimization techniques-joins?
  • Integration with Teradata- use case

Understanding of Hive

  • Hive as a Data Warehouse
  • Creating Tables for Analysis of data
  • Techniques of Loading Data into Tables
  • Difference between Internal and External Tables
  • Understanding Hive Data Types Joining,Union datasets
  • Join Optimizations
  • Partitions and Bucketing
  • Running a Spark SQL Application
  • Dataframes on a JSON file
  • Dataframes on hive tables
  • Dataframes on JSON
  • Querying operations dataframes
  • Hive Writing HSQL queries for data retrieval

Phase 4: NoSQL and Cluster Walkthrough (Day 5)

Know Kafka Tool and Spark Streaming

Introduction to Kafka

  • Kafka Overview
  • Salient Features of Kafka
  • Topics, Brokers and Partitions
  • Kafka Use cases

Kafka Connect and Spark Streaming

  • Kafka Connect
  • Hands-on Exercise

Structured Streaming

  • Structured Streaming Overview
  • How it is better than Kafka streaming?
  • Hands-on Exercises integrating with Kafka using Spark Streaming

 

 

Course Customization Options

To request a customized training for this course, please contact us to arrange.

Inquire now

Best selling courses

Duration: 5 days – 35 hrs   Overview The “SOC Network and Threat Detection and Analysis” training course is designed to equip Security Operations Center (SOC) analysts and IT security professionals with the skills and knowledge required to detect, analyze, and respond to network threats effectively. This comprehensive course covers essential topics such as threat...

Duration 1 day – 7 hrs   Overview   This 1-day training builds upon basic warehouse operations knowledge and introduces key logistics concepts involved in the movement and coordination of goods—especially wet and dry food items—within and outside the warehouse. Participants will explore transport logistics, inbound and outbound coordination, documentation practices, and cold chain considerations,...

Duration 2 days – 14 hrs   Overview   This hands-on course provides an introduction to Splunk, a powerful platform for searching, monitoring, and analyzing machine-generated data. The training focuses on how developers and QA professionals can leverage Splunk to gain insights from logs and metrics, improve application observability, detect anomalies, and support test validation....

Duration 3 days – 21 hrs   Overview.   This course is designed for fresh graduates aspiring to build a career in Data Science. It introduces the fundamentals of data science, focusing on data analysis, visualization, and basic machine learning concepts using Python. The course provides hands-on practice with real-world datasets, equipping participants with the...

Duration 5 days – 35 hrs   Overview   The Microsoft Certified: Azure Administrator Associate (AZ-104) Training Course is designed for IT professionals who want to develop expertise in managing Microsoft Azure environments. This course provides hands-on training on deploying, managing, and monitoring Azure services, including virtual machines, networking, storage, identity, and security.   Objectives...

ARTIFICIAL INTELLIGENCE / MACHINE LEARNING / DEEP LEARNING

Capstone Workshop: Build Your Own GenAI Toolkit TECH AI

GenAI Toolkit Training is a practical, hands-on capstone workshop designed to help participants bring their generative AI knowledge together into a personalized toolkit of AI tools, prompts, workflows, and practical solutions. Rather than simply learning individual AI platforms, participants will learn how to select and combine the right tools to address real-world technical and professional...

In this course, you will gain practical foundation level training that enables immediate and effective participation in big data and other analytics projects. Learn ways of storing data that allow for efficient processing and analysis, and gain the skills you need to store, manage, process, and analyze massive amounts of unstructured data to create an appropriate data lake.

Advanced Laravel Engineering Scalable Architecture, Performance, Testing, Queues & DevOps equips developers with the skills to design scalable Laravel applications, optimize performance, implement automated testing, manage queues, and integrate DevOps practices for reliable, production-ready deployments.   Duration 5 Days – 35 hrs   Overview This advanced-level bootcamp is designed for teams building large-scale or mission-critical...

// BG EARTH WHEN NOT PLAYING

We use cookies on our website to personalize your experience by storing your preferences and recognizing repeat visits. By clicking “Accept”, you agree to the use of all cookies. You can also select “Cookie Settings” to adjust your preferences and provide more specific consent. Cookie Policy