The Data Engineering on GCP with Apache Airflow, Python, SQL, and Modern Data Tools Training Course provides participants with practical knowledge and skills for designing, building, orchestrating, and maintaining modern data pipelines using Google Cloud Platform (GCP) and widely adopted data engineering technologies.
The course covers the complete data engineering lifecycle—from data ingestion, storage, transformation, and processing to workflow orchestration, data warehousing, monitoring, and pipeline optimization. Participants will use Python and SQL as core data engineering languages while working with technologies such as Google Cloud Storage, BigQuery, Pub/Sub, Dataflow, Dataproc, Apache Beam, Apache Airflow, and Cloud Composer.
Participants will learn how batch and streaming pipelines are designed, how Airflow DAGs automate complex workflows, and how cloud-native services can be integrated to build reliable and scalable data platforms. The course also introduces data quality, security, observability, performance optimization, and data engineering best practices.
The training is designed as a general vendor-focused GCP data engineering program suitable for organizations building or modernizing cloud-based analytics and data platforms.
Duration 5 Days – 35 hrs.
Objectives
- Explain modern data engineering concepts, architectures, and workflows.
- Understand the role of a data engineer within a modern data platform.
- Navigate the core GCP services commonly used for data engineering.
- Design scalable batch and streaming data pipelines.
- Use Python for data ingestion, transformation, validation, and automation.
- Write efficient SQL queries for data processing and analytics.
- Work with Google Cloud Storage for data lake implementations.
- Create and manage datasets and tables in BigQuery.
- Load, transform, query, and optimize data using BigQuery.
- Understand ETL and ELT architectural patterns.
- Use Apache Beam and Dataflow for scalable data processing.
- Understand the role of Dataproc and Apache Spark in large-scale data processing.
- Implement event-driven and streaming pipelines using Pub/Sub.
- Understand Apache Airflow architecture and workflow concepts.
- Build and manage Airflow Directed Acyclic Graphs (DAGs).
- Orchestrate GCP data pipelines using Apache Airflow and Cloud Composer.
- Implement dependencies, scheduling, retries, parameters, and error handling.
- Apply data quality and validation techniques within data pipelines.
- Monitor, troubleshoot, and optimize data engineering workloads.
- Apply security, IAM, governance, and cost-management principles.
- Design an end-to-end GCP data engineering solution.
Target Audience
- Data Engineers
- Junior and Aspiring Data Engineers
- Data Analysts transitioning into Data Engineering
- ETL/ELT Developers
- Database Developers
- SQL Developers
- Python Developers
- Cloud Engineers
- Big Data Engineers
- Business Intelligence Developers
- Analytics Engineers
- Data Warehouse Developers
- Software Engineers working with data platforms
- DevOps Engineers supporting data infrastructure
- Technical Consultants
- Solutions Architects
- IT Professionals responsible for data integration and processing
Prerequisites
- Basic understanding of databases and data concepts.
- Basic knowledge of SQL, including SELECT statements, joins, filtering, and aggregation.
- Basic programming knowledge; prior Python experience is beneficial.
- General understanding of cloud computing concepts.
- Familiarity with command-line environments is helpful.
- Basic understanding of data formats such as CSV and JSON is beneficial.
- No prior experience with Apache Airflow or GCP data engineering services is strictly required.
Course Outline
Day 1 – Data Engineering Foundations, GCP, Python, and SQL
Module 1: Introduction to Modern Data Engineering
- What is data engineering?
- Roles and responsibilities of a data engineer
- Data engineering vs. data analytics vs. data science
- Modern data platform architecture
- Data sources, ingestion, processing, storage, and consumption
- Structured, semi-structured, and unstructured data
- Batch processing vs. stream processing
- ETL vs. ELT
- Data lakes, data warehouses, and lakehouse concepts
- Data pipeline lifecycle
Module 2: Google Cloud Platform for Data Engineering
- GCP architecture and global infrastructure
- Projects, resources, regions, and zones
- GCP Console and Cloud Shell
- Service accounts
- Identity and Access Management fundamentals
- Overview of GCP data engineering services
- Cloud Storage
- BigQuery
- Pub/Sub
- Dataflow
- Dataproc
- Cloud Composer
- Integration between GCP data services
Module 3: Python for Data Engineering
- Python data types and structures
- Variables and operators
- Conditional statements and loops
- Functions and reusable processing logic
- Working with files
- Reading and writing CSV and JSON
- Exception handling
- Python modules and packages
- Working with APIs
- Introduction to pandas for data processing
- Data cleaning and transformation
- Connecting Python applications to cloud data services
Module 4: SQL for Data Engineers
- Relational database fundamentals
- SELECT, WHERE, ORDER BY, and GROUP BY
- JOIN operations
- Aggregations
- Subqueries
- Common Table Expressions
- CASE expressions
- Window functions
- Date and string functions
- Data manipulation concepts
- SQL for data transformation
- Query performance fundamentals
Day 2 – Cloud Storage, BigQuery, Data Warehousing, and ELT
Module 5: Building a Data Lake with Google Cloud Storage
- Data lake architecture
- Cloud Storage concepts
- Buckets and objects
- Storage classes
- Organizing data lake structures
- Uploading and downloading datasets
- Working with CSV, JSON, Avro, and Parquet
- Partitioning data
- Lifecycle management
- Access control
- Data lake design considerations
Module 6: Google BigQuery Fundamentals
- BigQuery architecture
- Serverless data warehousing
- Datasets, tables, and views
- Native and external tables
- Loading data into BigQuery
- Querying Cloud Storage data
- BigQuery SQL
- Creating and managing tables
- Schema definition and management
- Temporary tables
- Views and materialized views
Module 7: Advanced BigQuery and Data Transformation
- Partitioned tables
- Clustered tables
- Nested and repeated fields
- Working with arrays and structures
- Advanced analytical SQL
- Window functions
- BigQuery scripting
- Stored procedures
- Data transformation patterns
- ELT pipelines using BigQuery
- Query optimization
- Understanding query execution
- Controlling BigQuery costs
Module 8: Data Modeling for Analytics
- Operational vs. analytical data models
- Fact and dimension tables
- Star schema
- Snowflake schema
- Data normalization and denormalization
- Slowly changing dimensions
- Designing BigQuery analytical models
- Data warehouse design considerations
Day 3 – Data Processing with Dataflow, Apache Beam, Dataproc, and Streaming
Module 9: Data Processing Architecture
- Data processing requirements
- Batch processing architecture
- Streaming processing architecture
- Choosing appropriate GCP processing services
- Dataflow vs. Dataproc
- Distributed data processing concepts
- Scalability and fault tolerance
Module 10: Apache Beam and Google Cloud Dataflow
- Introduction to Apache Beam
- Beam programming model
- Pipelines
- PCollections
- PTransforms
- Sources and sinks
- Map, filter, and aggregation operations
- Windowing concepts
- Running pipelines locally
- Dataflow Runner
- Deploying pipelines to Dataflow
- Monitoring Dataflow jobs
- Error handling and troubleshooting
Module 11: Streaming Data with Pub/Sub
- Streaming data fundamentals
- Event-driven architecture
- Pub/Sub architecture
- Topics and subscriptions
- Publishers and subscribers
- Message acknowledgement
- Message delivery concepts
- Integrating Pub/Sub with Dataflow
- Pub/Sub to BigQuery pipelines
- Real-time ingestion architecture
Module 12: Dataproc and Apache Spark Fundamentals
- Introduction to Dataproc
- Apache Hadoop ecosystem overview
- Apache Spark architecture
- Spark DataFrames
- PySpark fundamentals
- Data transformations and actions
- Reading and writing cloud data
- Dataproc clusters
- Serverless Spark concepts
- Dataflow vs. Dataproc decision considerations
Day 4 – Apache Airflow and Cloud Composer
Module 13: Apache Airflow Fundamentals
- Introduction to workflow orchestration
- Why orchestration is required
- Apache Airflow architecture
- Scheduler
- Executor
- Metadata database
- Web interface
- Workers
- Directed Acyclic Graphs
- Tasks and task instances
- Operators
- Sensors
- Hooks and connections
Module 14: Building Airflow DAGs with Python
- DAG structure
- Creating DAGs
- Defining tasks
- Task dependencies
- Scheduling workflows
- Cron expressions
- Start dates and scheduling intervals
- Passing parameters
- Variables and connections
- TaskFlow concepts
- Inter-task communication
- Dynamic workflow concepts
- Reusable DAG development
Module 15: Production Airflow Workflow Management
- Task retries
- Retry delays
- Timeouts
- Trigger rules
- Branching workflows
- Sensors and external dependencies
- Backfills
- Catchup behavior
- Failure handling
- Logging
- Debugging DAGs
- Airflow development best practices
- Idempotent pipeline design
Module 16: Google Cloud Composer
- Introduction to Cloud Composer
- Managed Apache Airflow on GCP
- Composer environments
- Environment configuration
- Deploying DAGs
- Managing Python dependencies
- Airflow connections to GCP
- Orchestrating Cloud Storage workflows
- Orchestrating BigQuery jobs
- Orchestrating Dataflow pipelines
- Monitoring Composer environments
- Troubleshooting workflow failures
Day 5 – Production Data Pipelines, Quality, Security, Monitoring, and Optimization
Module 17: End-to-End Data Pipeline Design
- Identifying data sources and destinations
- Data ingestion patterns
- Batch ingestion
- Streaming ingestion
- Transformation layers
- Staging and curated datasets
- Pipeline dependencies
- Incremental data loading
- Change data concepts
- Designing reusable pipelines
- Pipeline configuration management
Module 18: Data Quality and Reliability
- Importance of data quality
- Completeness and accuracy
- Validity and consistency
- Duplicate detection
- Schema validation
- Null-value handling
- Data quality rules
- Pipeline validation
- Error handling
- Dead-letter patterns
- Retry strategies
- Building reliable and idempotent pipelines
Module 19: Security, Governance, and Monitoring
- IAM for data engineering workloads
- Principle of least privilege
- Service accounts
- Protecting sensitive data
- Encryption concepts
- Secrets and credential management
- Data governance fundamentals
- Data lineage concepts
- Audit logging
- Cloud Logging
- Cloud Monitoring
- Pipeline alerts and operational visibility
Module 20: Performance and Cost Optimization
- Optimizing BigQuery queries
- Partitioning and clustering strategies
- Reducing unnecessary data processing
- Dataflow performance considerations
- Airflow workflow optimization
- Resource selection
- Storage optimization
- Monitoring cloud consumption
- Cost-control practices
- Designing scalable and cost-efficient pipelines
Module 21: Integrated Data Engineering Solution
- Source data ingestion
- Landing data in Cloud Storage
- Python-based preprocessing
- SQL-based transformations
- Loading and transforming data in BigQuery
- Batch processing with Dataflow
- Streaming with Pub/Sub
- Workflow orchestration with Apache Airflow
- Managing workflows through Cloud Composer
- Data validation and quality controls
- Monitoring pipeline execution
- Troubleshooting failed workflows
- Reviewing an end-to-end production architecture
