Data Engineering on AWS with AWS DMS and Modern Data Services

Inquire now

Course Description

The Data Engineering on AWS with AWS DMS and Modern Data Services Training Course provides participants with practical knowledge of designing, building, migrating, processing, and managing modern data solutions using Amazon Web Services (AWS).

The course covers the end-to-end data engineering lifecycle, including data ingestion, database migration, data lakes, ETL/ELT processing, data warehousing, analytics, orchestration, monitoring, security, and performance optimization. A major focus is placed on AWS Database Migration Service (AWS DMS) for migrating and continuously replicating data between database platforms with minimal downtime.

Participants will explore commonly used AWS data engineering services including Amazon S3, AWS Glue, AWS DMS, Amazon RDS, Amazon Redshift, Amazon Athena, AWS Lambda, Amazon Kinesis, AWS Step Functions, Amazon CloudWatch, AWS IAM, and related services. The course also introduces architectural patterns for batch and streaming data pipelines and modern AWS data lake and analytics environments.

Duration 5 Days – 35 hrs.

Objectives

Explain the role and responsibilities of a data engineer in an AWS environment.
Understand AWS data engineering architecture and common data pipeline patterns.
Select appropriate AWS storage, database, processing, and analytics services.
Design scalable data ingestion and integration pipelines.
Build and manage data lakes using Amazon S3.
Understand ETL and ELT approaches for cloud-based data engineering.
Create data catalogs and ETL pipelines using AWS Glue.
Use AWS DMS to migrate and replicate databases.
Understand homogeneous and heterogeneous database migration scenarios.
Configure AWS DMS replication instances, endpoints, and migration tasks.
Implement full-load and Change Data Capture (CDC) migration strategies.
Work with relational databases through Amazon RDS and Amazon Aurora.
Design analytical data warehouse solutions using Amazon Redshift.
Query data directly from Amazon S3 using Amazon Athena.
Understand batch and real-time/streaming data processing architectures.
Use Amazon Kinesis for streaming data workloads.
Incorporate AWS Lambda and Step Functions into data workflows.
Apply security and access controls using AWS IAM and encryption.
Monitor data pipelines and AWS resources using Amazon CloudWatch.
Apply data quality, reliability, scalability, and performance best practices.
Design an end-to-end AWS data engineering solution.

Target Audience

Data Engineers
Database Administrators
Database Engineers
Data Architects
Cloud Engineers
Cloud Architects
ETL Developers
Data Warehouse Developers
Data Analysts with technical responsibilities
Business Intelligence Developers
Software Developers working with data platforms
DevOps Engineers supporting data workloads
Solutions Architects
System Engineers
IT Professionals responsible for database migration
Technical professionals transitioning into cloud data engineering

Prerequisites

Basic understanding of databases and relational database concepts.
Basic SQL knowledge, including SELECT, JOIN, GROUP BY, and data manipulation.
General understanding of ETL/ELT and data integration concepts.
Basic familiarity with cloud computing concepts.
Basic knowledge of data warehousing is helpful but not mandatory.
Familiarity with Python or another scripting language is beneficial.
Previous AWS experience is helpful but not required.

Course Outline

Day 1 – AWS Data Engineering Fundamentals and Data Storage

Module 1: Introduction to Modern Data Engineering

Role of a data engineer
Data engineering lifecycle
Structured, semi-structured, and unstructured data
Operational versus analytical workloads
Batch versus streaming processing
ETL versus ELT
Data pipelines and data integration
Data lakes, data warehouses, and lakehouse concepts

Module 2: AWS Architecture for Data Engineering

Overview of AWS global infrastructure
Regions and Availability Zones
AWS data engineering ecosystem
Compute, storage, database, integration, and analytics services
Designing highly available data architectures
Scalability and fault tolerance
Common AWS data engineering reference architectures

Module 3: Amazon S3 for Data Engineering

Amazon S3 architecture
Buckets and objects
Storage classes
Organizing data lake structures
Data partitioning strategies
File formats for data engineering
CSV, JSON, Parquet, and ORC
Compression considerations
Versioning and lifecycle management
S3 security and access control

Module 4: Building Data Lakes on AWS

Data lake architecture
Raw, processed, curated, and consumption layers
Data ingestion patterns
Data organization and partitioning
Metadata management
Data lake governance concepts
Designing scalable S3-based data platforms

Day 2 – AWS Database Migration Service and Database Integration

Module 5: AWS Database Services for Data Engineers

Amazon RDS overview
Amazon Aurora
Relational database engines
Amazon DynamoDB overview
Operational versus analytical databases
Selecting appropriate AWS database services
Database connectivity considerations

Module 6: Introduction to AWS Database Migration Service (AWS DMS)

AWS DMS architecture
Database migration concepts
Homogeneous migrations
Heterogeneous migrations
Source and target endpoints
Replication instances
Migration tasks
Supported migration patterns
Network and connectivity considerations

Module 7: Implementing Database Migration with AWS DMS

Preparing source and target databases
Creating replication instances
Configuring source endpoints
Configuring target endpoints
Testing endpoint connectivity
Creating migration tasks
Full-load migration
Change Data Capture (CDC)
Full load plus CDC
Table mappings and selection rules
Transformation rules

Module 8: Advanced AWS DMS Concepts

Continuous data replication
Migration monitoring
Validation and reconciliation
Logging and troubleshooting
Performance considerations
Handling large tables
Managing migration failures
Minimizing migration downtime
Migration cutover planning
AWS Schema Conversion Tool concepts
Schema conversion considerations for heterogeneous migrations

Day 3 – AWS Glue, ETL/ELT and Data Transformation

Module 9: Introduction to AWS Glue

AWS Glue architecture
AWS Glue Data Catalog
Databases and tables
Crawlers
Classifiers
Metadata discovery
Integrating Glue with Amazon S3
Schema management

Module 10: Building ETL Pipelines with AWS Glue

Creating ETL jobs
Data extraction
Data transformation
Data loading
Working with Glue Studio
Working with structured and semi-structured datasets
Filtering and cleansing data
Joining datasets
Data type conversion
Writing transformed data to Amazon S3

Module 11: Data Quality and Pipeline Reliability

Data validation concepts
Handling missing and invalid data
Duplicate detection
Schema consistency
Data quality rules
Error handling
Logging
Retry strategies
Idempotent pipeline concepts
Designing resilient data pipelines

Module 12: Serverless Data Processing

AWS Lambda fundamentals
Event-driven data processing
S3 event triggers
Integrating Lambda with data pipelines
AWS Step Functions overview
Workflow orchestration
Coordinating data engineering processes
Error handling and workflow control

Day 4 – Data Warehousing, Analytics and Streaming Data

Module 13: Amazon Redshift for Data Warehousing

Data warehouse architecture
Amazon Redshift concepts
Provisioned and serverless approaches
Loading data into Redshift
Data distribution concepts
Sort keys and data organization
Query performance considerations
Redshift Spectrum concepts
Integrating Redshift with Amazon S3

Module 14: Serverless Analytics with Amazon Athena

Amazon Athena architecture
Querying data stored in Amazon S3
Integrating Athena with AWS Glue Data Catalog
Working with partitions
Query optimization
Columnar data formats
Cost optimization
Common analytics use cases

Module 15: Real-Time Data Engineering with Amazon Kinesis

Streaming data fundamentals
Batch versus streaming architectures
Amazon Kinesis ecosystem
Kinesis Data Streams
Producers and consumers
Stream processing concepts
Data ingestion patterns
Real-time analytics use cases
Integrating streaming data with AWS data services

Module 16: Designing Batch and Streaming Pipelines

Batch ingestion architecture
Streaming ingestion architecture
Lambda architecture concepts
Event-driven architectures
Handling high-volume datasets
Selecting between batch and streaming
Combining historical and real-time data
Designing scalable ingestion pipelines

Day 5 – Security, Monitoring, Optimization and End-to-End Architecture

Module 17: Security for AWS Data Engineering

AWS Shared Responsibility Model
AWS IAM fundamentals
Users, roles, and policies
Least-privilege access
Service roles
Encryption at rest
Encryption in transit
AWS Key Management Service concepts
Securing S3 data
Securing databases and data pipelines
Secrets management concepts
Network security considerations

Module 18: Monitoring and Troubleshooting Data Pipelines

Amazon CloudWatch overview
Metrics and logs
Monitoring AWS DMS
Monitoring AWS Glue
Monitoring database workloads
Pipeline observability
Identifying failed jobs
Troubleshooting connectivity
Troubleshooting permissions
Troubleshooting data processing failures

Module 19: Performance and Cost Optimization

Storage optimization
Data partitioning
File format selection
Compression strategies
Query optimization
ETL performance considerations
AWS DMS performance considerations
Redshift optimization concepts
Serverless cost considerations
AWS data engineering cost-management practices

Module 20: End-to-End AWS Data Engineering Architecture

Identifying source systems
Data ingestion layer
Database migration and CDC layer
Data lake storage layer
Data catalog and governance layer
ETL/ELT transformation layer
Data warehouse and analytics layer
Batch and streaming integration
Security architecture
Monitoring and operational considerations
Designing an end-to-end AWS data pipeline
Reviewing production-ready data engineering patterns

Inquire now

Best selling courses

DATA ANALYTICS

Data Analytics with DAX, Python, SQL, and Business Intelligence Tools

The Data Analytics with DAX, Python, SQL, and Business Intelligence Tools Training Course provides participants with practical knowledge and skills for collecting, preparing, analyzing, modeling, and visualizing data to support informed business decision-making. The course combines key data analytics technologies and techniques, including Python, SQL, DAX (Data Analysis Expressions), Power BI, data modeling, data transformation,...

Inquire Now

PROJECT MANAGEMENT

Project Management – SDLC (Software Project Management)

The Project Management – SDLC (Software Project Management: The Complete Course) Training Course is a comprehensive program designed to equip project managers, software development leaders, business analysts, product owners, and IT professionals with the knowledge and practical skills required to successfully manage software development projects throughout the entire Software Development Life Cycle (SDLC). This course...

Inquire Now

LOGISTICS

Warehouse Operations and Inventory Management

Warehouse Operations and Inventory Management Training Course Duration 2 Days – 14 hrs. Overview This course provides a practical understanding of warehouse operations and inventory management for personnel who supervise, review, or audit warehouse processes. It covers the movement of goods from receiving through dispatch, with emphasis on warehouse layout, storage systems, picking strategies, stock rotation, inventory accuracy, and operational controls. Participants will learn to identify process weaknesses, investigate stock discrepancies, reduce shrinkage, and interpret warehouse productivity indicators. The course connects day-to-day warehouse activities with the records, controls, and performance measures needed for effective supervision and auditing. Objectives

Explain warehouse processes and their impact on inventory accuracy and service performance.
Evaluate warehouse layouts, storage systems, and product locations.
Identify appropriate controls for receiving, put-away, storage, picking, and dispatch.
Apply First-In, First-Out (FIFO) and First-Expired, First-Out (FEFO) principles.
Compare picking strategies based on order characteristics and operational requirements.
Identify causes of inventory discrepancies and recommend corrective actions.
Develop a cycle counting approach based on inventory value, movement, and risk.
Recognize sources of shrinkage and recommend preventive and detective controls.
Interpret warehouse productivity and inventory performance KPIs.
Review warehouse processes using supporting records, observations, and control evidence.

Target Audience

Warehouse supervisors, team leaders, and managers.
Inventory controllers and stock control personnel.
Logistics and distribution supervisors.
Internal auditors and operational compliance personnel.
Supply chain and operations professionals.
Personnel responsible for reviewing warehouse controls and performance.

Prerequisites

Basic familiarity with warehouse, logistics, or inventory activities.
General understanding of business processes and operational documentation.
Basic numerical skills, including percentages and simple calculations.
Basic spreadsheet skills are helpful.
No prior experience with a specific Warehouse Management System (WMS) is required.

Course Outline

Day 1: Warehouse Processes, Layout, and Storage Controls Module 1: Warehouse Operations and Process Controls

Warehouse roles within the supply chain.
End-to-end flow: receiving, put-away, storage, replenishment, picking, packing, and dispatch.
Physical stock movement and corresponding system transactions.
Responsibilities, authorization, and segregation of duties.
Common operational weaknesses and audit evidence.

Module 2: Warehouse Layout and Storage Systems

Functional zones, movement paths, and space utilization.
Receiving, storage, picking, staging, and dispatch areas.
Storage options: floor stacking, shelving, pallet racking, and bin storage.
Fixed and dynamic location systems.
Product slotting based on movement, size, handling needs, and compatibility.
Layout bottlenecks and their effects on productivity and inventory control.

Module 3: Receiving and Put-Away

Purchase orders, delivery documents, and expected receipts.
Quantity, condition, and item identification checks.
Recording shortages, overages, damage, and discrepancies.
Controls for acceptance, quarantine, and release.
Put-away rules and location confirmation.
Traceability through labels, barcodes, lot numbers, and serial numbers.
Timely recording of receipts and stock movements.

Module 4: Storage, Stock Rotation, and Replenishment

Storage discipline and location integrity.
FIFO and FEFO principles, applications, and limitations.
Batch, lot, and expiry-date management.
Controls for slow-moving, obsolete, damaged, and expired stock.
Replenishment of picking locations.
Supervision and review of stock rotation compliance.

Day 2: Order Fulfillment and Inventory Accuracy Module 5: Picking Strategies and Order Fulfillment

Discrete, batch, zone, and wave picking.
Selecting picking strategies for different order profiles.
Pick paths, location sequencing, and travel reduction.
Replenishment coordination and stock availability.
Picking verification and error prevention.
Packing, staging, and dispatch confirmation.
Handling short picks, substitutions, cancellations, and returns.

Module 6: Inventory Records and Accuracy Controls

Item master data, units of measure, and location records.
On-hand, available, allocated, and quarantined stock.
Transaction controls for receipts, transfers, issues, returns, and adjustments.
Common causes of physical and system stock mismatches.
Barcode scanning and WMS control points.
Inventory reconciliation and supporting documentation.
Measuring inventory accuracy and interpreting discrepancies.

Module 7: Cycle Counting and Physical Inventory

Cycle counting versus full physical inventory.
ABC classification and risk-based count priorities.
Count frequency based on value, movement, and discrepancy history.
Count preparation, blind counting, and movement controls.
Recounts, variance investigation, and adjustment approval.
Root cause analysis and corrective action follow-up.
Reviewing the effectiveness of the counting program.

Module 8: Shrinkage Reduction and Inventory Protection

Sources of shrinkage: loss, damage, theft, misplacement, and recording errors.
Access controls and accountability for stock movements.
Controls for high-value and sensitive inventory.
Returns, write-offs, scrap, and disposal controls.
Exception reporting and unusual transaction patterns.
Investigating recurring losses and unexplained adjustments.
Preventive and detective controls for shrinkage reduction.

Day 3: Performance Management, Process Review, and Improvement Module 9: Warehouse Productivity and Inventory KPIs

Receiving productivity and dock-to-stock time.
Put-away productivity and replenishment performance.
Picking productivity and order accuracy.
Inventory accuracy and cycle count variance.
Space utilization and storage capacity.
Shrinkage, damage, and expiry indicators.
Defining KPI formulas, data sources, and reporting boundaries.
Interpreting trends while balancing speed, accuracy, and service.

Module 10: Supervising and Auditing Warehouse Processes

Process mapping and identification of critical control points.
Reviewing procedures against actual warehouse practices.
Tracing transactions from documents to systems and physical stock.
Reviewing approvals, access rights, and segregation of duties.
Identifying control gaps and recurring process exceptions.
Documenting findings with clear supporting evidence.
Prioritizing corrective actions and tracking closure.

Module 11: Warehouse Improvement Planning

Identifying bottlenecks, unnecessary movement, and rework.
Improving layout, slotting, and picking arrangements.
Addressing inventory discrepancy root causes.
Standardizing transaction recording and exception handling.
Assigning improvement owners and measurable targets.
Monitoring results and sustaining operational improvements.

Module 12: Capstone — Warehouse Operations and Inventory Improvement Plan

Review a warehouse scenario containing layout, transaction, inventory, and performance information.
Identify weaknesses across receiving, put-away, storage, picking, and dispatch.
Recommend appropriate FIFO or FEFO controls.
Develop a cycle counting and inventory reconciliation plan.
Propose controls to reduce shrinkage and improve traceability.
Select relevant KPIs and define their calculation and reporting requirements.
Produce a prioritized improvement plan with responsibilities and expected outcomes.

Inquire Now

BUSINESS PROCESS MANAGEMENT

ServiceNow Certified System Administrator (CSA)

This course develops the foundational knowledge and practical skills needed to configure, administer, and maintain the ServiceNow platform. Participants explore platform navigation, instance configuration, application collaboration, self-service, automation, database management, security, and configuration migration. The proposed five-day duration accommodates foundational learning, configuration activities, and an integrated capstone. Duration 5 Days – 35 hrs. Objectives...

Inquire Now

CYBER SECURITY

SOC Network and Threat Detection and Analysis

Duration: 5 days – 35 hrs Overview The “SOC Network and Threat Detection and Analysis” training course is designed to equip Security Operations Center (SOC) analysts and IT security professionals with the skills and knowledge required to detect, analyze, and respond to network threats effectively. This comprehensive course covers essential topics such as threat...

Inquire Now

LOGISTICS

Logistics Operations Essentials: Transportation, Coordination & Documentation for Food Supply Chains

Duration 1 day – 7 hrs Overview This 1-day training builds upon basic warehouse operations knowledge and introduces key logistics concepts involved in the movement and coordination of goods—especially wet and dry food items—within and outside the warehouse. Participants will explore transport logistics, inbound and outbound coordination, documentation practices, and cold chain considerations,...

Inquire Now

DEV OPS / CONTAINERS

Splunk for Developers and QA Teams

Duration 2 days – 14 hrs Overview This hands-on course provides an introduction to Splunk, a powerful platform for searching, monitoring, and analyzing machine-generated data. The training focuses on how developers and QA professionals can leverage Splunk to gain insights from logs and metrics, improve application observability, detect anomalies, and support test validation....

Inquire Now

DATA ANALYTICS

Data Science for Beginners

Duration 3 days – 21 hrs Overview. This course is designed for fresh graduates aspiring to build a career in Data Science. It introduces the fundamentals of data science, focusing on data analysis, visualization, and basic machine learning concepts using Python. The course provides hands-on practice with real-world datasets, equipping participants with the...

Inquire Now

Data Engineering on AWS with AWS DMS and Modern Data Services

The Data Engineering on AWS with AWS DMS and Modern Data Services Training Course provides participants with practical knowledge of designing, building, migrating, processing, and managing modern data solutions using Amazon Web Services (AWS).

Related Courses

Best selling courses

Course Outline

Training Inquiry Information

Login