Abstract: In this workshop, we will cover data engineering best practices while using Azure Data Factory – Performance, Security, and Scalability being the key focus areas. We will build ETL pipelines as part of the workshop for hands-on learning.
1. Data pipeline overview
- What is data factory?
- Data engineering common scenario
- Getting started and environment setup
- Bringing data to data lake on Azure: On-premises and SaaS/PaaS datastore connectivity with a Copy activity
- Transforming data using data flows
- Govern data using Azure Purview Integration
- Scalability and Performance considerations
- External integrations with compute engines (Databricks, SProc)
- Continuous Integration and Continuous Deployment (CICD)
2. Data Integration: Connectivity and code-free transformation
3. Best practices for operationalizing data pipelines
Note: This recorded class is available in the format of a video course. Content is presented in modular videos. Learn more.