- Duration: 4 days
Overview
Master end-to-end data engineering with Azure Databricks and Unity Catalog.
This course takes participants from foundational Azure Databricks configuration through to production-ready data engineering and enterprise governance. Participants will learn how to configure Databricks environments, select and optimize compute resources, organize and secure data using Unity Catalog, and build scalable data ingestion and transformation pipelines.
The course provides practical knowledge of Apache Spark, Delta Lake, Lakeflow, Unity Catalog, data quality, pipeline orchestration, Git-based development, and workload optimization. Participants will also learn how to implement enterprise-grade security, governance, monitoring, and deployment practices.
By the end of the course, participants will have the practical skills required to implement, secure, operate, and maintain scalable lakehouse solutions using Azure Databricks.
What You’ll Learn
- Understand the fundamentals of data engineering and big data processing using Azure Databricks
- Explore Azure Databricks architecture, compute options, and core capabilities
- Configure and manage Azure Databricks environments
- Design and implement scalable data pipelines using Apache Spark
- Select and optimize compute resources for different data engineering workloads
- Organize and govern data using Unity Catalog
- Implement fine-grained security and access controls
- Ingest, cleanse, transform, and load large-scale datasets
- Implement batch and streaming data ingestion patterns
- Design effective data models and optimize table performance
- Implement data quality controls and schema management
- Build and orchestrate data pipelines using Lakeflow
- Implement reliable workflows using Lakeflow Jobs
- Apply Git-based development and CI/CD practices
- Use Databricks Asset Bundles for infrastructure-as-code deployments
- Monitor, troubleshoot, and optimize Azure Databricks workloads
- Apply enterprise best practices for data security, governance, and operational management
Who Should Attend
This course is designed for:
- Data Engineers
- Cloud Data Engineers
- Big Data Engineers
- Data Platform Engineers
- Analytics Engineers
- Data Architects
- Azure Data Professionals
- Technical professionals responsible for developing and managing lakehouse solutions
Participants should have a fundamental understanding of data analytics, cloud storage, data organization, and data engineering concepts. Familiarity with SQL, Python, Azure Databricks, Unity Catalog, Microsoft Entra ID, and Git is beneficial.
Prerequisites
Before attending this course, participants should have:
- Fundamental knowledge of data analytics concepts
- Basic understanding of cloud storage concepts
- Familiarity with SQL and data organization principles
- Basic Python programming experience is recommended
- Familiarity with Azure Databricks and Unity Catalog is recommended
- Basic understanding of data engineering and data warehouse concepts
- Basic knowledge of Microsoft Entra ID and Azure security
- Familiarity with Git version control fundamentals
Curriculum
Module 01: Set Up and Configure an Azure Databricks Environment
Build a strong foundation in Azure Databricks by exploring its architecture, integrations, compute options, and data organization capabilities.
Topics include:
- Azure Databricks fundamentals
- Workspace configuration
- Databricks architecture
- Azure service integrations
- Compute options
- Data organization
- Supporting data engineering, analytics, and AI workloads
Module 02: Explore Azure Databricks
Explore Azure Databricks as a scalable cloud platform for data engineering, analytics, and large-scale data processing using Apache Spark.
Topics include:
- Azure Databricks overview
- Apache Spark
- Databricks workspace
- Data engineering capabilities
- Analytics workloads
- Lakehouse architecture
Module 03: Understand Azure Databricks Architecture
Explore the architecture of Azure Databricks and understand how resources, compute, and storage are organized.
Topics include:
- Azure Databricks account hierarchy
- Control plane and compute plane
- Serverless compute
- Classic compute
- Workspace architecture
- Default storage
- External storage
- Unity Catalog managed storage
- Data organization and governance
Module 04: Understand Azure Databricks Integrations
Learn how Azure Databricks integrates with Microsoft services and development platforms to create comprehensive data and AI solutions.
Topics include:
- Microsoft Fabric
- Power BI
- Visual Studio Code
- Power Platform
- Copilot Studio
- Microsoft Purview
- Microsoft Foundry
- Data lakehouse integration
- Business intelligence integration
- Application and AI integration
Module 05: Select and Configure Compute in Azure Databricks
Learn how to select and configure the appropriate compute resources for different workloads while balancing performance, security, and cost.
Topics include:
- Azure Databricks compute options
- Serverless versus classic compute
- Compute configuration
- Performance settings
- Access permissions
- Library installation
- Cluster configuration
- Cost optimization
- Performance optimization
- Compute security best practices
Module 06: Create and Organize Objects in Unity Catalog
Learn how to organize data assets using the Unity Catalog three-level namespace.
Topics include:
- Unity Catalog fundamentals
- Catalogs
- Schemas
- Tables
- Views
- Volumes
- Environment isolation
- Foreign catalogs
- External database access
- Data naming conventions
- Data organization
- AI/BI Genie instructions
- Data discoverability
Module 07: Secure Unity Catalog Objects
Learn how to implement centralized security and fine-grained access controls for data assets in Unity Catalog.
Topics include:
- Access control
- Fine-grained permissions
- Table-level security
- Schema-level security
- Row filtering
- Column filtering
- Credential management
- Azure Key Vault integration
- Service principals
- Managed identities
- Authentication and authorization
Module 08: Govern Unity Catalog Objects
Explore governance capabilities that help organizations secure, monitor, and manage their data estate.
Topics include:
- Data governance
- Fine-grained access control
- Data lineage
- Audit logging
- Data monitoring
- Secure data sharing
- Governance best practices
- Enterprise data management
Module 09: Design and Implement Data Modeling with Azure Databricks
Learn how to design effective data models for scalable and high-performance lakehouse solutions.
Topics include:
- Data ingestion design
- Table and data formats
- Data granularity
- Partitioning strategies
- Slowly Changing Dimensions (SCD)
- Data modeling patterns
- Table optimization
- Clustering strategies
- Performance considerations with Unity Catalog
Module 10: Ingest Data into Unity Catalog
Learn different approaches for ingesting data into Unity Catalog-managed tables.
Topics include:
- Lakeflow Connect
- Managed connectors
- Custom ingestion using notebooks
- SQL-based batch ingestion
- Change Data Capture (CDC)
- Streaming ingestion
- Message bus integration
- Auto Loader
- Automatic file detection
- Lakeflow Spark Declarative Pipelines
- Ingestion workflow orchestration
Module 11: Cleanse, Transform, and Load Data into Unity Catalog
Learn how to transform raw data into clean, structured datasets suitable for analytics and downstream workloads.
Topics include:
- Data profiling
- Data quality analysis
- Column data types
- Duplicate management
- Null value handling
- Filtering
- Aggregation
- Joins
- Set operators
- Pivoting
- Denormalization
- Append operations
- Overwrite operations
- Merge operations
- Loading transformed datasets
Module 12: Implement and Manage Data Quality Constraints with Azure Databricks
Learn how to implement data quality controls and maintain data integrity throughout data pipelines.
Topics include:
- Data validation
- Data quality checks
- Schema enforcement
- Schema evolution
- Schema drift management
- Pipeline expectations
- Data integrity
- Data quality monitoring
Module 13: Design and Implement Data Pipelines with Azure Databricks
Learn how to design and implement robust, reusable data pipelines using Azure Databricks.
Topics include:
- Notebook-based pipelines
- Lakeflow Spark Declarative Pipelines
- Pipeline architecture
- Workflow orchestration
- Task dependencies
- Task logic
- Error handling
- Pipeline reliability
- Data pipeline best practices
Module 14: Implement Lakeflow Jobs with Azure Databricks
Learn how to automate and manage data engineering workloads using Lakeflow Jobs.
Topics include:
- Creating Lakeflow Jobs
- Job configuration
- Triggers
- Scheduling
- Task dependencies
- Alerts
- Automatic restarts
- Job monitoring
- Reliable pipeline execution
- Workflow management
Module 15: Implement Development Lifecycle Processes in Azure Databricks
Learn how to integrate Azure Databricks development with modern software engineering and deployment practices.
Topics include:
- Git folders
- Git version control
- Branching strategies
- Pull requests
- Development workflows
- Testing strategies
- Databricks Asset Bundles
- Infrastructure as Code (IaC)
- Databricks CLI
- Bundle deployment
- Multi-environment deployments
- Development lifecycle management
Module 16: Monitor, Troubleshoot, and Optimize Workloads in Azure Databricks
Learn how to monitor, troubleshoot, and optimize Azure Databricks workloads for reliability, performance, and cost efficiency.
Topics include:
- Cluster consumption metrics
- Lakeflow Jobs monitoring
- Job troubleshooting
- Spark job diagnostics
- Performance analysis
- Caching optimization
- Data skew
- Spill management
- Shuffle optimization
- Spark performance tuning
- Log streaming
- Azure Log Analytics
- Workload monitoring
- Cost and performance optimization
Course Outcome
Upon successful completion of this course, participants will be able to design, develop, secure, deploy, monitor, and optimize scalable data engineering solutions using Azure Databricks and Unity Catalog.
Participants will gain practical skills in lakehouse architecture, Apache Spark, data ingestion, transformation, data modeling, Unity Catalog governance, Lakeflow pipelines, workflow orchestration, Git-based development, infrastructure as code, and workload optimization, enabling them to build and maintain enterprise-ready Azure Databricks environments.