KIYANZA - Plateforme d'optimisation et de suivi des strategies de communication
Medallion architecture on Azure Databricks, PySpark, and Delta Lake
An enterprise-grade Lakehouse data pipeline ingesting multi-source transactional and telemetry data into Bronze, Silver, and Gold Delta Lake tables with automated schema enforcement, dbt transformations, and data quality observability.
Performance Metrics & Quantified Impact
The Challenge & Core Problem
Legacy batch jobs running on relational SQL data warehouses suffered from skyrocketing compute costs, frequent pipeline failures during schema drifts, and 6+ hour delays in delivering business intelligence reports.
Key Engineering Objectives
- Architect a scalable Medallion (Bronze/Silver/Gold) architecture on Azure Databricks.
- Build robust PySpark streaming and batch ingestion jobs with automatic schema evolution.
- Implement dbt models for dimensional modeling (Kimball Star Schema) in the Gold layer.
- Integrate automated data quality testing with Great Expectations and alerting webhooks.
System Architecture & Pipeline Design
Data is captured from relational DBs via Debezium CDC and IoT telemetry via Azure Event Hubs, landed in ADLS Gen2 Bronze raw format, standardized and deduplicated into Silver Delta tables, and aggregated into dimensional Gold marts via dbt.
Event Hubs / CDC Kafka → Bronze ADLS Gen2 (Raw Parquet/Delta) → PySpark Sanitization & Deduplication → Silver Layer (Enriched Delta) → dbt Dimensional Modeling & Great Expectations → Gold Layer (Star Schema Marts) → Power BI & Analytics Consumers
Bronze Ingestion with Auto Loader
Configured Databricks Auto Loader to incrementally ingest raw JSON and CSV files from Azure Data Lake Storage Gen2 with cloud notification mode and schema rescue columns.
Silver Layer Cleansing & CDC Merge
Developed PySpark jobs applying Delta Lake MERGE operations for Type 2 Slowly Changing Dimensions (SCD2), timestamp alignment, deduplication, and PII masking.
Gold Layer Modeling with dbt
Constructed Kimball dimensional models (Facts and Conformed Dimensions) using dbt-databricks, executing automated documentation generation and dependency DAG scheduling.
Data Observability & Orchestration
Implemented circuit breaker checks using Great Expectations to block corrupted batches, with orchestrations managed via Azure Data Factory and Databricks Asset Bundles.
Technical Obstacles & Engineering Solutions
Obstacle #1
Severe 'Small File Problem' in the Bronze layer causing slow read performance and excessive metadata overhead on the driver node.
Configured Delta Lake auto-compaction and optimized writes (`spark.databricks.delta.optimizeWrite.enabled = true`) combined with scheduled daily VACUUM and OPTIMIZE jobs with Z-Ordering on primary query predicates.
Obstacle #2
Unannounced schema changes from upstream source databases causing downstream pipeline crashes.
Leveraged Delta Schema Evolution with `_rescued_data` column capture in Auto Loader, gracefully routing unrecognized columns to staging quarantine tables without halting pipeline execution.
Obstacle #3
Spiking compute costs on Databricks clusters during uneven streaming traffic spikes.
Configured aggressive single-node cluster autoscaling with Azure spot VM instances for worker pools, driving a 38% reduction in monthly cloud infrastructure spend.
Impact & Key Takeaways
The Lakehouse architecture replaced legacy siloed warehouses with a unified, high-performance platform that slashed operational costs and accelerated business reporting cycles.
Interested in similar architectures for your team?
Let's discuss custom Data Pipelines, AI deployment, or high-throughput Python backends.