📍 Nairobi, Kenya | 🌍 Open to Remote, Hybrid & On-site Roles
Professional Summary
Data Engineer with 2+ years of experience building scalable data pipelines and orchestrating workflows across fintech and agri-commodity environments. Recently eliminated 10+ hours of manual auditing per week, cut daily API payloads by 97% (200KB to 5KB), and architected Bronze, Silver, and Gold Medallion layers on Delta Lake and Parquet in Microsoft Fabric OneLake.
Certifications & Education
Certifications:
• Microsoft Certified: Fabric Data Engineer Associate (DP-700)
• Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750)
Education:
• Data Engineering Bootcamp — Lux Academy
• BSc Statistics and Programming — Kenyatta University, Nairobi (Oct 2021 – Present)
Flagship Engineering Systems
1. Real-Time Crypto CDC & Streaming Engine (Debezium, Kafka, Spark Streaming, Cassandra):
Streamed 115,000+ market events across 3,700+ Binance trading pairs with zero data loss using Debezium CDC on PostgreSQL WAL and Spark Structured Streaming (30s micro-batch latency).
2. Distributed Cloud ELT & Analytics Platform (Airflow, dbt, BigQuery, Snowflake, Grafana):
Hybrid edge-to-cloud pipeline modeling 57,800+ NBA game/player rows across 3 seasons; cut daily API requests by >99% using async FastAPI polling and SQLite checkpointing.
3. Microsoft Fabric Medallion Lakehouse & Real-Time KQL Eventhouse:
End-to-end Bronze/Silver/Gold Lakehouse on OneLake using PySpark, Dataflow Gen2, KQL Eventhouse streaming, and Power BI Direct Lake semantic models.
Zero-Data-Loss Change Data Capture: Streamed 115,000+ market events across 3,700+ Binance trading pairs with zero data loss by deploying Debezium CDC on PostgreSQL Write-Ahead Logs (WAL) into Apache Kafka.
Low-Latency Stream Processing: Achieved 30-second micro-batch latency and 1-minute windowed spread/OHLCV aggregations by engineering Spark Structured Streaming pipelines writing to Apache Cassandra and FastAPI inside Docker Compose.
Distributed Edge-to-Cloud Extraction: Guaranteed daily ingestion across rate-limited REST APIs by engineering a distributed Apache Airflow and FastAPI pipeline over Tailscale VPN with dynamic rate-limiting, SQLite checkpointing, and retry logic (cutting daily player API calls from ~550 to 0–5).
Automated CI/CD & Dimensional Modeling: Caught schema-breaking changes in dimensional marts using dbt (SCD Type 2 player histories) coupled with automated `pytest` unit tests, dbt data quality checks, and GitHub Actions CI/CD workflows.
Microsoft Fabric (DP-700)OneLake Delta LakePySpark & KQL EventhousePower BI Direct Lake
Architecture: Raw Telemetry & Operational Feeds ➔ OneLake Bronze (Parquet) ➔ PySpark & Dataflow Gen2 Silver (Delta Lake) ➔ Gold Star Schema & KQL Eventhouse ➔ Power BI Direct Lake
Engineering Breakdown:
Medallion Lakehouse Engineering: Implemented Bronze, Silver, and Gold layers in Microsoft Fabric OneLake using PySpark notebooks and Delta Lake tables, enforcing schema validation and deduplication before surfacing Gold star schemas via Power BI Direct Lake.
Real-Time Intelligence: Configured KQL Eventhouse pipelines for low-latency event querying and anomaly detection across streaming operational logs.
Architecture: Raw Bandcamp Sales Events ➔ dbt Staging Views ➔ Standardized Currency/Transaction Models ➔ Automated SQL Data Quality Assertions
Engineering Breakdown:
Analytics Engineering & DQC: Built a modular dbt transformation pipeline (`bandcamp_transformation`) to standardize raw digital music e-commerce transactions and enforce custom singular SQL data quality tests (`amount_over_usd_is_positive.sql`).
SugarCube Africa | Data Engineer (Sep 2024 – Present) · Nairobi, Kenya
Built Bronze, Silver, and Gold Medallion layers on Delta Lake and Parquet in OneLake by writing PySpark and SQL ELT transformations that clean raw operational data and model it for Power BI.
Eliminated 10+ hours/week of manual auditing by enforcing data quality rules in PySpark across ~100 daily fulfillment logs and quarantining failing records before they reach reports.
Integrated data from the company's MySQL trading platform database, price feeds, and REST APIs by running incremental Apache Airflow loads that extract only changed rows instead of scanning full tables.
Cut API payload size by 97% (~200KB to 5KB) and met morning-report SLAs by building daily Apache Airflow ETL extraction DAGs with Celery/Redis caching workers.
Accelerated transaction clearance by 20% using CTEs and window functions to give analysts and finance teams a clear view of stalled KYC and order records.
Flagged price swings of 15% or more in 5-minute windows by running tumbling-window anomaly detection on a Medallion lakehouse.
Flamingo Commodities | Data Analyst Intern (Jan 2024 – Aug 2024) · Nairobi, Kenya
Eliminated duplicate records by developing PostgreSQL `MERGE` scripts to securely ingest, clean, and reconcile raw market data.
Saved 2+ hours of manual ingestion by writing automated Python and Bash file-watcher ingestion scripts.
Education
• Lux Academy | Data Engineering Bootcamp
• Kenyatta University, Nairobi | BSc Statistics and Programming (Oct 2021 – Present)
cliffe@ubuntu:~/experience$ _
Certifications
• Microsoft Certified: Fabric Data Engineer Associate (DP-700)
• Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750)
Core Skills
Azure & Lakehouse:
Azure, Azure Databricks, Microsoft Fabric, OneLake, Delta Lake, Parquet
The Blueprint for Modern Data Orchestration ↗ Platform: DEV Community | Highlights: Why cron jobs fail at scale and how DAG dependencies, retries, and backfills work in Apache Airflow.