Hello, I'm

Pradeep Kalluri

|

Building scalable cloud data platforms and production-grade pipelines at NatWest Bank, London.

0 Article Views
0 Merged PRs
0 Years Exp
View My Work Get in Touch Download CV
Scroll Down

About Me

I'm a Data Engineer with hands-on experience designing, developing, and deploying scalable data pipelines for analytics and business intelligence across enterprise organizations.

Currently at NatWest Bank, I work across modern data platforms building reliable data flows using Kafka, PySpark, Snowflake, and Airflow.

With experience spanning financial services and consulting, I've delivered data engineering solutions across cloud data platforms, real-time streaming systems, and advanced analytics environments.

☁️

Cloud Platforms

Azure Databricks, AWS, Snowflake, Microsoft Fabric

⚡

Data Pipelines

Kafka, PySpark, Airflow orchestration

🔧

ETL/ELT Solutions

Python, SQL, dbt, Azure Data Factory

📊

Analytics & BI

Tableau, Power BI, SAP BW integration

🎓 Education

MBA — Master in Business Administration

York St John University, London, UK

2023 – 2024

Bachelor of Internet & Communication Technology

Tor Vergata University, Rome, Italy

2020 – 2023

📍 Location

London, United Kingdom

🚀 Currently

  • Building production data pipelines at NatWest Bank
  • Writing about data engineering on Medium (71K+ views)
  • Contributing to Apache Airflow & dbt-core
  • Speaking at data engineering meetups
  • Pursuing UK Global Talent Visa in Digital Technology

Experience

Data Engineer

NatWest Bank
Sep 2025 – Present Greater London, UK

Retail Banking Data Quality and Pipeline Engineering. Building production data platforms processing millions of transactions daily.

  • Design and implement real-time data ingestion pipelines using Kafka and Amazon S3
  • Develop distributed data processing workflows with PySpark for transformation and validation
  • Build curated data models in Snowflake supporting downstream analytics and reporting
  • Orchestrate end-to-end pipelines using Apache Airflow DAGs with dependency management
  • Optimize SQL and PySpark workloads for performance and cost-efficiency
Kafka PySpark Amazon S3 Snowflake Airflow Tableau

Data Engineer

Accenture
Jul 2023 – Aug 2025

Enterprise Data Platform Modernisation. Delivered large-scale cloud data engineering solutions for Fortune 500 clients across multiple industries.

  • Designed and implemented scalable ETL/ELT pipelines using Azure Databricks, ADF, and Snowflake
  • Developed PySpark workflows for distributed data processing and transformation
  • Built reusable transformation layers with dbt ensuring consistent business logic
  • Automated pipeline deployments using CI/CD (GitHub Actions, Terraform)
  • Leveraged Microsoft Fabric for unified analytics workflows and lakehouse architecture
Azure Databricks Snowflake dbt Microsoft Fabric Terraform PySpark

Data Engineer

Dpoint Group
May 2022 – Jun 2023 Barcelona, Spain

SAP BW to Azure Migration & Power BI Reporting Modernisation. Developed BI and analytics solutions for manufacturing and logistics operations.

  • Developed and maintained ETL processes using SSIS to extract data from SAP BW
  • Created interactive Power BI dashboards for executive insights and KPI monitoring
  • Automated recurring reporting workflows using Python and Excel VBA
  • Supported migration of on-premise ETL processes to Azure Data Factory
SSIS SAP BW Power BI Azure Data Factory Python

Technical Skills

Python

PySpark, Pandas

SQL

T-SQL, PL/SQL

Shell

Bash Scripting

AWS

S3, Glue, Lambda

Azure

Databricks, ADF

Fabric

Microsoft Fabric

Kafka

Streaming & Events

Airflow

DAG Orchestration

Snowflake

Cloud Data Warehouse

PySpark

Distributed Processing

dbt

Transform Layer

PostgreSQL

Relational DB

MySQL

Relational DB

Redshift

Azure SQL

Docker

Containerization

Terraform

IaC

CI/CD

GitHub Actions

Tableau

Data Visualization

Power BI

Dashboards & KPIs

Projects

Production

Real-Time Data Pipeline Platform

NatWest Bank

Building production-grade data pipelines processing millions of transactions daily with real-time streaming and automated quality frameworks.

10K+ events/sec
40+ DAGs
6h → 30min recovery
Kafka PySpark Snowflake Airflow S3
Click for case study →
Production

Enterprise Cloud Data Platform

Accenture

Delivered large-scale cloud platforms for Fortune 500 clients using Azure Databricks, Snowflake, and Microsoft Fabric with data mesh architecture.

1000+ users
70% faster deploy
Azure Databricks Snowflake Fabric dbt Terraform
Click for case study →
Production

Business Intelligence Platform

Dpoint Group

Developed BI and analytics solutions for manufacturing and logistics operations, automating 30+ manual reporting processes.

30+ reports automated
Days → Hours time saved
SSIS Power BI SAP BW Azure Data Factory
Click for case study →
Open Source

Real-Time Data Quality Monitor

✅ Production-Ready

ML-powered real-time data quality monitoring system detecting anomalies in streaming data with sub-10ms latency using Isolation Forest.

332K+ orders
93% quality
<10ms latency
Kafka Spark Streaming scikit-learn PostgreSQL
Click for case study →
Open Source

Modern ETL / Data Platform

✅ Open Source

A full modern data engineering platform built from scratch. Cost-effective alternative to commercial tools, potentially saving companies £100K+ annually.

1K+ orders
£100K+ savings
Airflow Kafka dbt Spark Docker
Click for case study →
Open Source

E-commerce Data Pipeline

End-to-end data pipeline demonstrating modern data engineering practices with PySpark, Airflow, dbt, and comprehensive testing.

PySpark Airflow dbt Docker
Click for case study →

Certifications

🏅

Microsoft Fabric Data Engineer Associate

Microsoft

January 2026 Verify Credential →
❄️

SnowPro Core (COF-C03)

Snowflake

Score: 923 / 1000

February 2026 Verify Credential →
📚

AWS Solutions Architect

AWS

In Progress
📚

Azure Data Engineer Associate

Microsoft

In Progress

Writing & Speaking

📝 Technical Writing

71,000+ views across platforms

HackerNoon ⭐

Real-Time Data Quality Monitor: A Production ML Approach

March 2026 The New Stack ⭐

The Weekend Our Pipeline Processed the Same Data 47 Times

January 2026 DZone ⭐

How I Fixed a Silent Production Bug in Apache Airflow

February 2026 Apache Airflow Official

A Beginner's Guide to Contributing to Apache Airflow

February 2026

🎤 Speaking Engagements

✅ Completed

Oxford Microsoft Data Platform Group

Building Production Data Pipelines That Scale

February 2026

📋 Conference Proposals

13 proposals submitted to data engineering conferences across Europe

🧑‍🏫 Mentoring & Community

Active mentor on Topmate (Top 5% Mentor) and internal training lead at NatWest Group. Helping aspiring data engineers transition into the field.

Open Source Contributions

Apache Airflow

6+ Merged PRs + 15+ Reviews
✅ MERGED FastAPI: Lazy initialize flask_app in FAB #64908 ✅ MERGED Spark: add post_submit_commands to SparkSubmitHook #64391 ✅ MERGED Worker: ensure Celery tasks are registered at startup #63110 ✅ MERGED Bug fix contribution: migration stability #61005 ✅ MERGED Pool name validation fix #59938

dbt-core

5+ Merged PRs
✅ MERGED Fix: correct deprecation for top-level config #12618 ✅ MERGED Selection: use tuple membership check #12562 ✅ MERGED Fixed @requires.catalogs for compile #12388 ✅ MERGED dbt init UX: add directory change instruction #12232 🟡 Under Review Debug compilation error fix #12502

Confluent Kafka Python

1 Submitted
🟢 Submitted SSL configuration enhancement

What People Say

GitHub Activity

-- Public Repos
-- Followers
-- Following

Contribution Graph

Last 52 weeks of activity

Less
More
View Full Profile on GitHub →

Get in Touch

Passionate about building reliable, scalable data platforms that empower data-driven decision making.

📧

Email

kalluripradeep99@gmail.com

Click to copy
💼

LinkedIn

linkedin.com/in/pradeepkalluri
🐙

GitHub

github.com/kalluripradeep
✍️

Medium

@kalluripradeep99
👨‍💻

Dev.to

pradeep_kaalluri
🧑‍🏫

Topmate

Mentoring & Career Guidance
✓ Email copied to clipboard!

Pradeep Kalluri

Data Engineer | Cloud Platforms | DataOps

London, UK • kalluripradeep99@gmail.com • linkedin.com/in/pradeepkalluri

Professional Summary

Data Engineer with 3+ years of experience designing, developing, and deploying scalable data pipelines for analytics and business intelligence across enterprise organizations. Currently building production data platforms at NatWest Bank processing millions of transactions daily.

Experience

Data Engineer — NatWest BankSep 2025 – Present

Real-time data ingestion (Kafka, S3), distributed processing (PySpark), curated data models (Snowflake), orchestration (Airflow)

Data Engineer — AccentureJul 2023 – Aug 2025

Scalable ETL/ELT pipelines (Azure Databricks, ADF, Snowflake), dbt transformations, CI/CD automation, Microsoft Fabric

Data Engineer — Dpoint GroupMay 2022 – Jun 2023

ETL processes (SSIS, SAP BW), Power BI dashboards, Azure Data Factory migration

Core Skills

Python, PySpark, SQL, Kafka, Airflow, Snowflake, dbt, Azure Databricks, AWS, Microsoft Fabric, Docker, Terraform, Tableau, Power BI

Education

MBA — York St John University, London (2023–2024)

B.Sc. ICT — Tor Vergata University, Rome (2020–2023)

Certifications

Microsoft Fabric Data Engineer Associate • SnowPro Core (923/1000)

Open Source

Apache Airflow (6+ merged PRs) • dbt-core (5+ merged) • Confluent Kafka

⬇ Download PDF

🔍 The Problem

💡 The Solution

🏗️ Architecture

📊 Impact

🛠️ Tech Stack