I'm a Lead Software Engineer at Zeta Global and a recognized Data Lakes & Lakehouse Architecture Expert with 6+ years of hands-on experience building production-grade data platforms. I specialize in Apache Hudi, Apache Iceberg, AWS EMR, and Spark, architecting solutions that process terabytes of data daily while achieving significant cost reductions.
"Making sophisticated data engineering accessible to everyone through code, content, and community"
🏗️ Creator of LakeBoost — Production framework integrating Apache Hudi with AWS Glue ETL, powering 50+ tables with 466 daily jobs (11,832 monthly), achieving 40% operational overhead reduction and 4-5x cost savings
📝 Featured on AWS Storage Blog — How Zeta Global scales multi-tenant data ingestion with Amazon S3 Tables
💰 Delivered $200K+ Cost Savings — Led Lakehouse adoption achieving ~50% cost reduction while processing 55TB+/month with sub-10-minute refresh latency
⚡ Performance Optimization Expert — Reduced data processing from hours to minutes; achieved 80x faster searches through Elasticsearch optimization (4.4s → 125ms)
interface Engineer {
role: "Lead Software Engineer";
focus: ["Data Lakes", "Lakehouse", "Platform"];
stack: ["Hudi", "Iceberg", "Spark", "AWS"];
}Leading enterprise-scale data transformation:
|
metrics = {
"data_volume": "1.3TB daily | 55TB monthly",
"table_count": "10,000+ Iceberg tables",
"job_throughput": "466 daily | 11,832 monthly",
"cost_reduction": "~50% | $200K+ saved",
"performance": "2× faster ingestion",
"latency": "<10 min refresh"
}Core expertise:
|
| Platform | Reach & Engagement |
|---|---|
| 🎬 YouTube | 46,188 Subscribers | 432.9K Views | 16.4K Hours Watch Time |
| 📹 Videos Created | 1,800+ Tutorials (3.9M Impressions, 5.5% CTR) |
| ✍️ Medium Blog | 422 Followers | 18.4K Presentations | 5.2K Views | 3K Reads |
| 📝 Blog Posts | 100+ Articles on Data Engineering |
| 10,922 Followers | 1,000+ Posts | |
| 📂 GitHub | 300+ Open Source Repositories |
| 🌐 Website | Technical Resources & Portfolio |
| ⏱️ Avg View Duration | 2:21 minutes (High engagement) |
- 🏗️ Data Lake & Lakehouse Architecture — Apache Hudi, Iceberg, Delta Lake
- ☁️ AWS Big Data Engineering — Glue, EMR, S3, Step Functions, Lambda
- ⚡ Real-time Data Pipelines — Kafka, Kinesis, Streaming ETL
- 🔧 Performance & Cost Optimization — Query tuning, incremental processing
- 🎓 End-to-End Production Projects — From ingestion to analytics
- 💾 Data Management Best Practices — Table formats, partitioning, clustering
- 🐍 PySpark & Python — Distributed computing, big data processing
Creating 1,800+ videos and 100+ blog posts to democratize data engineering knowledge and help engineers worldwide build better data platforms.
How Zeta Global scales multi-tenant data ingestion with Amazon S3 Tables
Deep dive into production architecture handling 10,000+ Iceberg tables, 55TB+ monthly processing, and multi-tenant data ingestion at scale with sub-10-minute refresh latency
- Experiment: S3 Tables with Incremental Loads up to 520GB — Performance benchmarking at scale
- S3 Tables: Table Maintenance Flexibility with Spark — Advanced maintenance strategies
- Stream Real-Time Data to AWS S3 Tables using Kafka — Production streaming patterns
- Writing to S3 Tables Managed Iceberg Tables Using DuckDB — Interoperability demonstrations
- ElasticSearch Performance Tuning: 80X Faster Searches Case Study — 4.4s → 125ms optimization
| Degree | Field | Institution | Achievement |
|---|---|---|---|
| 🎓 M.S. | Electrical Engineering | University of Bridgeport | 4.0 GPA |
| 🎓 M.S. | Computer Engineering | University of Bridgeport | Best Academic Award |
| 🎓 B.S. | Electronic Engineering | K.J. Somaiya Institute |
🏅 Honors: Best Academic Achievement Award (4.0 GPA) • 3rd Place UB Hackathon • The Builder Award
Data Lakes & Lakehouses • Apache Hudi & Iceberg • AWS Data Platforms • Spark Optimization
Multi-tenant Architecture • Cost Optimization • Content Creation • Technical Speaking
Open Source Collaboration • Mentorship & Career Guidance
I'm actively seeking speaking engagements, technical collaborations, content partnerships, and consulting opportunities around:
- Data Engineering & Lakehouse Architecture
- AWS Big Data Technologies & Cost Optimization
- Building Production Data Platforms at Scale
- Content Creation & Developer Education
Location: New York City Metropolitan Area
- 📹 Created 1,800+ educational videos in a single year — that's averaging 5 videos every single day!
- 🎬 Accumulated 432.9K views and 16.4K hours of watch time helping engineers worldwide
- 📝 Authored 100+ technical blog posts reaching 18.4K presentations and 5.2K views
- 💻 Built 300+ GitHub repositories with production-ready code and tutorials
- 🌍 Teaching data engineering in 4 languages: English, Hindi, Gujarati, and Marathi
- 🏆 Turned data processing from hours to minutes and saved companies $200K+ in cloud costs
- 🚀 Managing 10,000+ Iceberg tables processing 55TB+ monthly in production
- 🎯 Created LakeBoost framework now running 466 jobs daily (11,832/month) in production
432.9K YouTube Views
16.4K Hours Watch Time
46,188 YouTube Subscribers
1,800+ Videos Published
10,922 LinkedIn Followers
5.2K Medium Blog Views
300+ GitHub Repositories
100+ Technical Blog Posts
55TB Data Processed Monthly
$200K+ Cost Savings Delivered
Made with ❤️ and countless hours of code by Soumil Shah
"Building data platforms by day, creating educational content by night, making data engineering accessible 24/7"
Last updated: January 2025