In this conversation, we go deep into the reality behind the AI hype, what makes engineers valuable (it’s not writing more code), and why the future of the field looks very different from what most people think.

We talk about the shift from coding to system thinking, why fundamentals matter more in the age of AI, and how junior engineers will need to evolve as tools like Claude and ChatGPT take over the “grunt work.”

Victor also shares hard-earned lessons from teaching, startups, consulting, and building systems at AWS, along with practical advice for engineers looking to stand out in a crowded, uncertain job market.

This is not a hype conversation. It’s a real look at where things are going.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

🔑 What We Cover

* Why AI is making fundamentals more important, not less

* The biggest mistake engineers make is chasing promotions

* How to actually become a high-impact engineer

* Why does doing more Jira tickets not matter

* What’s broken about today’s interview process

* The future of junior engineers in an AI world

* Tactical vs strategic engineering (and why it matters)

* Why most AI-generated code is still “low quality.”

* How to think about career growth in a weird job market

💡 Key Takeaway

The best engineers aren’t the ones writing the most code—they’re the ones who understand systems, think long-term, and can drive decisions.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Yordan’s story starts like many of mine, early programming, gaming, PHP, Linux servers—but what makes this conversation interesting is how he evolved from a generalist software engineer into a data engineering leader without even realizing it at first.

We spend a lot of time digging into what actually matters in modern data engineering, and it’s not the tools.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We talk about:

* Why the industry went too far into complexity and is now swinging back toward simplicity

* The reality of running a data platform at scale (and why most teams waste time maintaining tools instead of delivering value)

* How to think about migrations the right way without breaking everything

* The difference between junior, mid, and senior engineers—and why ambiguity tolerance and impact matter more than coding ability

* Why “perfect” engineering is a trap and how to actually ship work that matters

We also get into AI, and Yordan has one of the more grounded takes you’ll hear right now. Most companies aren’t even close to ready for AI, and the idea that it’s replacing engineers anytime soon misses the bigger problem: messy data, unclear metrics, and weak foundations.

Check out Yordan’s Substack below!

We also talk about:

* How AI is actually used on real teams today (not Twitter hype)

* Why juniors with AI can be risky without strong processes

* How to think about code reviews, testing, and slowing down when it matters

On top of that, we dig into content creation, Substack, and what it takes to stand out in a world full of generic AI-generated content. Yordan’s approach is simple: write from real experience or don’t write at all.

This is one of those conversations that cuts through a lot of noise and gets back to fundamentals, how to think, how to build, and how to grow as an engineer in a rapidly changing space.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Jacob didn’t start in tech the way most people expect. He began in accounting, working with Excel and financial systems, before slowly realizing that the real problem he loved solving wasn’t finance, it was data pipelines. That path eventually led him deep into SQL Server, data warehousing, and ultimately to DuckDB, a tool that fundamentally changed how he thought about processing data.

* What we get into is bigger than just tools, though.

We talk about why DuckDB exploded in popularity, what it gets right that traditional databases and even modern cloud warehouses struggle with, and why the industry may be swinging back toward simplicity after years of over-engineered “modern data stacks.”

There’s a really interesting thread here around how engineers accidentally created too much complexity, and now tools like DuckDB are winning by removing it.

We also go deep on the evolution of the data stack itself. From SQL Server’s “everything in one box” model, to the unbundled chaos of the modern stack, and now back toward a more unified, simpler approach. Jacob shares how MotherDuck is thinking about that shift and where things are headed next.

* One of the more important parts of this conversation is around AI.

There’s a strong argument here that AI doesn’t kill data engineering; it massively expands it. Instead of fewer queries being written, we may be heading toward a world where AI agents generate orders of magnitude more queries than humans ever could. That flips a lot of assumptions on their head, especially around things like data modeling, which suddenly becomes more important, not less.

We also talk about:

* Why most Spark workloads are overkill

* When single-node tools like DuckDB actually win

* The real tradeoffs behind Lakehouse architectures

* Why data modeling is still critical in an AI-driven world

* How engineers should think about building in 2026 and beyond

This is one of those conversations that helps you zoom out and see where things are actually going, not just what tools are trending this week.

If you’re building data platforms, experimenting with AI, or just trying to simplify your stack, this one is worth your time.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Ben shares how he went from working in kitchens… to data engineering… to Facebook… and eventually walking away from it all to build his own consulting business.

And yeah, it wasn’t all glamorous.

“I was making the same money as Facebook… and I hated my life.”

We get into the stuff most people don’t talk about:

* What it’s actually like working in Big Tech

* Why high-paying jobs can still burn you out

* How he transitioned into consulting (and what people get wrong)

* The reality of modern data stacks and tool sprawl

* Whether data engineering is changing because of AI

* Why fundamentals still matter more than ever

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We also go deep on where the industry is heading:

* Is the “modern data stack” breaking down?

* Are tools like DuckDB actually replacing warehouses?

* Is data modeling dead… or just not trendy anymore?

* What AI is really changing (and what it’s not)

If you’re trying to break into data, grow your career, or figure out where things are headed… this is one of the more honest conversations you’ll hear.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Ben also runs a course and community for those interested in getting into consulting.

In this episode, I sit down with Matthew Housley, a famed co-author of Data Engineering Fundamentals and former CTO of Ternary Data, to talk about what actually makes data teams successful and why so many organizations get it wrong despite having modern stacks, cloud platforms, and expensive dashboards.

* Matthew’s path is a little different than most. He started in academia as a mathematics instructor before moving into industry as a data scientist at Overstock.com, and eventually leading data strategy and analytics as a CTO. That mix of academic rigor and real-world execution gives him a very clear perspective on where things break down.

We get into the gap between data science and real business impact, why analytics foundations matter more than flashy models, and what companies consistently underestimate when building out data platforms. We also talk about what it actually looks like to transition from academia to industry, and how that shapes how you think about data problems at scale.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

If you’ve ever felt like your data stack should be delivering more value than it is, this conversation will probably hit close to home.

⏱️ Topics we cover:

* Why most analytics efforts fail before they even start

* The difference between “doing data” and delivering value

* Data science vs data engineering vs analytics reality

* Academic thinking vs industry execution

* What CTOs actually care about when it comes to data

* Building foundations that don’t fall apart six months later

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

In this episode of the Data Engineering Central Podcast, I sit down with Neil Roberts — a developer who’s been through every major wave of the web, from BASIC on an Atari to modern TypeScript, and now deep into LLMs and agentic workflows.

This is not another surface-level “AI will change everything” conversation. We get into what is actually happening right now, where it works, where it completely breaks, and what developers are getting wrong.

* We talk about why front-end and UX matter more than ever in an AI world, why most people misunderstand agents, and what real day-to-day workflows with LLMs actually look like.

* There’s also a hard look at who benefits from AI, who falls behind, and whether we are quietly building fragile systems that we don’t fully understand.

If you’re a developer trying to figure out where this is all going, this is one of those conversations worth paying attention to.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Expect to learn:

* Why AI is as much a UX problem as it is a backend problem

* What “agents” actually mean in practice, not in demos

* Where LLM workflows are useful today and where they fail hard

* Whether junior developers should be worried or excited

* How building apps changes when AI is part of the system

* What developers should actually be doing right now to stay relevant

Neil also has a podcast, The Skill Tree, on AI and agentic-specific topics.

We also get into a bigger question most people are avoiding:

* Are we heading toward AI-assisted coding… or AI-orchestrated systems where developers become supervisors?

* And maybe more importantly… which side of that shift do you want to be on?

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Andreas shares his full journey from industrial IoT and working at Bosch to building one of the largest data engineering education platforms in the world, training over 2,000 students and reaching more than 100,000 engineers globally. We get into what production data systems actually look like, why most learning paths are broken, and how AI is reshaping the role of the modern data engineer.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

* We also dig into the uncomfortable truths. AI can write code, but it cannot replace thinking. Most engineers focus too much on tools and not enough on problem-solving, system design, and communication. That gap is only getting bigger.

If you are trying to figure out how to stay relevant in data engineering, or you are just getting started and want to avoid years of wasted effort, this conversation will change how you think about your career.

Today’s podcast is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.

What we cover:

* Why most data engineers are learning the wrong things

* The shift from coding to problem-solving and system design

* How AI is actually changing data engineering workflows

* Why courses and tutorials are becoming less effective

* The difference between real production systems and “toy projects.”

* The future of data engineering jobs and whether AI will replace them

* Why fundamentals still matter more than ever

One of the biggest takeaways is simple. The tools will keep changing, but the problems stay the same. The engineers who win are those who understand systems, ask better questions, and connect business problems to real solutions.

Links:

* Learn Data Engineering Academy:

https://learndataengineering.com

* Andreas Kretz on LinkedIn

* Andreas Kretz on YouTube

* Sponsor: https://estuary.dev

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In this episode, Chris Gambill explains his storied career in tech and data through the years, dealing with data at Fortune 500 company scale, and breaking out on his own.

We cover career growth, what separates senior engineers from true strategic operators, and the biggest mistakes people make early on. We discuss the classic problems that have plagued data teams for decades and why it’s all still a struggle.

Today’s podcast is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.

We also dig into Databricks vs Snowflake, what matters and what doesn’t, and how to think about modern data architecture without falling for marketing hype.

* On the AI side, we talk about why most LLMs, in the context of developer lifecycles, have changed how we do data, and also about what human skills cannot be replaced.

If you care about leveling up beyond just building pipelines, this one is for you.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We break down what those systems actually looked like, including ingestion, modeling, and the realities of batch vs real-time in industrial environments.

We also dive into:

* AI Workflows for Developers

* His work as a committer on Delta Lake

* Where lakehouse architecture works and where it falls short

* The transition into Developer Relations at Databricks

This is a grounded, practical conversation about what actually matters when building data platforms.

Today’s podcast is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.

You can find Robert on LinkedIn and GitHub, below.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Come follow me on YouTube!!

* Not to jump to another job.

* Not to start a company.

* But to take a step back from everything.

Kevin shares his full journey—from growing up in the suburbs of Atlanta to building a career at Apple, and ultimately reaching the point where he could walk away financially and mentally.

You can follow along with Kevin below.

We dive deep into what it’s really like working in tech: the high salaries, the lifestyle creep, the pressure, and the surprising reality that even people making great money often have no clear financial plan.

This conversation also explores the rise of FIRE (Financial Independence, Retire Early), how Kevin discovered it through Mr. Money Mustache, and why his perspective on it has changed over time.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

What starts as a path to freedom can easily turn into a scarcity mindset—and that’s something most people don’t talk about.

We also get into:

* Why high income does not equal financial freedom

* The hidden trap of lifestyle inflation in tech

* The simple investing strategy that actually works (and why most people ignore it)

* Why many engineers are “close” to freedom—but never pull the trigger

* The psychology of money, status, and why people stay stuck

* How a failed project and burnout became a turning point

* And how Kevin went from overworked and unhealthy… to climbing mountains and preparing to backpack 1,000 miles

This is not your typical “get rich quick” or “retire at 30” conversation. It’s a grounded, honest look at money, work, and what it actually takes to build a life you don’t need to escape from.

If you work in tech, think about FIRE, or just feel like you’re stuck on the treadmill, this one will hit home.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We start with his early journey into tech—how he first discovered large-scale data systems and the lessons he learned from working with real-world Spark workloads.

* The conversation then turns toward the future of data engineering, particularly the growing role of AI in software development and data infrastructure. We discuss why generic AI coding assistants often struggle with complex distributed systems, whether AI will eventually be able to automatically optimize data pipelines, and how the role of the data engineer may evolve in the coming years.

We covered a lot of career advice for new and upcoming data professionals.

We also discuss the origin of DataFlint, a tool designed to help engineers better understand and optimize Spark workloads by analyzing execution plans, logs, and runtime context.

If you work with Spark, large-scale data pipelines, or modern data platforms, this conversation will give you a deeper look into how the data engineering landscape is evolving.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Matt has spent decades building and architecting data platforms across technologies such as SQL Server, Oracle, DB2, Hadoop, Redshift, and BigQuery, and now focuses on modern tools such as DuckDB and single-node analytics.

We discuss how the data industry has evolved, what actually makes data platforms succeed, and where tools like DuckDB, Polars, Databricks, and Snowflake fit into the future of analytics.

We also dive into the impact of AI on coding and data engineering, and whether distributed compute clusters will remain dominant — or if more workloads will move toward high-performance single-node systems.

Topics Covered

* Matt’s early career and journey into data engineering

* The evolution of data warehousing and ETL frameworks

* Traditional enterprise data systems vs modern cloud platforms

* DuckDB and the rise of single-node analytics

* Polars vs DuckDB: where each tool shines

* Databricks vs Snowflake

* AI-assisted coding and its impact on engineers

* The current data engineering job market

* Lessons learned from decades of building data systems

* Writing a book on DuckDB

We talk about how he first got into software, the early jobs and tools that shaped his thinking, and the massive technology shifts he’s witnessed across decades of engineering—from early stacks and tools to today’s AI-assisted workflows.

* We also dive into the difference between coding and real-world software engineering, what separates junior, senior, and principal engineers, and why many developers misunderstand what it takes to grow in this field.

* We discuss leadership vs individual contributor paths, the origin of his Coding Challenges platform, why algorithm puzzles dominate developer culture, and what actually makes engineers improve quickly.

Finally, we tackle the big question everyone is asking right now: how AI is reshaping software engineering, and what skills will matter most over the next decade.

We cover:

• How Yuki got into tech• Early career lessons and pivots• Tools and technologies he’s worked with over the years• How data engineering has evolved• The impact of AI on software development• What engineers should focus on right now• Advice for those building their careers in data

Yuki shares practical insights on navigating the industry, staying adaptable, and thinking long-term about your technical growth.

If you’re a data engineer, aspiring engineer, or just interested in where AI and modern software are going, this one’s for you.

Yuki writes on …

LinkedIn - https://www.linkedin.com/in/yukikakegawa/

https://yukikakegawa.me/#blog

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

🔔 Subscribe for more interviews with leaders in data engineering, AI, and modern data platforms.

We cover:

* How Bart got into tech and distributed systems

* His journey through different engineering roles

* Spark internals and why they still matter

* The realities of lakehouse architecture

* Streaming vs batch systems

* AI’s impact on data engineering

* What engineers should focus on in 2026

In a world obsessed with abstractions and AI tooling, we explore whether understanding the internals is still worth it — or if the game has fundamentally changed.

If you’re a data engineer, architect, or platform leader trying to navigate the next phase of the lakehouse era, this one’s for you.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

—

🎙️ Data Engineering Central PodcastHosted by Daniel Beach

If you’re a CTO or data leader looking for help building or optimizing your data platform, reach out — consulting inquiries welcome.

Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

This isn’t a buzzword-heavy DevOps chat. It’s a grounded, honest discussion between two engineers about how people really get into tech, how careers evolve over time, and why modern infrastructure is as much about systems thinking and human judgment as it is about tools.

We talk through Maxine’s journey from early technical curiosity to hands-on DevOps work, dealing with “ClickOps” to automation-first infrastructure, and how writing and teaching reshaped the way she thinks about engineering.

What we cover in this episode:

* 🛠️ From ClickOps to DevOps — what that transition actually looks like in the real world

* 🧠 Why DevOps is fundamentally about systems and people, not just pipelines and YAML

* 📚 How Maxine went from self-teaching to authoring practical guides like LLMs for Humans and The DevOps Career Switch Blueprint

* 🤯 Common mistakes engineers make when learning DevOps, cloud, and distributed systems

* 🔍 Testing failures, production realities, and where modern infrastructure still breaks down

* 🤖 What AI and LLMs actually change for engineers, and what’s mostly hype

* 🧭 Career advice for engineers without a traditional background

* 🔮 Where DevOps and platform engineering are heading over the next 3–5 years

Throughout the conversation, Maxine brings a refreshing, human-centered perspective to topics that are often over-abstracted or oversold. We dig into the tradeoffs behind tooling choices, the reality of production systems, and the importance of learning how to think, not just what to deploy.

If you’re navigating a DevOps or infrastructure career, wrestling with modern stacks, or trying to make sense of AI’s role in engineering, this episode offers clarity, context, and hard-won insight.

Learn more about Maxine’s work:

* Writing & guides:

* LinkedIn: https://www.linkedin.com/in/maxinemeurer/

* Gumroad resources: https://mameurer.gumroad.com

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We dive into:

* 🧠 How the role of software engineering has shifted with the rise of distributed, real-time systems

* 📊 Why event streaming and platforms like Kafka aren’t just messaging systems, but the backbone of modern data architectures

* 🚀 How the community’s tooling and mental models have had to evolve — from static databases and nightly jobs to continuous, always-on streaming applications

* 🤖 A candid look at how AI and real-time data are intersecting, shaping both tooling and expectations for the next decade

* 🔮 Robin’s perspective on where the industry is headed — beyond buzzwords toward real engineering maturity

Along the way, we get historical context, real-world lessons from conference stages and community forums, and a perspective on building resilient, scalable systems that power today’s data-rich applications.

If you’ve ever wondered how we got from batch jobs to continuous event streams, or what it really takes to build modern pipelines that support AI workflows, this conversation with Robin is a must-listen.

For more from Robin:

* 📍 His personal blog & talks:

https://rmoff.net/

* 🔗 LinkedIn profile: https://www.linkedin.com/in/robinmoffatt

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Tyler is a long-time open-source contributor to projects such as delta-rs. You can watch him on YouTube, read his blog, or work directly with him through his consultancy, Buoyant Data.

Tyler has spent years deep in the open-source data ecosystem, contributing to projects such as Delta Lake and thinking critically about how real-world data systems are built and maintained. This isn’t a hype-driven conversation—it’s a grounded discussion about what’s working, what’s breaking, and what’s coming next.

We dig into:

* What the Lakehouse architecture gets right—and where it still falls short

* Why multimodal data (text, images, audio, video, embeddings) changes everything

* How open table formats like Delta Lake fit into the next generation of data platforms

* The growing gap between data tooling hype and day-to-day data engineering reality

* What skills and architectural thinking will matter most for data engineers over the next decade

If you’re building or operating modern data platforms—and trying to separate real signal from noise—this episode is for you.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We talk candidly about:

* What actually matters in modern data and analytics

* Why so much “data content” misses the mark

* The difference between noise and real signal

* What works (and doesn’t) when building a technical audience

* Writing, consistency, and credibility in the data space

* Why opinions + experience beat trends and buzzwords

If you’re a data engineer, analyst, or technologist who’s curious about both building better data systems and communicating ideas that resonate, this episode goes deep on the lessons learned from doing both.

This is less about hacks—and more about craft, judgment, and long-term thinking.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Andy’s career didn’t start in software at all. It started with physical circuits, literally wiring systems as an electrician, before moving into programming, databases, and eventually decades of hands-on data engineering work.

This conversation isn’t about trends or hype cycles. It’s about how the fundamentals of data work have evolved, what hasn’t changed, and what you only learn after years of building, breaking, fixing, and rebuilding real systems.

We talk about how the industry got here, how tools have changed, where they haven’t helped as much as advertised, and what newer data engineers can learn from a long, practical career spent close to the metal.

If you’re interested in perspective, experience, and lessons earned the hard way — this one’s for you.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

I’m joined by Thomas Horton a longtime friend and one of the most well-rounded data professionals I know. Over the course of his career, Tom has worn just about every hat in data: developer, DBA, analyst, and everything in between. He’s lived through the era of on-prem databases, the rise of analytics, and the constant reinvention that defines modern data engineering today.

We talk about what’s changed, what hasn’t, and why many of the “new” problems in data feel oddly familiar. We also dig into lessons learned the hard way, lessons that are just as relevant for early-career data engineers as they are for seasoned practitioners navigating today’s ever-expanding stacks.

On a personal note, a huge portion of what I know about relational databases and analytics can be traced back to Tom. This conversation is part reflection, part history lesson, and part reality check on where the data industry is headed next.

* If you’re interested in the past, present, and future of data—and what really matters beneath all the tooling, this is an episode you won’t want to miss.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

We dig into open-source ecosystems, Lakehouse architectures, the evolution of Spark, streaming, what’s broken and what’s working in today’s data tooling, and the lessons Scott has learned scaling platforms at some of the biggest companies in the world.

If you care about data engineering, architecture, OSS, or the future of the modern data stack, you’ll love this one.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Make sure to follow Scott here on Substack, and over on GitHub.

* Cluster Fatigue

* The Death of Open Source

Going to be a great show, come along for the ride!

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

Hello! A new episode of the Data Engineering Central Podcast is dropping today, we will be covering a few hot topics!

* Apache Iceberg Catalogs

* new Boring Catalog

* new full Iceberg support from Databricks/Unity Catalog

* Databricks SQL Scripting

* DuckDB coming to a Lake House near you

* Lakebase from Databricks

Going to be a great show, come along for the ride!

Thanks …

Anywho, be that as it may, I didn’t want you to miss my pretty face, so here is a video of me ranting about Apache Iceberg, something I’ve had a lot of practice doing and enjoy quite thoroughly.

For all you free-loaders out there, you can get 20% off to celebrate Memorial Day.

https://dataengineeringcentral.substack.com/Merica

It’s time for another episode of the Data Engineering Central Podcast. In this episode, we cover …

* Rust-based tool called UV to replace pip and poetry etc

* Apache X-Table and the Future of the Lake House

* How is AI going to affect you?

Thanks for being a consumer of Data Engineering Central; your support means a lot. Please share this podcast with your friend…

* AWS Lambda + DuckDB and Delta Lake (Polars, Daft, etc).

* IAC - Long Live Terraform.

* Databricks Data Quality with DQX.

* Unity Catalog releases for DuckDB and Polars

* Bespoke vs Managed Data Platforms

* Delta Lake vs. Iceberg and UinFORM for a single table.

Thanks for b…

It's a good one, buckle up!

* Apache Airflow vs Databricks Workflows

* End-of-Year Engineering Planning for 2025

* 10 Billion Row Challenge with DuckDB vs Daft vs Polars

* Raw Data Ingestion.

As usual, the full episode is available to paid subscribers, and a shortened version to you free loaders out there, don’t worry, I still love you though.

* Should you use DuckDB or Polars?

* Small Engineering Changes (PR Reviews)

* Daft vs Spark on Databricks with Unity Catalog (Delta Lake)

* Primary and Foreign keys in the Lake House

Enjoy!

Welcome to Episode 02

In today’s episode, we will talk about the following topics from the Data Engineering perspective …

* Using OpenAI’s o1 Model to do Data Engineering work

* Lord Save us from more ETL tools

* Rust for the small things

* Hosted (SaaS) vs Build

Welcome to Episode 01

In today’s episode we will talk about the following topics from the Data Engineering perspective …

* Snowflake vs Databricks.

* Is Apache Spark being replaced??

* Notebooks in Production. Bad.