Apache Airflow and the Future of Data Engineering

Recently I had the opportunity to do a collaborative blog post with Maxime Beauchemin about Apache Airflow and the future of data engineering.

Maxime is a data engineer at Airbnb and creator of their data pipeline framework, Apache Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale.

Here's a teaser:

Introduction

Every once in a while I read a post about the future of tech that resonates with clarity.

A few weeks ago it was The Rise of the Data Engineer by Maxime Beauchemin, a data engineer at Airbnb and creator of their data pipeline framework, Apache Airflow. At Astronomer, Apache Airflow is at the very core of our tech stack: our integration workflows are defined by data pipelines built in Apache Airflow as directed acyclic graphs (DAGs). A post like that gives validation as to why right now is the best time for a company like Astronomer to exist.

After reading the post, I reached out to Max about doing an interview post, and to my delight he entertained the request with thoughtful answers to our questions about Apache Airflow and the future of data engineering. You’ll find his answers below, but first I’d like to add a little context.

You might be wondering, “What is data engineering and why does it matter?”

...

You can view the entire post here:

Apache Airflow and the Future of Data Engineering: A Q&A

Discuss this post on Hacker News


Special thanks to Maxime, Laurel Brunk, Chris Hendrixson, and the Astronomer team for their support with this post.


Updated July 18, 2017 - Changed Airflow references to Apache Airflow


If you enjoyed this post, enter your email below to subscribe to future posts.

Taylor D. Edmiston

Taylor is a startup software engineer, minimalist, and sneakerhead. He's in the top 15% of software developers for 2016 on Stack Overflow having reached over 125k people.

Cincinnati, Ohio https://www.tedmiston.com

Subscribe to Taylor D. Edmiston

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!