Tue, 13 December 202222:56:00 GMT
Why pre-cloud tools are quietly dying
In Why is everyone trying to kill Airflow, Daniel Beach argues that the demise of popular open-source orchestrator Apache Airflow is exaggerated, "calling no-go on the doomsday proclaimers who are out there peddling the end of Airflow as we know it is near."
He argues that while there are lots of things to be unhappy about (read the article for a long list of those), it's fundamentally a good system, there's a strong open-source community around it and some cloud vendors are even providing the system as a managed service.
The trouble is, Airflow is cut from the same cloth as other pre-cloud era tools (the initial release was in 2015, the same year that Kubernetes came out), being both hard to scale down and hard to scale up. As Daniel Beach mentions, Airflow has inspired a range of new tools that were all built on a cloud foundation (typically Kubernetes).
Scaling down
Scaling down means being able to use the same tool in the small. You might want to use Airflow for a small project, as just another cog in the wheel of an application for its sheer utility, or in a system for continous integration.
In Scaling down is hard to do (2000), Mauri Laitinen describes how this is actually fundamentally a difficult problem in software engineering, especially from the perspective of an organization. Running small workflows often amounts to tacit knowledge that's hard to encode in a standardized way.
In more practical terms, the computing overhead of running Airflow at small scale is significant. Systems that were designed to operate in the cloud such as AWS Step Functions or Azure Data Factory are much easier to run at low scale and consequently, are offered at a low price point.
Of course, those systems are not free software, but there is no reason a free software couldn't operate in the same way. Airflow could probably be reengineered to fit better into this computing framework, but it takes a lot of effort to both move a software package forward, introducing new abstractions, and maintain backwards compatibility.
Scaling up
Scaling up means using the same tool to solve bigger problems, different kinds of problems, or lots of small problems.
One way to solve bigger problems with Airflow is to call upon specialized services to do the heavy lifting, letting Airflow handle just the orchestration aspect. But where is the fun in that? And scaling down is still hard.
Astronomer, a commercial company that's betting big on Airflow and contributing significant resources to its development, is pushing for Airflow to do more, providing for example a Python library to make it easier to run ETL workflows within Airflow. There's a lot of competition in this area however, and perhaps users will choose software that doesn't put Airflow front and center, but merely offers it as one out of many backends. Also, the name "Astro" is used liberally by Astronomer to mean different things, suggesting that they're also not quite sure what this value proposition is all about.
Surviving the cloud
Ultimately, Airflow is no distributed compute engine. Activities in Airflow are just Python programs and there are no abstractions to enable pushdown optimizations, parallel processing or pluggable storage. This simplicity is a huge part of Airflow's success, but it's also what it keeps it from solving bigger problems, or more generic problems.
The way software progresses is to find useful abstractions that help solve a class of problems. Airflow is still lacking in this regard, probably deliberately so, but part of the story is that Airflow is written in Python, a language that wasn't designed to support large programs or powerful abstractions, running on top of an operating platform that doesn't provide much functionality.
Running workloads in a cloud computing environment means renting resources efficiently. If a resource isn't used, turn it off. The operating platform or cloud fabric needs to support this sort of orchestration, or you will have to reinvent it at a higher layer, with a substantial overhead.
Not really able to scale down an individual "instance" of Airflow, there is an ongoing effort to implement multi-tenancy, still in a draft state. This work could definitely extend Airflow's shelf life, allowing it to better scale to enterprise needs without the overhead, but the road is long and there are architectural shortcomings that need to be addressed as well.