Pine59 migrated its extensive monorepo of hundreds of DAGs from a legacy Airflow setup to Google Cloud’s Managed Airflow (Gen 3) running Airflow 3. The change delivered faster task scheduling, higher processing throughput, and a more stable execution environment, which directly affects engineers responsible for large‑scale data pipelines and ML workloads.
What Changed in the Orchestration Stack
The company replaced its previous orchestration layer with the newly released Managed Airflow (Gen 3) service. This service runs Apache Airflow 3 on a fully managed, abstracted infrastructure. In internal stress‑tests, the Gen 3 environment showed immediate gains in processing speed, task scheduling latency, and overall stability, prompting a full migration of production workloads.
Impact on MLOps and Pipeline Architecture
Data processing continues to rely on BigQuery for heavy‑lift analytics, while model inference was moved to a dedicated Google Kubernetes Engine (GKE) cluster optimized for that purpose. By separating orchestration (Managed Airflow) from inference compute (GKE), Pine59 achieved clearer responsibility boundaries and reduced contention between data transformation and model serving tasks.
Developer Experience and Extensibility
Airflow 3 introduced a revamped UI and a streamlined plugin authoring system. Pine59 quickly built custom plugins that integrate directly into the UI, such as a BigQuery auto‑linkify tool that turns table references in logs and XCom entries into clickable links to BigQuery Studio, and a DAG run configuration search form that simplifies locating specific runs. These extensions improve observability and reduce the time engineers spend debugging pipeline failures.
Operational Considerations
The migration required updating the monorepo to be compatible with Airflow 3’s DAG definition standards and redeploying the DAGs to the Managed Airflow environment. Because the service abstracts the underlying infrastructure, operational overhead for scaling and patching is reduced, but teams must monitor the managed service’s health metrics and ensure that custom plugins remain compatible with future Airflow releases. The separation of inference workloads onto GKE also introduces the need to manage GKE‑specific concerns such as node pool sizing and container image security.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
- Evaluate the performance profile of Managed Airflow (Gen 3) against existing orchestration solutions before committing to a migration.
- Consider isolating heavy ML inference workloads in a dedicated GKE cluster to avoid resource contention with data‑processing jobs.
- Leverage Airflow 3’s plugin framework to add context‑specific tooling that can accelerate debugging and observability.
- Plan for ongoing compatibility testing of custom plugins as Airflow evolves.
- Monitor managed‑service health dashboards to maintain operational awareness without direct infrastructure management.