NAVER details Python fork and GC issues from its Airflow 3.0 PoC, along with fixes and upstream contributions
Announced October 8, 2026
What happened
NAVER's data ingestion platform team published a two-part write-up on its PoC migrating from Airflow 2.10.2 to 3.0. It explained that both rising worker memory, caused by Airflow creating task processes via fork, and intermittent dag-processor restarts in a MySQL environment stemmed from the interaction of Python GC and fork. It also covered how the team traced the causes, fixed them, and contributed upstream.
Why it matters
It is a practical reference for teams looking to move large-scale batch pipelines to Airflow 3.