You may have been struggling to gain a clear, comprehensive view of your Apache Airflow workflows but the good news is that you can now address that with the Alex Apache Airflow Scanner. Tracking data lineage and understanding the relationships between your DAGs, tasks, and datasets is a common data engineering and data governance challenge, so you can imagine how excited we are to introduce a new Airflow Metadata Scanner.
Like all Alex Metadata Scanners, the Airflow Scanner is a powerful tool designed to bridge any gaps in your understanding of your data landscape where Airflow might be in use. It is designed to work with the Alex Solutions Data Catalog.
Why should you care?
Data technology landscapes can be very complex and data-driven businesses need a comprehensive understanding of data lineage in relation to all the data landscape components. This understanding is essential to understanding important data governance factors like data quality, how your business meets its compliance obligations, and effective data technology troubleshooting. The Airflow Metadata Scanner joins dozens of other metadata scanners in empowering you to:
-
Centralize your understanding of your Airflow Assets by automatically populating the Alex Catalog with your DAGs, datasets, and tasks, thus creating a unified view of your data ecosystem.
-
Visualize Data Lineage where Airflow is part of the data journey and thereby gain invaluable insights into how data flows through your Airflow workflows with clear lineage visualizations.
-
Enhance Airflow data asset searchability in as much as you can easily locate and analyze Airflow assets within the Alex platform using an Airflow related filter.
-
Bridge gaps in your understanding between Airflow and other data assets in your Data Catalog. The scanner will allow you to connect your Airflow environment directly to your Alex Solutions Data Catalog, ensuring consistent and continuously updated metadata.
Apache Airflow Dependency Graph
Alex Data Lineage Service View of Airflow
Contact Us to see how the scanner seamlessly integrates with the Alex platform, brings the ability for data lineage visualization, and learn how Airflow dependency graphs and data lineage graphs differ in their form and functionality.
Get Started with the Airflow Field Scanner today to unlock the full potential of your Airflow environment
Prerequisites:
-
Python 3.6 or later.
-
Required Python libraries (requests, airflow_client, scanner.py).
-
Access to an Apache Airflow instance with the REST API enabled.
-
Access to your own installation of the Alex Solutions Data Catalog.
