learning apache airflow
- What is Apache Airflow? For beginners
- Airflow en 2025: Orquestación de Datos sin Dolor | José Manuel Martínez
Airflow runs a local Web UI where you can monitor and trigger ingestions.
For Apache instances deployed on AWS I think there is a specific URL or UI you have to go to to monitor. [id].us-east-1.airflow.amazonaws.com/
concepts
DAG
Directed Acyclic Graph
secuencia de tareas conectadas entre si
ETL
Extract: DBs, APIs, Files...
Transform: clean, validate and transform
Load: load processed data onto destination systems; internal apps, warehouses, data lakes, dashboards
Operators
unidad basica de trabajo dentro de un DAG, nativo de airflow.
representa una tarea conectada
Hay muchos operadores
ejemplos:
- sensor de tiempo
- python operator ejecuta python
- bash operator ejecuta script bash
- SQL operators;
- external messaging operators: Slack, email...
- kubernetes op: one of the most powerful ones
allows you to run tasks that are in kubernetes pods. let's you use different programming languages and access to any sort of tool really