learning apache airflow

Airflow runs a local Web UI where you can monitor and trigger ingestions.
For Apache instances deployed on AWS I think there is a specific URL or UI you have to go to to monitor. [id].us-east-1.airflow.amazonaws.com/

concepts

DAG

Directed Acyclic Graph
secuencia de tareas conectadas entre si

ETL

Extract: DBs, APIs, Files...
Transform: clean, validate and transform
Load: load processed data onto destination systems; internal apps, warehouses, data lakes, dashboards

Operators

unidad basica de trabajo dentro de un DAG, nativo de airflow.
representa una tarea conectada
Hay muchos operadores
ejemplos:

allows you to run tasks that are in kubernetes pods. let's you use different programming languages and access to any sort of tool really