How to install Airflow locally using Docker

mkdir ~/airflow-docker
cd airflow-docker
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.2.3/docker-compose.yaml"
  • airflow-webserver - The webserver is available at http://localhost:8080
  • airflow-worker - The worker that executes the tasks given by the scheduler.
  • airflow-init - The initialization service.
  • flower - The flower app for monitoring the environment. It is available at http://localhost:5555.
  • postgres - The database.
  • redis - The redis - broker that forwards messages from scheduler to worker.
mkdir -p ./dags ./logs ./plugins
  • ./logs - contains logs from task execution and scheduler.
  • ./plugins - you can put your custom plugins here.
echo -e “AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0” > .env
AIRFLOW_UID=50000
docker-compose up airflow-init
docker-compose up# to check if airflow service is running
docker ps
http://localhost:8080
Airflow Signin screen
Airflow logon screen
Airflow Dashboard
Airflow Dashboard
docker exec 'airflow_docker_container_id' airflow version# airflow_docker_container_id -> lets use container id of docker web server
# version -> if you want to get version of airflow instance
# you will get container ID if you hit 'docker ps'
 curl -X GET --user 'airflow:airflow' 'http://localhost:8080/api/v1/dags'# at this point if you will get unauthorised error then make sure you have below environment variable set in docker-composer.yaml file# AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'# then run docker-compose down && docker-compose up# check if airflow service is up or not

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store