How to install Airflow locally using Docker

Install docker composer from here

To deploy Airflow on Docker Compose, you should fetch docker-compose.yaml.

mkdir ~/airflow-docker
cd airflow-docker
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.2.3/docker-compose.yaml"

This file contains several service definitions:

• airflow-scheduler - The scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.
• airflow-webserver - The webserver is available at http://localhost:8080
• airflow-worker - The worker that executes the tasks given by the scheduler.
• airflow-init - The initialization service.
• flower - The flower app for monitoring the environment. It is available at http://localhost:5555.
• postgres - The database.
• redis - The redis - broker that forwards messages from scheduler to worker.

All these services allow you to run Airflow with CeleryExecutor. For more information, see Architecture Overview.

mkdir -p ./dags ./logs ./plugins

Some directories in the container are mounted, which means that their contents are synchronized between your computer and the container.

• ./dags - you can put your DAG files here.
• ./logs - contains logs from task execution and scheduler.
• ./plugins - you can put your custom plugins here.

This file uses the latest Airflow image (apache/airflow). If you need to install a new Python library or system library, you can build your image.

If you are in macOS or Linux you need to make sure that user and group permissions are the same between those folders from your host and the folders in your docker container.

echo -e “AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0” > .env

For other operating systems, you will get warning that AIRFLOW_UID is not set, but you can ignore it. You can also manually create the .env file in the same folder your docker-compose.yaml is placed with this content to get rid of the warning:

AIRFLOW_UID=50000

Now we can initialize our airflow instance with service airflow-init

docker-compose up airflow-init

All the related services are now initialized. Now to run airflow

docker-compose up
# to check if airflow service is running
docker ps

Open a web browser and hit the below localhost URL

http://localhost:8080

Airflow logon screen

Airflow Dashboard

How to interact with Airflow command-line interface

docker exec 'airflow_docker_container_id' airflow version
# airflow_docker_container_id -> lets use container id of docker web server
# version -> if you want to get version of airflow instance
# you will get container ID if you hit 'docker ps'

In order to use API, let's try with some simple example

 curl -X GET --user 'airflow:airflow' 'http://localhost:8080/api/v1/dags'
# at this point if you will get unauthorised error then make sure you have below environment variable set in docker-composer.yaml file
# AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
# then run docker-compose down && docker-compose up
# check if airflow service is up or not

Reference
[1] https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html