How to install Airflow locally using Docker
Install docker composer from here
To deploy Airflow on Docker Compose, you should fetch docker-compose.yaml.
mkdir ~/airflow-docker
cd airflow-dockercurl -LfO "https://airflow.apache.org/docs/apache-airflow/2.2.3/docker-compose.yaml"
This file contains several service definitions:
airflow-scheduler
- The scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.airflow-webserver
- The webserver is available athttp://localhost:8080
airflow-worker
- The worker that executes the tasks given by the scheduler.airflow-init
- The initialization service.flower
- The flower app for monitoring the environment. It is available athttp://localhost:5555
.postgres
- The database.redis
- The redis - broker that forwards messages from scheduler to worker.
All these services allow you to run Airflow with CeleryExecutor. For more information, see Architecture Overview.
mkdir -p ./dags ./logs ./plugins
Some directories in the container are mounted, which means that their contents are synchronized between your computer and the container.
./dags
- you can put your DAG files here../logs
- contains logs from task execution and scheduler../plugins
- you can put your custom plugins here.
This file uses the latest Airflow image (apache/airflow). If you need to install a new Python library or system library, you can build your image.
If you are in macOS or Linux you need to make sure that user and group permissions are the same between those folders from your host and the folders in your docker container.
echo -e “AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0” > .env
For other operating systems, you will get warning that AIRFLOW_UID
is not set, but you can ignore it. You can also manually create the .env
file in the same folder your docker-compose.yaml
is placed with this content to get rid of the warning:
AIRFLOW_UID=50000
Now we can initialize our airflow instance with service airflow-init
docker-compose up airflow-init
All the related services are now initialized. Now to run airflow
docker-compose up# to check if airflow service is running
docker ps
Open a web browser and hit the below localhost URL
http://localhost:8080
How to interact with Airflow command-line interface
docker exec 'airflow_docker_container_id' airflow version# airflow_docker_container_id -> lets use container id of docker web server
# version -> if you want to get version of airflow instance
# you will get container ID if you hit 'docker ps'
In order to use API, let's try with some simple example
curl -X GET --user 'airflow:airflow' 'http://localhost:8080/api/v1/dags'# at this point if you will get unauthorised error then make sure you have below environment variable set in docker-composer.yaml file# AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'# then run docker-compose down && docker-compose up# check if airflow service is up or not
Reference
[1] https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html