Airflow 사용기 (1) - 설치

환경변수설정

export AIRFLOW_HOME=~/airflow

source ~/.bashrc

설치

pip install apache-airflow

docker로 설치하기

sudo docker pull puckel/docker-airflow
sudo docker run -d \
-p 8080:8080 \
--name airflow \
docker.io/puckel/docker-airflow

DB 세팅

CREATE DATABASE airflow;

vi airflow.cfg
----------------------------------------
executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@host/db
----------------------------------------
airflow db init

celery 세팅

pip install celery
pip install redis

vi airflow.cfg
----------------------------------------
executor = CeleryExecutor
broker_url = redis://redis_host:redis_port/0
result_backend = db+postgresql://airflow:airflow@host/db
----------------------------------------

airflow.cfg 기타 설정

# sample 제거
load_examples = False

# timezone 세팅
default_timezone = Asia/Seoul

실행

webserver 실행
airflow webserver -p 8080

scheduler 실행
airflow scheduler

celery 실행
airflow celery worker

celery flower 실행
airflow celery flower