Description

This module is an alternative Airflow XCom backend.

Details

Airflow XCom is a way Tasks exchange values in a DAG. By default the values are stored in the default Airflow database. This makes it unsuitable for sharing large volumes of data for two reasons:

• PostgreSQL has a limit of 1GB per XCom value
• Using the same database as other Airflow data may degrade performance

For these reasons Airflow allows for implementing a custom XCom backend. This module is one such implementation for use with Redis.

Requirements

Currently the module uses the Python Redis module which is already provided by Airflow.

The only requirements is for Airflow to have a Redis Connection defined called xcom_cache.

TODO

Possible future improvements:

• Get Connection ID from configuration or environment variables
• Splitting of values into different keys to avoid size limits