Updating Requirements Files

Procedure to validate updates to requirements.txt

Updating the dependency versions pinned in requirements.txt and notebook_requirements.txt might be trickier than it seems. Unlike the dependency specs in setup.py, requirements files are supposed to specify the complete environment, including indirect dependencies. Changing a dependency version might require cascades of other dependency updates (e.g. a newer version of requests might also require a newer version of chardet). In addition to possibly changing other existing requirements, a dependency update might also add new dependencies that aren't already listed in our requirements file. So to keep our requirements files complete and self-consistent, more care is needed than just bumping a single requirement and calling it a day.

These steps determine if any indirect dependencies are missing, or if they are present if they need their version bumped, to have the requirements files specify a complete environment:

1. Create and activate a new empty environment: conda create -n rdtools-install python=3.7 and conda activate rdtools-install
2. Bump the dependency version of interest in the requirements file.
3. Install the packages in the two requirements files: pip install -r requirements.txt and, if needed, pip install -r docs/notebook_requirements.txt
4. Run the check_reqs.py script below, which will summarize the differences between the package versions listed in the requirements files and the packages that are actually installed.

check_reqs.py

import subprocess
import os
import pandas as pd

def parse_requirements(text):
    lines = text.strip().split('\n')
    lines = [line.strip() for line in lines if not line.startswith('#')]
    requirements = dict([line.split("==") for line in lines])
    return requirements


def read_requirements(filename):
    with open(filename, 'r') as f:
        text = f.read()
    return parse_requirements(text)


def diff(old, new):
    changes = {}
    for key in set(old.keys()) | set(new.keys()):
        old_version = old.get(key, None)
        new_version = new.get(key, None)
        if old_version != new_version:
            changes[key] = (old_version, new_version)
    return changes


official_reqs = read_requirements('requirements.txt')
official_nb_reqs = read_requirements(os.path.join('docs', 'notebook_requirements.txt'))
official_reqs = {**official_reqs, **official_nb_reqs}

current_reqs = parse_requirements(subprocess.run(['pip', 'freeze'], capture_output=True, text=True).stdout)
df = pd.DataFrame(diff(official_reqs, installed_reqs), index=['requirements file(s)', 'installed'])

sort_insensitive = lambda index: pd.Index(map(str.lower, index))
df = df.T.sort_index(key=sort_insensitive)
print(df)