Solution to check if a requirement is met without pkg_resources

[HansBug]

Problem description

In the previous pkg_resources package, this issue can be solved with the following code (link):

import pkg_resources
from pkg_resources import DistributionNotFound, VersionConflict

# dependencies can be any iterable with strings, 
# e.g. file line-by-line iterator
dependencies = [
  'Werkzeug>=0.6.1',
  'Flask>=0.9',
]

# here, if a dependency is not met, a DistributionNotFound or VersionConflict
# exception is thrown. 
pkg_resources.require(dependencies)

However, pkg_resources has been deprecated, so this method is not recommended for continued use. In the migration guide provided in the importlib_metadata documentation, there is no new interface that is completely equivalent to pkg_resources.require. Therefore, we need to use the import_metadata library (native in Python 3.8 or higher) and the packaging library to achieve this functionality.

I have currently implemented a version, the code is here: https://github.com/HansBug/hbutils/blob/main/hbutils/system/python/package.py#L202, where the most critical part is the _yield_reqs_to_install function, which checks a requirement and its sub-requirements included in its extra one by one, and enumerates the unsatisfied requirements. By calling this iterator function, the check of whether a requirement is satisfied can be achieved. This is the core code of the function:

def _yield_reqs_to_install(req: Requirement, current_extra: str = ''):
    if req.marker and not req.marker.evaluate({'extra': current_extra}):
        return

    try:
        version = importlib_metadata.distribution(req.name).version
    except importlib_metadata.PackageNotFoundError:  # req not installed
        yield req
    else:
        if req.specifier.contains(version):
            for child_req in (importlib_metadata.metadata(req.name).get_all('Requires-Dist') or []):
                child_req_obj = Requirement(child_req)

                need_check, ext = False, None
                for extra in req.extras:
                    if child_req_obj.marker and child_req_obj.marker.evaluate({'extra': extra}):
                        need_check = True
                        ext = extra
                        break

                if need_check:  # check for extra reqs
                    yield from _yield_reqs_to_install(child_req_obj, ext)

        else:  # main version not match
            yield req

def check_reqs(reqs: List[str]) -> bool:
    """
    Overview:
        Check if the given requirements are all satisfied.

    :param reqs: List of requirements.
    :return satisfied: All the requirements in ``reqs`` satisfied or not.

    Examples::
        >>> from hbutils.system import check_reqs
        >>> check_reqs(['pip>=20.0'])
        True
        >>> check_reqs(['pip~=19.2'])
        False
        >>> check_reqs(['pip>=20.0', 'setuptools>=50.0'])
        True

    .. note::
        If a requirement's marker is not satisfied in this environment,
        **it will be ignored** instead of return ``False``.
    """
    return all(map(lambda x: _check_req(Requirement(x)), reqs))

Of course, if you need this function immediately, you can simply

pip install hbutils>=0.9.0

and then

from hbutils.system import check_reqs
print(check_reqs(['pip>=20.0']))
print(check_reqs(['pip~=19.2']))
print(check_reqs(['pip>=20.0', 'setuptools>=50.0']))

hbutils is a universal toolkit library containing various common functions, and the project address is https://github.com/HansBug/hbutils. It will be maintained for a long time.

[HansBug]

From python/importlib_metadata#450

[smheidrich]

@HansBug Hi! I'm currently trying to get some information about how to best detect & handle missing extras / optional dependencies at runtime into the PyPA's packaging guide (pypa/packaging.python.org#1606) and your check_reqs is the only modern alternative to pkg_resources.require I know of.

The current draft links to both this issue and your hbutils package, but it feels a bit weird to recommend a "giant toolbox of stuff"-type package for this one function. I'm wondering if you'd be open in principle to splitting it off into its own package? If you don't want to do it yourself, I could also do it & make you maintainer afterwards.

Not saying that it should be done now, because it's still possible that PyPA's reviewers will decide the guide shouldn't include this information at all, in which case it'd be pointless effort - but I thought I'd ask here & give a heads-up before making promises over there.

[sinoroc]

Maybe distlib can do this. Within the Python packaging ecosystem, it is a relatively well-known and definitely well-trusted library. If it can not do it out of the box, maybe the amount of custom code necessary would be smaller than what would be necessary when using the combination of importlib.metadata and packaging. I do not know for sure, but maybe it is worth investigating.

[smheidrich]

I had a look at distlib and it doesn't seem like it has anything to make this more comfortable than with importlib.metadata and packaging. I thought maybe something like this might work:

from distlib.locators import DistPathLocator, DependencyFinder
from distlib.database import DistributionPath

req = "typer-slim[standard]"
distpath = DistributionPath(include_egg=True)
locator = DistPathLocator(distpath)
print(locator.locate(req))

But unfortunately it seems to completely ignore the extra in the requirement and always returns a positive result whether it's installed or not.

I then tried

dfin = DependencyFinder(locator)
print(dfin.find(req))

But this goes in the other direction and reports anything as unsatisfied that's part of any extra, whether the extra is part of the requirement or not, plus some other weird stuff like dependencies whose marker is not even fulfilled...

[sinoroc]

I asked: pypa/distlib#234.

[sinoroc]

Cross link:

https://discuss.python.org/t/check-if-an-environments-packages-conflict/67511

[smheidrich]

By the way, another reason why even if this functionality was in distlib, a separate package might still be preferable: distlib's minimum download size is around 500 kB and its installed size almost 2 MB. In many cases that will be about as large or larger than the dependencies the package author is trying to make optional, defeating the purpose. Contrast this with the packaging package whose wheel is only 50 kB. (Neither package has any dependencies of its own so these sizes are total)

[sinoroc]

distlib's minimum download size is around 500 kB and its installed size almost 2 MB. In many cases that will be about as large or larger than the dependencies the package author is trying to make optional, defeating the purpose

Good point

[smheidrich]

@HansBug Another question: It's probably implied by you posting this snippet here for others to use, but just to be sure: Is it fine to include it (or modified versions of it) in the aforementioned guide for people to copy into their code without the restrictions imposed by the Apache 2 license under which hbutils is provided? If not, we'll have to add a license disclaimer on top of the snippet, which I've seen people do before, but looks a bit odd & makes it harder to use (also not sure if the guide's maintainers would accept this).

[vsajip]

distlib's minimum download size is around 500 kB and its installed size almost 2 MB

Yes, and that includes the launchers which are used on Windows only - those are around 750K in size.

I can look at the issue @sinoroc raised on distlib, but if the size is going to be a deal-breaker for your needs, might you not use it, anyway?

[smheidrich]

It would depend on the use case, e.g. if the optional dependency in question is NumPy or PyTorch, pulling in a 500 kB dependency should be worth it to make the experience better.

But just going by the description of distlib's purpose from its README stating

It is intended to be used as the basis for third-party packaging tools

it doesn't sound like the natural place for such functionality, which would be meant to be used in arbitrary packages, not just packaging tools.

[vsajip]

I posted a possible solution on the distlib issue.