pip install cyvlfeat doesn't work sometimes

[AbdealiLoKo]

When setting up travis, i realized pip install cyvlfeat doesn't just work.

It needs cython and numpy installed beforehand. While this is reasonable, it would be nice to be able to just run pip install cyvlfeat and not worry about such things ( That's what package managers are for :) )

setuptools has a setup_requires which I think is meant for this ? (I haven't used it before)
I found h5py's setup adds cython in setup_requires and also this SO question is something which tries to do the same with numpy.

[AbdealiLoKo]

I just realized that this works fine in conda (as mentioned in the readme), so this is only a pypi/pip thing

[patricksnape]

Yes, unfortunately this is a bit of a complicated issue. It's one of the many reasons that pip is really not a good venue for shipping compiled extensions, particularly when they require third party dependencies (such as vlfeat).

For example, numpy is shipped in wheels and is also distributed by conda. It is probably quite important to you that your numpy is optimised since its the cornerstone of the scientific Python stack. Unfortunately, if you use the various tricks that allow you install numpy as part of setup_requires, you will build numpy from scratch (you won't receive the wheel) and this build will likely be unoptimized.

Cython is something I could handle not requiring, though it makes the setup.py quite complicated. Numpy really is tough because you almost definitely don't want to build it from scratch.

Finally, satisfying the compiled vlfeat dependency is also quite complicated and would require the ability to pass arguments pertaining to the path of the library and include files. This I could also sort out. It still wouldn't just work, however, as you'd need to have downloaded and built vlfeat yourself beforehand.

[patricksnape]

List of changes required to allow easily pip installing cyvlfeat:

• Refactor extension building logic to allow for Cython to not be installed.
  • Ship c/c++ code with sdist
• Use the Numpy trick from pandas to allow numpy in the setup_requires

class build_ext(_build_ext):
    def build_extensions(self):
        numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')

        for ext in self.extensions:
            if hasattr(ext, 'include_dirs') and not numpy_incl in ext.include_dirs:
                ext.include_dirs.append(numpy_incl)
        _build_ext.build_extensions(self)

• Add either arguments --vlfeat-include-dir=/--vlfeat-library-dir or environment variables VLFEAT_INCLUDE_DIR/VLFEAT_LIBRARY_DIR to set the path to vlfeat include and library dirs.

[jabooth]

I made a start on grouping the logic for pandas-style setup.py into a single-file module that can be placed next to setup.py on this branch https://github.com/jabooth/menpo/tree/optcython

The idea being - we write this properly once, and then can reuse that logic across all packages (like versioneer). Perhaps cyvlfeat is a nice testing ground for it?

That branch is a little noisy as it includes a bunch of changes internal to menpo - the key thing to look at is just setup.py:

https://github.com/jabooth/menpo/blob/optcython/setup.py

and extmanager.py:

https://github.com/jabooth/menpo/blob/optcython/extmanager.py