[DO NOT LAND] Cherry-pick pip fixes from release/0.2 into main #3462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3462
Note: Links to docs will display an error until the docs builds have been completed. ❌ 4 New FailuresAs of commit 1ae8abe with merge base 5d2a17b ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
dbort
added
the
ciflow/binaries/all
Manually install build requirements because `python setup.py bdist_wheel` does not install them.
setup.py is sometimes run as root in docker containers. buck2 doesn't allow running as root unless $HOME is owned by root or does not exist. So temporarily undefine it while configuring cmake, which runs buck2 to get some source lists. Also, the buck2 daemon can sometimes get stuck on the CI workers. Try killing it before starting the build, ignoring any failures.
Some CI jobs can fail with "OS file watch limit reached" when running buck2. This section should reduce the number of files that it tries to watch.
Change the build-wheels workflow to only fetch the first layer of submodules. ExecuTorch only needs the first layer of submodules to build its pip package, but the `build_wheels_*.yaml` workflows will recursively fetch all submodules by default. Fetching all submodules can also cause `buck2` to fail because it will try to watch too many files. This change makes `buck2` work on the CI runners, speeds up the jobs, and reduces disk/network usage.
Always build the pybindings when building the pip wheel. Always link in XNNPACK.
libtorch.so builds with the old glibc ABI, so we need to as well, for any source files that include torch headers.
pip wheels will need to be able to find the torch libraries. On Linux, the .so has non-absolute dependencies on libs like "libtorch.so" without paths; as long as we `import torch` first, those dependencies will work. But Apple dylibs do not support non-absolute dependencies, so we need to tell the loader where to look for its libraries. The LC_LOAD_DYLIB entries for the torch libraries will look like "@rpath/libtorch.dylib", so we can add an LC_RPATH entry to look in a directory relative to the installed location of our _portable_lib.so file. To see these LC_* values, run `otool -l _portable_lib*.so`.
The executorch build system will ensure that .dylib/.so files have LC_LOAD_DYLIB and LC_RPATH entries that will work when they're installed. Delocating (i.e., making copies of the .dylibs that ET's libs depend on) will break any libs that depend on the torch libraries if users ever import both `torch` and the executorch library. Both import paths must load exactly the same file, not just a copy of it.
This script is run by CI after building the executorch wheel. Before running this, the job will install the matching torch package as well as the newly-built executorch package and its dependencies. For now we test the export of a simple model, and try executing it using the runtime pybindings. Test Plan: ``` ./install_requirements.sh python build/packaging/smoke_test.py ```
Use the logic from https://github.com/pytorch/torcharrow/blob/15a7f7124d4c73c8c541547aef072264baab63b7/setup.py#L21 to play nicely with the pytorch ecosystem CI build environment. Test Plan: ``` $ ./install_requirements.sh ... Successfully installed executorch-0.2.0a0+1ba292a $ python >>> from executorch import version >>> version.__version__ '0.2.0a0+1ba292a' >>> version.git_version '1ba292ae4071c4eede8ea14e8f10ffd973a085b4' >>> ^D $ grep Version /home/dbort/.conda/envs/executorch-tmp/lib/python3.10/site-packages/executorch-0.2.0a0+1ba292a.dist-info/METADATA Metadata-Version: 2.1 Version: 0.2.0a0+1ba292a ``` Temporarily commented out the call to `setup()` in `setup.py` then imported it. ``` $ python >>> from setup import Version >>> Version.string '0.2.0a0+1ba292a' >>> Version.git_hash '1ba292ae4071c4eede8ea14e8f10ffd973a085b4' >>> Version.write_to_python_file("/tmp/version.py") >>> ^D $ cat /tmp/version.py from typing import Optional __all__ = ["__version__", "git_version"] __version__ = "0.2.0a0+1ba292a" git_version: Optional[str] = '1ba292ae4071c4eede8ea14e8f10ffd973a085b4' ``` ``` $ BUILD_VERSION="5.5.5" python >>> from setup import Version >>> Version.string '5.5.5' ```
Fill out the recommended `project` keys, most of which will affect the web page that PyPI will render for the `executorch` package. See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#about-your-project for the latest guidance. Use https://github.com/pytorch/pytorch/blob/a21327e0b03cc18850a0608be2d9c5bd38fd4646/setup.py#L1394 as a guide for the actual values. Add a README-wheel.md file that will be included in the wheel, and will become the main page contents on PyPI. Test Plan: * Installed the package with `./install_requirements.sh` * Looked at the files under ~/miniconda3/envs/executorch/lib/python3.10/site-packages/executorch-0.2.0a0+1a499e0.dist-info. METADATA and LICENSE both contain the new metadata.
When installing the executorch pip package for CI jobs, look on the pytorch servers when resolving dependencies. This lets the executorch package depend on pytorch pre-release and nightly versions. Also run the llava setup with `-x` to make it easier to debug failures.
|
Obsoleted by the stack of PRs at #3477 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is just to test the full stack before importing into diffs.