Packing conda environments for the Raspberry Pi

[stephane-caron]

If you have configured your Upkie as a Wi-Fi access point, chances are, you have already faced the issue of installing conda packages without an Internet connection. One solution is to connect to the Internet via a separate Wi-Fi dongle, but it will become tedious if the need arises too often. In what follows, we propose a way to use the Internet connection of your host computer instead.

⚠️ Update: Packing pixi environments for the Raspberry Pi works better (details below).

With a Makefile

Here is a set of Makefile rules that:

• Download conda packages for the Raspberry Pi locally on your host computer
• Pack them into an archive
• Upload the archive to your Upkie
• SSH into your Upkie and unpack the archive to get a corresponding conda environment

Assumptions

• PROJECT_NAME is set (as in the main upkie Makefile)
• We use Micromamba on both sides (likely easy to port to Mamba but I haven't tested)

Host-side rules

clean:  ## clean up temporary files
	rm -f conda_env.tar.gz

conda_env.tar.gz:
	${MAMBA_EXE} env create -f environment.yaml -n raspios_$(PROJECT_NAME) --platform linux-aarch64 -y
	tar -zcf conda_env.tar.gz -C ${MAMBA_ROOT_PREFIX}/envs/raspios_$(PROJECT_NAME) .
	${MAMBA_EXE} env remove -n raspios_$(PROJECT_NAME) -y

.PHONY: pack_conda_env
pack_conda_env: conda_env.tar.gz  ## prepare conda environment to install it offline on your Upkie

Upkie-side rule

.PHONY: unpack_conda_env
unpack_conda_env:  ### unpack conda environment to remote conda path
	-${MAMBA_EXE} env list | grep $(PROJECT_NAME) > /dev/null && ${MAMBA_EXE} env remove -n $(PROJECT_NAME) -y
	mkdir -p ${MAMBA_ROOT_PREFIX}/envs/$(PROJECT_NAME)
	tar -zxf conda_env.tar.gz -C ${MAMBA_ROOT_PREFIX}/envs/$(PROJECT_NAME)

See also

This simple method is not as general as conda-pack, but conda-pack has the following caveat as of writing this post:

The OS where the environment was built must match the OS of the target. This means that environments built on Windows can’t be relocated to Linux.

Unpacking an environment allows it to run on the robot, but we are only creating a virtual environment there, which does not qualify as unleashing the full power of conda 😉 (for instance we don't sync packages).

[traversaro]

See also

This simple method is not as general as conda-pack, but conda-pack has the following caveat as of writing this post:

The OS where the environment was built must match the OS of the target. This means that environments built on Windows can’t be relocated to Linux.

I just found this discussion on Google researching similar topics, in case someone is familiar/used to pixi, a possible alternative for conda-pack is pixi-pack that instead explicitly supports, cross-platform packs see https://github.com/Quantco/pixi-pack/blob/v0.2.2/README.md#cross-platform-packs .

[stephane-caron]

Thank you @traversaro 👍 We are now using pixi-pack in the PPO balancer and it reduced the size of the compressed environment from 3.3 GB to 230 MB 🤩

Here is a separate note for further discussions: Packing pixi environments for the Raspberry Pi.