Proxmox Bug 6040 - Patch Documentation

This repository contains a functional demo, and a following discussion.

The intention is to demonstrate, and then explain the context for bug report 6040 and the patch linked to it.

Table of contents

• Instructions for Interactive Demo
• Discussion and Patch Rationale

Interactive Demo

The purpose of the demo is to demonstrate how exclude patterns work under tar.

Instructions

1. Clone the repository. I prefer to clone all of my repositories to $HOME/repos, which I am assuming below:

   cd ~
   git clone https://github.com/diraneyya/bug_proxmox_6040.git tar_extraction_demo
   cd tar_extraction_demo

2. After cloning the repository, type in the following commands:

   cd ~/repos/tar_extraction_demo
   tar tf file1.tar
   tar tf file2.tar

Tip

The t option stands for "Tree", which lists the file paths inside of the archive. The f option stands for "File", and allows the user to select the tar archive using its filename.

Note

The two files, file1.tar and file2.tar contain the same set of files and directories. Yet, they are different in an important way. Run the two commands above to figure out how.

3. Now is the time to spin up the demo using the demo.sh script:

   cd ~/repos/tar_extraction_demo
   # run the script
   ./demo.sh

4. The script might take a bit of time to run on the first time it is invoked, so please be patient. It might also offer instructions on how to install missing utilities (such as tree or tmux), or refuse to run if the docker command was not found on the local system.

Tip

Docker is used by the script to mimic the Debian-12 environment used by Proxmox when extracting archives.

Note

Unlike tree and tmux, the script does not try to offer advice on how to install Docker.

5. After launching the demo, you must see a horizontal pane at the top, and three vertical panes at the bottom. These are for:

   • The archive's original contents in the content folder. (bottom left)
   • The results of extracting file1.tar to the extracted1 folder using the test_tar_extraction helper function. (bottom middle)
   • The results of extracting file2.tar to the extracted2 folder using the test_tar_extraction helper function. (bottom right)

   

Tip

Initially, the bottom middle and bottom right panes, are empty. This will change as you start running the extraction commands at the top pane. Note that with every extraction, the contents of the folders are wiped out to simulate a fresh start. Hence, if the extraction command fails, then both directories will remain empty.

6. Now let us start by extracting both archives using no exclusion patterns at all:

   test_tar_extraction

Tip

The test_tar_extraction helper will take the arguments you supply to it and channel them to tar for the extraction of both archives (i.e. file1.tar and file2.tar).

Warning

Can you make sense of what you see after running this command? Are you able to see the full tar command being used underneath? was the extraction successful in this case?

7. Now let us try to supply an empty exclusion pattern, as follows:

   test_tar_extraction --exclude

Warning

Can you make sense of what you see running this command? Are you able to see the full tar command being used by test_tar_extraction? was the extraction successful in this case?

8. Now, try some more exclude patterns:

   test_tar_extraction --exclude *
   test_tar_extraction --exclude sample123      # what happened?
   test_tar_extraction --exclude sample123/*    # is this what we want?
   test_tar_extraction --exclude ./sample123    # what happened?
   test_tar_extraction --exclude ./sample123/*  # is this what we want?

Note

The goal here is to exclude the contents of the root sample123 folder in the archive, leaving an empty root directory after the extraction. Note that there is another nested directory with the same name (i.e. [./]sample789/sample123), which needs to be extracted as-is.

Tip

If you are wondering why supplying an exclusion pattern of just sample123 does not exclude the non-root (i.e. nested) directory, then try using the --no-anchored option:

test_tar_extraction --no-anchored --exclude sample123
# "tar xf file?.tar -C extracted? --anchored --no-anchored --exclude sample123"
# note that the supplied "--no-anchored" cancels the "--anchored" option built
# into the `test_tar_extraction` helper function.

9. Finally. Now you are able to take everything you learnt above and apply it to create a tar exclusion pattern that successfully excludes the root sample123 folder in both archives:

   test_tar_extraction --exclude sample123/* --exclude ./sample123/* # uses --anchored
   # a smarter alternative, which expands to the same
   test_tar_extraction --exclude={,./}sample123/*

Cleaning Up

Before moving on to the discussion part of this README file. You can exit the demo using Ctrl+B, followed by D. After that, you can clean the environment using the following clean-up command:

cd ~/repos/tar_extraction_demo
# run the cleaning script
./demo.sh clean

Discussion

Current Code

This relevant line of the current codebase excludes the root dev folder's contents when extracting the root filesystem for an LXC container in Proxmox:

    push @$cmd, '--exclude' , './dev/*';

The only issue here, is that it assumes that the tarball has paths starting with a dot slash (similar to file1.tar), rather than without a dot slash (similar to file2.tar).

Inquiry

Below is a table showing different root filesystem images and converters, and assessing their output's compatibility with Linux containers in Proxmox.

The idea here is that anything that resembles a Linux root-filesystem archive, deserves an opportunity to be leveraged as a Linux container template —should this be, at all, possible.

Project	Archive Path Prefix	Root dev Folder?	Before Patch	After Patch
Ubuntu1	None	Populated	❌	✅
Alpine2	./	Empty	✅	✅
Linux Containers3	./	Empty	✅	✅
docker2lxc4	./	Limited	✅	✅
sqfs2tar5	None	Populated	❌	✅

Warning

This table is a work in progress.

Rationale for the Patch

Tip

This section is improved by ChatGPT.

As virtualization evolved from VMs to containers, Proxmox users increasingly seeked to adapt existing VM workloads for Linux containers. A critical step in this process involves obtaining a root filesystem archive that encapsulates the base operating system. Such archives, however, vary depending on their origins:

1. Legacy Archives: Derived from VM-oriented systems or live system images (e.g., squashfs archives).

   • Tend to include fully populated root folders (e.g. /dev).
   • Often omit a leading ./ prefix in the paths within the archive.

2. Modern Archives: Designed with containerization in mind or for specific containerization technologies (e.g., Docker).

   • Typically contain minimal or empty root folders, including an empty or a minimally populated /dev.
   • Consistently include a ./ prefix in the paths within the archive.

These differences create challenges in Linux container creation, particularly with populated /dev directories and paths without a dot slash. Linux containers manage their own root /dev folder, and hence, a fully populated /dev in the archive's root leads to failure in container creation.

The Current Situation

The existing Proxmox codebase addresses this by excluding the /dev root directory during archive extraction. However, the current exclusion pattern (--exclude ./dev/*) assumes archive paths that begin with dot slash ./. While this works for modern archives (which, generally speaking, do not have fully populated root folders), it is ineffective for legacy archives lacking the dot slash ./ path prefix. Consequently, users attempting to repurpose such archives must manually repackage them —a labor-intensive and unnecessary process.

The Proposed Modification

The proposed edit is rather simple, which is to change the exclusion pattern from ./dev/* to dev/*, to accommodate the legacy archives in which this failure is most likely to occur.

Alternatively, it is possible (and cheap) to accommodate both archive types, by using two exclusion patterns (--exclude={,./}dev/*).

This approach eliminates the need for repackaging and ensures compatibility with a broader range of root filesystem archives.

The Patch

https://lore.proxmox.com/pve-devel/mailman.80.1736016466.441.pve-devel@lists.proxmox.com/T/#u

Footnotes

1. https://cloud-images.ubuntu.com/ (Look files ending with -root.tar.xz) ↩

2. https://alpinelinux.org/downloads/ (Look for Mini Root Filesystems) ↩

3. https://images.linuxcontainers.org/ (Look files named rootfs.tar.xz) ↩

4. Using docker2lxc it was possible to convert this 10GB universal:2 Microsoft Docker container to an LXC template, which worked. ↩

5. https://cdimage.debian.org/debian-cd/current-live/amd64. Using a live system ISO, the sqfs2tar utility was used to convert the squashfs filesystwm image to tar, which was then gzipped to create an LXC template. ↩