Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to run 32-bit armv7 / armhf OS #147

Open
mvgijssel opened this issue Dec 26, 2024 · 8 comments
Open

Unable to run 32-bit armv7 / armhf OS #147

mvgijssel opened this issue Dec 26, 2024 · 8 comments

Comments

@mvgijssel
Copy link

mvgijssel commented Dec 26, 2024 •
edited
Loading

Describe the bug

I'm trying to run a 32-bit arm OS. I changed the arch in the machine configuration as mentioned here but proxmox fails to start the vm with error

TASK ERROR: don't know how to emulate architecture 'arm'

Full vm settings:

arch: arm
bios: ovmf
boot: order=scsi2;net0
cores: 1
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=64M
memory: 1024
meta: creation-qemu=9.2.0,ctime=1735189312
name: pikvm
net0: virtio=BC:24:11:4F:97:E5,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi2: local:iso/debian-12.8.0-armhf-netinst.iso,media=cdrom,size=799020K
scsihw: virtio-scsi-single
smbios1: uuid=c04d3c22-ecb9-4637-8778-712056c23c4a
sockets: 1
unused0: local:100/vm-100-disk-1.raw

The necessary qemu binaries seem to be available

root@proxmox:/etc/pve/nodes/proxmox/qemu-server# ls -la /usr/bin | grep qemu
lrwxrwxrwx  1 root root          19 Dec 15 12:13 kvm -> qemu-system-aarch64
-rwxr-xr-x  1 root root     5246448 Dec 15 12:13 qemu-aarch64
-rwxr-xr-x  1 root root     5246448 Dec 15 12:13 qemu-aarch64_be
-rwxr-xr-x  1 root root     2048928 Dec 15 12:13 qemu-alpha
-rwxr-xr-x  1 root root     3370992 Dec 15 12:13 qemu-arm
-rwxr-xr-x  1 root root     3370992 Dec 15 12:13 qemu-armeb
-rwxr-xr-x  1 root root      662256 Dec 15 12:13 qemu-edid
-rwxr-xr-x  1 root root     4047888 Dec 15 12:13 qemu-hexagon
-rwxr-xr-x  1 root root     2113600 Dec 15 12:13 qemu-hppa
-rwxr-xr-x  1 root root     2746320 Dec 15 12:13 qemu-i386
-rwxr-xr-x  1 root root     2369936 Dec 15 12:13 qemu-img
-rwxr-xr-x  1 root root     2304592 Dec 15 12:13 qemu-io
-rwxr-xr-x  1 root root      989936 Dec 15 12:13 qemu-keymap
-rwxr-xr-x  1 root root     2688368 Dec 15 12:13 qemu-loongarch64
...

Expected behavior
I expect a 64-bit arm hypervisor like Proxmox to be able to run a 32-bit arm virtual machine

ENV (please complete the following information):

  • OS: Debian GNU/Linux 12 (bookworm)
  • ARCH: Linux proxmox 6.6.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux
  • Version [ do pveversion -v]
proxmox-ve: 8.3.0 (running kernel: 6.6.62+rpt-rpi-v8)
pve-manager: 8.3.1+port1 (running version: 8.3.1+port1/fb48e850ef9dde27)
proxmox-kernel-helper: 8.1.0
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.1
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9+port
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0+port
libpve-rs-perl: 0.9.0
libpve-storage-perl: 8.2.9
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.3.0-2
proxmox-backup-file-restore: 3.3.0-2
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.3.2
pve-cluster: 8.0.10
pve-container: 5.2.2+port1
pve-docs: 8.3.1
pve-edk2-firmware: not correctly installed
pve-firewall: 5.1.0
pve-firmware: not correctly installed
pve-ha-manager: 4.0.6
pve-i18n: 3.2.4
pve-qemu-kvm: 9.2.0-1+port2
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.0+port3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.9.0
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
@mvgijssel
Copy link
Author

Seems this was explicitly added in the pimox project: pimox/qemu-server@2d13f27

@mvgijssel
Copy link
Author

When modifying the command to invoke the vm generated by proxmox to use qemu-system-arm the vm actually starts:

From aarch64 (generated by proxmox)

/usr/bin/kvm -id 100 -name pikvm,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=c04d3c22-ecb9-4637-8778-712056c23c4a -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.ms.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-0.qcow2 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password=on -cpu host -m 1024 -readconfig /usr/share/qemu-server/pve-port.cfg -device qemu-xhci,id=ehci,bus=pcie.0,addr=0x1 -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pcie.1,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu-pci,id=vga,bus=pcie.0,addr=0x2 -device virtio-serial,id=spice,bus=pcie.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:eb4f53577652 -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/var/lib/vz/template/iso/debian-12.8.0-armhf-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=io_uring -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:4F:97:E5,netdev=net0,bus=pcie.0,addr=0xb,id=net0,bootindex=101 -machine type=virt-6.0+pve0,gic-version=host

To use 32-bit arm:

/usr/bin/qemu-system-arm -id 100 -name pikvm,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=c04d3c22-ecb9-4637-8778-712056c23c4a -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.ms.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-0.qcow2 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password=on -m 1024 -readconfig /usr/share/qemu-server/pve-port.cfg -device qemu-xhci,id=ehci,bus=pcie.0,addr=0x1 -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pcie.1,addr=0x1 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-kbd,id=keyboard,bus=ehci.0,port=2 -device virtio-gpu-pci,id=vga,bus=pcie.0,addr=0x2 -device virtio-serial,id=spice,bus=pcie.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:eb4f53577652 -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/var/lib/vz/template/iso/debian-12.8.0-armhf-netinst.iso,if=none,id=drive-scsi2,media=cdrom,aio=io_uring -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:4F:97:E5,netdev=net0,bus=pcie.0,addr=0xb,id=net0,bootindex=101 -machine type=virt

@jiangcuo
Copy link
Owner

If kernel enable CONFIG_COMPAT and you machine support 32bit .You can use qemu-system-aarch64 to run arm32 under arm64.

You can exec lscpu to check your cpu. If show 32,Congratulations！You can run KVM accelerated armhf virtual machines.

root@pvetest:~/pve-port# lscpu
Architecture:             aarch64
  CPU op-mode(s):         64-bit |32-bit

A good example is that the Raspberry Pi supports both arm64 and armhf armbian.

The principle is to run the armhf rootfs by starting the aarch64 kernel

@mvgijssel
Copy link
Author

Seems like both modes are supported by the CPU

root@proxmox:/etc/pve/nodes/proxmox/qemu-server# lscpu
Architecture:             aarch64
  CPU op-mode(s):         32-bit, 64-bit
  Byte Order:             Little Endian
CPU(s):                   4
  On-line CPU(s) list:    0-3
Vendor ID:                ARM
  Model name:             Cortex-A72
    Model:                3
    Thread(s) per core:   1
    Core(s) per cluster:  4
    Socket(s):            -
    Cluster(s):           1
    Stepping:             r0p3
    CPU(s) scaling MHz:   100%
    CPU max MHz:          1800.0000
    CPU min MHz:          600.0000
    BogoMIPS:             108.00
    Flags:                fp asimd evtstrm crc32 cpuid
Caches (sum of all):      
  L1d:                    128 KiB (4 instances)
  L1i:                    192 KiB (4 instances)
  L2:                     1 MiB (1 instance)
NUMA:                     
  NUMA node(s):           2
  NUMA node0 CPU(s):      0-3
  NUMA node1 CPU(s):      
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; __user pointer sanitization
  Spectre v2:             Vulnerable
  Srbds:                  Not affected
  Tsx async abort:        Not affected

@mvgijssel
Copy link
Author

mvgijssel commented Dec 26, 2024 •
edited
Loading

Trying to run a VM with the Debian armhf iso (https://cdimage.debian.org/debian-cd/current/armhf/iso-cd/) attached:

bios: ovmf
boot: order=scsi2;net0
cores: 1
cpu: host
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=64M
memory: 1024
meta: creation-qemu=9.2.0,ctime=1735189312
name: pikvm
net0: virtio=BC:24:11:4F:97:E5,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi2: local:iso/debian-12.8.0-armhf-netinst.iso,media=cdrom,size=799020K
scsihw: virtio-scsi-pci
smbios1: uuid=c04d3c22-ecb9-4637-8778-712056c23c4a
sockets: 1

results in the device not being found error:

BdsDxe: failed to load Boot0006 "UEFI QEMU QEMU CD-ROM " from PciRoot (0x0)/Pci (0x5,0)/Scsi (0x0, 0x2): Not found

image

I interpreted this as "the vm doesn't find the iso, because the iso is 32-bit and the vm is 64 bit".

@jiangcuo
Copy link
Owner

Yes.Only aarch64 efi or aarch64 kernel can be boot!
You can install aarch64 debian on /dev/sda . And deboostrap an minimal armhf base on /dev/sdb. Now you can boot to armhf with aarch64 grub.
However, a system compatible with 32-bit CPUs can be used chroot to the armhf rootfs.

@jiangcuo
Copy link
Owner

So using LXC will be a very convenient method

@mvgijssel
Copy link
Author

Yes.Only aarch64 efi or aarch64 kernel can be boot! You can install aarch64 debian on /dev/sda . And deboostrap an minimal armhf base on /dev/sdb. Now you can boot to armhf with aarch64 grub. However, a system compatible with 32-bit CPUs can be used chroot to the armhf rootfs.

Interesting! How does the aarch64 grub “know” how to run the armhf OS? Are there particular settings that need to be applied? I’m trying to boot/create a PiKVM vm image (https://pikvm.org/download/) , don’t think LXC is going to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mvgijssel @jiangcuo