-
Notifications
You must be signed in to change notification settings - Fork 2
Qemu
This document is not the ultimate version. Some of the referred links are to commits which were branch heads at the time of writing this document. It is more an internal documentation for the cloudsuite team to be informed of what was done to run the benchmarks inside Qemu.
It would be most like the instructions here but it has changed a bit and there are several additional things to be aware of.
- When installing the basic dependencies you may be prompted that your package manager is unable to install
python-software-propertieswhich you can ignore, - If you are using
ubuntu 18Hoss and I found that you can not build our Qemu using it, yet. You can use kind of virtual environment to achieve. I used the followingDockerfileto build a Docker image and build the Qemu inside it. All you have to do is to follow Docker documentation and mount the Qemu directory from our repository to the Docker container:
FROM ubuntu:16.04
RUN sed -Ei 's/^# deb-src/deb-src/' /etc/apt/sources.list
RUN apt-get update -qq
RUN apt-get install -y \
build-essential checkinstall wget sudo \
python-dev software-properties-common \
pkg-config python-software-properties zip zlib1g-dev unzip libbz2-dev \
libtool git-core
RUN apt-get --no-install-recommends -y build-dep qemu
RUN mkdir -p /root/Downloads
WORKDIR /root/Downloads
RUN wget https://github.com/git-lfs/git-lfs/releases/download/v2.5.1/git-lfs-linux-amd64-v2.5.1.tar.gz
RUN tar -xf git-lfs-linux-amd64-v2.5.1.tar.gz
RUN chmod +x install.sh && ./install.sh && git lfs install | grep -q 'Git LFS initialized.'
ENV GCC_VERSION "5"
RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test
RUN apt-get update -qq
RUN apt-get -y install gcc-${GCC_VERSION} g++-${GCC_VERSION}
RUN update-alternatives --install /usr/bin/gcc gcc \
/usr/bin/gcc-${GCC_VERSION} 20
RUN update-alternatives --install /usr/bin/g++ g++ \
/usr/bin/g++-${GCC_VERSION} 20
RUN update-alternatives --config gcc
RUN update-alternatives --config g++
ENV BOOST "boost_1_59_0"
ENV BOOST_VERSION "1.59.0"
RUN wget http://sourceforge.net/projects/boost/files/boost/${BOOST_VERSION}/${BOOST}.tar.bz2 -O /tmp/${BOOST}.tar.gz
RUN tar -xf /tmp/${BOOST}.tar.gz
WORKDIR /root/Downloads/${BOOST}
RUN ./bootstrap.sh --prefix=/usr/local
RUN ./b2 -j8
RUN ./b2 install
- When cloning the repository and building the Qemu you have to change the commands to the following:
git clone --branch dev git@github.com:parsa-epfl/qflex.git
cd qflex
git submodule update --init --recursive
cd images
git checkout master
git lfs fetch
git lfs pull
cd qemu
./configure --target-list=aarch64-softmmu \
--enable-flexus --disable-werror --enable-extsnap --disable-tpm
make -j
-
If everything goes OK you can find the built executable in
qemu/aarch64-softmmurelative to theqflexrepository root, -
I had to manually add the appropriate old apt repository and install
libiscsi-2in order to run the Qemu executable.
In order to run Docker inside Qemu you have to use the ubuntu image from here which is head of the branch install-docker-update as of today. You have to do git checkout to the specific version in images sub-module and then use git lfs fetch; git lfs pull; to pull the big image files.
Remember to use initrd.img-4.4.0-83-generic-DockerInstalled as the initrd option for Qemu. We will go through how to use the image files to run Qemu instances in next sections.
You can use the qemu-system-aarch64 executable and run the instance by passing the image files as arguments. But it is easier to use the preset scripts from our repository. You can follow the instructions here to use our preset scripts to run the Qemu instance, using this method you only need to configure a file to point to the Qemu executable and image files and then run the instance using a single command.
To run multiple Qemu instances you can use mrun which is a python script developed by @katetsu. It is highly under development so make sure to be synchronized with the team before using it. First we had problem with the edge version of mrun to run multiple instances with ns3 network simulation enabled. Hoss fixed the issue and we used the mrun version from here for the reset of the work. The version mentioned is the head of the branch mrun_fix as of today.
After you checked out the version mentioned above you have to configure mrun scripts to run multiple instances. mrun has a good help which you can find out by doing ./mrun -h. If this is the very first time you are using mrun you have to do several pip installs to install its dependencies. To generate the sample configuration files simply do ./mrun -g. There will be two files generated. One file which is the setup file is pointing to the other as the instance configuration. You can have as many instance configuration files as you want and point to them in the setup configuration file.
Here there are sample configuration files which I used for the rest of the work. You can see one setup file pointing to two instance files. We configured some port numbers for each of the instances to be different. You can see the serial port number set for the q1 instance to 5555 for example, which will cause mrun to run the instance with a serial console enabled listening on 5555 so you can use telnet localhost 5555 to connect to the console of the machine.
<!-- qemu-setup-sample-file.xml -->
<setup>
<instance>
<file>/home/aasgari/Documents/qflex/scripts/mrun/q1.xml</file>
</instance>
<instance>
<file>/home/aasgari/Documents/qflex/scripts/mrun/q2.xml</file>
</instance>
</setup>
<!-- /home/aasgari/Documents/qflex/scripts/mrun/q1.xml -->
<instance>
<executable>
<binary>/home/aasgari/Documents/qflex/qemu/aarch64-softmmu/qemu-system-aarch64</binary>
</executable>
<parameter>
<enabled>on</enabled>
<option>-machine</option>
<arg>virt</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-cpu</option>
<arg>cortex-a57</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-smp</option>
<arg>4
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-m</option>
<arg>2000</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-kernel</option>
<arg>/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/vmlinuz-4.4.0-83-generic
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-initrd</option>
<arg>/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/initrd.img-4.4.0-83-generic-DockerInstalled</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-append</option>
<arg>'root=/dev/sda2'
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-global</option>
<arg>virtio-blk-device.scsi=off</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>virtio-scsi-device,id=scsi
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-drive</option>
<arg>
file=/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/ubuntu-16.04-lts-blank.qcow2,id=rootimg,cache=unsafe,if=none
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>scsi-hd,drive=rootimg</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-rtc</option>
<arg>driftfix=slew</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-serial</option>
<arg>telnet:localhost:5555,server,nowait</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-name</option>
<arg>q1</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-accel</option>
<arg>tcg,thread=single</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-nographic</option>
<arg>
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-netdev</option>
<arg>user,id=net1,hostfwd=tcp::2222-:22</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>virtio-net-device,mac=52:54:00:00:02:12,netdev=net1</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-exton</option>
<arg></arg>
</parameter>
</instance>
<!-- /home/aasgari/Documents/qflex/scripts/mrun/q2.xml -->
<instance>
<executable>
<binary>/home/aasgari/Documents/qflex/qemu/aarch64-softmmu/qemu-system-aarch64</binary>
</executable>
<parameter>
<enabled>on</enabled>
<option>-machine</option>
<arg>virt</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-cpu</option>
<arg>cortex-a57</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-smp</option>
<arg>4
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-m</option>
<arg>2000</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-kernel</option>
<arg>/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/vmlinuz-4.4.0-83-generic
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-initrd</option>
<arg>/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/initrd.img-4.4.0-83-generic-DockerInstalled</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-append</option>
<arg>'root=/dev/sda2'
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-global</option>
<arg>virtio-blk-device.scsi=off</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>virtio-scsi-device,id=scsi
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-drive</option>
<arg>
file=/home/aasgari/Documents/qflex/images/ubuntu-16.04-blank/ubuntu-16.04-lts-blank.qcow2,id=rootimg,cache=unsafe,if=none
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>scsi-hd,drive=rootimg</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-rtc</option>
<arg>driftfix=slew</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-serial</option>
<arg>telnet:localhost:5556,server,nowait</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-name</option>
<arg>q2</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-accel</option>
<arg>tcg,thread=single</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-nographic</option>
<arg>
</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-netdev</option>
<arg>user,id=net1,hostfwd=tcp::2221-:22</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-device</option>
<arg>virtio-net-device,mac=52:54:00:00:02:13,netdev=net1</arg>
</parameter>
<parameter>
<enabled>on</enabled>
<option>-exton</option>
<arg></arg>
</parameter>
</instance>
Finally you can use the following command to run the multiple Qemu instances using mrun:
./mrun -r qemu-setup-sample-file.xml -qmp
First go to the ns3 directory which is 3rdparty/ns3 and then run the following commands. The first command may warn you of some disabled features but as long as the networking works it is fine.
./waf configure —enable-examples —enable-sudo
./waf
You have to run the following:
sudo ./mrun -r qemu-setup-sample-file.xml -qmp -ns /home/aasgari/Documents/qflex/3rdparty/ns3
in which /home/aasgari/Documents/qflex/3rdparty/ns3 should be the path to the ns3 directory in the repository. This is not updated in the mrun help as of today.
Remember to initialize eth0 interfaces for the Qemu instances from inside like this:
sudo ifconfig eth0 10.10.10.10/24 in one of them and sudo ifconfig eth0 10.10.10.11/24 in another.
To make sure the networking is working you can use netcat and telnet to see if you are able to establish TCP connections. In some cases your host machine may ban some network functionalities because we are using a special simulated network architecture. If your iptables deny TCP connections for example, you may get into trouble. More information can be found in this issue.
It is informative to show you the simulated network architecture for the simplest scenario:
+-----------------+ +-----------------+ +-------------------+
| | | | | |
| Instance 1 | | ns3 script | | Instance 2 |
| | | +--------+ | | |
+-eth0------------+ +----|--------|---+ +--------------eth0-+
+--|--------------++------------|-+ +-|------------++----------------|--+
| | || | | | | || | |
| | tap-0 || tap-0-ns | | | | tap-1-ns || tap-1 | |
| | || | | | | || | |
+--|--------------++------------|-+ +-|------------++----------------|--+
| +------+ | | +------+ |
| | | | | | | |
| | br-0 | | | | br-1 | |
+----------------------------+ +------------------------------+
| | | |
+------+ +------+
First we tried to use our docker installed ubuntu image to run the memcached benchmark and create an image with the updated cloudsuite. We had issues running it. One was the one mentioned before in this section. Another was that I was building an old version of qemu, so be careful not to do this again. And the last one was an issue with our code added to qemu which is described here. According to the last issue I will describe how to create the benchmark images using the qemu dev version and with ./configure --target-list=aarch64-softmmu --enable-extsnap configuration command.
According to what described in the above section, first compile qemu using this configuration command:
./configure --target-list=aarch64-softmmu --enable-extsnap.
There are two ways to create snapshots. One is the old way using a single Qemu node described in this example.
The second which is using mrun is described in this README.