Skip to content

Switch scalability test with idle Multinet switches

Jump to bottom
Konstantinos Papadopoulos edited this page Oct 28, 2017 · 45 revisions

Test description

This is a switch scalability test with idle switches emulated using multinet. Multinet is a mininet basis topology emulator, able to generate large topologies (order of magnitude ~104), using clustering. The switches are in idle mode, meaning that they do not initiate any traffic to the controller.

The goal is to explore the maximum number of switches the controller can sustain, and how a certain-sized topology should be connected to the controller so that the latter can successfully discover it at the minimum time.

In contrast to the switch scalability test with idle MT-Cbench switches the emulated switches can be arranged in a disconnected, linear, mesh, ring topology type allowing in this sense for more complex topology types. This feature, along with the extended protocol support (OF1.3), constitute a more realistic switch emulator compared to MT-Cbench.

Usage

A switch scalability test using Multinet topology emulator with idle switches, can be started by specifying the following options in NSTAT command line:

  • --test=sb_idle_scalability
  • --sb-generator-base-dir=<Multinet dir>

Under the stress_test/sample_test_confs/<controller_name>/ directory, the JSON files ending in _sb_idle_scalability_multinet can be handled as template configuration files for this kind of test scenario. You can specify them to the --json-config option to run a sample test. For larger-scale stress tests, have a look at the corresponding files under the stress_test/stress_test_confs/<controller_name>/ directory.

Nodes deployment

For this test, the minimum number of nodes needed, are 3.

  • NSTAT node
  • controller node
  • SouthBound emulator node (Multinet). In this case we may have more than one nodes and we use one as a master node, which is the reference node for the communication between NSTAT node and the rest of the nodes of Multinet topology cluster, the worker nodes

In order to deploy these nodes, based on docker containers, we have two options

  • download the prebuilt environment from DockerHub
  • build your own container locally using the provided Dockerfiles for proxy and no-proxy environments, under the path deploy/docker

In both cases, docker has to be installed and any user that will manipulate docker containers, must be added to the docker group. To deploy the required nodes, see installation wiki.

After deployment of docker nodes, update the NSTAT repository using the following steps

  • open a new terminal and execute the command

    docker ps -a

    the output of the above command will be similar to the following

     CONTAINER ID       IMAGE                                  COMMAND               CREATED              STATUS              PORTS    NAMES

 4c05473bb7c8       intracom/nstat-sdn-controllers:proxy   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   controller
 72e4572878e2       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-01
 13d191e6ef75       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-02
 60db64735a26       intracom/nstat:proxy                   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   nstat

    get the container names of all docker containers you created

  • for each docker name execute the following command

       WAIT_UNTIL_RETRY=2
   docker exec -i $container_name /bin/bash -c "rm -rf /opt/nstat; \
      cd /opt; \
      until git clone https://github.com/intracom-telecom-sdn/nstat.git -b master; do \
          echo 'Fail git clone NSTAT. Sleep for $WAIT_UNTIL_RETRY and retry'; \
      done; \
      if [[ $container_name =~ mn ]]; then \
          until service openvswitch-switch start; do \
              echo 'Fail starting openvswitch service. Sleep for $WAIT_UNTIL_RETRY and retry'; \
              sleep $WAIT_UNTIL_RETRY; \
          done \
      fi"

    where you should replace the $container_name with the container names of the corresponded docker node, acquired from previous step. As we can observe all multinet nodes have the mn prefix in their names. In this case we have 2 multinet nodes.

Download the prebuild environment

Edit json test configuration file

The IP addresses of all deployed VMs and the credentials to open SSH connections, must be configured in the json configuration file of the sample test we want to run. This action must be done in nstat node.

The IP addresses of all deployed VMs and the credentials to open SSH connections, must be configured in the json configuration file of the sample test we want to run. This action must be done in nstat_node.

  • Run the command

    docker ps -a

    to get container names of

    • NSTAT node
    • Controller node
    • SouthBound emulator nodes (all multinet nodes Master and worker nodes)

  • Get IP Addresses of all nodes

    docker exec -i $container_name /bin/bash -c "ifconfig"

  • SSH into nstat_node

    ssh root@<NSTAT_node_ip>

    the password to connect is root123.

  • Edit json file /opt/nstat/stress_test/sample_test_confs/boron/boron_sb_active_scalability_multinet.json and change the following lines changing IP addresses and SSH credentials:

    "nstat_node_ip":"<NSTAT_node_ip>",
"nstat_node_ssh_port":"22",
"nstat_node_username":"root",
"nstat_node_password":"root123",

"controller_node_ip":"<Controller_node_ip>",
"controller_node_ssh_port":"22",
"controller_node_username":"root",
"controller_node_password":"root123",

"sb_emulator_name":"MULTINET",
"sb_emulator_node_ip":"<Multinet_master_node_ip>",
"sb_emulator_node_ssh_port":22,
"sb_emulator_node_username":"root",
"sb_emulator_node_password":"root123",

    For the sb_emulator_node_ip we can select one ip address from the SouthBound emulator nodes. The Master can also participate in the list of the worker nodes multinet_worker_ip_list and have a dual role. The only requirement for this setup is to place the master process to listen on a different port for REST requests than the worker process running on the same node with the master. The REST port of the master process is defined from topology_rest_server_port key of the configuration file. For example if we have the following configuration

    "multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[<REST_port_worker_1>, <REST_port_worker_2>, ... ],

    and we assume that sb_emulator_node_ip is equal to the <Worker_1_ip> then topology_rest_server_port should not have the same value with the first element in the list of multinet_worker_port_list

    In json file of the test, we must configure the values of the IP address list and the port list of REST interface of Multinet workers. Multinet workers addresses, are the addresses of SouthBound emulator nodes. There must be one-to-one relation between the multinet_worker_ip_list and the multinet_worker_port_list.

    "multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[3333, 3333, ... ],

    In case of starting more than one worker processes on the same node, we separate them by changing the port number. For example if we want to start 2 worker processes on localhost we would do the following configuration

    "multinet_worker_ip_list":["127.0.0.1", "127.0.0.1", ... ],
"multinet_worker_port_list":[3333, 3334, ... ],

Run the test

In order to run the test

  • Open a new terminal and execute the following command

    docker exec -i nstat /bin/bash -c "export PYTHONPATH=/opt/nstat; source /opt/venv_nstat/bin/activate; \
python3.4 /opt/nstat/stress_test/nstat.py \
  --test=sb_idle_scalability \
  --ctrl-base-dir=/opt/nstat/controllers/odl_boron_pb/ \
  --sb-emulator-base-dir=/opt/nstat/emulators/multinet/ \
  --json-config=/opt/nstat/stress_test/sample_test_confs/boron/boron_sb_idle_scalability_multinet.json \
  --json-output=/opt/nstat/results.json \
  --html-report=/opt/nstat/report.html \
  --output-dir=/opt/nstat/results_boron_sb_idle_scalability_multinet/"

Inspect results

Once test execution is over, inspect the results under

/opt/nstat/results_boron_sb_idle_scalability_multinet

Configuration keys

The configuration keys that must be specified in the JSON configuration file are:

config key type description
nstat_node_ip string IP Address of the NSTAT node
nstat_node_ssh_port string the ssh port of the NSTAT node
nstat_node_username string username for ssh login in the NSTAT node
nstat_node_password string password for ssh login in the NSTAT node
controller_name string name of the used controller. This value is used in Controller Factory method to return the appropriate controller object. For this test it should be ODL
controller_node_ip string IP Address of the Controller node
controller_node_ssh_port string The ssh port of the Controller node
controller_node_username string Username for ssh login in the Controller node
controller_node_password string Password for ssh login in the Controller node
sb_emulator_name string The name of SouthBound emulator. This value is used in Generator Factory method to return the appropriate SouthBound emulator object. For this test it should be MULTINET
sb_emulator_node_ip string IP Address of the Multinet node. Based on the multinet documentation, this is the IP address of the Multinet master node. This configuration key currently is not in use and is reserved for future releases of NSTAT.
sb_emulator_node_ssh_port string The ssh port of the Multinet node
sb_emulator_node_username string username for ssh login in the Multinet node
sb_emulator_node_password string password for ssh login in the Multinet node
sb_emulator_clean_handler string executable that cleans up locally Multinet files cloned from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
sb_emulator_build_handler string executable that clones locally Multinet files from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
controller_build_handler string executable for building controller (relative to --ctrl-base-dir command line parameter)
controller_clean_handler string executable for cleaning up controller directory (relative to --ctrl-base-dir command line parameter)
controller_start_handler string executable for starting controller (relative to --ctrl-base-dir command line parameter)
controller_stop_handler string executable for stopping controller (relative to --ctrl-base-dir command line parameter)
controller_status_handler string executable for querying controller status (relative to --ctrl-base-dir command line parameter)
controller_statistics_handler string executable for changing the period that the controller collects topology statistics (relative to --ctrl-base-dir command line parameter)
controller_persistent_handler string disables persistence of controller. This can be acchieved by adding the attribute persistent=false in file <controller_base_dir>/etc/org.opendaylight.controller.cluster.datastore.cfg
controller_oper_hosts_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of hosts from the operational datastore
controller_oper_links_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of links from the operational datastore
controller_oper_switches_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of switches from the operational datastore
controller_oper_flows_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of flows from the operational datastore
controller_flowmods_conf_handler string configures the controller plugins to respond with flow modifications on any PacketIN message with ARP payload
controller_logs_dir string controllers logs directory (relative to --ctrl-base-dir command line parameter)
controller_get_handler string executable for downloading the controller prebuild version from its repository and extracts it.
controller_port number controller port number where OF switches should connect
controller_restconf_port number controller RESTCONF port number
controller_restconf_user string controller RESTCONF user name
controller_restconf_password string controller RESTCONF password
controller_statistics_period_ms array of numbers controller different statistics period values (in (ms))
topology_rest_server_boot string executable that boots up all REST servers on Multinet master and workers nodes. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_rest_server_stop string executable that stop's all REST servers initiated by Multinet on master and workers nodes. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_server_rest_port number the port that Multinet server will listen to
topology_init_handler string executable that initializes a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_start_switches_handler string executable that stops a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_stop_switches_handler string executable that stops a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_get_switches_handler string executable that retrieves the number of booted switches in a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
multinet_topo_size array of numbers number of Multinet switches per Worker. Total switches is equal to topology_size * number_of_workers
multinet_topo_type array of strings type of Multinet topology {RingTopo, LinearTopo, DisconnectedTopo}
multinet_topo_hosts_per_switch array of numbers number of Multinet hosts per switch
multinet_topo_group_size array of numbers size of a group of switches
multinet_topo_group_delay_ms array of numbers delay between different switches groups (in milliseconds)
multinet_switch_type string the type of software switch, that will be used from our Multinet topology
multinet_worker_ip_list array of strings a list of all IP addresses of worker nodes
multinet_worker_port_list array of numbers a list of port numbers of all REST servers on worker nodes
multinet_traffic_gen_duration_ms number the duration in miliseconds, during which traffic will be generated from topology switches to the controller. This traffic is PacketIN OpenFlow messages which contain ARP payload and they trigger controller to respond with FlowMod OpenFlow messages
multinet_interpacket_delay_ms number the delay between transmited PacketIN OpenFlow messages
java_opts array of strings java options parameters (we use this usually to define java VM memory and garbage collector configuration)
plots array of plot objects configurations for plots to be produced after the test

** All IP addresses described for the configuration keys in the previous table, must be in the same subnet. Also all the ssh users must be able to execute commands in privileged mode passwordless.**

The array-valued configuration keys shown in bold are the test dimensions of the test scenario. The stress test will be repeated over all possible combinations of their values.

The most important configuration keys, which affect the stressing level on the controller, are:

  • topology_size: defines the total number of switches in the network (controls switch scalability).
  • topology_group_size: defines the batch size of switches to be started
  • topology_group_delay_ms: defines the delay between batches of switches

With the parameters of topology_group_size and topology_group_delay_ms we can gradually start a topology of switches and not start them all in once (actually this is the case where topology_group_delay_ms=0)

Plot configuration

See the plotting page.

Result keys

The result keys produced by this kind of test scenario and which can be used subsequently to generate custom plots, are the following:

Result key type description
global_sample_id number unique (serial) ID for this sample
timestamp number unique timestamp for this sample
date string date this sample was taken
discovered_switches_error_code number identification of what went wrong during switch discovery (0 successful switch discovery, 201 maximum polling deadline was reached without any new switch discovery. This deadline resets to its initial value if new switches are discovered from the polling function)
successful_bootup_time number has bootup times of successful test iterations. This is helper to make easy ploting of successful cases. For unsuccessful cases it has -1 value.
bootup_time_secs number time in secs in order to discover switches. We can see the number of discovered switches in discovered_switches result key.
max_discovered_switches number the maximum number of discovered switches recorded in the controller datastore from the polling function.
discovered_switches number the final number of recorded switches recorded from the polling function. If this number is 0 it means that the controller was collapsed before the topology is booted up. Polling function starts after topology bootup. The time we get for bootup_time_secs in this case is the sum of time it took for the topology to bootup plus the discovery deadline time.
multinet_size number number of Multinet switches connected to the controller
multinet_worker_topo_size number number of switches created within a single worker node
multinet_workers number total number of worker nodes, deployed by Multinet
multinet_topology_type string Multinet network topology type {ring, linear, disconnected, mesh}
multinet_hosts_per_switch number number of Multinet hosts per switch
multinet_group_size number size of a group of switches
multinet_group_delay_ms number delay between different switches groups (in (ms))
controller_node_ip string controller IP address where OF switches were connected
controller_port number controller port number where OF switches should connect
controller_java_xopts array of strings controller Java optimization flags (-X)
one_minute_load number one-minute average system load
five_minute_load number five-minute average system load
fifteen_minute_load number fifteen-minute average system load
used_memory_bytes number system used memory in bytes
total_memory_bytes number system total memory in bytes
controller_cpu_shares number the percentage of CPU resources of the physical machine, allocated to the controller process
controller_cpu_system_time number CPU system time for controller
controller_cpu_user_time number CPU user time for controller
controller_num_threads number number of controller threads measured when this sample was taken
controller_num_fds number number of open file descriptors measured when this sample was taken
controller_statistics_period_ms number the interval (in (ms)) of the statistics period of the controller

The result key in bold (successful_bootup_time) is the main performance metric produced by this test scenario. Another important parameter is multinet_size. In cases of successful topology bootup the number of discovered_switches is the same with multinet_size.

Sample experimental results

Sample experimental results for this test are shown below

Intro

Stress Tests

Emulators

Monitoring tools

  • OpenFlow monitoring tools

Design

Releases

ODL stress tests performance reports

Sample Performance Results

Clone this wiki locally