Skip to content

Flow scalability test with idle Multinet switches

Jump to bottom
Konstantinos Papadopoulos edited this page Oct 28, 2017 · 33 revisions

Test description

This is a flow scalability test, with NorthBound clients (apps) installing flows on the switches of a Multinet topology. With this test one can investigate both capacity and timing aspects of flows installation via the controller NB (RESTCONF) interface. Specifically, the targets metrics that can be explored are:

  • maximum number of flows the controller can handle from the NB interface
  • time required to install a certain number of flows (or alternatively, the flow installation throughput)

This test uses the NorthBound flow generator to create flows in a scalable and configurable manner (number of flows, delay between flow creation). To emulate conditions of an ever increasing stressing towards the controller NB interface, the generator issues flows concurrently from multiple worker threads, which mimic real-world NorthBound clients. The number of these workers is also configurable.

The flows are being written to the controller configuration DataStore via its NB (RESTCONF) interface, and then forwarded to an underlying OpenFlow switch topology as flow modifications. The test verifies the success or failure of flow operations via the controller's operational DataStore.

The supported types of flow operations are

  • flow additions
  • flow deletions

Flow additions are performed if flow_delete_flag is not used as a command option, so it takes its default value, False. Otherwise, flow deletions are performed. Flow deletion operations requires the existence of installed flows on the switches, so an operation of flow addition must always preceded the activation of flow_delete_flag. For more details on configuration options of this type of test, see section of Configuration keys .

For the generation of network topology on which the flow operations will be performed, multinet is used.

Usage

A flow scalability test can be started by specifying the following options in NSTAT command line:

  • --test=nb_active_scalability
  • --nb-generator-base-dir=<NorthBound flow generator base dir>
  • --sb-generator-base-dir=<Multinet base dir>

Under the stress_test/sample_test_confs/<controller_name>/ directory, the JSON files ending in _nb_active_scalability_multinet can be handled as template configuration files for this kind of test scenario. You can specify them to the --json-config option to run a sample test. For larger-scale stress tests, have a look at the corresponding files under the stress_test/stress_test_confs/<controller_name>/ directory.

Nodes deployment

For this test, the minimum number of nodes needed, are 4.

  • NSTAT node
  • controller node
  • NorthBound generator node (nbgen)
  • SouthBound emulator node (Multinet). In this case we may have more than one nodes and we use one as a master node, which is the reference node for the communication between NSTAT node and the rest of the nodes of Multinet topology cluster, the worker nodes

In order to deploy these nodes, based on docker containers, we have two options

  • download the prebuilt environment from DockerHub
  • build your own container locally using the provided Dockerfiles for proxy and no-proxy environments, under the path deploy/docker

In both cases, docker has to be installed and any user that will manipulate docker containers, must be added to the docker group. To deploy the required nodes, see installation wiki.

After deployment of docker nodes, update the NSTAT repository using the following steps

  • open a new terminal and execute the command

    docker ps -a

    the output of the above command will be similar to the following

     CONTAINER ID       IMAGE                                  COMMAND               CREATED              STATUS              PORTS    NAMES

 4c05473bb7c8       intracom/nstat-sdn-controllers:proxy   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   controller
 72e4572878e2       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-01
 13d191e6ef75       intracom/multinet:proxy                "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   mn-02
 60db64735a26       intracom/nstat:proxy                   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   nstat
 2b7ca294d1b4       intracom/nbgen:proxy                   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp   nbgen

    get the container names of all docker containers you created

  • for each docker name execute the following command

       WAIT_UNTIL_RETRY=2
   docker exec -i $container_name /bin/bash -c "rm -rf /opt/nstat; \
      cd /opt; \
      until git clone https://github.com/intracom-telecom-sdn/nstat.git -b master; do \
          echo 'Fail git clone NSTAT. Sleep for $WAIT_UNTIL_RETRY and retry'; \
      done; \
      if [[ $container_name =~ mn ]]; then \
          until service openvswitch-switch start; do \
              echo 'Fail starting openvswitch service. Sleep for $WAIT_UNTIL_RETRY and retry'; \
              sleep $WAIT_UNTIL_RETRY; \
          done \
      fi"

    where you should replace the $container_name with the container names of the corresponded docker node, acquired from previous step. As we can observe all multinet nodes have the mn prefix in their names. In this case we have 2 multinet nodes.

Download the prebuild environment

Edit json test configuration file

The IP addresses of all deployed VMs and the credentials to open SSH connections, must be configured in the json configuration file of the sample test we want to run. This action must be done in nstat node.

The IP addresses of all deployed VMs and the credentials to open SSH connections, must be configured in the json configuration file of the sample test we want to run. This action must be done in nstat_node.

  • Run the command

    docker ps -a

    to get container names of

    • NSTAT node
    • Controller node
    • Nbgen node (NorthBound generator)
    • SouthBound emulator nodes (all multinet nodes, Master and worker nodes)

  • Get IP Addresses of all nodes

    docker exec -i $container_name /bin/bash -c "ifconfig"

  • SSH into nstat_node

    ssh root@<NSTAT_node_ip>

    the password to connect is root123.

  • Edit json file /opt/nstat/stress_test/sample_test_confs/boron/boron_nb_active_scalability_multinet.json and change the following lines changing IP addresses and SSH credentials:

    "nstat_node_ip":"<NSTAT_node_ip>",
"nstat_node_ssh_port":"22",
"nstat_node_username":"root",
"nstat_node_password":"root123",

"controller_node_ip":"<Controller_node_ip>",
"controller_node_ssh_port":"22",
"controller_node_username":"root",
"controller_node_password":"root123",

"sb_emulator_name":"MULTINET",
"sb_emulator_node_ip":"<Multinet_master_node_ip>",
"sb_emulator_node_ssh_port":22,
"sb_emulator_node_username":"root",
"sb_emulator_node_password":"root123",

"nb_emulator_name":"NB-GENERATOR",
"nb_emulator_node_ip":"<Nbgen_node_ip>",
"nb_emulator_node_ssh_port":22,
"nb_emulator_node_username":"root",
"nb_emulator_node_password":"root123",

    For the sb_emulator_node_ip we can select one ip address from the SouthBound emulator nodes. The Master can also participate in the list of the worker nodes multinet_worker_ip_list and have a dual role. The only requirement for this setup is to place the master process to listen on a different port for REST requests than the worker process running on the same node with the master. The REST port of the master process is defined from topology_rest_server_port key of the configuration file. For example if we have the following configuration

    "multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[<REST_port_worker_1>, <REST_port_worker_2>, ... ],

    and we assume that sb_emulator_node_ip is equal to the <Worker_1_ip> then topology_rest_server_port should not have the same value with the first element in the list of multinet_worker_port_list

    In json file of the test, we must configure the values of the IP address list and the port list of REST interface of Multinet workers. Multinet workers addresses, are the addresses of SouthBound emulator nodes. There must be one-to-one relation between the multinet_worker_ip_list and the multinet_worker_port_list.

    "multinet_worker_ip_list":["<Worker_1_ip>", "<Worker_2_ip>", ... ],
"multinet_worker_port_list":[3333, 3333, ... ],

    In case of starting more than one worker processes on the same node, we separate them by changing the port number. For example if we want to start 2 worker processes on localhost we would do the following configuration

    "multinet_worker_ip_list":["127.0.0.1", "127.0.0.1", ... ],
"multinet_worker_port_list":[3333, 3334, ... ],

Run the test

In order to run the test

  • Open a new terminal and execute the following command

    docker exec -i nstat /bin/bash -c "export PYTHONPATH=/opt/nstat; source /opt/venv_nstat/bin/activate; \
python3.4 /opt/nstat/stress_test/nstat.py \
  --test=nb_idle_scalability \
  --ctrl-base-dir=/opt/nstat/controllers/odl_boron_pb/ \
  --sb-emulator-base-dir=/opt/nstat/emulators/multinet/ \
  --nb-emulator-base-dir=/opt/nstat/emulators/nb_generator/ \
  --json-config=/opt/nstat/stress_test/sample_test_confs/boron/boron_sb_idle_scalability_multinet.json \
  --json-output=/opt/nstat/results.json \
  --html-report=/opt/nstat/report.html \
  --output-dir=/opt/nstat/results_boron_nb_active_scalability_multinet/"

Inspect results

Once test execution is over, inspect the results under

/home/vagrant/nstat/results_boron_nb_active_scalability_multinet

Configuration keys

The configuration keys that must be specified in the JSON configuration file are:

config key type description
nstat_node_ip string IP Address of the NSTAT node
nstat_node_ssh_port string the ssh port of the NSTAT node
nstat_node_username string username for ssh login in the NSTAT node
nstat_node_password string password for ssh login in the NSTAT node
controller_name string name of the used controller. This value is used in Controller Factory method to return the appropriate controller object. For this test it should be ODL
controller_node_ip string IP Address of the Controller node
controller_node_ssh_port string The ssh port of the Controller node
controller_node_username string Username for ssh login in the Controller node
controller_node_password string Password for ssh login in the Controller node
sb_emulator_name string The name of SouthBound emulator. This value is used in Generator Factory method to return the appropriate SouthBound emulator object. For this test it should be MULTINET
sb_emulator_node_ip string IP Address of the Multinet node. Based on the multinet documentation, this is the IP address of the Multinet master node. This configuration key currently is not in use and is reserved for future releases of NSTAT.
sb_emulator_node_ssh_port string The ssh port of the Multinet node
sb_emulator_node_username string username for ssh login in the Multinet node
sb_emulator_node_password string password for ssh login in the Multinet node
sb_emulator_clean_handler string executable that cleans up locally Multinet files cloned from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
sb_emulator_build_handler string executable that clones locally Multinet files from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
nb_emulator_name string
nb_emulator_node_ip string IP Address of the Northbound traffic generator VM
nb_emulator_node_ssh_port string the ssh port of the Northbound traffic generator VM
nb_emulator_node_username string username for ssh login in the Northbound traffic generator VM
nb_emulator_node_password string password for ssh login in the Northbound traffic generator VM
nb_emulator_run_handler string executable for starting Northbound traffic generator (relative to --nb-generator-base-dir command line parameter)
nb_emulator_get_oper_ds_handler string this is an NorthBound generator handler. It is used in order to identify flows, inserted by NorthBound generator, inside the operational datastore of the Controller
nb_emulator_build_handler string executable for cloning Northbound traffic generator from its repository. The root path of this executable, is defined by --nb-generator-base-dir command line parameter
nb_emulator_clean_handler string executable to clean up and leave to its initial state the root directory of Northbound traffic generator, defined by --nb-generator-base-dir command line parameter
controller_build_handler string executable for building controller (relative to --ctrl-base-dir command line parameter)
controller_clean_handler string executable for cleaning up controller directory (relative to --ctrl-base-dir command line parameter)
controller_start_handler string executable for starting controller (relative to --ctrl-base-dir command line parameter)
controller_stop_handler string executable for stopping controller (relative to --ctrl-base-dir command line parameter)
controller_status_handler string executable for querying controller status (relative to --ctrl-base-dir command line parameter)
controller_statistics_handler string executable for changing the period that the controller collects topology statistics (relative to --ctrl-base-dir command line parameter)
controller_persistent_handler string disables persistence of controller. This can be acchieved by adding the attribute persistent=false in file <controller_base_dir>/etc/org.opendaylight.controller.cluster.datastore.cfg
controller_oper_hosts_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of hosts from the operational datastore
controller_oper_links_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of links from the operational datastore
controller_oper_switches_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of switches from the operational datastore
controller_oper_flows_handler string makes a RESTCALL to the NorthBound interface of the controller in order to get the number of flows from the operational datastore
controller_flowmods_conf_handler string configures the controller plugins to respond with flow modifications on any PacketIN message with ARP payload
controller_logs_dir string controllers logs directory (relative to --ctrl-base-dir command line parameter)
controller_get_handler string executable for downloading the controller prebuild version from its repository and extracts it.
controller_port number controller port number where OF switches should connect
controller_restconf_port number controller RESTCONF port number
controller_restconf_user string controller RESTCONF user name
controller_restconf_password string controller RESTCONF password
controller_statistics_period_ms array of numbers controller different statistics period values (in (ms))
topology_rest_server_boot string executable that boots up a Multinet server at a remote Multinet VM (relative to --sb-generator-base-dir command line parameter)
topology_stop_switches_handler string executable that stops a Multinet topology
topology_get_switches_handler string executable that retrieves the number of booted switches in a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_get_flows_handler string executable that retrieves the number of installed flows on switches in a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_start_topo_handler string executable that starts switches in a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_init_topo_handler string executable that initializes a Multinet topology. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
topology_server_rest_port number the port that Multinet server will listen to
topology_size array of numbers number of Multinet switches
topology_topology_type array of strings type of Multinet topology {RingTopo, LinearTopo, DisconnectedTopo}
topology_hosts_per_switch array of numbers number of Multinet hosts per switch
topology_group_size array of numbers size of a group of switches
topology_group_delay_ms array of numbers delay between different switches groups (in milliseconds)
multinet_clean_handler string executable that cleans up locally Multinet files cloned from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
multinet_build_handler string executable that clones locally Multinet files from Multinet repository. The root path of this executable, is defined by --sb-generator-base-dir command line parameter
multinet_worker_ip_list array of strings a list of all IP addresses of worker nodes
multinet_worker_port_list array of numbers a list of port numbers of all REST servers on worker nodes
multinet_switch_type string the type of software switch, that will be used from our Multinet topology
flow_workers array of numbers number of worker threads that will issue simultaneous flow operations
total_flows array of numbers total flows that will be added to the network
flow_operations_delay_ms array of numbers delay between flow operations issued by workers
flow_delete_flags boolean if true workers will also perform the corresponding flow deletions and measure the deletion time
java_opts array of strings options for the JVM
plots array of plot objects configurations for plots to be produced after the test

The array-valued configuration keys shown in bold are the test dimensions of the test scenario. The stress test will be repeated over all possible combinations of their values.

The most important configuration keys are

  • total_flows: defines the total number of flow operations that will be issued as RESTCONF calls from the NorthBound interface
  • flow_workers: defines the number of worker threads that will concurrently issue RESTCONF calls
  • flow_operations_delay_ms: the delay (in ms) between RESTCONF calls

The values of flow_workers and flow_operations_delay_ms define how aggressive the transmission rate of RESTCONF calls will be.

Plot configuration

See the plotting page.

Result keys

The result keys produced by this kind of test scenario and which can be used subsequently to generate custom plots, are the following:

Result key type description
global_sample_id number unique (serial) ID for this sample
timestamp number unique timestamp for this sample
date string date this sample was taken
total_flows number number of total flows intended to be added/deleted in DS
failed_flow_operations number number of total flow operations on controller's datastore that have failed. Success or failure is defined by the return REST status code, we get from a flow operation (addition or deleteon).
add_controller_time number time in seconds for all rest requests for flow additions to be sent and their response to be received
add_controller_rate number number of flows divided by the add_controller_time [Flows/s]
remove_controller_time number time in seconds for all rest requests for flow deletions to be sent and their response to be received
remove_controller_rate number number of flows divided by the remove_controller_time [Flows/s]
add_switch_time number time from the first rest request for flow addition, transmited to the NorthBound interface of the controller, until all flows are installed on OpenFlow switches of the topology
add_switch_rate number number of flows divided by the add_switch_time [Flows/s]
remove_switch_time number time from the first rest request for flow deletion, transmited to the NorthBound interface of the controller, until all flows are deleted on OpenFlow switches of the topology
remove_switch_rate number number of flows divided by the remove_switch_time [Flows/s]
add_confirm_time number time from the last rest request confirmation received for flow addition, until all flows are present in the operational datastore of the controller
add_confirm_rate number number of flows divided by the add_confirm_time [Flows/s]
remove_confirm_time number time from the last rest request confirmation received for flow deletion, until all flows are removed from the operational datastore of the controller
remove_confirm_rate number number of flows divided by the remove_confirm_time [Flows/s]
end_to_end_installation_time number time in seconds from the first flow installation request until all flows have been installed and become visible in the controller's operational datastore
end_to_end_installation_rate number number of flows divided by the end_to_end_installation_time [Flows/s]
end_to_end_remove_time number time in seconds from the first flow deletion request until all flows have been deleted from controller's operational datastore
end_to_end_remove_rate number number of flows divided by the end_to_end_remove_time [Flows/s]
flow_operation_delay_ms number delay between flow operations (in (ms))
flow_workers number number of worker threads issuing simultaneously flow operations
flow_delete_flag boolean flag that will indicate whether or not to delete flows nad measure the deletion time
multinet_size number number of Multinet switches connected to the controller
multinet_worker_topo_size number number of switches created within a single worker node
multinet_topology_type string Multinet network topology {ring, linear, disconnected, mesh}
multinet_hosts_per_switch number number of Multinet hosts per switch
multinet_group_size number size of a group of switches
multinet_group_delay_ms number delay between different switches groups (in (ms))
controller_node_ip string controller IP address where OF switches were connected
controller_port number controller port number where OF switches should connect
controller_vm_size number controller virtual machine size
controller_java_xopts array of strings controller Java optimization flags (-X)
free_memory_bytes number system free memory in bytes
used_memory_bytes number system used memory in bytes
total_memory_bytes number system total memory in bytes
one_minute_load number one-minute average system load
five_minute_load number five-minute average system load
fifteen_minute_load number fifteen-minute average system load
controller_cpu_shares number the percentage of CPU resources of the physical machine, allocated to the controller process
controller_cpu_system_time number CPU system time for controller
controller_cpu_user_time number CPU user time for controller
controller_num_threads number number of controller threads measured when this sample was taken
controller_num_fds number number of open file descriptors measured when this sample was taken
controller_statistics_period_ms number the interval (in (ms)) of the statistics period of the controller

The result keys in bold (all keys with suffix _time and _rate) are the main performance metrics produced by this test scenario.

Sample experimental results

Intro

Stress Tests

Emulators

Monitoring tools

  • OpenFlow monitoring tools

Design

Releases

ODL stress tests performance reports

Sample Performance Results

Clone this wiki locally