Previous Releases

Previous releases

Release 1.3 ("Eunomia") features

• Multinet based southBound idle stability tests: in the current NSTAT release this kind of tests are performed after utilizing the advantages offered by the multinet tool. Therefore, large scale topologies can be created in a fast and resource efficient way giving NSTAT the ability to stress an SDN controller, in a much wider range in terms of switch scalability.

• Multinet based southBound active stability tests: the current NSTAT release supports initiating OpenFlow traffic transmission from Mininet switches to the controller's southbound interface. Each switch initiates traffic by having one of its attached hosts to generate traffic (Mausezahn, etc) towards another attached host on the same switch. Controller is able to learn the switches topology and forwards the packets along with their corresponding flows via the L2swicth feature. In comparison to MT-Cbench active switches, NSTAT now performs active stress tests with a more realistic OF switch implementation (OF 1.3) giving the ability for more accurate results.

• Traffic monitoring: the current release of NSTAT allows for monitoring the controller response throughput to the SouthBound devices connected to it. The rationale is to be able to track the throughput at a given time while a stress test still runs. This is achieved by sniffing the OF packets sent by the controller to the switches using the oftraf tool which utilizes external tools i.e pcap library etc.

Release 1.2 ("Eunomia") features

• Multi-node test execution: NSTAT allows the execution of its basic components (controller, SouthBound emulator, NorthBound generator) on different nodes. Every component can be built, started, executed and monitored on a node (e.g. physical or virtual machine) specified in the configuration file, which is assumed to be already up and running. NSTAT itself may run on a separate node from which it orchestrates and monitors all other components. To make this possible, nodes are assumed to support SSH connectivity and have already checked out an identical instance of NSTAT source tree.

• NorthBound flow generation handlers: NorthBound traffic generation logic is decoupled from the flow scalability test. Separate NorthBound handlers are supported and can be executed from a separate node ("NorthBound generator node"), as it happens with every other NSTAT component.

Release 1.1 ("Eunomia") features

• Custom Mininet topologies**: NSTAT provides logic for configurable boot-up and management of well-known Mininet topologies (disconnected, linear, mesh). This handling logic can be used in all stress test scenarios involving Mininet. (removed from after Release 1.3. Switch scalability tests using Mininet have been replaced by the deployment of the Multinet emulator)

• Flow scalability stress test**: a new test scenario has been implemented for flow scalability. In this case, a configurable number of NorthBound clients (apps) install flows on the switches of a Mininet topology and at a configurable rate. Further details can be found at Northbound active scalability tests.

• Vagrant setup environment**: a Vagrantfile along with provisioning scripts are provided as a means to offer an easy to configure, reproducible and portable build- and runtime environment for NSTAT.

Release 1.0 ("Eunomia") features

• Fully automated, end-to-end testing: The key components in NSTAT tests are the SDN controller and the southbound OpenFlow traffic generator. NSTAT is responsible for automating and sequencing every single step of the testing procedure, including:
  • component lifecycle management and orchestration: build, start, stop, etc., and synchronize the controller and generator,
  • test dimensioning: iterate the test over a large space of experimental parameters,
  • online monitoring and sampling: additional threads are being launched in the background to monitor component progress and collect performance results and runtime statistics,
  • reporting: report generation and result plotting

• Exhaustive stress testing: within a single run, NSTAT repeats the test over an extensive experimental space defined by the cartesian product of all iterable test configuration parameters (the test dimensions, e.g. the number of switches simulated by the SB emulator, the number of simulated hosts, etc.). In this way the user is able to express and execute multiple test cases into a single run.

• Scalable traffic generation: NSTAT provides MT-Cbench, a multi-threaded implementation of Cbench, aiming at emulating large-scale SDN topologies with thousands of switches and bring controller performance to its limits.

• Unification of stress tests: NSTAT unifies existing and new test cases and tools under a common environment. The tests currently implemented are:

  • idle southbound scalability test: boots up a topology at idle mode and counts installed switches and best installation times
  • active southbound scalability test: sends OpenFlow traffic from a booted topology and measures controller performance as the topology scales
  • active southbound scalability test: sends OpenFlow traffic from a booted topology for a large time window and measures controller performance in time
  • active northbound scalability test: sends flow operations from the northbound interface of the controller and measures the time that it takes to the controller to perform these flow operations on the connected OpenFlow switches it controls

• Easy and rich configuration system: NSTAT allows the user to easily specify the test configuration parameters using a single JSON file. Nearly all test components and procedures from component orchestration to plotting are largely described by these configuration parameters.

• Comprehensive reporting and flexible plotting system: NSTAT provides comprehensive output in a single HTML report, including all the information required to fully describe a test. It is also supported by a flexible plot generation methodology (within the configuration parameters JSON file) which allows the user to easily specify plot types and parameters.

• Modular and extensible architecture: for most cases, NSTAT is not tightly coupled to a specific controller or emulator implementation or version. Rather, it is pluggable to any controller/emulator provided it adheres to certain conventions. Furthermore, extending the existing test dimensions and result statistics is rather easy and doesn't affect existing logic.