GettingStarted

Getting started with BoaTestRunner - just next test framework.

Introduction

BoaTestRunner is very simple testing framework written in Python. It is an extension to standard Python test framework PyUnit.

In this document is described how to download and install BoaTestRunner and how to created and developed test cases.

General information

Prerequisites

Host machine should have Python installed. Almost every Linux distribution has Python as a part of a standard configuration.

The framework was tested with Python 2.4. Was not tested with Python 3.x (future).

Instalation

Checkout: http://code.google.com/p/boatester/source/checkout

Following directory structure should be created.

 boatester-read-only/BoaHarness
 boatester-read-only/BoaHarness/src
 boatester-read-only/BoaHarness/src/testharness
 boatester-read-only/BoaHarness/src/testharness/TestCaseHelper.py
 boatester-read-only/BoaHarness/src/testharness/TestCaseHelper.pyc
 boatester-read-only/BoaHarness/src/testharness/__init__.py
 boatester-read-only/BoaHarness/src/testharness/TestBoa.py
 boatester-read-only/BoaHarness/src/testharness/__init__.pyc
 boatester-read-only/BoaHarness/src/testharness/TestBoa.pyc

 boatester-read-only/TestBoaHarness
 boatester-read-only/TestBoaHarness/src
  ...

Two packages are extracted. BoaHarness contains BoaTestRunner executable codes. TestBoaHarness contains some tests and samples. To execute tests only BoaHarness package is sufficient.

To run samples:

On Linux (assuming that packages were checked-out into current directory)

export PYTHONPATH=boatester-read-only/BoaHarnes/src/testharness
python boatester-read-only/TestBoaHarness/src/testboaharness.py boatester-read-only/TestBoaHarness/src/TestResource /tmp/rundir all x

On Windows

set PYTHONPATH=C:\TEMP\boatester-read-only\BoaHarness\src\testharness
C:\python26\python boatester-read-only\TestBoaHarness\src\testboaharness.py boatester-read-only\TestBoaHarness\src\TestResource c:\temp\rundir all x

Output (if successful )

dirres  run.sh
setUp
testCase1
setUp
Test number 2
testCase2
=========================
TEST PASSED  number of tests : 3

Vocabulary

Following terms will be used:

• BoaTestRunner : test framework, test harness. Executes tests and evaluates the result.

• Test, test case : executed by BoaTestRunner. Should provide test result: passed or failed.

• Test case resource : directory containing all data, scripts, files necessary for the test case to be run. Should be platform independent. Should be also "directory independent", cannot assume any absolute path.

• Test suite : collection (subdirectories) of test case resources.

• Common test resources : a collection of data used by more than one test case. Using of any common resource (subdirectory) is declarative.

• Running test directory : the directory where the test is performed. The directory is cleared at the beginning.

• Test case method, test case handler, test case type : the reusable method for running test. Two methods are predefined (Python TestSuite object and command). Custom test case handler can be defined.

How tests are organized

Every test (test case) should be independent and repeatable. BoaTestRunner does not support test dependency e.g. next test case is using the output of the previous test. The test could be run in any order.

Every test is a subdirectory in TestSuite directory. Test subdirectory should contain all resource needed to run a test and elaborate test result. It is possible (although not necessary) to have a directory with common resources used by more than one test.

Test case subdirectory name should follow a pattern: test{number} e.g test1, test2 .. test12 ... BoaTestRunner is looking for test.properties file containing some information on the test.

Example:

TestResource
  Test1
    test.properties
    run.sh
    sample
      file1
      file2
  Test2
    test.properties
    testdata.txt
    testinput.log
  SomeData
    subdir1
    subdir2

In the directory structure above two test cases will be run: Test1 and Test2. Directory SomeData will be omitted.

Only Test1\test.properties and Test2\test.properties files are required by BoaTestRunner framework. Additional files in this directories are used by test case only.

SomeData could contain additional resources used by tests. Usage of this resource should be declared in test.properties file. By default only files in test subdirectory are used.

To add new test case simply create next directory (e.g Test3), lay down proper test.properties file and all files and directory necessary to run the test. The new test case will be automatically discovered by BoaTestRunner.

How test suite is executed

1. Analyze test suite directory. Subdirectories having test in their names are taken and test.properties file is read. List of test case candidates is created.
2. Executes test cases: basing on parameters all tests can be run, only one test or tests starting from some point. Gathers result of every test cases.
3. Print test suite result: number of tests executed, passed, failed.

How one test case is executed

1. Clears running test directory.
2. If using of common test resource is declared copies that resource to running test directory
3. Copies all test case resource to running test directory
4. Runs test case using proper test case handler.
5. Propagate test result (passed, failed) to BoaTestRunner.

Creating first test suite

Example

We want to create two test cases. First uses one test resource: file and test if this file is copied properly, the second is using also one common resource (another file). The expected result for the first test is to have one file as a resource, the expected result for the second test case is to have two files.

Plan your test suite structure

• Create test launcher (Python module)
• Create a shell script to start the test.
• Select running test directory. Keep in mind that this directory is cleaned every time. Assume directory: /tmp/rundir
• Select test suite directory. Should contain two subdirectories: test1 and test2. Assume directory {HOME}/mytestsuite
• Create test script for every test (here simple shell script testing if the file exists).
• Select a directory for a common resource. It could be the same directory as test suite directory {HOME}/mytestsuite/resource/

Create test launcher (Python module)

Copy testboaharness.py module (from directory: boatester-read-only/TestBoaHarness/src). Assume that this file name is runtest.py. This module assumes that test suite resource is the same as a common resource directory.

Additional information: one can get more detailed information by uncommenting line with logging settings.

import logging
from TestBoa import *
from TestCaseHelper import *

#LOG_FILENAME = '/tmp/testlogging.out'
#logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)
#logging.basicConfig(level=logging.INFO)
#logging.basicConfig(level=logging.DEBUG)

def printhelp():
    print "Usage:"
    print "runtest /res dir/ /run dir/ /spec/ /testid/"

if __name__ == "__main__":
    if len(sys.argv) != 5:
        printhelp()
        sys.exit(4)
    resource = sys.argv[1]
    rundir = sys.argv[2]
    testspec = sys.argv[3]
    testid = sys.argv[4]
    globresource = resource
    propfile = None
    factory = TestCaseFactory()
    suiteparam = RunSuiteParam(factory, None, propfile, globresource, resource, rundir)
    try:
        runSuite(suiteparam, testspec, testid)
    except Exception, e:
        print e

Create shell script to start tests

Assume that BoasTestRunner executable are extracted into $HOME/checkout directory. testrun.py module file is created in $HOME directory

#!/bin/sh
export PYTHONPATH=$HOME/checkout/boatester-read-only/BoaHarness/src/testharness
python $HOME/runtest.py $HOME/mytestsuite /tmp/rundir all x

Create first test case

Create test case directory

Create $HOME/mytestsuit/test1 and dir subdirectory Create empty $HOME/mytestsuit/test1/dir/file1.txt file

Create test.properties file

Create $HOME/mytestsuit/test1/test.properties file

[defaults]
descr=Test one file
copytestres=dir:out
linux.command=test.sh
windows.command=test.cmd

Explanation:

• descr : text

  • Test name displayed while running

• copytestres : {from}:{to}

  • Copies test resource (directory) from {from} directory to {to} directory in running test directory.
  • Example dir:out
    • $HOME/mytestsuite/test/dir/file1.txt file
    • is copied to:
    • /tmp/rundir/out/file1.txt

• linux.command

  • Defines script (command) to run for Linux platform. The script should exit with code 0 if success and nonzero if failure. The script is copied to running test directory and launched there.

• windows.command

  • The same for windows system

Create test script

Create $HOME/mytestsuite/test1/dir/test.sh script file.

#!/bin/sh
if ! test -d out;  then exit 1; fi
if test -f out/file1.txt; then exit 0; fi
exit 1

Self-descriptive!

Summary

Test resource directory should look like:

$HOME/mytestsuite/test1
                     test.sh
                     test.properties
                   - dir
                      file.txt

Before launching test script test running directory looks like:

/tmp/rundir
       test.sh
       test.properties
     - out
         file1.txt

Run your first test suite !

./run.sh

=========================
TEST PASSED  number of tests : 1

Congratulations !

Creating second test case

Create test case directory

This second test is very like to first but, this time, we want to use the common resource.

So let's create one common resource:

  $HOME/mytestsuite/resource/dir/file2.txt

Create test.properties file

Create $HOME/mytestsuit/test2/test.properties file

[defaults]
descr=Test one file
copytestres=dir:out
copycommonres=resource/dir1:out
linux.command=test.sh
windows.command=test.cmd

This file is exactly the same as before, only one line was added:

copycommonres=resource/dir1:out

This line copies resource/dir1 directory to out a directory of test running directory.

Create test script

Create $HOME/mytestsuite/test2/dir/test.sh script file.

#!/bin/sh
if ! test -d out;  then exit 1; fi
if test -f out/file1.txt; then exit 0; fi
if test -f out/file2.txt; then exit 0; fi
exit 1

Run both tests

=========================
TEST PASSED  number of tests : 2

Creating third test case

Information

This test is exactly like the first but instead of shell script Python Test Case object is used.

Create test.properties file

Create $HOME/mytestsuit/test3/test.properties file

[defaults]
descr=Test one file
copytestres=dir:out
testcase=Test3Case

Property file contains:

testcase=Test3Case

The testcase property indicates that Python module Test3Case.py will be used to run a test.

Create Python Test Case object

Create $HOME/mytestsuit/test3/Test3Case.py file

import unittest
import Test3Case
import TestCaseHelper
import os

def injectParam(param,tepar) :
    Test3Case.param = param
    Test3Case.tepar = tepar

class TestSuite(unittest.TestCase):

    def setUp(self):
       TestCaseHelper.prepareRunDir(Test3Case.param, Test3Case.tepar)
       self.d = TestCaseHelper.ChangeDir(Test3Case.param)

    def testCase1(self):
        res = os.path.exists("out/file1.txt")
        self.assertTrue(res)         

    def tearDown(self) :
        self.d.restore()

This Python module is quite different than shell script in the previous examples.

def injectParam(param,tepar) :
    Test3Case.param = param
    Test3Case.tepar = tepar

This method is mandatory. BoaTestRunner injects 'TestParam' and 'OneTestParam' container containing common and test case specific parameters.

    def setUp(self):
       TestCaseHelper.prepareRunDir(Test3Case.param, Test3Case.tepar)
       self.d = TestCaseHelper.ChangeDir(Test3Case.param)

Unlike shell script the Python module should on its own to copy resources (prepareRunDir method) and change current directory to test running directory (ChangeDir object)

    def testCase1(self):
        res = os.path.exists("out/file1.txt")
        self.assertTrue(res)         

It is the test case in the flesh.

Very important: test result is propagated by TestCase assertion statement.

    def tearDown(self) :
        self.d.restore()

Restores current directory.

Creating fourth test case

This test case is almost identical as the third one but uses also a common resource.

Create $HOME/mytestsuit/test4/test.properties file

[defaults]
descr=Test one file
copytestres=dir:out
copycommonres=resource/dir1:out
testcase=Test4Case

Create $HOME/mytestsuit/test4/Test4Case.py file

import unittest
import Test4Case
import TestCaseHelper
import os

def injectParam(param,tepar) :
    Test4Case.param = param
    Test4Case.tepar = tepar

class TestSuite(unittest.TestCase):

    def setUp(self):
       TestCaseHelper.prepareRunDir(Test4Case.param, Test4Case.tepar)
       self.d = TestCaseHelper.ChangeDir(Test4Case.param)

    def testCase1(self):
        res1 = os.path.exists("out/file1.txt")
        res2 = os.path.exists("out/file2.txt")
        self.assertTrue(res1 and res2)         

    def tearDown(self) :
        self.d.restore()

Final, rule and bound them all

=========================
TEST PASSED  number of tests : 4