AdvancedBoaTester

More advanced features of BoaTestRunner

Introduction

In this document it is described how to create custom test case types and API available.

Custom test cases

Introduction

Assume that we have a series of test cases which follows the same common pattern.

For instance:

• Run shell script (binary program). This script (program) is conducting very important reports which are placed in directory output as report1.txt report2.txt ... etc
• Test is passed if all output reports are exactly as expected. The number and content of this files should be exactly the same as reports kept in test case resource directory.

Prepare test properties files

test.property file should contain some property to make it different than the other test cases. Cannot contain keys: test case (Python Test Case object) nor command (command test case)

test.properties

[defaults]
descr=Test custom test case
runcommand=prog.sh

runcommand property contains a command to run.

Test case property could contain reference to some environment variable: Details on property file: http://www.python.org/doc//current/library/configparser.html Only three variables are supported:

Name	Meaning
globresdir	Directory for global resources
resdir	Directory for current test case resources
rundir	Working directory for test execution

Prepare custom test case factory

Custom test case factory should be registered. This factory:

1. Should recognize that test case given should be run by custom handler
2. If yes then return TestCase object running this test.

class MyFactory:

    def constructTestCase(self, param, tepar):
        runcommand = tepar.getPar('runcommand')
        if runcommand == None : return None
        teca = ReportTestCase(param, tepar)
        return [teca]

Factory class should have 'constructTestCase' method. It this test case (test.property file is passed as tepar parameter) has runcommand property than factory returns 'ReportTestCase' object. Otherwise, return None value.

Prepare custom TestCase object

class ReportTestCase(unittest.TestCase):

    def __init__(self, param, teparam):
        unittest.TestCase.__init__(self, 'runTest')
        self.param = param
        self.teparam = teparam

    def setUp(self):
        prepareRunDir(self.param, self.teparam)

    def runTest(self):
        runcommand = self.teparam.getPar('runcommand')
        TestCaseHelper.copyFile(self.param, self.teparam, runcommand)
        res = runBin(self.param, "./" + runcommand, runcommand)
        self.assertEqual(0,res)
        res = compareFiles(self.param, self.teparam, "out", ".lst", "out")
        self.assertTrue(res)

It should extend unittest.TestCase object.

    def setUp(self):
        prepareRunDir(self.param, self.teparam)

Copies resources to the test running directory (in this case do nothing but can be added for future extension)

        runcommand = self.teparam.getPar('runcommand')
        TestCaseHelper.copyFile(self.param, self.teparam, runcommand)

Copies command from test case directory to test running directory.

        res = runBin(self.param, "./" + runcommand, runcommand)
        self.assertEqual(0,res)

Runs command and make sure that command exits with code 0

        res = compareFiles(self.param, self.teparam, "out", ".lst", "out")
        self.assertTrue(res)

Compares all files in out in the test case directory with the out directory in test running directory. The test fails if any difference has been found.

Test script to run

#!/bin/sh
echo "I'm here"
mkdir out
echo "Report result" > out/report1.lst

Test case directory structure

  <test case>
     test.properties
     prog.sh
     <out>
      report1.lst

API description

There is some useful API available to custom test case. It is documented in detail via 'doc' objects:

https://github.com/stanislawbartkowski/boatester/blob/master/BoaHarness/src/testharness/TestBoa.py

https://github.com/stanislawbartkowski/boatester/blob/master/BoaHarness/src/testharness/TestCaseHelper.py

TestCaseHelper

What	Name	Description
method	osPrefix	return "windows" or "linux"
method	isLinux
method	prepareRunCommand
object	changeDir	change current directory to running directory
method	runBin	run external program, script
object	TestException
object	OneTestParam	container for test.properties for test case
object	TestParam	container for global properties
method	removeDir
method	copyDir
method	copyFile
method	clearRunDir
method	prepareRunDir	prepares test running directory
method	compareFiles	compare file contents

TestBoa

What	Name	Description
object	TestCaseFactory	container for registering custom test case types
object	RunSuiteParam	container with parameters
method	runSuite	entry point