Testing

This page is dedicated to testing of Invenio and the B2Share overlay. More information from Invenio about the tests-suite can be found in their hacking section: http://invenio-demo.cern.ch/help/hacking/test-suite

Testing basics

Software testing is about testing specific behaviours of a system. A system is typically subdivided into units (subroutines/ functions), which have an certain expected behaviour. Testing each unit is called unit-testing. In such tests a specification of a given unit is programmatically covered. Each valid input should result in an expected output. Important is to very edge cases (input that is valid, but uncommon).

Integration testing combines many already tested units, and verifies if combining these will have the expected results. Basically the same as unit-testing, but abstracted a level.

Regression testing tests a system by loading many configurations and many types of data (which it should support), and trying to break a system over many iterations of data manipulation.

Often in testing programmers use fixtures. These fixtures provide some system state baseline, for example: a created user, an uploaded document etc.

NOTE: all tests result in ERROR (internally broken), OK (all valid assertions) or Failed (incorrect assertions)

Invenio Tests

Invenio have split their tests into five categories: unit-tests, js-unit-tests, regression-tests, web-tests, and flask-tests. Tests are executed via inveniocfg. Because Invenio is installed as Apache user, running these tests requires to execute as the Apache user.

Running Tests

Unit-tests, and js-unit-tests are more or less like unit-tests. Flask-tests and web-tests behave more like integration-tests.

• NOTE (1): The regression-tests require a clean reinitialised database (like documented here: http://invenio-demo.cern.ch/help/hacking/test-suite#3.3)

• NOTE (2): Web-tests depend on Selenium, which loads a browser (Firefox) on the system Invenio is installed, effectively the VM. Because Invenio must use the Apache user to run tests, effectively the Apache should be able to launch a X-session.

sudo su -c "sudo -u apache /opt/invenio/bin/inveniocfg --run-unit-tests"
sudo su -c "sudo -u apache /opt/invenio/bin/inveniocfg --run-js-unit-tests"
sudo su -c "sudo -u apache /opt/invenio/bin/inveniocfg --run-flask-tests"
# NOTE: uses selenium (which requires the apache user to use X)
sudo su -c "sudo -u apache /opt/invenio/bin/inveniocfg --run-web-tests"
# NOTE: load invenio demo-set (will brake B2share and old records!!)
sudo su -c "sudo -u apache /opt/invenio/bin/inveniocfg --run-regression-tests"

Run individual tests

INVENIO_DIR="/opt/invenio"
sudo su -c "sudo -u apache python $INVENIO_DIR/lib/python/invenio/XXX_unit_tests.py"
# spawns Firefox Selenium
sudo su -c "sudo -u apache python $INVENIO_DIR/lib/python/invenio/XXX_web_tests.py"
sudo su -c "sudo -u apache python $INVENIO_DIR/lib/python/invenio/XXX_regression_tests.py"
# NOTE: flask and js do not execute directly from cli

Inner working

The subroutine call path that is being executed from inveniocfg to execute --run-XX-tests is described below:

• invenio/modules/miscutil/lib/inveniocfg.py
  • main(argv)
    • --run-unit-tests -> build_and_run_unit_test_suite()
    • --run-js-unit-tests -> build_and_run_js_unit_test_suite()
    • --run-regression-tests -> build_and_run_regression_test_suite()
    • --run-web-tests -> build_and_run_web_test_suite()
    • --run-flask-tests -> build_and_run_flask_test_suite()

All tests are redirected to the testutils.py module:

• invenio/modules/miscutil/lib/testutils.py
  • build_and_run_unit_test_suite()
    • collects all *_unit_tests.py files within 'invenio' recursive
    • inherits: InvenioTestCase -> invenio/modules/miscutil/lib/testutils.py
  • build_and_run_js_unit_test_suite()
    • collects all *_tests.js files within 'invenio' recursive
    • dependent on JsTestDriver.jar for execution
    • requires a web-interface: http://localhost:9876
  • build_and_run_regression_test_suite()
    • collects all *_regression_tests.py files within 'invenio' recursive
    • inherits: InvenioTestCase -> invenio/modules/miscutil/lib/testutils.py
  • build_and_run_web_test_suite()
    • collects all *_web_tests.py files within 'invenio' recursive
    • inherits: InvenioWebTestCase -> invenio/modules/miscutil/lib/testutils.py
    • dependent on Selenium, and Firefox
    • requires the executing user (Apache) to forward X over SSH
  • build_and_run_flask_test_suite()
    • collects all *_flask_tests.py files within 'invenio' recursive
    • inherits: FlaskSQLAlchemyTest -> flask dependency

Enable X-Forwarding

• stub

B2Share Overlay Tests

B2share is an extension on Invenio...

• stub