Openapitest Framework

The purpose of this openapitest framework is to simplify authoring, organizing, executing, and reporting result of API tests using API specification. It runs on node.js and distributed via npm. Add this module to your node.js project as a development dependency and start writing API or Integration tests using YAML language.

Installation

---

npm install --save-dev openapitest

Writing tests for your project

---

See below example for one of several ways tests can be added and organized. The test-integration folder is where all YAML test suites name with .spec.yaml, relevant resources like shared and test specific data are organized into sub-folders. The shared-data folder is optional, used only to store common configuration and test data that are used by more than one test suites. Test-suite specific data are stored next to test spec file. The test data file can be .data.js or .data.yaml.

ProjectCoolThing
├─ test-integration
   ├─ config
   |   ├─ common.config.js
   |   └─ testenvironment.config.yaml
   ├─ data
   |   ├─ testusers.data.js
   |   └─ testenvironment.data.yaml
   └─ test-suites
       ├─ cool-features-group1
       |   ├─ coolTest1.spec.yaml
       |   ├─ coolTest1.data.yaml
       |   ├─ coolTest2.spec.yaml
       |   ├─ coolTest2.data.js
       ├─ cool-features-group2
           ├─ coolTestg2.spec.yml
           └─ coolTestg2.data.yaml
           
├─ test-unit
├─ test-e2e
├─ server
├─ api-spec
├─ README.md
├─ package.json
├─ .gitignore
├─ ...

Executing tests

---

npm script can be added to execute all tests, or different suite of tests to the package.json:

"test-int": "openapitest -o api-spec/pct-api.json -t test-integration -s test-integration/shared-data -u  https://localhost:9000"

Run the tests by calling : npm run test-int . The test results are are saved to the root directory by default.

More Help:

openapitest --help

Options:
  -V, --version            Output the version number
  -o, --openapi  [path]    Open API relative/ absolute path. e.g: <path>/openapi.json
  -t, --testDir  [path]    Test folder relative/ absolute path.  e.g: <path>/test-spec
  -d, --dataDir  [path]    Test data folder. Defaults to a folder called "data" that is a sibling to the test-suites folder
  -s, --sharedir [path]    Shared Test data folder relative/ absolute path.  e.g: <path>/share_data
  -c, --dataConfig [path]  Common Test data config folder relative/ absolute path.  e.g: <path>/config
  -g, --globalConfig [path]  Global Test data config folder relative/ absolute path.  e.g: <path>/global-config
  -u, --url [url]          Server URL. e.g: http://localhost:9000
  -r, --report <n>         Will generate the html report or not. Default 0; e.g: 1 or 0
  -h, --help               output usage information

Documentation

File naming convention:

---

Test spec file name: should be ending with .spec.yaml.

Example: user.spec.yaml

Test data file name: should be ending with .data.yaml or .data.js.

Example: login.data.yaml

Config priority:

---

We have 3 types of config support here:

• Config: This one will stay inside the test folder. If we have same config name, framework will take this value as first priority.
• Data Config: This one will stay outside of test folder. If we have same config name, framework will take this value as second priority.
• Global Config: Any where. If we have same config name, framework will take this value as last priority.

Tags:

---

• apiCalls
  • name
  • swagger
    • name
    • call
    • parameters
    • header
    • query
    • data
      • $file
    • basicAuth
      • $file
    • conditions
    • save
    • print
    • only
    • skip
    • repeat
    • before
    • after
    • expect
      • json
      • headers
      • status
      • error
• tests
  • name
  • expects

---

apiCalls

The purpose of apiCalls is to call endpoint and return response. Under the apiCalls, we can use two tag such as name and swagger.

name:

The value of the name tag will be test suite name. Example:

name: testing user services

Swagger:

The purpose of swagger is to call all endpoints one by one and return response of the endpoint. Supported tags: name, call, header, query, data, basicAuth, save and expect

name: The value of the name tag will be endpoint name. Example:

name: calling login endpoint - expecting success

call: The keys of the call can be operationId of the endpoint. We can find endpoints with operationId in openapi.json (openapi specification file). Example:

call: post_users_login

parameters: We can define parameter under the parameters tag. There is no file support here. Example:

parameters: 	
    userId: 123456
    otherId: ${otherId}
    other: $config.common.other

header: If we need to send any header information during the endpoint call, we can define under the header tag. There is no file support here. Example:

header:
  Accept: application/json
  Authorization: Bearer sessionToken
  other: $config.common.other

query: We can define query/ get value under query tag. There is no file support here. Example:

query:
  id: 1212
  otherId: 1234
  other: $config.common.other

data: data is responsible to send request data or form data to endpoint. Value of the data can be 2 types like as key-value pair and file base. We can create 2 types of file with .yaml and .js extension.

key-value pair example:

data:
  email: test@localhost.com
  password: "123"
  other: $config.common.other

file example:

data:
   $file: login

YAML base: example: login.data.yaml

email: test@localhost.com
password: "123"

JS base: example: login.data.js

module.exports = {
  "email": "test@localhost.com",
  "password": "123"
}

basicAuth: We can define the basic auth information here. We can set basic auth with 2 format such as key-value pair and file same as data tag.

key-value pair example:

basicAuth:
   email: test@localhost.com
   password: "123"
   other: $config.common.other

file example:

basicAuth:
   $file: basicAuth

YAML base: example: basicAuth.data.yaml

basicAuth:
   email: test@localhost.com
   password: "123"

JS base: example: basicAuth.data.js

module.exports = {
  "email": "test@localhost.com",
  "password": "123"
}

conditions: If we want to give any condition for response/ result value then we can do that by using conditions tag. Example:

conditions:
  util: 
    expect:
      json:
        - status: to.be.eql complete
      headers:
        - server: to.be.eql nginx 
      interval: 5000
      limit: 10    

Here we added until condition. This endpoint will be called until status is complete and server is nginx. We can set condition for json response and headers.

limit: Failure limit, representing the maximum number of falsey returns from result or response that will be permitted before invocation is deemed to have failed. A negative number indicates that the attempt should never fail, instead continuing for as long as result or response have returned truthy values.

interval: The retry interval, in milliseconds. A negative number indicates that each subsequent retry should wait for twice the interval from the preceding iteration (i.e. exponential backoff). The default value is -1000, signifying that the initial retry interval should be one second and that each subsequent attempt should wait for double the length of the previous interval.

print: If we want to print the response or error then we can do that by using print tag. Example:

print: 1

only: If you want to run only a subset of tests by using only tag. Example:

only: true

skip: If you want to skip one or more tests, use the skip Example:

skip: true

repeat: If you want to run a test more than once without repeating the all test data you can use the repeat Example:

repeat: 10 

In the case above the same test would be repeated 10 times

before: With its default "BDD"-style interface, openapitest, provides the hooks before, these should be used to set up preconditions or debug before your tests.

/**
*
* @param testData holds all the test data you can use in before code
* @param {expect} testData.expect chaijs expect API e.g: `expec(true).to.be.equal(false);//Will fail`, for full documentation see `https://www.chaijs.com/api/bdd/
* @param {array[Spec]} testData.specs array of specs in the running suite e.g: `[ [ { name: 'Login - success', only: false, save: [Object] },...]`, can be used to modify other tests behavior
* @param {object} testData.op operation definition comming from open api spec file, e.g: `{ tags: [ 'User' ], summary: 'login', requestBody: {...} responses: {...}'
* @param {object|undefined} testData.body the request body to be sent, in get request it may be undefined
* @param testData.basicAuth auth information to be used in the auth header
* 
*/
before: !!js/function "function(testData){  /** runs before the test */;   }"

after: With its default "BDD"-style interface, openapitest, provides the hooks after, these should be used to clean up after your tests or doing custum expecs that the openapitest APi does not provide.

/**
*
* @param testData holds all the test data you can use in after code
* @param {expect} testData.expect chaijs expect API e.g: `expec(true).to.be.equal(false);//Will fail`, for full documentation see `https://www.chaijs.com/api/bdd/
* @param {array[Spec]} testData.specs array of specs in the running suite e.g: `[ [ { name: 'Login - success', only: false, save: [Object] },...]`, can be used to modify other tests behavior
* @param {object} testData.op operation definition comming from open api spec file, e.g: `{ tags: [ 'User' ], summary: 'login', requestBody: {...} responses: {...}'
* @param {object|undefined} testData.body the request body to be sent, in get request it may be undefined
* @param {object} testData.basicAuth auth information to be used in the auth header
* @param {SuperagentResponse} testData.res server response to full documentation check `http://visionmedia.github.io/superagent/#response-properties`. If get the error, testData.res value will be empty object
* @param {object} testData.error error object. For successful response, error value will be empty object and other case will get error object
*
*/
after: !!js/function "function(testData){  /** runs after the test */;   }"

save: After calling the endpoint, If we want to save response for next endpoint or for assert the test then we can do that by set under save tag. The response type can be text or json. Example: Saving session token

save:
   sessionToken: json.sessionToken
   other: $config.common.other

Example: Returning full response

save:
   userResponse: json

We can save value from file as well. Example: save username from basicAuth file. From config folder. File name will be common.config.(js/yaml)

save:
   username: $file.basicAuth.email
   other: $config.common.other

We can parse any value from response using regular expression.

Format:

<which.response.value.want.to.parse>: <$regex regularExpression>

Example: parsing user id from url: http://host:port/users/12345

save:
   userId: 
      json.user.url: $regex ([a-z\d]+)$

expect: We can do some assertion test here like status, response, headers and error text. To test response, we do under json/ text tag and for headers, we do under headers tag and to test status, we use status tag. Status's value can be list or single value.

Format:

<response.value (actual value)>: <operator> <expected value>

Here <operator> is optional. If we do not provide <operator> then default <operator> will be equal. Example:

expect:
  json:
    - user: to.be.an object
    - user: to.have.key _id
    - sessionToken: to.contain session
  headers:
    - content-type: application/json; charset=utf-8
    - x-powered-by: Express
  status:
    - 200
    - 201
  error: Forbidden

or

expect:
  status: 200

Example of swagger section: more than one endpoint:

swagger:
- name: Calling login endpoint - expecting success
  call: post_users_login
  data:
    $file: login
  header:
    Accept: application/json
  save:
    sessionToken: json.sessionToken
  expect:
    json:
      - user: to.be.an object
      - user: to.have.key _id
      - sessionToken: to.contain session:
    headers:
      - content-type: application/json; charset=utf-8
      - x-powered-by: Express
    status: 200
    
- name: call demployment post endpoint - expecting success      
- call: post_company
  header:
    Accept: application/json
    Authorization: Bearer {$sessionToken}
  data:
    $file: compnay
  save:
    responseObj: json
  expect:
    json:
      - user: to.be.an object
      - user: to.have.key _id
      - sessionToken: to.contain session:
    headers:
      - content-type: application/json; charset=utf-8
      - x-powered-by: Express
    status: 200

Here, we can see that login endpoint (post_users_login) returning sessionToken and in the deployment section, we are using the sessionToken's value under Authentication key.

tests:

We will write all test assertion rules in here. Each test start with "-" dash sign. Under the tests, we can use tags like name, expects.

name: name tag will be name of the test/ step. Example:

name: session token exists

expects: Expecting assertion will go here. This value will come from endpoint's response under the save tag.

Example:

expects: 
   - user: to.have.key _id

Format:

<response.value (actual value)>: <operator> <expected value>

Full example of tests:

tests:
- name: user is an object
  expects: 
    - sessionResponseObj.1.user.href: ${userResponse.user.href}
    - user: to.be.an object
    - user: to.have.key href
    
- name: user session and logout url check 
  expects: 
    - sessionToken: to.contain session
    - logout.href: to.contain /users/logout

Full Example: user.spec.yaml

apiCalls:
  name: login to the system
  swagger:
  - name: Login fail - try with empty credentials 
    call: post_users_login
    data:
      email: ''
      password: ''
    query:
      allow-redirect: 0
    expect:
      status: 400
      error: email must be a valid email or username

  - name: Login fail - try with wrong password
    call: post_users_login
    data:
      $file: wrongPassword
    query:
      allow-redirect: 0
    expect:
      status: 403
      error: Forbidden

  - name: Login fail - try with without allow-redirect
    call: post_users_login
    data:
      $file: login
    expect:
      status: 
      - 200
      - 403
      error: Forbidden

  - name: Login - success
    call: post_users_login
    data:
      $file: login
    query:
      allow-redirect: 0
    expect:
      headers:
        - content-type: application/json; charset=utf-8
        - x-powered-by: Express

      json:
        - user: to.be.an object
        - user: to.have.key _id
        
      status: 200
        
tests:  
- name: user session and logout url check 
  expects: 
    - sessionToken: to.contain session
    - logout.href: to.contain /users/logout

Fake data

Sometime you may want to randomize the tests to create more realistic Test Scenarios. For that, we provide an Out of the box Faker.js API.

Faker is a massive amount of fake data generator. All you need to do is to add in any field the type !faker ["faker.api.you.want", "scope"], and it will be replaced by the randomized value. Next, we are going to present some examples, if you need the complete documentation, please read Faker.js docs.

apiCalls:
  name: login to the system
  swagger:
  - name: Login fail - try with wrong password
    call: post_users_login
    data:
      email: !faker ["internet.email"]
      password: !faker ["internet.password"]
    query:
      allow-redirect: 0
    expect:
      status: 403
      error: Forbidden

In the example above, the fields, email and password will be diferent for each test execution, onde Faker.js will generate a diferent valid email for each time it runs.

Fake data scope

There are tree scopes for the !faker calls, they are global, file or test. If you do not provide a scope, the framework will use global as default.

• Global scope will be evaluated once the file is loaded, and therefore will not change in the all test execution
• File scope will be evaluated for before start the test, so it will have a different value for each test execution
• Test scope will be evaluated for before start the it execution, so it will have a different value for each spec