-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is-13 add basic API tests #842
Open
pkeroulas
wants to merge
15
commits into
AMWA-TV:master
Choose a base branch
from
pkeroulas:is-13
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
1902876
IS-13: add basic API tests
pkeroulas 8a4c1bf
Output more explicit error messages
pkeroulas 9d92412
IS-13: add tests for self & devices resources
pkeroulas 9c3f078
IS-13: test consistency with IS-04
pkeroulas 35a2c78
is-13: cover senders and receivers
pkeroulas 1f3b948
IS-13: strip grouphint tag which pollutes the test
pkeroulas 3c8bbf4
IS-13: improve error msg
pkeroulas 6664276
IS-13: add a pause to accomodate the update propagation
pkeroulas 5fafebb
is-13: add test suite description
pkeroulas e5d6cbe
IS-13: rename functions, variables and constants
pkeroulas 6080619
IS-13: remove logger
pkeroulas 1886026
IS-13: refactor results and errors
pkeroulas 0a2f6fd
IS-13: add service announcement test
pkeroulas 24a9817
IS-13: list uncovered tests
pkeroulas 8269c65
IS-13: make the propagation delay configurable
pkeroulas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,298 @@ | ||
# Copyright (C) 2023 Advanced Media Workflow Association | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
|
||
""" | ||
The script implements the IS-13 test suite according to the AMWA IS-13 NMOS Annotation | ||
Specification (https://specs.amwa.tv/is-13/). At the end of the test, the initial state | ||
of the tested unit is supposed to be restored but this cannot be garanteed. | ||
|
||
Not covered yet: | ||
* Persistency: https://specs.amwa.tv/is-13/branches/v1.0-dev/docs/Behaviour.html#persistence-of-updates | ||
* Read only Tags: https://specs.amwa.tv/is-13/branches/v1.0-dev/docs/Behaviour.html#read-only-tags | ||
* Individual Tag reset | ||
* 500 reponses are ignored | ||
|
||
Terminology: | ||
* `resource` refers to self, devices, senders or receivers endpoints | ||
* `annotation_property` refers to annotable objects: label, description or tags | ||
""" | ||
|
||
from ..GenericTest import GenericTest, NMOSTestException | ||
|
||
from ..import Config as CONFIG | ||
from ..import TestHelper | ||
import re | ||
import copy | ||
import time | ||
|
||
IS13_SPEC_VERSION = "v1.0-dev" | ||
IS13_SPEC_URL = f"https://specs.amwa.tv/is-13/branches/{IS13_SPEC_VERSION}/docs" | ||
|
||
ANNOTATION_API_KEY = "annotation" | ||
NODE_API_KEY = "node" | ||
|
||
# Constants for label and description related tests | ||
STRING_LENGTH_OVER_MAX_VALUE = ''.join(['X' for i in range(100)]) | ||
STRING_LENGTH_MAX_VALUE = STRING_LENGTH_OVER_MAX_VALUE[:64] # this is the max length tolerated | ||
|
||
# Constants for tags related tests | ||
TAGS_LENGTH_OVER_MAX_VALUE = {'location': ['underground'], 'studio': ['42'], 'tech': ['John', 'Mike']} | ||
TAGS_LENGTH_MAX_VALUE = TAGS_LENGTH_OVER_MAX_VALUE.copy() | ||
TAGS_LENGTH_MAX_VALUE.pop('tech') # must have a max of 5 | ||
TAGS_TO_BE_STRIPPED = 'urn:x-nmos:tag:grouphint/v1.0' | ||
|
||
|
||
def get_ts_from_version(version): | ||
""" Convert the 'version' object (string) into float """ | ||
return float(re.sub(':', '.', version)) | ||
|
||
|
||
def strip_tags(tags): | ||
if TAGS_TO_BE_STRIPPED in list(tags.keys()): | ||
tags.pop(TAGS_TO_BE_STRIPPED) | ||
return tags | ||
|
||
|
||
class IS1301Test(GenericTest): | ||
""" | ||
Runs IS-13-Test | ||
""" | ||
|
||
def __init__(self, apis, **kwargs): | ||
GenericTest.__init__(self, apis, **kwargs) | ||
self.annotation_url = self.apis[ANNOTATION_API_KEY]["url"] | ||
self.node_url = f"{self.apis[ANNOTATION_API_KEY]['base_url']}/x-nmos/node/{self.apis[NODE_API_KEY]['version']}/" | ||
|
||
def get_resource(self, url): | ||
""" Get a resource """ | ||
|
||
valid, r = self.do_request("GET", url) | ||
if valid and r.status_code == 200: | ||
try: | ||
return r.json() | ||
except Exception as e: | ||
raise(f"Can't parse response as json {e.msg}") | ||
else: | ||
return None | ||
|
||
def compare_resource(self, annotation_property, value1, value2): | ||
""" Compare strings (or dict for 'tags') """ | ||
|
||
if value2[annotation_property] is None: # this is a reset, value2 is null, skip | ||
return True | ||
|
||
if annotation_property == "tags": # tags needs to be stripped | ||
value2[annotation_property] = strip_tags(value2[annotation_property]) | ||
return TestHelper.compare_json(value2[annotation_property], value1[annotation_property]) | ||
|
||
return value2[annotation_property] == value1[annotation_property] | ||
|
||
def set_resource(self, test, url, node_url, value, prev, expected, msg, link): | ||
""" Patch a resource with one ore several object values """ | ||
|
||
self.log(f" {msg}") | ||
|
||
annotation_property = list(value.keys())[0] | ||
valid, resp = self.do_request("PATCH", url, json=value) | ||
if not valid: | ||
raise NMOSTestException(test.FAIL("PATCH Request FAIL", link=f"{IS13_SPEC_URL}/Behaviour.html#setting-values")) | ||
# TODO: if put_response.status_code == 500: | ||
|
||
# pause to accomodate update propagation | ||
time.sleep(CONFIG.API_PROCESSING_TIMEOUT) | ||
|
||
# re-GET | ||
resp = self.get_resource(url) | ||
if not resp: | ||
raise NMOSTestException(test.FAIL(f"GET /{ANNOTATION_API_KEY} FAIL")) | ||
# check PATCH == GET | ||
if not self.compare_resource(annotation_property, resp, expected): | ||
raise NMOSTestException(test.FAIL(f"Compare req vs expect FAIL - {msg}", link=link)) | ||
# check that the version (timestamp) has increased | ||
if get_ts_from_version(prev['version']) >= get_ts_from_version(resp['version']): | ||
raise NMOSTestException(test.FAIL(f"Version update FAIL \ | ||
({get_ts_from_version(prev['version'])} !>= {get_ts_from_version(resp['version'])})", \ | ||
link=f"{IS13_SPEC_URL}/Behaviour.html#successful-response>")) | ||
|
||
# validate that it is reflected in IS04 | ||
node_resp = self.get_resource(node_url) | ||
if not node_resp: | ||
raise NMOSTestException(test.FAIL(f"GET /{NODE_API_KEY} FAIL")) | ||
# check PATCH == GET | ||
if not self.compare_resource(annotation_property, resp, expected): | ||
raise NMOSTestException(test.FAIL(f"Compare /annotation vs /node FAIL {msg}", | ||
link=f"{IS13_SPEC_URL}/Interoperability_-_IS-04.html#consistent-resources")) | ||
# check that the version (timestamp) has increased | ||
if get_ts_from_version(node_resp['version']) != get_ts_from_version(resp['version']): | ||
raise NMOSTestException(test.FAIL(f"Compare /annotation Version vs /node FAIL {msg}", | ||
link=f"{IS13_SPEC_URL}/Interoperability_-_IS-04.html#version-increments")) | ||
|
||
return resp | ||
|
||
def log(self, msg): | ||
""" | ||
Enable for quick debug only | ||
""" | ||
# print(msg) | ||
return | ||
|
||
def create_url(self, test, base_url, resource): | ||
""" | ||
Build the url for both annotation and node APIs which behaves differently. | ||
For iterables resources (devices, senders, receivers), return the 1st element. | ||
""" | ||
|
||
url = f"{base_url}{resource}" | ||
r = self.get_resource(url) | ||
if r: | ||
if resource != "self": | ||
if isinstance(r[0], str): # in annotation api | ||
index = r[0] | ||
elif isinstance(r[0], dict): # in node api | ||
index = r[0]['id'] | ||
else: | ||
raise NMOSTestException(test.FAIL(f"Unexpected resource found @ {url}")) | ||
url = f"{url}/{index}" | ||
else: | ||
raise NMOSTestException(test.FAIL(f"No resource found @ {url}")) | ||
|
||
return url | ||
|
||
def copy_resource(self, resource): | ||
""" Strip and copy resource """ | ||
|
||
r = copy.copy(resource) | ||
r.pop('id') | ||
r.pop('version') | ||
r['tags'] = strip_tags(r['tags']) | ||
|
||
return r | ||
|
||
def do_test_sequence(self, test, resource, annotation_property): | ||
""" | ||
In addition to the basic annotation API tests, this suite includes the test sequence: | ||
For each resource: | ||
For each annotation_property: | ||
- Read initial value | ||
- store | ||
- Reset default value by sending null | ||
- check value + timestamp + is-14/value + is-04 timestamp | ||
- store | ||
- Write max-length | ||
- check value + timestamp + is-14/value + is-04 timestamp | ||
- Write >max-length | ||
- check value + timestamp + is-14/value + is-04 timestamp | ||
- Reset default value again | ||
- check value + timestamp + is-14/value + is-04 timestamp | ||
- compare with 1st reset | ||
- Restore initial value | ||
""" | ||
|
||
url = self.create_url(test, f"{self.annotation_url}node/", resource) | ||
node_url = self.create_url(test, self.node_url, resource) | ||
|
||
# Save initial resource value | ||
resp = self.get_resource(url) | ||
initial = self.copy_resource(resp) | ||
|
||
msg = "Reset to default and save." | ||
link = f"{IS13_SPEC_URL}/Behaviour.html#resetting-values" | ||
default = resp = self.set_resource(test, url, node_url, {annotation_property: None}, resp, | ||
{annotation_property: None}, msg, link) | ||
|
||
msg = "Set max-length value and return complete response." | ||
link = f"{IS13_SPEC_URL}/Behaviour.html#setting-values" | ||
value = TAGS_LENGTH_MAX_VALUE if annotation_property == "tags" else STRING_LENGTH_MAX_VALUE | ||
resp = self.set_resource(test, url, node_url, {annotation_property: value}, resp, | ||
{annotation_property: value}, msg, link) | ||
|
||
msg = "Exceed max-length value and return truncated response" | ||
link = f"{IS13_SPEC_URL}/Behaviour.html#additional-limitations" | ||
value = TAGS_LENGTH_OVER_MAX_VALUE if annotation_property == "tags" else STRING_LENGTH_OVER_MAX_VALUE | ||
expected = TAGS_LENGTH_MAX_VALUE if annotation_property == "tags" else STRING_LENGTH_MAX_VALUE | ||
resp = self.set_resource(test, url, node_url, {annotation_property: value}, resp, | ||
{annotation_property: expected}, msg, link) | ||
|
||
msg = "Reset again and compare with default." | ||
link = f"{IS13_SPEC_URL}/Behaviour.html#resetting-values" | ||
resp = self.set_resource(test, url, node_url, {annotation_property: None}, resp, default, | ||
msg, link) | ||
|
||
msg = "Restore initial values." | ||
link = "{IS13_SPEC_URL}/Behaviour.html#setting-values" | ||
self.set_resource(test, url, node_url, initial, resp, msg, link) | ||
|
||
return test.PASS() | ||
|
||
def test_00_01(self, test): | ||
""" Annotation service must be announced """ | ||
r = self. get_resource(f"{self.node_url}self/") | ||
print(self.annotation_url) | ||
try: | ||
for s in r['services']: | ||
if 'urn:x-nmos:service:annotation' in s['type'] and s['href'] + '/' == self.annotation_url: | ||
return test.PASS() | ||
except Exception as e: | ||
return test.FAIL(f"Could't parse services in '/node/self' {str(e)}") | ||
return test.FAIL("Could't found 'annotation' as a service in '/node/self'", | ||
link=f"{IS13_SPEC_URL}/Interoperability_-_IS-04.html#discovery") | ||
|
||
def test_01_01(self, test): | ||
""" Annotation test: self/label (reset to default, set 64-byte value, set >64-byte, check IS04+version)""" | ||
return self.do_test_sequence(test, "self", "label") | ||
|
||
def test_01_02(self, test): | ||
""" Annotation test: self/description (reset to default, set 64-byte value, set >64-byte, check IS04+version)""" | ||
return self.do_test_sequence(test, "self", "description") | ||
|
||
def test_01_03(self, test): | ||
""" Annotation test: self/tags (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
return self.do_test_sequence(test, "self", "tags") | ||
|
||
def test_02_01(self, test): | ||
""" Annotation test: devices/../label (reset to default, set 64-byte value, set >64-byte, check IS04+version)""" | ||
return self.do_test_sequence(test, "devices", "label") | ||
|
||
def test_02_02(self, test): | ||
"""Annotation test: devices/../description (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
return self.do_test_sequence(test, "devices", "description") | ||
|
||
def test_02_03(self, test): | ||
"""Annotation test: devices/../tags (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
return self.do_test_sequence(test, "devices", "tags") | ||
|
||
def test_03_01(self, test): | ||
""" Annotation test: senders/../label (reset to default, set 64-byte value, set >64-byte, check IS04+version)""" | ||
return self.do_test_sequence(test, "senders", "label") | ||
|
||
def test_03_02(self, test): | ||
"""Annotation test: senders/../description (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
return self.do_test_sequence(test, "senders", "description") | ||
|
||
def test_03_03(self, test): | ||
"""Annotation test: sender/sevices/../tags (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typo |
||
return self.do_test_sequence(test, "senders", "tags") | ||
|
||
def test_04_01(self, test): | ||
""" Annotation test: receivers/../label (reset to default, set 64-byte value, set >64-byte, check IS04+version)""" | ||
return self.do_test_sequence(test, "receivers", "label") | ||
|
||
def test_04_02(self, test): | ||
"""Annotation test: receivers/../description (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
return self.do_test_sequence(test, "receivers", "description") | ||
|
||
def test_04_03(self, test): | ||
"""Annotation test: receivers/sevices/../tags (reset to default, set 5 tags, set >5 tags, check IS04+version)""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typo |
||
return self.do_test_sequence(test, "receivers", "tags") |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These strings appear in the UI to describe the test, so let's wordsmith somewhat, @jonathan-r-thorpe?