-
Notifications
You must be signed in to change notification settings - Fork 12
/
Copy pathTest_HOWTO.txt
278 lines (234 loc) · 12.5 KB
/
Test_HOWTO.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
HOW TO WRITE A TEST
===================
See document qpid-interop-test-devel-overview.txt for an overview of
Tests and Shims, and the relationship between them.
0. Introduction
===============
The Test is the top-level entry point for a particular test objective
in which the clients are to be tested against each other. The overall
idea is to have the clients send and receive messages in such a way as
to test their interoperability. The shims are stand-alone executables
written in the client API / language which send or recieve the messages
so as to achieve the test objectives. The Test calls the shims to
perform the sends and receives, and evaluates the data it received in
order to pass or fail the test objective.
Each test has one or more Shims written to perform the specific
messaging task for that test. Each shim is a stand-alone executable
binary or script which can be called by the Test, and the parameters
sent to the shims and the data received from the shims is
specific to that Test. The practice of using JSON for sending
data to and from the shims has been adopted, as it is fairly widely
available, and can handle the necessary data structure.
1. Test name and location
=========================
Each Test is a single stand-alone Python program located in the
src/python/qpid_interop_test directory. The test is named
<test-name>_test.py
2. Common modules
=================
The following modules are available to Tests:
2.1 qit_broker_props.py
-----------------------
Opens a connection to a broker and reads the connection properties in
order to return a string containing the broker name. This is useful
if you need to know the broker name for any reason. One of the common
uses of this is to conditionally skip certain tests which fail based
on the broker against which they are running.
2.2 qit_common.py
-----------------
Code and classes common to most tests.
This module includes class QitTestTypeMap which contains empty maps
useful for storing test data vs data type for some tests. The class
containing this map is subclasses and populated with test data for
each test. This implies that the test data is saved as literals
within the Test itself (which is current practice). However, using
this technique is optional, and test data can be obtained by any
useful means, so long as it is easy to access and maintain.
NOTE: All tests except amqp_complex_types_test use this technique.
NOTE: Test amqp_complex_types_test stores its test data in JSON
files, then uses a code generator to emit test code for each client.
2.3 qit_errors.py
-----------------
Defines error classes for use with tests. Currently this only
defines a single error class (InteropTestError) which accepts
a text error message, but more complex error classes may
be added if needed.
2.4 qit_jms_types.py
--------------------
Used by JMS tests, this module defines common QpidJms values and
operations.
2.5 qit_shim.py
---------------
This module defines the interface for calling, sending parameters
to and receiving data from Shims.
The Test will use separate worker threads for launching the sender
and receiver shims (so that they are running simultaneously) so that
the broker will have both connections open at the same time. Some
"brokers" (like the Qpid Dispatch Router, which has no message storage
ability) require that the receiver is launched before the sender in
order that there is something to consume the message immediately
they are received.
Each shim object defines how the JSON parameters (and if necessary,
other parameters - such as Java classpath) are sent to the shim for
both the send and receive cases.
+-- Sender
Thread <-- ShimWorkerThread <--+
+-- Receiver
Every client API under test must have a shim object in shims.py. Note
that this is shared by all Tests using this cleint API. The class
structure is as follows:
+-- ProtonPythonShim
|
+-- ProtonCppShim
|
+-- QpidJmsShim
Shim <--+
+-- <client_4>
|
+-- <client_5>
|
...
Each shim object contains an instance of Sender and Receiver when it is
created, and which can be launched on its own thread. When the Sender
and Receiver instance are run, they call the corresponding client Shim
executable with the correct parameters for the test. When the Shim
executable finishes, any data in stdout and stderr are piped back to
the shim object for processing.
HINT: There are some useful debug print statements in the shim Sender
and Receiver run() methods. If enabled during shim development, these
will print out the full parameters containing the JSON data for each
shim as it is launched. In addition, other statements will print
returned data from each shim. Once you have this, it is easy to run
the shim independently from the test using these parameters. This is
especially useful if you need to use a debugger on the shim.
2.5 qit_xunit_log.py
--------------------
This module allows test results to be stored in xunit format.
3. Command-line options
=======================
The Tests should accept the following command-line options:
parameter | action | default | metavar | help
------------------+------------+------------------+--------------+------
--sender | store | localhost:5672 | IP-ADDR:PORT | Node to which test suite will send messages
--receiver | store | localhost:5672 | IP-ADDR:PORT | Node from which test suite will receive messages
--no-skip | store_true | | | Do not skip tests that are excluded by default for reasons of a known bug [1]
--broker-type | store | | BROKER_NAME | Disable test of broker type (using connection properties) by specifying the broker name, or "None" [2]
--timeout | store | varies per test | SEC | Timeout for test in seconds (%d sec). If test is not complete in this time, it will be terminated
--include-shim | append | | SHIM-NAME | Name of shim to include [3][4]
--exclude-shim | append | | SHIM-NAME | Name of shim to exclude [3][5]
--xunit-log | store_true | | | Enable xUnit logging of test results
--xunit-log-dir | store | cwd()/xunit_logs | LOG-DIR-PATH | Default xUnit log directory where xUnit logs are written [xunit_logs dir in current directory (%s)]
--description | store | | DESCR | Detailed description of test, used in xUnit logs
--broker-topology | store | | DESCR | Detailed description of broker topology used in test, used in xUnit logs
Notes:
[1] --no-skip: Only applies if you are using the broke name (through
qit_broker_props.py) to exclude some tests bassed on the broker
name.
[2] --broker-type: Should skip checking for the broker name (through
qit_broker_props.py) and use the given name. "None" may have
specific meaning to some tests for broker that don't return
connection properties.
[3] --include-shim, --exclude-shim: These are mutually exclusive.
[4] --include-shim: This option generally clears the list of known
shims and uses those supplied by this argument. This argument may
be used multiple times.
[5] --exclude-shim: This option generally uses the list of known
shims but deletes this argument from the list. This argument may
be used multiple times.
Parameters for allowing for filtering of test cases should also be
provided so that by a combination of the above shim parameters and
the test data parameters, a test can be narrowed down to a single
test case. For example, in a type test, mutually exclusive parameters
which include or exclude individual types would be appropriate.
4. General Test layout
======================
Tests perform the following tasks:
4.1 Define test cases (and data)
--------------------------------
This is generally done by subclassing class QitTestTypeMap (see 2.2
above). However, any method to obtain a map containing test cases as
the index and the test data for that case as the value is appropriate.
If known test failures exist for certain brokers, then define a
BROKER_SKIP map in which the affected test cases are the index,
and each value consists of a sub-map with the broker name as the
index and a text reason for the skip as the value.
If a client has a known failure or if it does not support certain
types, then define a CLIENT_SKIP map in which the affected test cases
are the index, and each value consists of a sub-map with the client
name as the index and a text reason for the skip as the value.
NOTE: Every test skipped in this manner should reference a JIRA issue
for why that test is failing. This makes it easier to track and
re-enable the tests when the issue is resolved. It also encourages
reporting of issues.
4.2 Define your test class
--------------------------
Derive your test class from class QitTestCase. Make sure it has a run_test()
method, and which should receive as parameters the send and receive shim
objects being used for this test case. This method should perform the
following tasks:
* Create a unique queue name for this test;
* Create a Sender object by calling the shim create_sender() method;
* Create a Receiver object by calling the shim create_receiver() method;
* Run the Sender and Receiver by calling the start() method on each object.
This will cause the Shims to be called on separate threads through the OS.
* Wait for the Sender and Receiver shims to finish executing by calling
join().
* Process any returned errors by failing the test.
* If no errors are present, process the returned data and make a decision
to pass or fail the test based on the received data values. Usually these
are compared with the sent values to determine correctness.
4.3 Create your test cases
--------------------------
These can be created either statically or dynamically. How you choose to do it
depends on the type of test and how repetitive it is. If the tests are very
different and specific, then using the static method will work better
4.3.1 Static tests methods
--------------------------
Create one method per test labeled test<test_name> in the test class. See
https://docs.python.org/3.8/library/unittest.html for details. Each test should
accomplish a single objective using a single or limited set of similar
messages.
4.3.2 Dynamically created test methods
--------------------------------------
Were tests consist of the permutations and combinations of a number of test
shims and test values, it is possible to create the test methods dynamically
which saves a lot of error and repetition. An example may be found in
amqp_types-test, where the method which creates the test is
create_testcase_class(amqp_type, shim_product, timeout).
4.4 Add test options
--------------------
Use argparse.ArgumentParser to add test arguments. See section 3 above for
recommended options. Individual tests may also define additional test-specific
options as needed to control/modify that test.
4.5 Create a shim map
---------------------
In the main section of the test, create a shim map as follows:
{<shim_name>: <shim_instance>,
...
}
the product of which is used to run the shims against each other when
dynamically creating test cases.
5. Write the shims
==================
Each test needs to call one or more shims to perform the task of sending and
receiving messages. If only one shim exists, then it will test sending and
receiving against itself. See Shim_HOWTO for details on shim writing.
6. Test
=======
Get it working. 'Nuff said.
7. Add to the suite at Qpid Interop Test
========================================
If you have added a useful test, consider adding it to Qpid Interop Test.
Submit a JIRA at https://issues.apache.org/jira/browse/QPIDIT/.