twister: recording: Allow multiple patterns & Thread-Metric benchmark data collection

[golowanow]

1. Extend Twister Harness 'recording' feature to allow multiple regular expression patterns to extract different types of records from test output.
   Add 'merge' recording mode to collect all extracted data fields into a single record object of the test instance.
   Export to CSV file now takes all field names occurred in the collected records, sort it alphabetically, and then use it for columns instead of using only the first record's fields. This is done to address possible situation when records have different set of fields.
   Adjust Twister documentation and test suite to the above changes.

2. Adjust testcase.yaml files to changes in Twister schema which now allows multiple recording patterns ('record: regex:').

3. Extentd benchmark.thread_metric (tests/benchmarks/thread_metric) test suite to collect benchmark measurements into Twister reports as recordings parsed from the test's output: time period values as well as errors.
   Additionally, each test is executed until it makes at least 3 measurements to estimate variance.

[golowanow]

Thread-MetricX log snippet with invalid counter error artificially introduced to demonstrate recordings (test suite PASSED though):

twister - DEBUG - Spawning QEMUHandler Thread for qemu_x86/atom/tests/benchmarks/thread_metric/benchmark.thread_metric.preemptive
twister - DEBUG - QEMU (852855): SeaBIOS (version zephyr-v1.0.0-0-g31d4e0e-dirty-20200714_234759-fv-az50-zephyr)
twister - DEBUG - QEMU (852855): Booting from ROM..
twister - DEBUG - QEMU (852855): *** Booting Zephyr OS build v4.0.0-2367-g303aa9ecf9f5 ***
twister - DEBUG - QEMU (852855): **** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
twister - DEBUG - HARNESS:Console:EXPECTED(1/6):'(.*) Thread-Metric(.+) Relative Time:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855): ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
twister - DEBUG - QEMU (852855):    Average: 1777871, 0: 355574, 1: 355574, 2: 355574, 3: 355574, 4: 355575
twister - DEBUG - QEMU (852855): Time Period Total:  1777871
twister - DEBUG - HARNESS:Console:EXPECTED(2/6):'(.*)Time Period Total:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855):.
twister - DEBUG - QEMU (852855): **** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
twister - DEBUG - HARNESS:Console:EXPECTED(3/6):'(.*) Thread-Metric(.+) Relative Time:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855): ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
twister - DEBUG - QEMU (852855):    Average: 3555990, 0: 711198, 1: 711198, 2: 711198, 3: 711198, 4: 711198
twister - DEBUG - QEMU (852855): Time Period Total:  1778119
twister - DEBUG - HARNESS:Console:EXPECTED(4/6):'(.*)Time Period Total:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855):.
twister - DEBUG - QEMU (852855): **** Thread-Metric Preemptive Scheduling Test **** Relative Time: 90
twister - DEBUG - HARNESS:Console:EXPECTED(5/6):'(.*) Thread-Metric(.+) Relative Time:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855): ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
twister - DEBUG - QEMU (852855):    Average: 5334110, 0: 1066822, 1: 1066822, 2: 1066822, 3: 1066822, 4: 1066822
twister - DEBUG - QEMU (852855): Time Period Total:  1778120
twister - DEBUG - HARNESS:Console:EXPECTED(6/6):'(.*)Time Period Total:[ ]*[0-9]+(.*)'
twister - DEBUG - QEMU (852855):.
twister - DEBUG - QEMU (852855) complete with passed (None) after 5.891331195831299 seconds
twister - DEBUG - No timeout, return code from QEMU (None): 0
twister - DEBUG - return code from QEMU (None): 0
twister - DEBUG - run status: qemu_x86/atom/tests/benchmarks/thread_metric/benchmark.thread_metric.preemptive passed
twister - INFO - 2/8 qemu_x86/atom             tests/benchmarks/thread_metric/benchmark.thread_metric.preemptive PASSED (qemu 5.891s)
twister - INFO -                                    benchmark.thread_metric.preemptive                                          PASSED

resulting JSON entries:

        {
            "name":"tests/benchmarks/thread_metric/benchmark.thread_metric.preemptive",
            "platform":"qemu_x86/atom",
            "status":"passed",
...
            "recording":[
                {
                    "error_message":[
                        "Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!",
                        "Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!",
                        "Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!"
                    ],
                    "error_details":[
                        "1777871, 0: 355574, 1: 355574, 2: 355574, 3: 355574, 4: 355575",
                        "3555990, 0: 711198, 1: 711198, 2: 711198, 3: 711198, 4: 711198",
                        "5334110, 0: 1066822, 1: 1066822, 2: 1066822, 3: 1066822, 4: 1066822"
                    ],
                    "total_time_period":[
                        "1777871",
                        "1778119",
                        "1778120"
                    ]
                }
            ]
        },

[golowanow]

it seems on CI it needs very close to 300 sec. just for one measurement at benchmark.thread_metric.preemptive
This platform was also noted in #79570 (comment)

[nashif]

ThreadX benchmark ...

We have no threadx benchmarks in zephyr, you probably mean "thread metric" benchmark.

[golowanow]

ThreadX benchmark ...

We have no threadx benchmarks in zephyr, you probably mean "thread metric" benchmark.

right, the PR's title changed to Thread-Metric