twister: add integration_toolchains support #82515

nashif · 2024-12-03T22:22:10Z

Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.

This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.

nashif · 2024-12-03T22:22:54Z

./scripts/twister -p native_sim -s  tests/kernel/common/kernel.common.toolchain -v
Renaming output directory to /home/nashif/zephyrproject/zephyr/twister-out.4
INFO    - Using Ninja..
INFO    - Zephyr version: v4.0.0-1436-gd9f057ded860
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/nashif/zephyrproject/zephyr/twister-out/testplan.json
INFO    - JOBS: 8
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
INFO    - 1/2 native_sim/native         tests/kernel/common/kernel.common.toolchain        PASSED (native 0.042s (host))
INFO    - 2/2 native_sim/native         tests/kernel/common/kernel.common.toolchain        PASSED (native 0.045s (llvm))```

nashif · 2024-12-11T13:41:44Z

with the sample diff in the test config and ./scripts/twister -vv -ll DEBUG -j 1 -p qemu_x86 -T tests/kernel/sleep in summary kernel.common.timing is counted as filtered test suites (static) and the corresponding log is
DEBUG   - qemu_x86/atom             tests/kernel/sleep/kernel.common.timing            SKIPPED: Not supported by the toolchain

this is now addressed

cfriedt

LGTM - so is this an alternative to filtering based on toolchain or is it intended to replace filtering based on toolchain? Does it produce fewer skipped / filtered test results?

nashif · 2024-12-11T23:34:39Z

LGTM - so is this an alternative to filtering based on toolchain or is it intended to replace filtering based on toolchain? Does it produce fewer skipped / filtered test results?

this adds another vector for testing where desired. Right now a test configurations is based on

test scenario -> platforms1 -> toolchain1
test scenario -> platforms2 -> toolchain1

where toolchain1 is whatever is configured in the environment.

with this, it is possible to have:

test scenario -> platforms1 -> toolchain1
test scenario -> platforms1 -> toolchain2
test scenario -> platforms2 -> toolchain1
test scenario -> platforms2 -> toolchain2

nashif · 2024-12-14T15:35:53Z

added documentation

golowanow · 2024-12-16T08:08:55Z

scripts/pylib/twister/twisterlib/reports.py

@@ -355,6 +355,8 @@ def json_report(self, filename, version="NA", platform=None, filters=None):
                suite["used_rom"] = used_rom

            suite['retries'] = instance.retries
+            if instance.toolchain:


isn't instance.toolchain always set to some value ?

golowanow · 2024-12-16T08:17:01Z

scripts/pylib/twister/twisterlib/testplan.py

+                if itoolchain:
+                    toolchain = itoolchain
+                else:
+                    default_toolchain = "zephyr" if not self.env.toolchain else self.env.toolchain


I think this logic is worth to mention in doc/develop/test/twister.rst

golowanow · 2024-12-16T09:07:40Z

with this artificial attempt to use the new attribute, for native_sim there are ERROR CMake build failure (native <zephyr>), whereas just SKIPPED: Not supported by the toolchain for qemu_x86.

diff --git a/tests/kernel/sleep/testcase.yaml b/tests/kernel/sleep/testcase.yaml   
index eceb053d082..6472dbae001 100644                                              
--- a/tests/kernel/sleep/testcase.yaml                                             
+++ b/tests/kernel/sleep/testcase.yaml                                             
@@ -1,9 +1,13 @@                                                                   
 tests:                                                                            
   kernel.common.timing:                                                           
+    integration_toolchains:                                                       
+      - host                                                                      
     tags:                                                                         
       - kernel                                                                    
       - sleep                                                                     
   kernel.common.timing.minimallibc:                                               
+    integration_toolchains:                                                       
+      - zephyr                                                                    
     filter: CONFIG_MINIMAL_LIBC_SUPPORTED                                         
     tags:                                                                         
       - kernel

with the same probe, I've noticed some confusing log output, e.g. twister -vv -ll DEBUG -T tests/kernel/sleep -p native_sim:

INFO    - Using 'zephyr' toolchain.                                                                                                        
...
DEBUG   - platform filter: ['native_sim']                                                                                                  
...
DEBUG   - Checking platform filter: ['native_sim']                                                                                         
...
DEBUG   - native_sim/native         tests/kernel/sleep/kernel.common.timing.minimallibc FILTERED: Not supported by the toolchain: zephyr

better to change to something like:

INFO    - Toolchains available: 'host', 'zephyr'.                                                                                                        
...
DEBUG   - native_sim/native         tests/kernel/sleep/kernel.common.timing.minimallibc FILTERED: Not supported for the toolchain: 'host'
or
DEBUG   - native_sim/native         tests/kernel/sleep/kernel.common.timing.minimallibc FILTERED: Needs toolchain(s) 'zephyr' not applicable for the platform.

Added integration_toolchains to allow building/testing with multiple toolchains available in the environment. This changes the output structure and adds another level in the path under twister_out signifying the toolchain used. The toolchain used (variant) is also part of the json output now. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Use new keyword integration_toolchains to build test using additional toolchains. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Document the new option and how to use integration_toolchains. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif mentioned this pull request Dec 3, 2024

ci: remove clang workflow, use main twister workflow to build using llvm #82354

Closed

nashif force-pushed the topic/twister/integration_toolhcains branch 4 times, most recently from f5723a1 to c721948 Compare December 6, 2024 20:09

nashif marked this pull request as ready for review December 6, 2024 20:10

zephyrbot added area: Twister Twister area: Kernel labels Dec 6, 2024

zephyrbot requested review from andyross, ceolin, cfriedt, dcpleung, gchwier, golowanow, hakehuang, KamilxPaszkiet, LukaszMrugala, npitre, PerMac, peter-mitsis and TaiJuWu December 6, 2024 20:10

zephyrbot assigned andyross and peter-mitsis Dec 6, 2024

nashif assigned nashif and unassigned andyross and peter-mitsis Dec 6, 2024

nashif force-pushed the topic/twister/integration_toolhcains branch 3 times, most recently from 5f6e67c to 55730a9 Compare December 9, 2024 18:39

nashif force-pushed the topic/twister/integration_toolhcains branch 2 times, most recently from 9e91b51 to 182a7c8 Compare December 11, 2024 12:42

cfriedt previously approved these changes Dec 11, 2024

View reviewed changes

nashif dismissed cfriedt’s stale review via 4659768 December 13, 2024 01:07

nashif force-pushed the topic/twister/integration_toolhcains branch 5 times, most recently from 934987b to 31f3ff8 Compare December 13, 2024 21:34

hakehuang previously approved these changes Dec 14, 2024

View reviewed changes

cfriedt previously approved these changes Dec 14, 2024

View reviewed changes

nashif dismissed stale reviews from cfriedt and hakehuang via f731d98 December 14, 2024 15:34

nashif force-pushed the topic/twister/integration_toolhcains branch from 31f3ff8 to f731d98 Compare December 14, 2024 15:34

zephyrbot added the area: Testsuite Testsuite label Dec 14, 2024

zephyrbot requested a review from aaronemassey December 14, 2024 15:34

golowanow reviewed Dec 16, 2024

View reviewed changes

nashif added 3 commits January 7, 2025 09:29

tests: kernel: common: use integration_toolchains

7bcb67e

Use new keyword integration_toolchains to build test using additional toolchains. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

doc: twister: add documentation of integration_toolchains

d4f940a

Document the new option and how to use integration_toolchains. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif force-pushed the topic/twister/integration_toolhcains branch from f731d98 to d4f940a Compare January 7, 2025 15:50

hakehuang self-requested a review January 8, 2025 03:56

hakehuang approved these changes Jan 8, 2025

View reviewed changes

cfriedt approved these changes Jan 8, 2025

View reviewed changes

kartben merged commit 53486c9 into zephyrproject-rtos:main Jan 8, 2025
37 checks passed

keith-zephyr mentioned this pull request Jan 10, 2025

Twister ignores ZEPHYR_TOOLCHAIN_VARIANT environment variable for posix architecture #83766

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

twister: add integration_toolchains support #82515

twister: add integration_toolchains support #82515

nashif commented Dec 3, 2024 •

edited

Loading

nashif commented Dec 3, 2024

nashif commented Dec 11, 2024

cfriedt left a comment

nashif commented Dec 11, 2024 •

edited

Loading

nashif commented Dec 14, 2024

golowanow Dec 16, 2024

golowanow Dec 16, 2024

golowanow commented Dec 16, 2024 •

edited

Loading

twister: add integration_toolchains support #82515

twister: add integration_toolchains support #82515

Conversation

nashif commented Dec 3, 2024 • edited Loading

nashif commented Dec 3, 2024

nashif commented Dec 11, 2024

cfriedt left a comment

Choose a reason for hiding this comment

nashif commented Dec 11, 2024 • edited Loading

nashif commented Dec 14, 2024

golowanow Dec 16, 2024

Choose a reason for hiding this comment

golowanow Dec 16, 2024

Choose a reason for hiding this comment

golowanow commented Dec 16, 2024 • edited Loading

nashif commented Dec 3, 2024 •

edited

Loading

nashif commented Dec 11, 2024 •

edited

Loading

golowanow commented Dec 16, 2024 •

edited

Loading