-
Notifications
You must be signed in to change notification settings - Fork 30.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_runner: add level-based diagnostic handling for reporter #55964
base: main
Are you sure you want to change the base?
Conversation
Review requested:
|
Hi @pmarchini, Please review and let me know if you want me to change anything. |
Hey @hpatel292-seneca, I won't be able to review until Monday. I've also requested other reviews in the meantime. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #55964 +/- ##
==========================================
+ Coverage 87.99% 88.54% +0.54%
==========================================
Files 653 657 +4
Lines 188091 190215 +2124
Branches 35941 36529 +588
==========================================
+ Hits 165516 168431 +2915
+ Misses 15751 14976 -775
+ Partials 6824 6808 -16
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Requesting changes since this already has an approval. This also needs docs and tests.
get 'debug'() { | ||
return colors.gray; | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's remove the debug level. The reporter stream is not a generic logger, and we have other ways (NODE_DEBUG
) of adding debug output.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will remove that. And I have a question so this CI https://github.com/nodejs/node/actions/runs/11983534043/job/33427085217?pr=55964 failed and it's for First commit message adheres to guidelines / lint-commit-message (pull_request)
so should I change commit history??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test_runner: add level to diagnostics
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I am asking if I should change the commit history and force push.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah. Yes, please. You'll need to avoid merge commits too, as the tooling does not handle them well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pmarchini Ok so I am taking reference from this tests
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.mjs
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.snapshot
and based on that I wrote this test.
fixtures/test-runner/output/spec_reporter_diagnostic_levels.mjs
// Flags: --test-reporter=spec
import { test } from 'node:test';
import { TestsStream } from '../../../lib/internal/test_runner/tests_stream.js';
process.env.FORCE_COLOR = '3';
const testsStream = new TestsStream();
test('Diagnostic Levels Color Output', () => {
testsStream.diagnostic(1, {}, 'Info-level message', 'info');
testsStream.diagnostic(1, {}, 'Warning-level message', 'warn');
testsStream.diagnostic(1, {}, 'Error-level message', 'error');
});
and add this in test-runner-output.mjs
{
name: 'test-runner/output/spec_reporter_diagnostic_levels.mjs',
transform: specTransform,
tty: true,
}
now I am running tools/test.py test/parallel/test-runner-output.mjs --snapshot
but it's not creating snap for added test.
How I can create snaps??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot you were working in a Windows environment. Considering this, I would suggest picking a different approach. You could add a test to node/test/parallel/test-runner-coverage-thresholds.js
that forces the colors via the environment variable in the spawn
, and check that the error message 'coverage does not meet...' is displayed in red.
Note: you need to set the reporter to spec
.
This is also because test-runner-output.mjs
is intended for 'e2e' testing of the test runner's output under specific circumstances.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pmarchini @cjihrig Thank you so much for your help. It wouldn't be possible without your help.
I ran test cases and here is output
I printed raw output in test case and here is output
and here is test case:
test(`test failing ${coverage.flag} with red color`, () => {
const result = spawnSync(process.execPath, [
'--test',
'--experimental-test-coverage',
`${coverage.flag}=99`,
'--test-reporter', 'spec',
fixture,
], {
env: { ...process.env, FORCE_COLOR: '3' },
});
const stdout = result.stdout.toString();
const redColorRegex = /\u001b\[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/;
assert.match(stdout, redColorRegex, 'Expected red color code not found in diagnostic message');
assert.strictEqual(result.status, 1);
assert(!findCoverageFileForPid(result.pid));
});
If it looks good to you I can commit it and you can review the whole PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @pmarchini @cjihrig @jasnell @MoLow
Could we please land this PR if no change is required??
Added a parameter to allow severity-based formatting for diagnostic messages. Defaults to 'info'. This update enables better control over message presentation (e.g., coloring) based on severity levels such as 'info', 'warn', and 'error'. Refs: nodejs#55922
Updated to process the parameter for events. Messages are now formatted with colors based on the (e.g., 'info', 'warn', 'error'). This change ensures diagnostic messages are visually distinct, improving clarity and reducing debugging effort during test runs. Refs: nodejs#55922
Enhanced to include colors for the following diagnostic levels: : blue - info : yellow - warn : red - error Refs: nodejs#55922
Updated coverage threshold checks in to use the parameter when calling. Errors now use the 'error' level for red-colored formatting. This ensures coverage errors are highlighted effectively in the output. Fixes: nodejs#55922
implemented requested change by removing debug from reporterColorMap Refs: nodejs#55964 (review)
188f357
to
038b0f8
Compare
updated the documentation for the 'test:diagnostic' event to include the new level parameter. clarified its purpose, default value, and possible severity levels ('info', 'warn', 'error'). Fixes: nodejs#55922
Add a test in to verify that the diagnostic error messages about unmet coverage thresholds are displayed in red when using the spec reporter. Fixes: nodejs#55922
Hi @pmarchini @cjihrig, |
Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Should I go ahead and send commit? |
Sure 🚀 |
Added eslint-disable comment to bypass no-control-regex. This allows testing ANSI escape sequences for red color in error messages without triggering lint errors. Fixes: nodejs#55922
@pmarchini I did. Thanks |
Hi @pmarchini @cjihrig, |
Updated the description of the parameter to note that color output is specific to the spec reporter. This helps users understand its behavior and create custom reporters with accurate expectations. Fixes: nodejs#55922
Is it possible to add or update a test that verifies the data added to the event. I'm not sure that validating red text coming from the spec reporter is thorough enough. |
How I can test that? Could you please elaborate?? |
The |
Hi @cjihrig, node/test/parallel/test-runner-run.mjs Line 28 in 96cd2a6
I figured out that I should test like this it('should emit diagnostic events with correct level and message', async () => {
const diagnosticEvents = [];
const stream = run({
files: ['test-file.js'], // I am confused here!!
reporter: 'spec',
});
stream.on('test:diagnostic', (event) => {
diagnosticEvents.push(event.data);
});
for await (const _ of stream);
assert(diagnosticEvents.length > 0, 'No diagnostic events were emitted');
const errorEvent = diagnosticEvents.find((e) => e.level === 'error');
assert(errorEvent, 'No error-level diagnostic events found');
assert.match(
errorEvent.message,
"Error Message",
'Diagnostic message format mismatch'
);
}); I am not sure how to write that test-file.js which I want to run in this test. |
I think you'd want to do something like this. That will let you run the same test fixture that you're running in your test from |
So I tried that like this it('should emit diagnostic events with correct level and message', async () => {
const diagnosticEvents = [];
// Run the test suite and capture events
const stream = run({
files: [join(testFixtures, 'coverage.js')],
reporter: 'spec',
});
stream.on('test:diagnostic', (event) => {
diagnosticEvents.push(event);
});
for await (const _ of stream);
console.log(diagnosticEvents) // here I am printing the events
// Assertions on diagnostic events
assert(diagnosticEvents.length > 0, 'No diagnostic events were emitted');
const errorEvent = diagnosticEvents.find((e) => e.level === 'error');
}); So I think I am able to capture
As you can see from the output diagnostic events contain the const result = spawnSync(process.execPath, [
'--test',
'--experimental-test-coverage',
`${coverage.flag}=99`,
'--test-reporter', 'spec',
fixture,
], {
env: { ...process.env, FORCE_COLOR: '3' },
});
```
I tried adding flags in run api call but it's not working.
|
I think that's fine. We don't need to make sure the level is set to |
add a test to ensure that diagnostic events emitted by the test runner contain level parameter. Refs: nodejs#55964
Hi @cjihrig, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Added eslint-disable-next-line to bypass no-unused-vars check ref: nodejs#55964
@@ -88,6 +88,25 @@ for (const coverage of coverages) { | |||
assert(!findCoverageFileForPid(result.pid)); | |||
}); | |||
|
|||
test(`test failing ${coverage.flag} with red color`, () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @hpatel292-seneca, it seems this test is failing in the CI.
Could you please take a look at it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @pmarchini, I tried reading logs on failed tests and it seems like this regex
is not matching with the output. But as we are not logging anything I am not sure what we are getting output here.
But this test is running as expected and passing locally and in GitHub Action CI.
Github Action run: https://github.com/nodejs/node/actions/runs/12320594158/job/34394971556
Below is the SS of the locally run test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I saw this CI failure but I am not sure what is causing the failure as this test is running as expected on my local machine. I tried reading logs on CI but I can't find anything. Is there any way I can debug those CI runs??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @hpatel292-seneca, if you follow the link I've shared, you will find the description of the error.
It seems that the expected red error does not match any part of the output.
I've restarted many times, and the error systematically appears across different OS.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @pmarchini, I got your point but the test is running as expected with red color on my machine.
For example: I am printing output in the test like this
test(`test failing ${coverage.flag} with red color`, () => {
const result = spawnSync(process.execPath, [
'--test',
'--experimental-test-coverage',
`${coverage.flag}=99`,
'--test-reporter', 'spec',
fixture,
], {
env: { ...process.env, FORCE_COLOR: '3' },
});
const stdout = result.stdout.toString();
console.log(stdout); \\ Here I am printing output
// eslint-disable-next-line no-control-regex
const redColorRegex = /\u001b\[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/;
assert.match(stdout, redColorRegex, 'Expected red color code not found in diagnostic message');
assert.strictEqual(result.status, 1);
assert(!findCoverageFileForPid(result.pid));
});
And this is the output I am getting:
And here is the logs of failed pipeline run
actual: 'invalid tap output
✔ a test (3.3287ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 304.826895',
expected:
/\u001b[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/,
So basically, it is not even printing the coverage error which I think it shoud print if I am using this flags --experimental-test-coverage
, ${coverage.flag}=99
, and --test-reporter', 'spec
which I am using in the test.
Do you think it is a environment or configuration issue??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would say yes but in the CI it's also failing under Windows OS
This fixes #55922
Change summary
Updated the reporter.diagnostic to accept level parameter like this
Then I updated
#handleEvent
like thisAnd I am Updated
reporterColorMap
like thisand color already contain logic for this colors
I also set the reporter.diagnostic call from test.js like this (level="Error")
Here is Demo output:
![image](https://private-user-images.githubusercontent.com/100322816/389002448-6e5ed480-90a9-46b8-82f0-cfbe1025bab9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxNzk3MjAsIm5iZiI6MTczOTE3OTQyMCwicGF0aCI6Ii8xMDAzMjI4MTYvMzg5MDAyNDQ4LTZlNWVkNDgwLTkwYTktNDZiOC04MmYwLWNmYmUxMDI1YmFiOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjEwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIxMFQwOTIzNDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02N2QzMjFmNGU2N2MyNTA4MzE4YmQxNTY2ZDBlZDkyMjRlYzc1YTRkZWU2Njk1NDQ5ZDlkNDgzOWNhZTg1MjkyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.OFb96OJ2sb6FgpNtpEkUen4BKFrLNyFCOW3iuqLGO4w)