Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner: add level-based diagnostic handling for reporter #55964

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

hpatel292-seneca
Copy link

@hpatel292-seneca hpatel292-seneca commented Nov 23, 2024

This fixes #55922

Change summary

Updated the reporter.diagnostic to accept level parameter like this

  diagnostic(nesting, loc, message, level = 'info') {
    this[kEmitMessage]('test:diagnostic', {
      __proto__: null,
      nesting,
      message,
      level,
      ...loc,
    });
  }

Then I updated #handleEvent like this

 #handleEvent({ type, data }) {
    switch (type) {
      case 'test:fail':
        if (data.details?.error?.failureType !== kSubtestsFailed) {
          ArrayPrototypePush(this.#failedTests, data);
        }
        return this.#handleTestReportEvent(type, data);
      case 'test:pass':
        return this.#handleTestReportEvent(type, data);
      case 'test:start':
        ArrayPrototypeUnshift(this.#stack, { __proto__: null, data, type });
        break;
      case 'test:stderr':
      case 'test:stdout':
        return data.message;
      case 'test:diagnostic':  // Here I added logic
        const diagnosticColor =
          reporterColorMap[data.level] || reporterColorMap['test:diagnostic'];
        return `${diagnosticColor}${indent(data.nesting)}${
          reporterUnicodeSymbolMap[type]
        }${data.message}${colors.white}\n`;
      case 'test:coverage':
        return getCoverageReport(
          indent(data.nesting),
          data.summary,
          reporterUnicodeSymbolMap['test:coverage'],
          colors.blue,
          true
        );
    }
  }

And I am Updated reporterColorMap like this

const reporterColorMap = {
  __proto__: null,
  get 'test:fail'() {
    return colors.red;
  },
  get 'test:pass'() {
    return colors.green;
  },
  get 'test:diagnostic'() {
    return colors.blue;
  },
  get info() { // Here I added logic
    return colors.blue;
  },
  get debug() { 
    return colors.gray;
  },
  get warn() { 
    return colors.yellow;
  },
  get error() { 
    return colors.red;
  },
};

and color already contain logic for this colors

I also set the reporter.diagnostic call from test.js like this (level="Error")

if (actual < threshold) {
            harness.success = false;
            process.exitCode = kGenericUserError;
            reporter.diagnostic(
              nesting,
              loc,
              `Error: ${NumberPrototypeToFixed(
                actual,
                2
              )}% ${name} coverage does not meet threshold of ${threshold}%.`,
              'error'  // Level is set to error for red color
            );
          }

Here is Demo output:
image

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/test_runner

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Nov 23, 2024
@hpatel292-seneca hpatel292-seneca marked this pull request as ready for review November 23, 2024 03:05
@hpatel292-seneca
Copy link
Author

Hi @pmarchini, Please review and let me know if you want me to change anything.

@hpatel292-seneca hpatel292-seneca marked this pull request as draft November 23, 2024 03:33
@pmarchini pmarchini requested review from cjihrig and MoLow November 23, 2024 03:34
@pmarchini
Copy link
Member

Hey @hpatel292-seneca, I won't be able to review until Monday.
In the meantime, I suggest you take a look at the test runner output tests; we definitely need to add tests there.

I've also requested other reviews in the meantime.

Copy link

codecov bot commented Nov 23, 2024

Codecov Report

Attention: Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.54%. Comparing base (e92499c) to head (c06e9e1).
Report is 270 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/test_runner/reporter/utils.js 77.77% 2 Missing ⚠️
lib/internal/test_runner/test.js 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #55964      +/-   ##
==========================================
+ Coverage   87.99%   88.54%   +0.54%     
==========================================
  Files         653      657       +4     
  Lines      188091   190215    +2124     
  Branches    35941    36529     +588     
==========================================
+ Hits       165516   168431    +2915     
+ Misses      15751    14976     -775     
+ Partials     6824     6808      -16     
Files with missing lines Coverage Δ
lib/internal/test_runner/reporter/spec.js 96.26% <100.00%> (+0.07%) ⬆️
lib/internal/test_runner/tests_stream.js 91.66% <100.00%> (+0.04%) ⬆️
lib/internal/test_runner/test.js 96.95% <0.00%> (-0.02%) ⬇️
lib/internal/test_runner/reporter/utils.js 95.00% <77.77%> (-1.71%) ⬇️

... and 154 files with indirect coverage changes

Copy link
Contributor

@cjihrig cjihrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes since this already has an approval. This also needs docs and tests.

Comment on lines 43 to 45
get 'debug'() {
return colors.gray;
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the debug level. The reporter stream is not a generic logger, and we have other ways (NODE_DEBUG) of adding debug output.

Copy link
Author

@hpatel292-seneca hpatel292-seneca Nov 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove that. And I have a question so this CI https://github.com/nodejs/node/actions/runs/11983534043/job/33427085217?pr=55964 failed and it's for First commit message adheres to guidelines / lint-commit-message (pull_request) so should I change commit history??

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_runner: add level to diagnostics

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I am asking if I should change the commit history and force push.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Yes, please. You'll need to avoid merge commits too, as the tooling does not handle them well.

Copy link
Author

@hpatel292-seneca hpatel292-seneca Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmarchini Ok so I am taking reference from this tests
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.mjs
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.snapshot

and based on that I wrote this test.
fixtures/test-runner/output/spec_reporter_diagnostic_levels.mjs

// Flags: --test-reporter=spec

import { test } from 'node:test';
import { TestsStream } from '../../../lib/internal/test_runner/tests_stream.js';

process.env.FORCE_COLOR = '3';

const testsStream = new TestsStream();

test('Diagnostic Levels Color Output', () => {
  testsStream.diagnostic(1, {}, 'Info-level message', 'info');
  testsStream.diagnostic(1, {}, 'Warning-level message', 'warn');
  testsStream.diagnostic(1, {}, 'Error-level message', 'error');
});

and add this in test-runner-output.mjs

{
  name: 'test-runner/output/spec_reporter_diagnostic_levels.mjs',
  transform: specTransform,
  tty: true,
}

now I am running tools/test.py test/parallel/test-runner-output.mjs --snapshot but it's not creating snap for added test.
How I can create snaps??

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got this when I tried using test with release/node
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot you were working in a Windows environment. Considering this, I would suggest picking a different approach. You could add a test to node/test/parallel/test-runner-coverage-thresholds.js that forces the colors via the environment variable in the spawn, and check that the error message 'coverage does not meet...' is displayed in red.
Note: you need to set the reporter to spec.

This is also because test-runner-output.mjs is intended for 'e2e' testing of the test runner's output under specific circumstances.

Copy link
Author

@hpatel292-seneca hpatel292-seneca Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmarchini @cjihrig Thank you so much for your help. It wouldn't be possible without your help.
I ran test cases and here is output
I printed raw output in test case and here is output
image
and here is test case:
image

test(`test failing ${coverage.flag} with red color`, () => {
    const result = spawnSync(process.execPath, [
      '--test',
      '--experimental-test-coverage',
      `${coverage.flag}=99`,
      '--test-reporter', 'spec',
      fixture,
    ], {
      env: { ...process.env, FORCE_COLOR: '3' },
    });

    const stdout = result.stdout.toString();
    const redColorRegex = /\u001b\[31m Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/;
    assert.match(stdout, redColorRegex, 'Expected red color code not found in diagnostic message');
    assert.strictEqual(result.status, 1);
    assert(!findCoverageFileForPid(result.pid));
  });

If it looks good to you I can commit it and you can review the whole PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @pmarchini @cjihrig @jasnell @MoLow
Could we please land this PR if no change is required??

Added a parameter to allow severity-based formatting for
diagnostic messages. Defaults to 'info'.
This update enables better control over message presentation
(e.g., coloring) based on severity levels such as 'info', 'warn',
and 'error'.

Refs: nodejs#55922
Updated to process the parameter for events. Messages
are now formatted with colors based on the
(e.g., 'info', 'warn', 'error').
This change ensures diagnostic messages are visually distinct,
improving clarity and reducing debugging effort during test runs.

Refs: nodejs#55922
Enhanced  to include colors for the following
diagnostic levels:
 : blue - info
 : yellow - warn
 : red - error

Refs: nodejs#55922
Updated coverage threshold checks in to use the
parameter when calling. Errors now use the
'error' level for red-colored formatting.
This ensures coverage errors are highlighted effectively in the output.

Fixes: nodejs#55922
implemented requested change by removing debug
from reporterColorMap

Refs: nodejs#55964 (review)
updated the documentation for the 'test:diagnostic' event to
include the new level parameter. clarified its purpose, default
value, and possible severity levels ('info', 'warn',
'error').

Fixes: nodejs#55922
Add a test in  to verify that
the diagnostic error messages about unmet coverage thresholds
are displayed in red when using the spec reporter.

Fixes: nodejs#55922
@hpatel292-seneca
Copy link
Author

Hi @pmarchini @cjihrig,
The PR is ready for review.

@hpatel292-seneca
Copy link
Author

hpatel292-seneca commented Nov 27, 2024

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

@pmarchini
Copy link
Member

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

Sure 🚀

Added eslint-disable comment to bypass no-control-regex.
This allows testing ANSI escape sequences for red color
in error messages without triggering lint errors.

Fixes: nodejs#55922
@hpatel292-seneca
Copy link
Author

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

Sure 🚀

@pmarchini I did. Thanks

@hpatel292-seneca
Copy link
Author

Hi @pmarchini @cjihrig,
PR is ready for review and all CI passed. Please let me know if anything needs to be changed.
Thanks.

Updated the description of the  parameter to note
that color output is specific to the spec reporter. This
helps users understand its behavior and create custom
reporters with accurate expectations.

Fixes: nodejs#55922
@cjihrig cjihrig dismissed their stale review December 5, 2024 02:29

Dismissing my objection

@cjihrig
Copy link
Contributor

cjihrig commented Dec 5, 2024

Is it possible to add or update a test that verifies the data added to the event. I'm not sure that validating red text coming from the spec reporter is thorough enough.

@hpatel292-seneca
Copy link
Author

Is it possible to add or update a test that verifies the data added to the event. I'm not sure that validating red text coming from the spec reporter is thorough enough.

How I can test that? Could you please elaborate??

@cjihrig
Copy link
Contributor

cjihrig commented Dec 7, 2024

The run() API returns the stream that emits these events. You could call run() and perform some assertions on the diagnostic events coming out of the stream.

@hpatel292-seneca
Copy link
Author

hpatel292-seneca commented Dec 12, 2024

Hi @cjihrig,
Based on the this tests:

it('should succeed with a file', async () => {

I figured out that I should test like this

it('should emit diagnostic events with correct level and message', async () => {
    const diagnosticEvents = [];

    const stream = run({
      files: ['test-file.js'], // I am confused here!!
      reporter: 'spec',
    });

    stream.on('test:diagnostic', (event) => {
      diagnosticEvents.push(event.data);
    });

    for await (const _ of stream);

    assert(diagnosticEvents.length > 0, 'No diagnostic events were emitted');
    const errorEvent = diagnosticEvents.find((e) => e.level === 'error');
    assert(errorEvent, 'No error-level diagnostic events found');
    assert.match(
      errorEvent.message,
     "Error Message",
      'Diagnostic message format mismatch'
    );
  });

I am not sure how to write that test-file.js which I want to run in this test.

@cjihrig
Copy link
Contributor

cjihrig commented Dec 13, 2024

I am not sure how to write that test-file.js which I want to run in this test.

I think you'd want to do something like this. That will let you run the same test fixture that you're running in your test from test-runner-coverage-thresholds.js.

@hpatel292-seneca
Copy link
Author

I am not sure how to write that test-file.js which I want to run in this test.

I think you'd want to do something like this. That will let you run the same test fixture that you're running in your test from test-runner-coverage-thresholds.js.

So I tried that like this

it('should emit diagnostic events with correct level and message', async () => {
    const diagnosticEvents = [];

    // Run the test suite and capture events
    const stream = run({
      files: [join(testFixtures, 'coverage.js')],
      reporter: 'spec',
    });

    stream.on('test:diagnostic', (event) => {
      diagnosticEvents.push(event);
    });

    for await (const _ of stream);
    console.log(diagnosticEvents) // here I am printing the events
    // Assertions on diagnostic events
    assert(diagnosticEvents.length > 0, 'No diagnostic events were emitted');
    const errorEvent = diagnosticEvents.find((e) => e.level === 'error');
  });

So I think I am able to capture test:diagnostic events and as you can see from above test I am also printing the events and here how it looks

[
 [Object: null prototype] {
   nesting: 0,
   message: 'tests 1',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'suites 0',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'pass 1',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'fail 0',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'cancelled 0',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'skipped 0',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'todo 0',
   level: 'info'
 },
 [Object: null prototype] {
   nesting: 0,
   message: 'duration_ms 1277.1178',
   level: 'info'
 }
]

As you can see from the output diagnostic events contain the level parameter which is good but it's not emitting level: error may be because I am passing the flags like I did in my test in test-runner-coverage-thresholds.js. This flags:

const result = spawnSync(process.execPath, [
      '--test',
      '--experimental-test-coverage',
      `${coverage.flag}=99`,
      '--test-reporter', 'spec',
      fixture,
    ], {
      env: { ...process.env, FORCE_COLOR: '3' },
    });
    ```
I tried adding flags in run api call but it's not working. 
 

@cjihrig
Copy link
Contributor

cjihrig commented Dec 13, 2024

I think that's fine. We don't need to make sure the level is set to 'error'. I think 'info' is good enough. We mostly just want to test the event to make sure that the red text coming from the reporter is not due to something else.

add a test to ensure that diagnostic events emitted by the test
runner contain level parameter.

Refs: nodejs#55964
@hpatel292-seneca
Copy link
Author

Hi @cjihrig,
I have added the test. Please review and let me know if any changes are required. Thanks

Copy link
Contributor

@cjihrig cjihrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cjihrig cjihrig added request-ci Add this label to start a Jenkins CI on a PR. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Dec 13, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 13, 2024
@nodejs-github-bot
Copy link
Collaborator

Added eslint-disable-next-line to
bypass no-unused-vars check

ref: nodejs#55964
@cjihrig cjihrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 13, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 13, 2024
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@cjihrig cjihrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 17, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 17, 2024
@nodejs-github-bot
Copy link
Collaborator

@@ -88,6 +88,25 @@ for (const coverage of coverages) {
assert(!findCoverageFileForPid(result.pid));
});

test(`test failing ${coverage.flag} with red color`, () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @hpatel292-seneca, it seems this test is failing in the CI.
Could you please take a look at it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pmarchini, I tried reading logs on failed tests and it seems like this regex
image
is not matching with the output. But as we are not logging anything I am not sure what we are getting output here.

But this test is running as expected and passing locally and in GitHub Action CI.
Github Action run: https://github.com/nodejs/node/actions/runs/12320594158/job/34394971556
Below is the SS of the locally run test
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I saw this CI failure but I am not sure what is causing the failure as this test is running as expected on my local machine. I tried reading logs on CI but I can't find anything. Is there any way I can debug those CI runs??

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @hpatel292-seneca, if you follow the link I've shared, you will find the description of the error.
It seems that the expected red error does not match any part of the output.
I've restarted many times, and the error systematically appears across different OS.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pmarchini, I got your point but the test is running as expected with red color on my machine.
For example: I am printing output in the test like this

  test(`test failing ${coverage.flag} with red color`, () => {
    const result = spawnSync(process.execPath, [
      '--test',
      '--experimental-test-coverage',
      `${coverage.flag}=99`,
      '--test-reporter', 'spec',
      fixture,
    ], {
      env: { ...process.env, FORCE_COLOR: '3' },
    });

    const stdout = result.stdout.toString();
    console.log(stdout);  \\ Here I am printing output
    // eslint-disable-next-line no-control-regex
    const redColorRegex = /\u001b\[31m Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/;
    assert.match(stdout, redColorRegex, 'Expected red color code not found in diagnostic message');
    assert.strictEqual(result.status, 1);
    assert(!findCoverageFileForPid(result.pid));
  });

And this is the output I am getting:
image

And here is the logs of failed pipeline run

actual: 'invalid tap output
✔ a test (3.3287ms)

ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 304.826895',

expected: 
/\u001b[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/,

So basically, it is not even printing the coverage error which I think it shoud print if I am using this flags --experimental-test-coverage, ${coverage.flag}=99, and --test-reporter', 'spec which I am using in the test.

Do you think it is a environment or configuration issue??

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say yes but in the CI it's also failing under Windows OS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a level parameter to test runner diagnostics
6 participants