Feature/Cucumber Messages #233
Conversation
|
FYI, github Build is failing b/c one produced component (FileSinkPlugin) gets published only to my local Nuget store. It can then not be found by the build pipeline in github. |
|
First end-to-end working scenario is operational.
Next:
|
clrudolphi
force-pushed
the
feature-CucumberMessages
branch
from
364e466 to
1f2f1e6
Compare
|
Build is failing b/c I have a private build of Cucumber Messages nuget package. When the next release of that comes out, I will patch up the nuget references in this branch. |
This is fine. We have feed https://www.myget.org/feed/Packages/reqnroll-unstable where we could publish interim releases (of dependencies as well) and I think this feed is added to the nuget sources, so the build would be able to resolve it though them. But I never tested this model yet. |
…mber Messages Publisher and Broker to the initialization sequence provided by the Plugin architecture.
…est project. The File Sink is hooked in as a nuget package. Next: exploratory debugging to confirm the event listeners are wired up correctly at Feature start time.
Need to figure out why trace logging also not working.
…check of whether it had been initialized (temp). (Temp) - added Debugger.Launch to FileSinkPlugin to force launch of debugger.
…Publisher from the FileSinkPlugIn completely.
…the use of a BlockingCollection. Eliminated base class for Sinks. Eliminated use of Async for MessageSinks. Sinks have to ensure their own thread-safety and non-blocking behavior. May want to revisit that.
…ke Messages (meta, Source, GherkinDocument, and Pickles).
… information into Messages. First example is transforming StepBinding classes into messages of type StepDefinition.
…pport for CucumberMessages for that feature.
…e, then all are published.
…ioState emit sequences of Envelopes for consumption by the Publisher.
…fter scenario completion so that we have full binding Match info available to create the TestCase message.
… simple scenarios.
minimal and pending scenarios added.
… which I don't think apply to the .NET implementation. Need to add support for images and documents as embedded resources to be added to generated test projects before attachments and hooks can be implemented and tested. Not yet added infrastructure to compare output ndjon with expected.
…ly handle binary files.
Fails on undefined steps (but not specific to Hooks). Made change to TestExecutionEngine to add Feature, Scenario, and Step context to the HookBinding events.
… count for the test case is incremented on the testCaseStarted message. Added a Test Scenario for this, leveraging the xRetry.Reqnroll plugin from nuget. For this test, the test framework is switched from MSTest to XUnit. Not yet fully compliant b/c xRetry doesn't behave properly when a step fails b/c it is UNDEFINED.
|
First of all, those are great finds. Good job on such thorough investigation. Onto the problem itself: ideally, a good API design prevents invalid states, but in this instance, I would suggest we allow plug-ins to publish messages that would be considered invalid. We're treating Cucumber Messages as a mechanism for providing visibility of test execution and if the plugin causes invalid messages to be published, it's on the plugin to decide whether it's important to its users to be fully compatible with the spec. It may be completely compatible with the tools in use by the plugin users! The alternative would be to fail, either loudly causing a runtime failure for a user of the retry plugin, or silently and make the whole problem really opaque. I don't believe either of those are desirable failure modes, as both cause issues for the end-user that either stop them running tests or give them mysteriously incomplete output. |
An alternative that occurred to me is to create an Interface that retry plugins must implement that we would call upon during handling of ScenarioFinished event. The interface would allow the retry plugins the opportunity to tell us whether a failed test case would be retried. I would implement a Null implementation of this interface and register it in the global container. Retry plugins would implement the interface and replace the null impl with their own in the global container. Thoughts? Too complicated? EDIT: resolved - see comment below |
Updated the configuration.md file. Modified the constants file to make use of 'Directory' consistent across the configuration and constants.
…uired refactoring and duplication of the retry feature and expected results.
…o that actual Messages are sorted by PickleID of the test case(Scenario). This will ensure that Attempt counts are properly compared.
|
Merged in changes that provide for support of Retried test cases. These changes refactored how execution progress is tracked and accounts for retried scenarios. Message generation is deferred until the entire Feature is finished. This allows us to identify that a scenario has been executed multiple times and adjust the 'willBeRetried' flag on prior executions. |
TODO: Need proper ordering; instead of direct publish, these should go in to the Messages cache for sorting by timestamp. TODO: Enhance Validator to support these message types.
…w Message types of TestHookStarted and Finished. Enhanced the validation test code to support these two Message types when building cross-reference indexes.
|
Added support for two new Message types (TestRunHookStarted and Finished). These correspond to Reqnroll's Before/After TestRun and Before/After Feature hooks. |
…urbed by Messages.
…cifying '[BaseDirectory]'
|
This has nothing to do with the PR work per se, but the only way I can help as is, is by showing my appreciation of your efforts! Keep on rockin'!! :) @clrudolphi @gasparnagy |
…a test (that has derived from SystemTest). Relocated all embedded resource files to the conventional location. Moved the External Data Interop basic test's resources to the Samples directory to align them with the pattern.
Adds support for creation of Cucumber Messages during the execution of Reqnroll generated tests.
See Cucumber Message for information about the output format.
🤔 What's changed?
Changes are in three main area:
Messages are strings serialized from the CucumberMessage Envelope format and stored one per line in an ".ndjson" result file.
Configuration support is added as a new section of the reqnroll.json configuration file. (Without interfering with the existing configuration schema and config reading mechanisms). Configuration settings can also be overridden by use of environment variables.
⚡️ What's your motivation?
To provide future compatibility with the broader Cucumber eco-system of tooling.
To provide the input to a future implementation of a Reqnroll replacement to LivingDocs.
🏷️ What kind of change is this?
♻️ Anything particular you want feedback on?
Anything.
📋 Checklist: