-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add new test suite for context merging #293
base: main
Are you sure you want to change the base?
feat: Add new test suite for context merging #293
Conversation
Signed-off-by: christian.lutnik <christian.lutnik@dynatrace.com>
| Before Hooks | | ||
|
||
Scenario: A context entry is added to each level with different keys | ||
Given A context entry with key "API" and value "API value" is added to the "API" level |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you prefer the same line with different values each time, or a data table for the Given
and Then
parts of this scenario?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A datatable could here be beneficial, can we tag on datatables, because we do have the same problem as above, what if not all levels are supported by the sdk, how can we limit this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apparently tags can only enable/disable a whole table, not individual lines in a table. So we would need a table for every permutation of possible implementations if I understood that correctly.
Alternatively I could add a tag for each of the levels, or at least for transaction
and hook
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i am not sure tags work on steps, normally they only work on scenarios or examples. from the docs at https://cucumber.io/docs/cucumber/api#tags:
Tags can be placed above the following Gherkin elements:
Feature
Rule
Scenario
Scenario Outline
Examples
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, that's a problem. So how can we easily enable/disable these steps? Should we really duplicate all tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As long as we only need to duplicate gherkin tests, and not have additional effort in code, i think that is a feasible solution. Gherkin scenarios are cheap, not a lot of complexity and easy to grasp, there is no code to understand nor bigger complexity to grasp.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried something new by dublicating most of the tests. I don't really like this solution that much, but we should now at least be able to support implementations with and without hooks and/or transactions. Refactoring this into a Datatable would make the code implementation a bit more complicated I think, so I left it for now as individual steps
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, i think those gherkin tests will make our live easier in the future. But we have to be a little bit more careful how we enable certain tests with tagging. not all sdks support the whole feature set, and we want them to use this files and limit the usage. We should think about how to properly tag certain tests, or move them in separate scenarios. eg. transaction context merging
When Some flag was evaluated | ||
Then The merged context contains an entry with key "key" and value "value" | ||
|
||
Examples: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we might want to start adding tags here, not all sdks support eg. hooks or transaction - hence i think maybe adding a tag for context merges and the level would be good something
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, but here we also have the problem that tags can only be used on a whole table, not on individual lines inside a table
| Before Hooks | | ||
|
||
Scenario: A context entry is added to each level with different keys | ||
Given A context entry with key "API" and value "API value" is added to the "API" level |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A datatable could here be beneficial, can we tag on datatables, because we do have the same problem as above, what if not all levels are supported by the sdk, how can we limit this?
Signed-off-by: christian.lutnik <christian.lutnik@dynatrace.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good, we might want to have also one test with @Hooks @Transaction
testing both together - i know it is another duplication, and to be honest, i do not have a hard feeling for or against them. Thank you for improving our gherkin tests.
This PR
Adds gherkin tests to verify the merging order of contexts: https://openfeature.dev/specification/sections/evaluation-context/#requirement-323
See open-feature/flagd-testbed#13