Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add new test suite for context merging #293

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

chrfwow
Copy link
Contributor

@chrfwow chrfwow commented Feb 6, 2025

This PR

Adds gherkin tests to verify the merging order of contexts: https://openfeature.dev/specification/sections/evaluation-context/#requirement-323

See open-feature/flagd-testbed#13

Signed-off-by: christian.lutnik <christian.lutnik@dynatrace.com>
@chrfwow chrfwow requested a review from toddbaert as a code owner February 6, 2025 13:25
| Before Hooks |

Scenario: A context entry is added to each level with different keys
Given A context entry with key "API" and value "API value" is added to the "API" level
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you prefer the same line with different values each time, or a data table for the Given and Then parts of this scenario?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A datatable could here be beneficial, can we tag on datatables, because we do have the same problem as above, what if not all levels are supported by the sdk, how can we limit this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently tags can only enable/disable a whole table, not individual lines in a table. So we would need a table for every permutation of possible implementations if I understood that correctly.
Alternatively I could add a tag for each of the levels, or at least for transaction and hook

Copy link
Member

@aepfli aepfli Feb 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am not sure tags work on steps, normally they only work on scenarios or examples. from the docs at https://cucumber.io/docs/cucumber/api#tags:

Tags can be placed above the following Gherkin elements:

    Feature
    Rule
    Scenario
    Scenario Outline
    Examples

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a problem. So how can we easily enable/disable these steps? Should we really duplicate all tests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as we only need to duplicate gherkin tests, and not have additional effort in code, i think that is a feasible solution. Gherkin scenarios are cheap, not a lot of complexity and easy to grasp, there is no code to understand nor bigger complexity to grasp.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried something new by dublicating most of the tests. I don't really like this solution that much, but we should now at least be able to support implementations with and without hooks and/or transactions. Refactoring this into a Datatable would make the code implementation a bit more complicated I think, so I left it for now as individual steps

Copy link
Member

@aepfli aepfli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, i think those gherkin tests will make our live easier in the future. But we have to be a little bit more careful how we enable certain tests with tagging. not all sdks support the whole feature set, and we want them to use this files and limit the usage. We should think about how to properly tag certain tests, or move them in separate scenarios. eg. transaction context merging

When Some flag was evaluated
Then The merged context contains an entry with key "key" and value "value"

Examples:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might want to start adding tags here, not all sdks support eg. hooks or transaction - hence i think maybe adding a tag for context merges and the level would be good something

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but here we also have the problem that tags can only be used on a whole table, not on individual lines inside a table

| Before Hooks |

Scenario: A context entry is added to each level with different keys
Given A context entry with key "API" and value "API value" is added to the "API" level
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A datatable could here be beneficial, can we tag on datatables, because we do have the same problem as above, what if not all levels are supported by the sdk, how can we limit this?

Signed-off-by: christian.lutnik <christian.lutnik@dynatrace.com>
@chrfwow chrfwow requested a review from aepfli February 26, 2025 07:19
Signed-off-by: christian.lutnik <christian.lutnik@dynatrace.com>
Copy link
Member

@aepfli aepfli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, we might want to have also one test with @Hooks @Transaction testing both together - i know it is another duplication, and to be honest, i do not have a hard feeling for or against them. Thank you for improving our gherkin tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants