Add support for informative tests #793
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some tests are required to be executed, but their results do not have to be successful. They are meant to inform the user that their results need some attention, but they are non-blocking. An example of such test is a certain code scanner which may produce a high number of false positives. These tests are referred to as informative. This commit modifies the `test` package to more easily support these types of tests. Also, since `test.no_skipped_tests` has been modified to emit a violation instead of a warning, an effective date to a month from now has been added. This should allow users to gradually address any issues before they become blocking. Ref: EC-224 Signed-off-by: Luiz Carvalho <[email protected]>
zregvart
approved these changes
Nov 8, 2023
zregvart
added a commit
to zregvart/ec-cli
that referenced
this pull request
Jan 11, 2024
With EC-224 and the change in [1] we time-bombed the acceptance tests. This modifies the scenario slightly so a different rule, one emitting a warning, is used. I believe the original intent of the test is maintained. [1] conforma/policy#793
zregvart
added a commit
to zregvart/ec-cli
that referenced
this pull request
Jan 11, 2024
With EC-224 and the change in [1] we time-bombed the acceptance tests. This modifies the scenario slightly so a different rule, one emitting a warning, is used. I believe the original intent of the test is maintained. [1] conforma/policy#793
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some tests are required to be executed, but their results do not have to be successful. They are meant to inform the user that their results need some attention, but they are non-blocking. An example of such test is a certain code scanner which may produce a high number of false positives. These tests are referred to as informative.
This commit modifies the
testpackage to more easily support these types of tests.Also, since
test.no_skipped_testshas been modified to emit a violation instead of a warning, an effective date to a month from now has been added. This should allow users to gradually address any issues before they become blocking.Ref: EC-224