-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup test fixtures for resolvers #5317
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some other comments:
- Put
.yaml
and.csv
files into a separateruntime/resolvers/testdata
directory - Add support for ClickHouse and Druid tests
- Do you think it makes sense to have some kind of
--update
flag that will write out resolver results to the test files? To make it easier to author new tests. E.g. see this for an example: https://ieftimov.com/posts/testing-in-go-golden-files/
runtime/security.go
Outdated
type SecurityClaims struct { | ||
// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}. | ||
UserAttributes map[string]any | ||
UserAttributes map[string]any `yaml:"user_attributes"` | ||
// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource. | ||
// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens. | ||
AdditionalRules []*runtimev1.SecurityRule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we remove the yaml
parsing from this type (can be handled with a separate type in the tests)? It makes it look like this type can be serialized/deserialized to YAML, which it can't (because the proto type runtimev1.SecurityRule
is not YAML compatible)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder why we cannot do it YAML compatible, more copies of structs - more structs to maintain - harder to maintain when 2 structs are almost equal (finding the distinction is harder) - more error-prone.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As written above, the *runtimev1.SecurityRule
proto-generated type is not safe for use with YAML, so we don't want to give the impression that the SecurityClaims
type is YAML-serializable. Additionally, it's just a bad practice to hack an internal type that used in many places for a single specific use case (in this case, test fixtures) – it creates unnecessary coupling.
In this case, UserAttributes
is the only field we need to be able to populate from the tests, so it is simple to parse it from the test fixture and create a SecurityClaims
value for it.
runtime/resolvers/resolvers_test.go
Outdated
type Project struct { | ||
Sources map[string]yaml.Node | ||
Models map[string]yaml.Node | ||
Dashboards map[string]yaml.Node | ||
APIs map[string]yaml.Node | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of hard-coding the types, can we just use full paths as key types? We support quite a lot more resource types than just these four now. See this comment:
#5218 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally don't see much difference:
sources:
1.yaml
2.yaml
dashboards:
1.yaml
2.yaml
- sources/1.yaml:
- sources/2.yaml:
- dashboards/1.yaml:
- dashboards/2.yaml:
We can easily add a new field to Project struct if more resources is required.
That could make sense if you'd like to process all files equally with a single for-loop and it can save a few lines of code but you'll have to repeat yourself if you need multiple similar-type entries in the future, ie:
- sources/1.yaml:
- sources/2.yaml:
- dashboards/1.yaml:
- dashboards/2.yaml:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO reducing nesting one level for all the test files definitely improves readability here. The very slight repetition of the directory name does not hurt readabilty IMO, and anyway we don't require specific directory names anymore, so the files can just be created at the root level.
This also has the additional benefit of simplifying the test fixture parsing code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
options: | ||
claims: | ||
user_attributes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the sake of cleaner tests, I think we should flatten the nesting here and just have user_attributes
as a root property of the test type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It relates to the previous argument If you like to have more test structures then it's less production code coverage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand that we would like to see a more concise schema but it's already have a diverse structure, simplifying it is short term goal - eventually we will need to test more production fields anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't impact production code coverage at all – it just makes the test fixture simpler to read/write. Internally the test fixture parser would create a SecurityClaims
value, so the production code coverage is exactly the same.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@egor-ryashin : Any updates on this PR ? |
@nishantmonu51 Most of comments I'll resolve in 1d, the first one "updates" can take longer. |
|
Merge conflicts resolved. |
Updates having this issue go-yaml/yaml#430 |
|
go.mod
Outdated
@@ -261,6 +261,7 @@ require ( | |||
github.com/mattn/go-isatty v0.0.20 // indirect | |||
github.com/mattn/go-runewidth v0.0.15 // indirect | |||
github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect | |||
github.com/maxatome/go-testdeep v1.14.0 // indirect |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indirect dependency added without change to a direct dependency – try running go mod tidy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
if strings.Contains(err.Error(), c.dsn) { // avoid returning the actual DSN with the password which will be logged | ||
return nil, fmt.Errorf("invalid dsn") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps return the original error, but with something like strings.Replace(errMsg, c.dsn, "<masked>")
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -0,0 +1,91 @@ | |||
project: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please move the .yaml
test fixtures into a separate directory at runtime/resolvers/testdata
since the Go docs recommend putting non-Go test files in a separate testdata
directory
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
runtime/testruntime/testruntime.go
Outdated
// logger := zap.NewNop() | ||
// nolint | ||
// logger, err := zap.NewDevelopment() | ||
// require.NoError(t, err) | ||
logger, err := zap.NewDevelopment() | ||
require.NoError(t, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please revert before merging – makes the CI test output difficult to read
runtime/resolvers/resolvers_test.go
Outdated
func TestResolvers(t *testing.T) { | ||
|
||
entries, err := os.ReadDir("./") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: redundant newline
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
runtime/testruntime/testruntime.go
Outdated
InstanceOptions | ||
OLAPDriver string | ||
OLAPDSN string | ||
TempDir string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This appears unused
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
runtime/security.go
Outdated
type SecurityClaims struct { | ||
// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}. | ||
UserAttributes map[string]any | ||
UserAttributes map[string]any `yaml:"user_attributes"` | ||
// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource. | ||
// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens. | ||
AdditionalRules []*runtimev1.SecurityRule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As written above, the *runtimev1.SecurityRule
proto-generated type is not safe for use with YAML, so we don't want to give the impression that the SecurityClaims
type is YAML-serializable. Additionally, it's just a bad practice to hack an internal type that used in many places for a single specific use case (in this case, test fixtures) – it creates unnecessary coupling.
In this case, UserAttributes
is the only field we need to be able to populate from the tests, so it is simple to parse it from the test fixture and create a SecurityClaims
value for it.
runtime/resolvers/resolvers_test.go
Outdated
type Project struct { | ||
Sources map[string]yaml.Node | ||
Models map[string]yaml.Node | ||
Dashboards map[string]yaml.Node | ||
APIs map[string]yaml.Node | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO reducing nesting one level for all the test files definitely improves readability here. The very slight repetition of the directory name does not hurt readabilty IMO, and anyway we don't require specific directory names anymore, so the files can just be created at the root level.
This also has the additional benefit of simplifying the test fixture parsing code.
options: | ||
claims: | ||
user_attributes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't impact production code coverage at all – it just makes the test fixture simpler to read/write. Internally the test fixture parser would create a SecurityClaims
value, so the production code coverage is exactly the same.
runtime/resolvers/resolvers_test.go
Outdated
type Resolvers struct { | ||
Project Project | ||
Connectors map[string]*testruntime.InstanceOptionsForResolvers | ||
Tests map[string]*Test | ||
} | ||
|
||
type Test struct { | ||
Resolver string | ||
Options runtime.ResolveOptions | ||
Result []map[string]any | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I strongly believe it would simplify the code and readability here to combine this into a single struct with anonymous sub-structs to represent the YAML structure – and avoid using internal types.
This is also what we do for the resource YAML parsing, and it serves us well. Look at the AlertYAML
type here as an example:
type AlertYAML struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, but I'll need to copy all the data to ResolveOptions to call a method here:
res, err := rt.Resolve(context.Background(), &ropts)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see SecurityClaims
have a oneof
field, now it makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
All done. |
Can you take a look at the CI failures? |
There's a weird error currently
It's not reproduced locally with the same docker image. |
There's a new exception emerged after the merge:
I'm figuring that out. |
Solved here #5591 |
All done. |
No description provided.