Repository navigation
Conversation
bbdd587 to
9367008
Compare
joshuabaird
left a comment
There was a problem hiding this comment.
Thanks for this, the approach looks good. I checked out the branch and go test ./apis/... and go vet both pass. I also ran make manifests, make generate and make docs-update, and they produce no diff, so the hand-edited CRDs, chart copies, setup.yaml and docs match the generator output.
The single filters: header change in RenderMainConfigInYaml is simpler than the old branching and behaves the same. The default ordering is unchanged.
A few comments, none blocking:
afterNamespaceFiltersshows up in the namespacedFilterCRD and is silently ignored there. Someone could set it on aFilterand get no error. Would it be worth rejecting it for namespaced Filters (webhook or a CEL rule)? If not, I'd reword theFilterCRD description, since it currently talks about ClusterFilters.- In
TestRenderMainConfig_ClusterFilterAfterNamespaceFilters,strings.Index(config, "namespaced")matches the first occurrence of the word anywhere in the config. It works today, but a more distinctivesetvalue would be sturdier. - The last block of that test unmarshals into the same
parsedmap as the earlier block. Unmarshalling into an existing map merges keys, so I'd start from a fresh map. A short comment on whyafterOnlyis passed as the cluster filter list would also help. - The
ordinalcomment could mention that ordering applies within the before and after groups, and a short usage example in the docs would help. SplitByNamespaceFiltersis exported but only used internally. Unexport it unless you expect external callers.
Overall this looks good to me.
|
Thx for checking the generated files! Took all five in 169761f. Namespaced Filters now reject afterNamespaceFilters with a CEL rule on the Filter spec, so I updated the description to say that. The test uses distinct stage values and a fresh map with a comment on afterOnly, the ordinal comment explains the two groups, there's a short example in docs/best-practice/misc.md, and the split helper is unexported. I added the CEL rule and the description changes to the CRDs by hand, so it'd be worth rerunning make manifests to confirm they still match. |
joshuabaird
left a comment
There was a problem hiding this comment.
Thanks, the changes look good. I tested the CEL rule against envtest and it rejects afterNamespaceFilters: true on both create and update of a Filter, while ClusterFilters still accept it. "Basic test and verify" is failing. Locally, make manifests re-wraps the new ordinal description in the CRDs and setup.yaml, which would fail the git diff --exit-code check. Could you run make manifests and push the result? I'll approve once CI is green.
|
Thx for testing the CEL rule! 071630e wraps the ordinal description the way the generator does, matching the diff from the verify job. |
Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
…nd tidy the ordering docs and test Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
071630e to
7ab01de
Compare
|
@joshuabaird the verify job is green now with the rewrapped descriptions. The push dismissed your earlier review, so could you take another look when you get a chance? thx |
|
Also ran make generate manifests docs-update locally on 7ab01de and it comes out clean, no diff. |
joshuabaird
left a comment
There was a problem hiding this comment.
Thanks for the updates, the earlier comments all look addressed. I found one problem with how this behaves at runtime though, so I'm requesting changes.
Namespaced Filters only match namespaced tags. GenerateNamespacedMatchExpr turns kube.* into <md5(ns)>.kube.*, and the namespace-routing rewrite_tag filter re-tags records to <md5(ns)>.$TAG and drops the original. The re-tagged record then goes back through the filter chain from the start.
That means a ClusterFilter with match: kube.* and afterNamespaceFilters: true (the example in the PR description and in docs/best-practice/misc.md) never sees the records that went through namespaced Filters, because kube.* doesn't match <md5>.kube.foo. It only processes records from namespaces with no namespaced pipeline, where the ordering makes no difference. To get the intended behavior today a user would need match: "*" or a matchRegex that accounts for the md5 prefix.
The new test doesn't catch this because it only checks text order in the rendered config and passes no rewriteTagConfigs.
Could you make an "after" ClusterFilter also match the namespaced form of its tag? For example, generate a Match_Regex that allows an optional [0-9a-f]{32}\. prefix, or render it once per namespace that has a FluentBitConfig. A test that includes the rewrite_tag config and checks the after filter's match against the re-tagged form would cover it.
If you'd rather not change the matching, then at least the docs and example need to say the filter must match the namespaced tag, but I think fixing the match is what #1387 is really asking for.
Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
…nd tidy the ordering docs and test Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
7ab01de to
b85bd82
Compare
…:cvas22/fluent-operator into feat/1387-clusterfilter-after-namespaced
…ter namespaced Filters Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
|
Good catch, thx. ca5b90d renders an after-namespace ClusterFilter's match as a Match_Regex with an optional |
…filter-after-namespaced # Conflicts: # apis/fluentbit/v1alpha2/clusterfilter_types_test.go
|
@joshuabaird when you get a chance, could you take another look? ca5b90d should cover the re-tagged records case you found. thx |
What this PR does / why we need it:
ClusterFilters always go before the namespaced Filters in the generated config, and
ordinalonly sorts within one kind, so there was no way to run a cluster-wide filter on records after the namespaced Filters had processed them. This addsafterNamespaceFiltersto the ClusterFilter spec, which is the shape suggested in the issue. When it's true, the filter goes after the namespaced Filters, in both the classic and YAML config formats. The default stays the same, so existing configs render as they do today.Which issue(s) this PR fixes:
Fixes #1387
Does this PR introduced a user-facing change?
Additional documentation, usage docs, etc.:
Notes
The field lives in
FilterSpec, which namespaced Filters share, so it shows up in the Filter CRD too. Namespaced Filters ignore it, and the field description says so. I went this way rather than giving ClusterFilter its own spec type, because a new type would break every Go caller that builds aClusterFilter{Spec: FilterSpec{...}}. Happy to change that if you'd rather keep it out of the Filter CRD.For YAML, the cluster, rewrite_tag and namespaced filters now share a single
filters:header written in one place, which replaces the old branch that decided which section should write it. I added a test that checks the order in both formats, and that an "after" filter with a lower ordinal still lands after the namespaced ones. I hand-edited the CRD copies and docs/fluentbit.md becausemake manifestsand the doc generator don't run cleanly on my machine.@CharlieR-o-o-t I saw you'd picked this up in January. Since nothing had landed I went ahead, but let me know if you have something in progress.