Repository navigation
feat(fluentbit): add the geoip2 filter - #2147
NopeCharon wants to merge 1 commit into
Conversation
Add a geoip2 field to the ClusterFilter and Filter items for Fluent
Bit's geoip2 filter plugin, which looks up IP addresses of a record in a
MaxMind GeoIP2 or GeoLite2 database and appends location information
such as the country or city. Until now it was only usable through
customPlugin.
The filter renders Database, one Lookup_Key per lookup key and one
Record "KEY LOOKUP_KEY %{path}" per record.
Fluent Bit refuses to start without a database, a lookup key and a
record, so the CRD requires all three. It also rejects keys that contain
whitespace, and values that are not %{path} with a path of 1 to 63
characters: Fluent Bit copies the path into a 64 byte buffer and aborts
with "buffer overflow detected" on the first record when the value is
not wrapped in %{...}.
Regenerate the deepcopy code, CRDs, setup.yaml and API docs, and list
the filter in docs/plugins/fluentbit/index.md.
Fixes fluent#2146
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
|
Hi, I'm Kim from IP2Location. Nice addition, and the structured One small docs suggestion: Fluent Bit's geoip2 filter reads any MMDB file through libmaxminddb, so the filter isn't limited to MaxMind databases. For example, the free IP2Location LITE DB11 MMDB ( Would you be open to wording the IP2Location LITE downloads (free sign-up): https://www.ip2location.com/database/lite |
What this PR does / why we need it:
Adds a
geoip2filter to ClusterFilter and Filter, for Fluent Bit'sgeoip2plugin. It looks up IP addresses from a record in a MaxMind GeoIP2 or GeoLite2 database and adds location fields such as country, city or coordinates. Until now it was only usable throughcustomPlugin.This renders as:
Each record is its own object (
key,lookupKey,value) rather than a free-formKEY LOOKUP_KEY VALUEstring, so the CRD can validate each part:database, at least one lookup key and at least one record are required. Fluent Bit refuses to start without them.valuemust be%{path}with a path of 1 to 63 characters. This one matters most: Fluent Bit copies the path into a 64-byte buffer. With a value likecity.names.en(no%{...}),fluent/fluent-bit:5.2.0aborts with*** buffer overflow detected ***on the first matching record. I reproduced that locally.The database file can be mounted with the existing
volumes/volumeMountsof the FluentBit resource, and thedatabasedescription says so.The deepcopy code, CRDs,
setup.yamland API docs are regenerated, and the filter is listed indocs/plugins/fluentbit/index.md.Which issue(s) this PR fixes:
Fixes #2146
Does this PR introduced a user-facing change?
Additional documentation, usage docs, etc.:
Testing
TestFilter_GeoIP2_Paramschecks the rendered parameters with several lookup keys and records.TestClusterFilterList_Load_GeoIP2checks the classic and YAML output of a ClusterFilter, where the records become a YAML list.make generate manifests docs-updateleaves no diff,make lintreports 0 issues, andgo test ./apis/... ./pkg/...passes.fluent/fluent-bit:5.2.0with adummyinput and MaxMind's publicGeoIP2-City-Test.mmdb. Both enriched the record as expected:{"remote_addr":"81.2.69.142","country":"United Kingdom","city":"London","latitude":51.5142}.