Skip to content

feat(fluentbit): add the geoip2 filter - #2147

Open
NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2146-fluentbit-geoip2-filter
Open

NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2146-fluentbit-geoip2-filter

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds a geoip2 filter to ClusterFilter and Filter, for Fluent Bit's geoip2 plugin. It looks up IP addresses from a record in a MaxMind GeoIP2 or GeoLite2 database and adds location fields such as country, city or coordinates. Until now it was only usable through customPlugin.

spec:
  match: "nginx.*"
  filters:
    - geoip2:
        database: /fluent-bit/geoip/GeoLite2-City.mmdb
        lookupKeys:
          - remote_addr
        records:
          - key: country
            lookupKey: remote_addr
            value: "%{country.names.en}"
          - key: city
            lookupKey: remote_addr
            value: "%{city.names.en}"

This renders as:

[Filter]
    Name    geoip2
    Match    nginx.*
    Database    /fluent-bit/geoip/GeoLite2-City.mmdb
    Lookup_Key    remote_addr
    Record    country remote_addr %{country.names.en}
    Record    city remote_addr %{city.names.en}

Each record is its own object (key, lookupKey, value) rather than a free-form KEY LOOKUP_KEY VALUE string, so the CRD can validate each part:

  • database, at least one lookup key and at least one record are required. Fluent Bit refuses to start without them.
  • Keys can't contain whitespace.
  • value must be %{path} with a path of 1 to 63 characters. This one matters most: Fluent Bit copies the path into a 64-byte buffer. With a value like city.names.en (no %{...}), fluent/fluent-bit:5.2.0 aborts with *** buffer overflow detected *** on the first matching record. I reproduced that locally.

The database file can be mounted with the existing volumes / volumeMounts of the FluentBit resource, and the database description says so.

The deepcopy code, CRDs, setup.yaml and API docs are regenerated, and the filter is listed in docs/plugins/fluentbit/index.md.

Which issue(s) this PR fixes:

Fixes #2146

Does this PR introduced a user-facing change?

Add the geoip2 filter plugin to the Fluent Bit ClusterFilter and Filter.

Additional documentation, usage docs, etc.:

New docs/plugins/fluentbit/filter/geoip2.md

Testing

  • TestFilter_GeoIP2_Params checks the rendered parameters with several lookup keys and records.
  • TestClusterFilterList_Load_GeoIP2 checks the classic and YAML output of a ClusterFilter, where the records become a YAML list.
  • make generate manifests docs-update leaves no diff, make lint reports 0 issues, and go test ./apis/... ./pkg/... passes.
  • I ran both rendered configs (classic and YAML) on fluent/fluent-bit:5.2.0 with a dummy input and MaxMind's public GeoIP2-City-Test.mmdb. Both enriched the record as expected: {"remote_addr":"81.2.69.142","country":"United Kingdom","city":"London","latitude":51.5142}.

Add a geoip2 field to the ClusterFilter and Filter items for Fluent
Bit's geoip2 filter plugin, which looks up IP addresses of a record in a
MaxMind GeoIP2 or GeoLite2 database and appends location information
such as the country or city. Until now it was only usable through
customPlugin.

The filter renders Database, one Lookup_Key per lookup key and one
Record "KEY LOOKUP_KEY %{path}" per record.

Fluent Bit refuses to start without a database, a lookup key and a
record, so the CRD requires all three. It also rejects keys that contain
whitespace, and values that are not %{path} with a path of 1 to 63
characters: Fluent Bit copies the path into a 64 byte buffer and aborts
with "buffer overflow detected" on the first record when the value is
not wrapped in %{...}.

Regenerate the deepcopy code, CRDs, setup.yaml and API docs, and list
the filter in docs/plugins/fluentbit/index.md.

Fixes fluent#2146

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
@kim-ip2location

Copy link
Copy Markdown

Hi, I'm Kim from IP2Location.

Nice addition, and the structured records with the %{path} validation is a good guard against that buffer overflow.

One small docs suggestion: Fluent Bit's geoip2 filter reads any MMDB file through libmaxminddb, so the filter isn't limited to MaxMind databases. For example, the free IP2Location LITE DB11 MMDB (DB11LITEMMDB) uses the same GeoLite2-City field layout, so the example in this PR works unchanged with just a different file:

filters:
  - geoip2:
      database: /fluent-bit/geoip/IP2LOCATION-LITE-DB11.MMDB
      lookupKeys:
        - remote_addr
      records:
        - key: country
          lookupKey: remote_addr
          value: "%{country.names.en}"
        - key: city
          lookupKey: remote_addr
          value: "%{city.names.en}"

Would you be open to wording the database description and geoip2.md as "an MMDB database (e.g. MaxMind GeoIP2/GeoLite2 or IP2Location LITE)"? Happy to send the exact text as a suggestion if that helps.

IP2Location LITE downloads (free sign-up): https://www.ip2location.com/database/lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support the Fluent Bit geoip2 filter

2 participants