Skip to content

feat(fluentbit): add the prometheus_remote_write input - #2145

Open
NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2144-fluentbit-prometheus-remote-write-input
Open

NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2144-fluentbit-prometheus-remote-write-input

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds a prometheusRemoteWrite input to ClusterInput, for Fluent Bit's prometheus_remote_write input plugin. It receives metrics pushed with the Prometheus remote write protocol, for example by Prometheus, Grafana Alloy or Fluent Bit's own prometheus_remote_write output. Until now it was only usable through customPlugin.

spec:
  prometheusRemoteWrite:
    tag: prom.rw
    port: 8080
    uri: /api/prom/push

This renders as:

[Input]
    Name    prometheus_remote_write
    tag    prom.rw
    port    8080
    uri    /api/prom/push

The input covers listen, port, uri, bufferMaxSize, bufferChunkSize, successfulResponseCode and threaded. It also takes the shared tls settings that the http input uses, so the listener can serve HTTPS, with keyPassword read from a Secret.

The CRD catches two mistakes that otherwise only show up at runtime:

  • successfulResponseCode is limited to 200, 201 and 204. Fluent Bit logs an error for any other value and falls back to 201.
  • uri must start with /. Fluent Bit compares it exactly against the request path, so without the slash every request is rejected with 400 invalid endpoint.

I left out tag_from_uri: in 5.2.0 the plugin registers it but never reads it. The port description reminds users to add the port to the FluentBit ports too, so the Service exposes it.

The deepcopy code, CRDs, setup.yaml and API docs are regenerated, and the input is listed in docs/plugins/fluentbit/index.md.

Which issue(s) this PR fixes:

Fixes #2144

Does this PR introduced a user-facing change?

Add the prometheus_remote_write input plugin to the Fluent Bit ClusterInput.

Additional documentation, usage docs, etc.:

New docs/plugins/fluentbit/input/prometheus_remote_write.md

Testing

  • TestInput_PrometheusRemoteWrite_Params and TestInput_PrometheusRemoteWrite_ParamsOnlySetFields check the rendered parameters with every field set, with none and with some.

  • TestInput_PrometheusRemoteWrite_ParamsTLS checks the TLS parameters, including a key password read from a Secret and the error when that key is missing.

  • TestClusterInputList_Load_PrometheusRemoteWrite checks the classic and YAML output of a ClusterInput.

  • make generate manifests docs-update leaves no diff, make lint reports 0 issues, and go test ./apis/fluentbit/... passes.

  • I ran the rendered configs on fluent/fluent-bit:5.2.0, with a second Fluent Bit sending fluentbit_metrics through its prometheus_remote_write output:

    • Classic config with every field set, including TLS on 8443, successfulResponseCode: 204 and threaded: true.
    • YAML config over plain HTTP.

    In both cases the receiver printed the incoming metrics (e.g. fluentbit_uptime{hostname="..."} = 1) and the sender logged no errors.

This PR and #2143 (kafka input) each add a field next to execWasi in InputSpec. Whichever merges second will need its generated files regenerated.

Add a prometheusRemoteWrite field to the ClusterInput spec for Fluent
Bit's prometheus_remote_write input plugin, which receives metrics sent
with the Prometheus remote write protocol. Until now it was only usable
through customPlugin.

The input renders tag, listen, port, uri, buffer_max_size,
buffer_chunk_size, successful_response_code and threaded, and reuses
the shared TLS settings like the http input, so the listener can serve
HTTPS with a key password read from a Secret.

The CRD restricts successfulResponseCode to 200, 201 and 204, the only
codes Fluent Bit accepts, and requires uri to start with a slash, since
Fluent Bit compares it with the request path and rejects every request
otherwise. tag_from_uri is left out because the plugin registers it but
never reads it.

Regenerate the deepcopy code, CRDs, setup.yaml and API docs, and list
the input in docs/plugins/fluentbit/index.md.

Fixes fluent#2144

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support the Fluent Bit prometheus_remote_write input

1 participant