Skip to content

bug: fluent-bit keeps the previous config after a config change (watcher reloads before the Secret volume is switched) #2141

Open

Description

@CharlieR-o-o-t

Describe the issue

After the fluent-bit config Secret changes, fluent-bit pods often reload the previous config and stay on it until the next change. The pod is Ready and logs [reload] successful reload done., so nothing shows that it is one version behind. With namespaced FluentBitConfig resources a newly created route is then missing on part of the nodes, and the logs of that namespace are not delivered from them.

fluent-bit-watcher sends SIGHUP on the first Create/Write event in /fluent-bit/config (cmd/fluent-watcher/fluentbit/main.go, isValidEvent). The kubelet updates a Secret volume atomically, which produces a burst of events:

  1. create a new ..<timestamp> directory - first event
  2. write the files into it
  3. create ..data_tmp and rename it to ..data - more events
  4. remove the old directory

The first event arrives before ..data points to the new directory, so fluent-bit starts the hot reload and reads the old content. The events of step 3 follow a fraction of a millisecond later and are dropped because a reload is already running:

level=info msg="Config file changed, reloading..."
[engine] caught signal (SIGHUP)
[ info] reloading instance pid=14 tid=0x7ff8ee586e80
[engine] caught signal (SIGHUP)
[engine] caught signal (SIGHUP)
level=info msg="Config file changed, reloading..."
level=info msg="Config file changed, reloading..."
[error] reloading in progress, aborting.
[error] reloading in progress, aborting.
[ info] [reload] stop everything of the old context
[ info] [reload] start everything
[ info] [reload] successful reload done.

When the old main config is read together with files that are already switched (for example the Lua script of a removed filter), the same race shows up as [filter_lua] filter cannot be loaded / [reload] reloaded config is invalid. Reloading is halted.

On a cluster with 120 fluent-bit pods, after one namespaced route was deleted and re-created, about 70% of the pods that reloaded were running the config from before the change (visible in the [input:emitter:<name>] initializing lines fluent-bit prints after each reload).

To Reproduce

In a cluster: change anything that changes the rendered config (for example create a namespaced FluentBitConfig with a Filter/Output), wait for the pods to log successful reload done, then compare what they loaded with the Secret. Part of the pods run the previous version.

Without a cluster, with docker (a volume updated the way the kubelet does it):

cat > update.sh <<'EOS'
#!/bin/sh
n=$1; d="..ts_$n"; old=$(readlink /c/..data 2>/dev/null)
mkdir "/c/$d"
printf '[SERVICE]\n    Flush 1\n    Grace 1\n[INPUT]\n    Name dummy\n    Tag v%s\n    Rate 2\n[OUTPUT]\n    Name stdout\n    Match *\n' "$n" > "/c/$d/fluent-bit.conf"
ln -s "$d" /c/..data_tmp
mv -T /c/..data_tmp /c/..data
[ -e /c/fluent-bit.conf ] || ln -s ..data/fluent-bit.conf /c/fluent-bit.conf
[ -n "$old" ] && rm -rf "/c/$old"
exit 0
EOS
docker volume create fbvol
docker run -d --name upd -v fbvol:/c -v "$PWD/update.sh:/update.sh:ro" busybox:1.36 sleep 3600
docker exec upd sh /update.sh 0
docker run -d --name fb -v fbvol:/fluent-bit/config ghcr.io/fluent/fluent-operator/fluent-bit:5.1.0
sleep 4
docker exec upd sh /update.sh 1; sleep 7
docker logs --tail 5 fb    # records are still tagged v0
docker exec upd sh /update.sh 2; sleep 7
docker logs --tail 5 fb    # now v1, one version behind

Result with ghcr.io/fluent/fluent-operator/fluent-bit:5.1.0: 8 of 8 updates ended on the previous config. With the watcher flag -poll: 0 of 10.

Expected behavior

After a config change every pod runs the new config, with one reload per change.

Your Environment

- Fluent Operator version: v3.10.0 (chart 4.3.0), fluent-bit image 5.1.0; the watcher code is the same on master
- Container Runtime: containerd
- Operating system: Amazon Linux 2023 (EKS, Kubernetes 1.34); also reproduced with docker as above
- Kernel version: not relevant to the race

How did you install fluent operator?

Helm chart fluent-operator 4.3.0.

Additional context

Workaround: run the watcher with -poll (FluentBit.spec.args). A fix is proposed in #2140.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions