Repository navigation
Description
Describe the issue
After the fluent-bit config Secret changes, fluent-bit pods often reload the previous config and stay on it until the next change. The pod is Ready and logs [reload] successful reload done., so nothing shows that it is one version behind. With namespaced FluentBitConfig resources a newly created route is then missing on part of the nodes, and the logs of that namespace are not delivered from them.
fluent-bit-watcher sends SIGHUP on the first Create/Write event in /fluent-bit/config (cmd/fluent-watcher/fluentbit/main.go, isValidEvent). The kubelet updates a Secret volume atomically, which produces a burst of events:
- create a new
..<timestamp>directory - first event - write the files into it
- create
..data_tmpand rename it to..data- more events - remove the old directory
The first event arrives before ..data points to the new directory, so fluent-bit starts the hot reload and reads the old content. The events of step 3 follow a fraction of a millisecond later and are dropped because a reload is already running:
level=info msg="Config file changed, reloading..."
[engine] caught signal (SIGHUP)
[ info] reloading instance pid=14 tid=0x7ff8ee586e80
[engine] caught signal (SIGHUP)
[engine] caught signal (SIGHUP)
level=info msg="Config file changed, reloading..."
level=info msg="Config file changed, reloading..."
[error] reloading in progress, aborting.
[error] reloading in progress, aborting.
[ info] [reload] stop everything of the old context
[ info] [reload] start everything
[ info] [reload] successful reload done.
When the old main config is read together with files that are already switched (for example the Lua script of a removed filter), the same race shows up as [filter_lua] filter cannot be loaded / [reload] reloaded config is invalid. Reloading is halted.
On a cluster with 120 fluent-bit pods, after one namespaced route was deleted and re-created, about 70% of the pods that reloaded were running the config from before the change (visible in the [input:emitter:<name>] initializing lines fluent-bit prints after each reload).
To Reproduce
In a cluster: change anything that changes the rendered config (for example create a namespaced FluentBitConfig with a Filter/Output), wait for the pods to log successful reload done, then compare what they loaded with the Secret. Part of the pods run the previous version.
Without a cluster, with docker (a volume updated the way the kubelet does it):
cat > update.sh <<'EOS'
#!/bin/sh
n=$1; d="..ts_$n"; old=$(readlink /c/..data 2>/dev/null)
mkdir "/c/$d"
printf '[SERVICE]\n Flush 1\n Grace 1\n[INPUT]\n Name dummy\n Tag v%s\n Rate 2\n[OUTPUT]\n Name stdout\n Match *\n' "$n" > "/c/$d/fluent-bit.conf"
ln -s "$d" /c/..data_tmp
mv -T /c/..data_tmp /c/..data
[ -e /c/fluent-bit.conf ] || ln -s ..data/fluent-bit.conf /c/fluent-bit.conf
[ -n "$old" ] && rm -rf "/c/$old"
exit 0
EOS
docker volume create fbvol
docker run -d --name upd -v fbvol:/c -v "$PWD/update.sh:/update.sh:ro" busybox:1.36 sleep 3600
docker exec upd sh /update.sh 0
docker run -d --name fb -v fbvol:/fluent-bit/config ghcr.io/fluent/fluent-operator/fluent-bit:5.1.0
sleep 4
docker exec upd sh /update.sh 1; sleep 7
docker logs --tail 5 fb # records are still tagged v0
docker exec upd sh /update.sh 2; sleep 7
docker logs --tail 5 fb # now v1, one version behindResult with ghcr.io/fluent/fluent-operator/fluent-bit:5.1.0: 8 of 8 updates ended on the previous config. With the watcher flag -poll: 0 of 10.
Expected behavior
After a config change every pod runs the new config, with one reload per change.
Your Environment
- Fluent Operator version: v3.10.0 (chart 4.3.0), fluent-bit image 5.1.0; the watcher code is the same on master
- Container Runtime: containerd
- Operating system: Amazon Linux 2023 (EKS, Kubernetes 1.34); also reproduced with docker as above
- Kernel version: not relevant to the raceHow did you install fluent operator?
Helm chart fluent-operator 4.3.0.
Additional context
Workaround: run the watcher with -poll (FluentBit.spec.args). A fix is proposed in #2140.
Activity