Composite `by:` group-by leaks the hash table on every execution (v2.5.14)

[alexey-milovidov]

Summary

A select with a composite by: key leaks memory proportional to the number
of groups, on every execution. The memory is never returned, so running the same
query repeatedly in one process grows RSS linearly until the process OOMs.

A single-column by: over the same number of groups does not leak, so this is
specific to the multi-key group path rather than to grouping in general.

Found while benchmarking Rayforce on ClickBench
(ClickHouse/ClickBench#1375),
where it is the reason several queries degrade or kill the server.

Repro

leak.rfl — 20M rows, 20M distinct [a b] pairs, the same query six times:

(set n 20000000)
(set idx (til n))
(.db.splayed.set "t" (table [a b] (list idx (% idx 1000))))
(set t (.db.splayed.get "t"))

(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(set r (select {from: t by: [a b] c: (count a) desc: c take: 10}))
(exit 0)

$ /usr/bin/time -v rayforce leak.rfl

Only one result is live at a time (each set r drops the previous one), and the
result is 10 rows.

Numbers

Peak RSS as a function of how many times the identical query runs in one process:

repeats	by: [a b] (20M groups)	by: a (20M groups)	by: b (1000 groups)	no grouping
1	6.6 GB	0.80 GB	0.17 GB	0.17 GB
2	9.3 GB	0.80 GB	0.17 GB	—
3	12.5 GB	0.79 GB	0.17 GB	—
6	22.7 GB	0.80 GB	0.17 GB	0.17 GB

by: [a b] grows by ~3.2 GB per execution and never comes back down. The other
three are flat. Note the second column: same 20M groups, same aggregate, single
key — no leak. Nothing here is retained by the script.

It scales with the group count, not the row count:

rows / groups	peak after 1	peak after 4	leaked per execution
5M	2.2 GB	5.7 GB	1.16 GB (~233 B/group)
10M	3.7 GB	9.5 GB	1.90 GB (~195 B/group)
20M	6.6 GB	16.0 GB	3.09 GB (~158 B/group)

Against a server (rayforce -p 5000) the same thing happens across IPC requests,
so a long-lived server accumulates it until queries start failing:

try 1: 2.83 s  RSS 4.5 GB
try 2: 1.86 s  RSS 7.9 GB
try 3: 2.60 s  RSS 11.2 GB
try 4: 3.49 s  RSS 14.4 GB
try 5: 1.82 s  RSS 17.6 GB
try 6: 1.79 s  RSS 20.8 GB

A cheap query issued afterwards runs fine and frees nothing (RSS stays at
20.8 GB), so it is not the last result being held.

-m does not contain it. With -m 200M RSS still climbed to 19 GB over five
executions before error: oom, and the spill path made one execution take
101 s instead of ~1 s.

Why it matters

On ClickBench (100M rows, 43 queries, 3 runs each) the queries with composite
keys and ~100M groups are exactly the ones that misbehave, and the extrapolated
leak (~16 GB per execution at 100M groups) matches what happens on a 32 GB
machine:

• by: [MobilePhone MobilePhoneModel] with (count (distinct UserID)) — server
  dies during the first run (error: io: connection closed)
• by: [UserID SearchPhrase] — run 1 is 4.30 s, runs 2 and 3 are 2.42 s and
  52.07 s; a second full benchmark on an identical machine got 11.39 s and
  92.71 s for the same two runs
• by: {UserID (minute EventTime) SearchPhrase} — 8.65 s, then 54.51 s and
  62.03 s
• by: [WatchID ClientIP] — 88 s, then error: oom twice

Run 1 of each query is reproducible to three significant digits across
independent benchmark runs, because the benchmark restarts the server before
every query. Runs 2 and 3 are not, because they execute in a process the first
run has already poisoned.

Environment

• Rayforce v2.5.14 (6067f3a), built from source with make release
• Linux aarch64 (also observed on x86-64 — the ClickBench runs above are
  c6a.4xlarge and c8g.4xlarge)