Skip to content

pkg/clickhouse, pkg/profile: bounds-check the location decoders - #6416

Open
guettlibot wants to merge 2 commits into
parca-dev:mainfrom
guettli:upstream-pr-decoder-bounds
Open

guettlibot wants to merge 2 commits into
parca-dev:mainfrom
guettli:upstream-pr-decoder-bounds

Conversation

@guettlibot

Copy link
Copy Markdown

Stacked on #6415 — review that one first; this branch contains its commit as a parent. The diff specific to this PR is the second commit.

What

The ClickHouse ingest path decodes every encoded location twice — profile.DecodeSymbolizationInfo for the address and mapping, then clickhouse.decodeLineInfo for the line and function — and neither had a single length check.

Feeding every prefix of one real encoded location to them:

decoder truncations that panic after
profile.DecodeSymbolizationInfo 19 of 66 0
clickhouse.decodeLineInfo 55 of 66 0

grep -rn 'recover()' pkg cmd finds nothing on this path, so a malformed record is a dead server rather than a failed request.

Three ways they could fault

  • data[offset] for the flag bytes, past the end of a short record.
  • data[offset:] for every varint, likewise.
  • data[offset : offset+int(length)] for the strings. The length is unsigned: one above MaxInt converts to a negative int, so offset+int(length) lands below offset. That satisfies a naive offset+int(length) > len(data) guard and then panics on a slice whose high bound is below its low one — so the check has to be made in the space the length was read in.

Panicking was not the worst outcome

In production these bytes come from an Arrow dictionary buffer, and array.Binary.Value returns a two-index slice whose cap runs to the end of the whole buffer. A read past the end of one entry therefore does not fault — it silently returns the bytes of the next location, which then get stored as this location's function filename. Plausible-looking, in a column nothing would flag. Reproduced with a real array.BinaryBuilder: of 71 truncations of entry 0, 20 silently copied neighbouring bytes into FunctionFilename.

A failed read now yields "" — never a prefix, never a neighbour.

Partial, but not silent

A record that runs out yields whatever was decoded before it did: that is the honest answer for a truncated record, and it keeps one bad location from discarding the rest of the batch. But silence here would be its own bug — a systematic drift would degrade every profile to unnamed frames, indistinguishable from legitimately unsymbolized ones, which is the same silent failure #6415 is about. So decodeLineInfo reports malformed records, extractStacktraceData counts them, and Ingest logs one warning per batch with the count. numLines == 0 and a line with no function are valid shapes and are not counted.

Verification

Behaviour on well-formed input is unchanged. A differential harness ran the old and new implementations over 20,864 generated valid records — hasMapping × numLines ∈ {0,1,2,3} × hasFunction × string lengths {0, 1, 127, 128, 300} × numeric widths up to MaxUint64, plus 20,000 seeded-random records — with zero divergences on any LineInfo field.

The truncation sweeps run over six record shapes, because each reaches a different set of reads: the unguarded decoder faulted on 13 of 14 prefixes of a no-function record and 34 of 43 of a no-mapping one, paths a single full-record sweep never visits. Oversized lengths (1<<20, 1<<63, MaxUint64) are covered at the function and mapping string positions.

The sweeps slice encoded[:i:i], not encoded[:i]: a two-index slice drops the original cap, and Go does not panic reading past len while still inside cap, so the cheaper spelling silently passes on inputs that really do over-read — it reports 45 where the capped slice reports 55. Given the Arrow cap behaviour above, the capped number is the one that describes production.

go test (including -race), go vet, gofmt -s and the license check are clean; the new tests fail on the pre-change code.

Deliberately left out

Both pre-existing, both happy to file separately:

  1. profile.decodeString and profile.DecodeInto are unguarded in the same style with the same unsigned-length hazard, reached from pkg/parcacol/querier.go:1172/:1566 and pkg/symbolizer. decodeString has 14 call sites across two packages, so guarding it is a wider change than this one.
  2. EncodeArrowLocation writes the hasFunction flag unconditionally (pkg/profile/encode.go:362) while serializedArrowLocationSize budgets the function block only if lineFunctionName.IsValid(i) — so a line with an invalid function name undersizes the buffer and the encoder overruns it.

Thomas Guettler added 2 commits October 2, 2026 22:29
A Go service's goroutine profile came back from a server running the
ClickHouse backend with every frame unsymbolized, while scraping the very
same /debug/pprof endpoint directly named all of them. The frame count
survived ingestion; the names did not.

The location encoders write a column between the line number and the
hasFunction flag. decodeLineInfo did not read it, so it read the column's
first byte as the flag. For pprof, which carries no column information, that
byte is a uvarint zero, so the flag read as false and the whole function
block -- name, system name, filename, start line -- was skipped.

Nothing failed while this happened. The address, the mapping and the line
number all decoded, the row was written, and the profile came back nameless.
Measured on a live server before this change: 3,632,819 stored frames, every
address and every line number decoded, zero function names.

A non-zero column was worse than lossy. OTLP and Arrow locations carry real
column numbers, and there the misread desynchronised the rest of the record:
column 1 decoded a corrupted name ("\tmain.main"), and a larger start line
walked the offset off the end of the buffer and panicked. The ClickHouse
ingest path has no recover(), so that is a dead server rather than a failed
request.

Only profiles that arrive already symbolized lose names -- anything scraped
from a Go /debug/pprof endpoint, and any other pre-symbolized upload. Those
are also the ones the symbolizer cannot rescue afterwards, because they
usually carry no build ID to look up, so the result reads like a debuginfo
coverage gap rather than a decoder that could not parse what it was handed.

profile.DecodeInto, the canonical decoder, reads line then column then the
flag; every encoder in pkg/profile writes them in that order. decodeLineInfo
was the only parser in the tree that disagreed.

The tests drive the real encoders rather than hand-built blobs, because a
blob written by hand would be written from whatever the decoder happened to
do and so would agree with the bug it existed to catch. They cover the
mapping and no-mapping branches, a line with no function, and non-zero
columns through EncodeOtelLocation -- including multi-byte ones, since a zero
column cannot tell a uvarint read apart from a bare offset++.
The ingest path decodes each encoded location twice -- DecodeSymbolizationInfo
for the address and mapping, decodeLineInfo for the line and function -- and
neither had a single length check. Feeding every prefix of one real encoded
location to them, 19 of 66 truncations panic in the first and 55 of 66 in the
second. There is no recovery interceptor on this path, so that is not a failed
request, it is a dead server. Both are now checked, and both are 0.

The bytes are produced by this server's own encoders, so they are
self-consistent in normal operation. That is what makes the failure mode worth
closing rather than dismissing: the realistic way to reach a malformed record
is encoder/decoder drift, which is exactly the state in which a decoder is
already walking the record wrongly, and one field read at the wrong offset
carries the next read further into territory that was indexed unchecked.

Three distinct ways they could fault:

  - data[offset] for the flag bytes, past the end of a short record.
  - data[offset:] for every varint, likewise.
  - data[offset : offset+int(length)] for the strings. The length is unsigned:
    one above MaxInt converts to a NEGATIVE int, so offset+int(length) lands
    BELOW offset. That satisfies a naive "offset+int(length) > len(data)" guard
    and then panics on a slice whose high bound is below its low one, so the
    check has to be made in the space the length was read in.

Panicking was not even the worst outcome. In production these bytes come from
an Arrow dictionary buffer, and array.Binary.Value slices with cap running to
the end of that whole buffer -- so a read past len but inside cap does not
fault, it silently returns the bytes of the NEXT location. A truncated entry
could therefore store a neighbouring location's bytes as this location's
function filename, as a plausible-looking value, in a column nothing would
flag. A failed read now yields "", never a prefix and never a neighbour.

Each read reports whether it succeeded, and a record that runs out yields
whatever had been decoded before it did. Partial information is the honest
answer for a truncated record and keeps one bad location from discarding the
rest of the batch -- but it must not be silent, or a systematic drift degrades
every profile to unnamed frames while looking exactly like legitimately
unsymbolized ones. decodeLineInfo now reports malformed records,
extractStacktraceData counts them, and Ingest logs one warning per batch with
the count. numLines == 0 and a line with no function are valid shapes and are
not counted.

Behaviour on well-formed input is unchanged; the round-trip tests pin that.
The truncation sweep runs over six record shapes, because each reaches a
different set of reads -- the unguarded decoder faulted on 13 of 14 prefixes of
a no-function record and 34 of 43 of a no-mapping one, paths a single
full-record sweep never visits.

The sweeps slice with encoded[:i:i] rather than encoded[:i]: a two-index slice
drops the original cap, and Go does not panic reading past len while still
inside cap, so the cheaper spelling silently passes on inputs that really do
over-read. It reports 45 where the capped slice reports 55 -- and given the
Arrow cap behaviour above, the capped number is the one that describes
production.

Left for follow-ups, both pre-existing: profile.decodeString and
profile.DecodeInto are unguarded in the same style, with the same unsigned
length hazard, reached from pkg/parcacol and pkg/symbolizer; and
EncodeArrowLocation writes the hasFunction flag unconditionally while
serializedArrowLocationSize budgets the function block only for a valid
function name, so the encoder itself can overrun its buffer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant