fix panic in stream push when context deadline expires during request marshalling#7541
Open
SungJin1212 wants to merge 3 commits into
Open
fix panic in stream push when context deadline expires during request marshalling#7541SungJin1212 wants to merge 3 commits into
SungJin1212 wants to merge 3 commits into
Conversation
dosubot
Bot
added
component/distributor
go
SungJin1212
changed the title
Member
Author
|
@alexqyle |
… marshalling Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
SungJin1212
force-pushed
the
fix-push-stream-out-of-range-panic
branch
from
93b0a3e to
c93d079
Compare
friedrichg
reviewed
May 26, 2026
| } | ||
| c.streamPushChan <- job | ||
| <-reqCtx.Done() | ||
| <-job.sendDone |
Member
There was a problem hiding this comment.
Suggested change
| <-job.sendDone | |
| select { | |
| case <-job.sendDone: | |
| return job.resp, job.err | |
| case <-ctx.Done(): | |
| <-job.sendDone | |
| return nil, ctx.Err() | |
| } |
| defer pushCancel() | ||
|
|
||
| // PushStreamConnection blocks until Send()+Recv() complete. | ||
| client.PushStreamConnection(pushCtx, writeReq) //nolint:errcheck |
Member
There was a problem hiding this comment.
Suggested change
| client.PushStreamConnection(pushCtx, writeReq) //nolint:errcheck | |
| _, pushErr := client.PushStreamConnection(pushCtx, writeReq) | |
| require.ErrorIs(t, pushErr, context.DeadlineExceeded, | |
| "caller should observe its own context deadline") |
friedrichg
reviewed
May 26, 2026
Comment on lines
+236
to
+258
| err = stream.Send(job.req) | ||
| if err == io.EOF { | ||
| job.resp = &cortexpb.WriteResponse{} | ||
| job.cancel() | ||
| close(job.sendDone) | ||
| return | ||
| } | ||
| if err != nil { | ||
| job.err = err | ||
| job.cancel() | ||
| close(job.sendDone) | ||
| continue | ||
| } | ||
| resp, err := stream.Recv() | ||
| if err == io.EOF { | ||
| job.resp = &cortexpb.WriteResponse{} | ||
| job.cancel() | ||
| close(job.sendDone) | ||
| return | ||
| } | ||
| job.resp = resp | ||
| job.err = err | ||
| if err == nil && job.resp.Code != http.StatusOK { | ||
| job.err = httpgrpc.Errorf(int(job.resp.Code), "%s", job.resp.Message) | ||
| } | ||
| job.cancel() | ||
| close(job.sendDone) |
Member
There was a problem hiding this comment.
close(job.sendDone) is repeated four times across all exit paths. With the new model, missing one of these through a panic, or someone adding a new return path in a future refactor would deadlock the caller forever on <-job.sendDone
Maybe it makes sense to do the close in a defer command
Member
Author
There was a problem hiding this comment.
Yeah I extracted the processing logic as a dedicated func and added a defer
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Distributor slice bounds out of range panic with gRPC push stream
Which issue(s) this PR fixes:
Fixes #7540
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags