Is your feature request related to a problem? Please describe.
ActivityCompletionClient.heartbeat fails on the first transient error from the server — most often RESOURCE_EXHAUSTED from namespace rate limiting, but also DEADLINE_EXCEEDED and UNAVAILABLE. For async completion, where an external process holds the task token and the heartbeat is the only thing keeping a long-running activity alive, a single rate-limited
heartbeat can cost us the activity via heartbeat timeout.
The worker-side path already handles this. HeartbeatContextImpl.doHeartBeatLocked catches StatusRuntimeException, logs a warning and reschedules after HEARTBEAT_RETRY_WAIT_MILLIS instead of failing the activity.
Describe the solution you'd like
Wrap recordHeartbeat in the same GrpcRetryer the other three reply calls in
ManualActivityCompletionClientImpl already use — complete, fail and reportCancellation
all run inside grpcRetryer.retry(...) with replyGrpcRetryerOptions, and recordHeartbeat is
the only one that doesn't. That gives heartbeat the same RESOURCE_EXHAUSTED handling
(including the message-too-large carve-out) for free.
A shorter expiration than replyGrpcRetryerOptions may make more sense here, since retrying past
the next heartbeat interval is pointless.
Open to another approach if you have one in mind — making the retryability decision public so
callers can apply their own policy would also unblock us.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
ActivityCompletionClient.heartbeatfails on the first transient error from the server — most oftenRESOURCE_EXHAUSTEDfrom namespace rate limiting, but alsoDEADLINE_EXCEEDEDandUNAVAILABLE. For async completion, where an external process holds the task token and the heartbeat is the only thing keeping a long-running activity alive, a single rate-limitedheartbeat can cost us the activity via heartbeat timeout.
The worker-side path already handles this.
HeartbeatContextImpl.doHeartBeatLockedcatchesStatusRuntimeException, logs a warning and reschedules afterHEARTBEAT_RETRY_WAIT_MILLISinstead of failing the activity.Describe the solution you'd like
Wrap
recordHeartbeatin the sameGrpcRetryerthe other three reply calls inManualActivityCompletionClientImplalready use —complete,failandreportCancellationall run inside
grpcRetryer.retry(...)withreplyGrpcRetryerOptions, andrecordHeartbeatisthe only one that doesn't. That gives heartbeat the same
RESOURCE_EXHAUSTEDhandling(including the message-too-large carve-out) for free.
A shorter expiration than
replyGrpcRetryerOptionsmay make more sense here, since retrying pastthe next heartbeat interval is pointless.
Open to another approach if you have one in mind — making the retryability decision public so
callers can apply their own policy would also unblock us.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.