BigQueryAgentAnalyticsPlugin can duplicate rows when writes are retried

[addenergyx]

Problem

BigQueryAgentAnalyticsPlugin can write duplicate rows when its internal BigQuery Storage Write API request is retried after an ambiguous failure.

This concerns retries of the BigQuery write, not retries by the agent, model, or tools.

The plugin writes to BigQuery's _default stream. BatchProcessor._write_rows_with_retry constructs one AppendRowsRequest and resends the same serialized batch following transient errors or its 30-second
client-side timeout.

If BigQuery accepts the first append but its acknowledgement is lost or delayed, the plugin retries the batch. Because _default provides at-least-once delivery, BigQuery can accept it again and produce duplicate rows.

Relevant code:

• BatchProcessor._write_rows_with_retry
• Default-stream selection

Impact

We observed byte-identical duplicate analytics rows in production, including identical timestamps, trace/span IDs, token usage, status, and content.

The issue is rare. Over seven days, six duplicate LLM_RESPONSE rows inflated a naive SUM(usage.total) by 182,555 tokens, approximately 0.31% of the total. This can silently over-count tokens, costs, latency, and other metrics derived from the analytics table. However this was for a poc agent with few users and can easily inflate with more use.

Proposed fix

BigQuery does not allow append offsets on _default, and BigQueryWriteAsyncClient has no exactly_once option.

Exactly-once append behaviour requires an application-created COMMITTED stream with offsets:

1. Create one committed stream per BatchProcessor.
2. Include the current offset in each AppendRowsRequest.
3. Retry an ambiguous append using the same stream and offset.
4. Advance the offset after success.
5. Treat ALREADY_EXISTS at that offset as confirmation that the earlier
   attempt succeeded.
6. Handle unexpected OUT_OF_RANGE without blindly resending on a new stream.

A committed stream preserves immediate row visibility while making Storage Write API retries idempotent.

This fix would prevent duplicates caused by BigQuery write retries. It would not deduplicate genuinely separate events emitted by the agent or application.

For anyone familiar with Apache beam this is similar at a high level to their use_at_least_once=False option for BigQuery Storage Write API sinks. Beam provides the stream, offset, and durable checkpoint management behind that abstraction; because ADK uses BigQueryWriteAsyncClient directly, the plugin must manage its committed stream and append offsets itself.

References:

• BigQuery streaming guarantees
• AppendRowsRequest.offset

[sanketpatil06]

Hi @addenergyx , Thank you for providing these details. To help us fully understand the issue, could you please share a minimal reproducible example? This will allow us to recreate the problem exactly as you are experiencing it.