Skip to content

Disk polling: investigate tail latency under concurrent writes #4178

Description

@diegomrsantos

Part of #4174, which records the compared revisions, workload and statistical method.

Investigate the unresolved tail behavior under concurrent writes. Start with archived timelines; collect more diagnostics only when a recurring delay or an explicit tail requirement makes the decision useful.

Case Change in run p99 Pointwise 95% interval
Explicit offset −0.02% [−2.79%, +2.88%]
Explicit offset with writes +3.40% [−0.62%, +7.63%]
Next −0.84% [−2.37%, +0.69%]
Consumer group +0.86% [−0.89%, +2.65%]
Resident −2.33% [−6.29%, +1.23%]

Positive values mean higher p99. Estimates are geometric means of paired ratios of run p99 values, not pooled request percentiles. They use the same pair bootstrap described in #4174. The interval with writes includes a small improvement and a meaningful increase, so it neither confirms a tail regression nor resolves a 3% margin. Mean elapsed time is a separate metric, reported in the parent.

The preceding campaign with the same binaries reported Next p99 +6.79% and group p99 +6.06%; those increases were not reproduced. Keep the campaigns separate. The rerun followed reported high Docker memory consumption and a guest restart, but memory pressure was not independently controlled and continuously measured. No container CPU throttling or memory limit events were recorded; those checks cannot exclude host pressure or establish a memory cause.

  1. Inspect the published raw timelines and resource records, including run order and valid slow operations. Look for clustering around a stage or resource condition.
  2. Reuse the parent's diagnostics and timestamps from Disk polling: measure owner completion service under concurrent polling #4177. Add sparse stage timestamps, scheduler waits or queue observations only where needed to locate the delay. CPU profiles alone cannot locate waiting.
  3. State the latency boundary, load model and worthwhile effect before fresh confirmation. Follow the parent's fixed budget and arrival model rules, including scheduling delays, errors and timeouts.

Tracing can create the tail being studied. Preserve valid slow runs and stop according to the declared budget. Repeating measurements on the same VM cannot remove systematic environment bias or establish physical device latency. A change should address a supported mechanism, not merely a difference between campaigns.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceChange impacts performanceserveriggy-server related change

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions