Skip to content

1.5.x cloud benchmarks#3615

Open
kmatasfp wants to merge 97 commits into
1.5.xfrom
1.5.x-cloud-benchmarks
Open

1.5.x cloud benchmarks#3615
kmatasfp wants to merge 97 commits into
1.5.xfrom
1.5.x-cloud-benchmarks

Conversation

@kmatasfp

@kmatasfp kmatasfp commented Jun 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Also contains replacement for memory semaphore plus fixes eviction deadlock.

for memory semaphore replacement see:

  • golem-worker-executor/src/services/active_workers/admission/mod.rs
  • golem-worker-executor/src/services/active_workers/component_charge/mod.rs
  • golem-worker-executor/src/services/active_workers/memory_probe.rs

@kmatasfp kmatasfp requested a review from a team June 5, 2026 00:18
@kmatasfp
fix: make ProvidedShardManager kill/restart no-ops instead of panics
4294bdb 
kill_all() is called after cloud_preflight_warmup completes. ProvidedShardManager
wraps an already-running process we don't own, so neither kill nor restart should
crash the binary. Both are now silent no-ops, matching UnavailableShardManager.
kmatasfp added 24 commits June 5, 2026 10:44
@kmatasfp
feat(benchmark): enable all tests
5b9902b
@kmatasfp
feat: retry connectivity to shard manager
742a669
@kmatasfp
chore: fmt
18d5af6
@kmatasfp
investigation: run echo test first to see if they get stuck again
395bcd2
@kmatasfp
feat(benchmark): lower number of conccurent live apps
dac3c69
@kmatasfp
feat: more observability, make memory component coefficient configurable
2256623
@kmatasfp
feat(benchmark): run only throughput-echo test
02e527a
@kmatasfp
feat(bench): try 200 apps after tuning
faeb651
@kmatasfp
feat: try 250 again
f8dd565
@kmatasfp
feat(benchmark): run all the tests again
1bf0063
@kmatasfp
fix: metric description
2e53af6
@kmatasfp
feat: proper load for our cluster
32ef9e5
@kmatasfp
feat(benchmark): run only benchmark tests
bc11779
@kmatasfp
feat: enable all tests again
5347626
@kmatasfp
feat(benchmark): increase max number of concurrent compilations
9e582a2
@kmatasfp
feat(worker-executor): add measured-headroom memory admission gate
e7b44bf
@kmatasfp
feat(worker-executor): charge component module size once per resident…
817c672 
… component
@kmatasfp
fix(worker-executor): disable measured admission when executor does n…
35874d3 
…ot own its memory environment
@kmatasfp
feat(benchmark): add throughput-under-memory-saturation benchmarks
acb9968
@kmatasfp
test(worker-executor): exercise admission reserve under maximum concu…
bfe1b14 
…rrent overlap
@kmatasfp
feat(benchmark): longer sustained load, bumpt the number of agents
c3af739
@kmatasfp
fix: add empty workspace
7dcb2d3
@kmatasfp
fix: use snake case as method names
139aed5
@kmatasfp
chore: 300 already saturates, no need for 500
442c1c5
kmatasfp added 10 commits June 17, 2026 15:21
@kmatasfp
feat(benchmark): make agent-density ramp configurable via suite yaml
3ab002a
@kmatasfp
feat(benchmark): drop executor /metrics scraping from agent-density v1
ef83208
@kmatasfp
fix(benchmark): cap per-agent components at 999 to stay within plan l…
897877c 
…imit
@kmatasfp
fix(benchmark): give each per-agent component its own environment to …
53c8f6a 
…avoid agent-type ambiguity
@kmatasfp
feat(benchmark): cap per-agent-component ramp at 1000 with finer low-…
b1477d8 
…end steps
@kmatasfp
fix(benchmark): inline per-agent ramp instead of yaml anchor for port…
3b5fad9 
…able yq parsing
@kmatasfp
perf(benchmark): create agents concurrently within a ramp step
675e02a
@kmatasfp
feat(benchmark): record invoke-latency percentiles per cell and add i…
7a46e0e 
…teration-pooling merge script
@kmatasfp
fix(benchmark): treat send-failure floods as connection-lost so a wed…
e6f6bab 
…ged executor trips catastrophic
@kmatasfp
add tokio task-dump debug image for diagnosing worker-executor hangs
9f8e6c1
@kmatasfp kmatasfp force-pushed the 1.5.x-cloud-benchmarks branch from d6e35f4 to 9f8e6c1 Compare June 18, 2026 07:57
kmatasfp added 19 commits June 18, 2026 01:14
@kmatasfp
fix task-dump build: Tasks has no len(), count via iterator
c864085
@kmatasfp
fix debug-image packaging: tar relative to target/ so Dockerfile COPY…
102bbe2 
… path resolves
@kmatasfp
Address perf flags review
1838be0
@kmatasfp
Clarify arm64 image labels
e7d6bf2
@kmatasfp
remove tokio task-dump debug scaffolding and restore full density suite
4c51921
@kmatasfp
delete created agents after each density cell
2e7c7d4
@kmatasfp
count executor restarts since cell start instead of pod lifetime
091c3f0
@kmatasfp
skip cleanup deletion for ephemeral density agents
1ccd6c4
@kmatasfp
skip agent deletion on catastrophic cells and bound cleanup time
71d51c1
@kmatasfp
measure concurrent ephemeral residency and fix iteration merge schema
98c37a7
@kmatasfp
sample active-cell load multiple times per ramp step
520c838
@kmatasfp
isolate shared-component cells: remove per-agent cells from density s…
ea0d191 
…uite
@kmatasfp
reduce per-agent prep components to 2 for fast prep
03a8067
@kmatasfp
unify density cells on per-step round load model (one concurrent invo…
bc0202a 
…ke per agent)
@kmatasfp
isolation suite: durable vs ephemeral concurrent-active under round m…
5714c0b 
…odel
@kmatasfp
treat sustained 503s as catastrophic overload; break invoke latency d…
86b2bc8 
…own per ramp step
@kmatasfp
log readable density cell summary with named terminated reason
5c5bf61
@kmatasfp
collapse single-value count results to one column in benchmark table
e8522aa
@kmatasfp
feat: better ramp steps
3e4a4aa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mschuwalow mschuwalow mschuwalow approved these changes

@vigoo vigoo Awaiting requested review from vigoo

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kmatasfp @vigoo @mschuwalow