Refresh Large_Scale_Tasks_Recipe.py for the current Task Group API

[elijahgjacob]

Summary

The Large-Scale Tasks recipe was written a year ago against client.beta.task_group with a hard-coded product-matching spec, and it no longer reflects how we run big batches. This replaces it with a generic, resumable runner on the current SDK surface. Same filename so existing links keep working.

What the new recipe does

• plan: counts runs, requests, and groups and estimates enqueue time from your quota. No API calls.
• submit: adds runs 1,000 per request with refresh_status=False, paced at 90% of --rate-limit (runs per minute), sharded into Task Groups. Every add_runs response is appended to runs.jsonl before the next request, so a crash or re-run never resubmits a paid run. Any CSV works: all columns become string input fields, or one column can be sent as JSON.
• status: polls each group summary, optionally until no group is active. Its output is how you see the running count plateau, which is what a throughput pilot should measure.
• export: streams each group's runs to JSONL with output and per-field basis, then checks every input row came back exactly once. Exits 2 if anything is missing, duplicated, or unexpected.

The module docstring covers what people get wrong on large jobs: the rate limit governs intake not throughput, runs cannot be cancelled once created, and a pilot should be at least 5k runs measured after concurrency plateaus. Wording matches the "Sizing a Large Job" section proposed in shapleyai/documentation#748.

Also updates the recipe descriptions in the root and python-recipes READMEs.

Tested against the live API

2,000 lite runs, 2026-09-22, group tgrp_4dd7cbbb4a3541ce883cecaac94c1e7d, about $10:

Command	Result
plan --input rows.csv	2 requests, 1 group, 1.1 min enqueue at the 2,000/min default
submit	Two 1,000-run requests, paced, 67 s wall clock
submit again	"2000 already submitted, 0 to go", no API writes
status --wait	Polled every 60 s; all 2,000 completed in about 8 min, 0 failed
export	2,000 rows with output and basis; validation ok: true, exit 0

The running count climbed 267 → 400 → 495 → 652 over the first four minutes before draining, the same ramp-up seen in the docs pilot. That is why the docstring says to pilot at least 5k runs and measure after the plateau rather than end to end.

Failure path, earlier 3-run check: a fake row appended to runs.jsonl made export report missing: 1, ok: false, exit 2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BqEaXBEuFVsj4Cz5iHyy2q