Repository navigation
Conversation
janbuchar
commented
Aug 20, 2026
janbuchar
commented
- counterpart to fix!: Don't empty the request queue between run() calls crawlee#4056 (comment)
… fill in the functionality
janbuchar
added
adhoc
|
See more at https://github.com/apify/apify-sdk-js/actions/runs/32372865287#summary-96437235858 |
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Thank you @janbuchar !
A few ideas - the first two are nits, the last one (dropping & migration) I'm actually curious about.
| private readonly client: KeyValueStoreClient, | ||
| private readonly onDropped?: () => void, |
There was a problem hiding this comment.
Let's do the private / #identifer switch in SDK v4 as well?
| do { | ||
| const { items, isTruncated, nextExclusiveStartKey } = await this.client.listKeys({ exclusiveStartKey }); | ||
|
|
||
| // Sequentially: a store with many records would otherwise fire a whole page of deletes at once. |
There was a problem hiding this comment.
// Sequentially: a store with many records would otherwise fire a whole page of deletes at once.
We could have some reasonable-sized batches for this, but perf updates can always come later, this is strictly internal anyway.
| /** Storages this backend dropped; the run's configuration and the Actor's schema still point at them. */ | ||
| private readonly droppedStorageIds = new Set<string>(); |
There was a problem hiding this comment.
What happens if
- we drop a storage (it's possible to call
.drop()on its own, right?) - the run migrates
- we try to open the dropped storage
droppedStorageIds seems ephemeral (it might disappear with the migration), no?
|
The tests fail because the crawlee part is not yet released. |
4 participants
