Skip to content

feat(pins): add ipfs_pin_count tool and limit/offset pagination for ipfs_list_pins - #15

Open
karawitan wants to merge 1 commit into
IPFS-Meshkit:mainfrom
karawitan:feat/pin-count-and-pagination
Open

feat(pins): add ipfs_pin_count tool and limit/offset pagination for ipfs_list_pins#15
karawitan wants to merge 1 commit into
IPFS-Meshkit:mainfrom
karawitan:feat/pin-count-and-pagination

Conversation

@karawitan

Copy link
Copy Markdown

Summary

Nodes with large pinsets (hundreds of thousands to millions of pins) make the current ipfs_list_pins tool impractical: it materializes the full pinset and returns every CID through the MCP channel — huge memory spikes, tool timeouts, and blown agent context.

  • New ipfs_pin_count MCP tool — streams Kubo pin/ls --type=all --stream and returns only counts by type (direct, recursive, indirect, total). Constant memory: no CIDs are accumulated, safe for pinsets of any size.
  • ipfs_list_pins pagination — new optional limit/offset params. When limit is set, iteration stops early instead of draining the stream. Response now includes count (and limit/offset when given).
  • Core additions, exposed through the whole stack:
    • MeshkitClient.countPins(): Promise<PinCount> (new PinCount type)
    • MeshkitClient.listPins(options?: ListPinsOptions)
    • Meshkit.countPins() / Meshkit.listPins(options) (primary node)
    • fil.one/S3 client: countPins = stored object count, pagination via slice
    • exported countPinsViaRpc / applyPinLsLine helpers from @ipfs-meshkit/core

Implementation notes

  • countPinsViaRpc POSTs /api/v0/pin/ls?type=all&stream=true and tallies the NDJSON stream line-by-line (handles the streamed {"Cid":...,"Type":...} format and the legacy Keys mapping; the legacy type-less Pins array is ignored). Optional auth headers from the node config are forwarded.
  • listPins({limit, offset}) on Kubo skips offset entries then stops the async iterator as soon as limit CIDs are collected.

Test plan

  • 224 unit tests pass (vitest run --project unit), including new tests: pin-count.test.ts (line tallying, chunk-boundary splits, legacy formats, HTTP errors), create-client.test.ts (early-stop pagination, offset skip, count via client), meshkit.test.ts (primary-node-only, option passthrough), storage.test.ts (MCP handlers)
  • tsc --noEmit clean; npm run build + mcp build succeed
  • Live-tested against a Kubo node with a 1M+ pinset: ipfs_list_pins {limit:2, offset:1} returns instantly; ipfs_pin_count streams without unbounded memory growth

Generated with Devin

…pfs_list_pins

Nodes with large pinsets (hundreds of thousands to millions of pins) make
the current ipfs_list_pins tool impractical: it materializes the full
pinset and returns every CID through the MCP channel, blowing up context
and timing out.

- Add core countPins(): streams Kubo `pin/ls --type=all --stream` and
  tallies counts by type (direct/recursive/indirect/total) with constant
  memory — no CIDs are accumulated.
- Add listPins({limit, offset}) pagination; when limit is set, iteration
  stops early instead of draining the stream.
- Expose both on the Meshkit facade (primary node) and the fil.one/S3
  client (count = stored objects, pagination via slice).
- MCP: new ipfs_pin_count tool; ipfs_list_pins now accepts optional
  limit/offset and returns count metadata.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant