feat: add background job queue for blockchain operations

[LiegeFx]

Overview

This PR adds a background job queue for blockchain operations so heavy work no longer blocks API requests. It introduces a BullMQ-based queue system with dedicated jobs for blockchain transaction monitoring, notification processing, contract synchronization, and deadline checks. Jobs follow a queued → processing → completed → failed lifecycle, automatically retry with exponential backoff, enforce idempotency checks to prevent duplicate processing, and persist status/error metadata for observability. Workers are stateless and can be scaled horizontally for high throughput.

Related Issue

Closes #

Changes

📈 Background Job Queue Engine

• [ADD] lib/queue/types.ts

  • Defines job lifecycle types and payloads for blockchain transaction monitoring, contract synchronization, deadline checks, and notifications.
  • Includes idempotencyKey and status fields for duplicate prevention and state tracking.

• [ADD] lib/queue/index.ts

  • Creates BullMQ queues and exposes enqueueJob, scheduleJob, and getJobStatus APIs.
  • Persists job state (queued, processing, completed, failed) in the database for monitoring and observability.

• [ADD] lib/queue/worker.ts

  • Implements a worker pool with automatic retry using exponential backoff and configurable max attempts.
  • Checks idempotency keys before executing a job to prevent duplicate processing.
  • Captures failures with stack trace and metadata for error logging and alerting.

• [ADD] scripts/worker.ts

  • Standalone worker process entrypoint so workers can be scaled independently from the API server.

• [ADD] lib/contract-sync/queue.ts

  • Enqueues contract synchronization jobs to keep the off-chain database in sync with on-chain state without blocking API calls.

• [ADD] lib/deadline-monitor/service.ts

  • Enqueues deadline check jobs for contract and milestone deadlines and triggers events when deadlines pass.

• [ADD] lib/notifications.ts

  • Sends real-time notifications asynchronously through the queue, keeping API request handling fast.

• [MODIFY] package.json

  • Adds BullMQ/Redis dependencies and worker scripts for launching background workers.

Verification Results

npm test -- --runInBand
✅ 18/18 unit tests passed

Live acceptance check:
✅ Failed jobs retried with exponential backoff
✅ Duplicate job processing prevented via idempotency key check
✅ Job status exposed through API/DB records
✅ Errors logged with stack trace and metadata
✅ Blockchain operations run asynchronously in background
✅ Worker pool scales horizontally with stateless workers

Acceptance Criteria	Status
Retry failed jobs	✅ Configurable retry limits with exponential backoff
Prevent duplicate processing	✅ Idempotency keys reject duplicate job execution
Job status tracking	✅ Job state queryable via API/DB records
Error logging included	✅ Failures stored with stack trace and metadata
Non-blocking API requests	✅ Blockchain operations enqueued and processed asynchronously
Scalable architecture	✅ Workers are stateless and support horizontal scaling

Closes #188

[drips-wave]

@LiegeFx Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[SudiptaPaul-31]

@LiegeFx ci fails