-
Deployment
Deploy background workers without dropping jobs
Ship worker processes with graceful shutdown, retries, and monitoring so redeploys don't lose in-flight work.
Free Prompt
Make my background workers deploy-safe. Right now I have workers processing jobs from a queue, but I've never checked what happens to in-flight jobs when I redeploy or when the worker crashes.
Do each of these:
1. Separate the deploy: workers should be a distinct process type from the web app so I can restart or scale them independently. Don't run workers inside the web process; a slow job will block requests.
2. Graceful shutdown: on SIGTERM, the worker should stop pulling new jobs, finish the ones in flight (up to a timeout), and only then exit. If a job is still running past the timeout, the queue library should either extend the visibility timeout or requeue the job on the next worker.
3. Idempotency: every job should be safe to run twice. Redelivery happens (a worker dies mid-job, a network hiccup ACKs late). Store an idempotency key or check 'is this already done' at the start of the handler so the second run doesn't double-charge, double-send, or corrupt state.
4. Retry policy: bounded retries with exponential backoff and jitter. Permanent failures go to a dead-letter queue I actually check. Do not retry forever; that's how one poisoned job burns a worker for a week.
5. Monitoring: queue depth per queue, jobs processed per minute, failure rate, and age of the oldest job. Alert on 'oldest job older than N minutes' rather than just queue depth; deep queues that drain quickly aren't a problem, but old jobs are.
6. Concurrency limits: cap workers per queue so a spike doesn't hammer the database or a third-party API. Different queues can have different concurrency.
Don't skip idempotency 'because retries are rare'; they aren't. The deliverable is workers running as their own process, graceful-shutdown handlers implemented, idempotency guards on the critical handlers (payments, emails, external calls), retries bounded with a dead-letter queue in place, and monitoring wired up.
What This Does / How This Helps
This makes workers safe to redeploy: graceful shutdown, idempotent handlers, bounded retries with a dead-letter queue, and monitoring on job age instead of raw depth. Worker bugs are quiet. A dropped job doesn't return an error to a user; a receipt just never sends. A non-idempotent handler doesn't crash; it charges a customer twice. Setting the safety net up before real traffic is on the queue is how you avoid finding these bugs in support tickets a week after launch.
Want to skip doing this by hand?
Fortivibe audits your app for all of the areas these prompts cover (and more).