Petunia Operations Runbook

Version: 1.0.0 Last Updated: 2025-12-22 Classification: Operations / On-Call Reference

This runbook provides operations engineers with everything needed to diagnose, triage, and resolve production incidents. Keep this document open during on-call shifts.

---

Table of Contents

1. Emergency Contacts & Escalation
2. Health Check Reference
3. Kill Switches & Feature Flags
4. Third-Party Status Pages
5. Common Incident Runbooks
6. Cron Job Monitoring
7. Queue System Operations
8. Circuit Breaker Operations
9. Graceful Degradation
10. Rate Limit Bypass (Emergency Only)
11. Rollback Procedures
12. Communication Templates
13. Recovery Procedures

---

Emergency Contacts & Escalation

Escalation Matrix

Severity	Response Time	Who to Contact	Action
SEV-1 (Production Down)	< 15 min	On-call → Engineering Lead → CTO	Page immediately, all hands
SEV-2 (Major Feature Broken)	< 1 hour	On-call → Engineering Lead	Page on-call, notify team
SEV-3 (Degraded Performance)	< 4 hours	On-call	Slack notification
SEV-4 (Minor Issue)	Next business day	Create ticket	Document only

Severity Definitions

SEV-1 - Critical (Production Down)

• Complete service outage
• Authentication system failure
• Data corruption or loss
• Payment processing failure affecting all users
• Voice system completely unavailable

SEV-2 - Major (Feature Broken)

• Single major feature unavailable (voice, inbox, dashboard)
• Significant performance degradation (>5x latency)
• Partial payment failures (>10% transactions)
• Elevated error rates (>5% of requests)

SEV-3 - Degraded

• Non-critical feature issues
• Minor performance degradation
• Isolated user reports
• Third-party service degradation (non-critical)

SEV-4 - Minor

• Cosmetic issues
• Feature requests misrouted as bugs
• Single-user edge cases
• Documentation errors

Contact Channels

Channel	Purpose	Response Time
PagerDuty	SEV-1/SEV-2 incidents	Immediate
#alerts-critical (Slack)	Automated alerts	< 5 min
#engineering (Slack)	Team coordination	Business hours
Email: ops@petunia.ai	Non-urgent issues	24 hours

---

Health Check Reference

Health Endpoints

Endpoint	Purpose	Expected Response
GET /api/health	Basic liveness	200 OK + { status: "healthy" }
GET /api/health/status	Comprehensive status	200 OK + detailed component status
GET /api/ready	Kubernetes readiness	200 OK when ready to serve traffic

Health Check Components

The /api/health/status endpoint returns status for each component:

{
  "status": "healthy | degraded | unhealthy",
  "timestamp": "ISO-8601",
  "components": {
    "database": { "status": "up", "latency_ms": 5 },
    "redis": { "status": "up", "latency_ms": 2 },
    "voice_primary": { "status": "up", "provider": "elevenlabs" },
    "voice_fallback": { "status": "up", "provider": "twilio" },
    "email": { "status": "up" },
    "webhooks": { "status": "up" },
    "websocket": { "status": "up" }
  }
}

Interpreting Health Status

Status	Meaning	Action Required
healthy	All systems operational	None
degraded	Some components impaired	Monitor, prepare for escalation
unhealthy	Critical component failure	Immediate investigation

Component-Specific Health Checks

Database Health (lib/services/core/database-health.ts)

# Manual check via psql
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -d postgres -c "SELECT 1"

# Check connection pool
curl -s http://localhost:3000/api/health/status | jq '.components.database'

Redis Health

# Manual check
redis-cli -h $REDIS_HOST ping

# Check from application
curl -s http://localhost:3000/api/health/status | jq '.components.redis'

Voice Provider Health (lib/services/voice/voiceHealthCheck.ts)

# Check all voice providers
curl -s http://localhost:3000/api/health/status | jq '.components | with_entries(select(.key | startswith("voice")))'

---

Kill Switches & Feature Flags

Emergency Kill Switches

These environment variables can disable problematic features without deployment:

Variable	Effect	Recovery
VOICE_ENABLED=false	Disable all voice features	Set to true
AI_ENABLED=false	Disable AI response generation	Set to true
AUTORESPONDER_ENABLED=false	Disable autoresponders	Set to true
WEBHOOKS_ENABLED=false	Disable outbound webhooks	Set to true
BILLING_ENABLED=false	Disable billing operations	Set to true

Feature Flags (Runtime)

Feature flags are stored in the database and can be toggled via admin API:

# Disable a feature
curl -X POST /api/admin/features/voice-transcription \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": false}'

# Check feature status
curl /api/admin/features | jq '.features'

Circuit Breaker Manual Controls

Voice circuit breakers can be manually controlled:

// Force open (block all requests to provider)
voiceCircuitBreakerRegistry.getBreaker('elevenlabs').forceOpen('Manual intervention: API degraded');

// Reset to closed (allow requests)
voiceCircuitBreakerRegistry.getBreaker('elevenlabs').reset();

// Check current state
voiceCircuitBreakerRegistry.getAllMetrics();

---

Third-Party Status Pages

Bookmark these for incident correlation:

Service	Status Page	Critical For
ElevenLabs	https://status.elevenlabs.io	Voice synthesis
OpenAI	https://status.openai.com	AI responses
Anthropic	https://status.anthropic.com	AI responses
Twilio	https://status.twilio.com	Voice/SMS fallback
Stripe	https://status.stripe.com	Payments
Supabase	https://status.supabase.com	Database/Auth
Vercel	https://www.vercel-status.com	Hosting
Sentry	https://status.sentry.io	Error tracking
Redis (Upstash)	https://status.upstash.com	Caching/Queues

Checking Third-Party Health Programmatically

# Quick status check script
for service in "status.elevenlabs.io" "status.openai.com" "status.stripe.com"; do
  echo -n "$service: "
  curl -s -o /dev/null -w "%{http_code}" "https://$service" && echo " OK" || echo " FAIL"
done

---

Common Incident Runbooks

1. Database Connection Failures

Symptoms:

• Error: ECONNREFUSED, CONNECTION_TIMEOUT, POOL_EXHAUSTED
• Health check shows database: down
• Elevated 500 errors across all endpoints

Investigation:

# Check database connectivity
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -c "SELECT 1"

# Check connection pool status
curl -s localhost:3000/api/health/status | jq '.components.database'

# Check for connection leaks
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -c \
  "SELECT count(*), state FROM pg_stat_activity WHERE datname='postgres' GROUP BY state"

Resolution:

1. If Supabase is down → Check https://status.supabase.com
2. If connection pool exhausted → Restart application pods
3. If credentials invalid → Verify DATABASE_URL in Vercel dashboard
4. If network issue → Check Vercel/Supabase region connectivity

Recovery Verification:

curl -s localhost:3000/api/health | jq '.status'
# Expected: "healthy"

---

2. Voice Provider Failures

Symptoms:

• Error: VOICE_SYNTHESIS_FAILED, VOICE_PROVIDER_UNAVAILABLE
• Circuit breaker in OPEN state
• Calls dropping or silent audio

Investigation:

# Check circuit breaker states
curl -s localhost:3000/api/health/status | jq '.components | with_entries(select(.key | startswith("voice")))'

# Check ElevenLabs status
curl -s https://api.elevenlabs.io/v1/user -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check Twilio fallback
curl -s "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json" \
  -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN"

Resolution:

1. If ElevenLabs down:

   • Failover should be automatic to Twilio
   • Verify: TWILIO_VOICE_ENABLED=true
   • Manual override: Set VOICE_PRIMARY_PROVIDER=twilio

2. If both providers down:

   • Enable voice disable kill switch: VOICE_ENABLED=false
   • Notify users via status page
   • Monitor third-party status pages

3. If circuit breaker stuck open:

   // Reset circuit breaker
   voiceCircuitBreakerRegistry.getBreaker('elevenlabs').reset();
   

Recovery Verification:

# Test voice synthesis
curl -X POST localhost:3000/api/voice/test-synthesis \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -d '{"text": "Hello world"}'

---

3. Payment Processing Failures

Symptoms:

• Error: BILLING_CHARGE_FAILED, WEBHOOK_SIGNATURE_INVALID
• Stripe webhooks failing
• Users unable to subscribe/pay

Investigation:

# Check Stripe API connectivity
curl -s https://api.stripe.com/v1/balance \
  -u "$STRIPE_SECRET_KEY:"

# Check webhook endpoint status
curl -s https://api.stripe.com/v1/webhook_endpoints \
  -u "$STRIPE_SECRET_KEY:" | jq '.data[].status'

# Check failed webhook queue
curl -s localhost:3000/api/admin/webhooks/failed | jq '.pending'

Resolution:

1. If Stripe API down:

   • Check https://status.stripe.com
   • Failed charges will retry automatically
   • Consider enabling BILLING_ENABLED=false for severe outages

2. If webhook signature failures:

   • Verify STRIPE_WEBHOOK_SECRET matches Stripe dashboard
   • Check for clock drift between servers

3. If webhook processing failures:

   • Failed webhooks are queued for retry
   • Check: GET /api/admin/webhooks/failed
   • Manual retry: POST /api/admin/webhooks/retry/{eventId}

Recovery Verification:

# Verify Stripe connectivity
curl -s localhost:3000/api/payments/health

# Check webhook queue is draining
curl -s localhost:3000/api/admin/webhooks/stats

---

4. Authentication Failures

Symptoms:

• Error: AUTH_SESSION_EXPIRED, AUTH_TOKEN_INVALID
• Users unable to log in
• Session-related 401 errors spike

Investigation:

# Check Supabase Auth service
curl -s "$NEXT_PUBLIC_SUPABASE_URL/auth/v1/health"

# Check session endpoint
curl -s localhost:3000/api/auth/session \
  -H "Cookie: sb-access-token=..."

# Check for JWT secret mismatch
echo $SUPABASE_JWT_SECRET | cut -c1-10

Resolution:

1. If Supabase Auth down:

   • Check https://status.supabase.com
   • Sessions remain valid until expiry
   • New logins will fail - prepare user communication

2. If JWT validation failing:

   • Verify SUPABASE_JWT_SECRET matches project settings
   • Check for environment variable propagation issues

3. If session storage issues:

   • Clear Redis session cache if corrupted
   • Force re-authentication via cookie invalidation

Recovery Verification:

# Test authentication flow
curl -X POST localhost:3000/api/auth/signin \
  -d '{"email":"test@example.com","password":"test"}'

---

5. AI Response Failures

Symptoms:

• Error: AI_GENERATION_FAILED, AI_RATE_LIMITED
• Autoresponders not generating responses
• Knowledge base queries timing out

Investigation:

# Check OpenAI status
curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" | head -1

# Check Anthropic status
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01"

# Check rate limit headers from last request
curl -I https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Resolution:

1. If rate limited:

   • AI requests use exponential backoff automatically
   • Check quota in provider dashboards
   • Consider enabling AI_ENABLED=false temporarily

2. If provider down:

   • Failover to secondary provider (OpenAI ↔ Anthropic)
   • Set AI_PRIMARY_PROVIDER=anthropic (or openai)

3. If responses slow:

   • Check for prompt size issues
   • Monitor token usage
   • Consider reducing AI_MAX_TOKENS

Recovery Verification:

# Test AI generation
curl -X POST localhost:3000/api/ai/test \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"prompt": "Say hello"}'

---

6. WebSocket Connection Issues

Symptoms:

• Real-time updates not working
• Inbox messages delayed
• Error: WEBSOCKET_CONNECTION_FAILED

Investigation:

# Check WebSocket server status
curl -s localhost:3000/api/health/status | jq '.components.websocket'

# Check active connections
curl -s localhost:3000/api/admin/websocket/stats

# Test WebSocket connectivity
wscat -c "wss://app.petunia.ai/api/websocket?token=..."

Resolution:

1. If WebSocket server crashed:

   • Check pod/container logs
   • Restart WebSocket service
   • Clients will auto-reconnect

2. If connection limits reached:

   • Scale WebSocket pods horizontally
   • Check for connection leaks (zombie connections)

3. If authentication issues:

   • Verify token validation logic
   • Check CORS configuration

Recovery Verification:

# Monitor reconnections
curl -s localhost:3000/api/admin/websocket/stats | jq '.reconnections_per_minute'

---

Cron Job Monitoring

Active Cron Jobs

Job	Schedule	Purpose	Failure Impact
dunning	Every 4 hours	Payment retry	Revenue loss
review-aggregation	Daily 2 AM	Aggregate reviews	Stale data
webhook-retry	Every 5 min	Retry failed webhooks	Event loss
overage-settlement	Hourly	Settle billing overages	Billing delays
data-retention	Daily 3 AM	Clean old data	Storage growth
lead-scoring	Every 15 min	Update lead scores	Stale scores
campaign-scheduler	Every minute	Schedule campaigns	Missed sends
analytics-rollup	Hourly	Aggregate metrics	Dashboard gaps
health-check	Every minute	System monitoring	Blind spots
session-cleanup	Every 6 hours	Clean expired sessions	Memory growth
cache-warmup	On deploy	Pre-warm caches	Cold start latency
subscription-sync	Every 30 min	Sync Stripe data	Billing mismatch

Cron Job Health Monitoring

# Check cron job execution history
curl -s localhost:3000/api/admin/cron/history | jq '.[0:5]'

# Check for failed jobs
curl -s localhost:3000/api/admin/cron/failed

# Manually trigger a job
curl -X POST localhost:3000/api/cron/dunning \
  -H "Authorization: Bearer $CRON_SECRET"

Cron Failure Recovery

If a cron job is failing repeatedly:

1. Check logs for error details
2. Verify environment variables are set
3. Check for database lock contention
4. Verify external service connectivity
5. Manual execution to test:

   curl -X POST localhost:3000/api/cron/{job-name} \
     -H "Authorization: Bearer $CRON_SECRET"
   

Critical Cron Dependencies:

Job	Requires	Recovery if Missing
dunning	Stripe API	Payment retries delayed
webhook-retry	Database	Events may be lost
analytics-rollup	Database, Redis	Manual re-aggregation
lead-scoring	AI API	Scores become stale

---

Queue System Operations

Queue Architecture

Petunia uses two queue systems:

1. BullMQ (Redis-backed) - High-throughput, ephemeral jobs

   • Voice processing
   • Email sending
   • Real-time notifications

2. Database-backed Queues - Durable, audit-required jobs

   • Webhook delivery
   • Billing operations
   • Critical notifications

Queue Monitoring

# BullMQ queue stats
curl -s localhost:3000/api/admin/queues/bullmq | jq '.'

# Database queue stats
curl -s localhost:3000/api/admin/queues/database | jq '.'

# Check for stuck jobs
curl -s localhost:3000/api/admin/queues/stuck

Queue Recovery Procedures

If BullMQ queue is backed up:

# Check queue depth
redis-cli -h $REDIS_HOST LLEN bull:voice-processing:wait

# If queue is stuck, clear stale jobs
curl -X POST localhost:3000/api/admin/queues/bullmq/voice-processing/clean \
  -H "Authorization: Bearer $ADMIN_TOKEN"

If database queue has failed jobs:

# View failed jobs
curl -s localhost:3000/api/admin/queues/database/failed

# Retry specific job
curl -X POST localhost:3000/api/admin/queues/database/retry/{jobId} \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Retry all failed jobs
curl -X POST localhost:3000/api/admin/queues/database/retry-all \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Queue Job States

State	Meaning	Action
waiting	Queued, awaiting processing	Normal
active	Currently being processed	Normal
completed	Successfully processed	Auto-cleaned
failed	Processing failed, may retry	Check errors
delayed	Scheduled for future	Normal
stalled	Worker died mid-processing	Auto-retry

---

Circuit Breaker Operations

Understanding Circuit States

State	Behavior	Trigger
CLOSED	Normal operation	Success threshold met
OPEN	All requests rejected	Failure threshold exceeded
HALF_OPEN	Limited test requests	Recovery time elapsed

Circuit Breaker Configuration

// Default configuration (circuitBreaker.ts:53-59)
{
  failureThreshold: 5,      // Failures before opening
  successThreshold: 2,      // Successes to close from half-open
  recoveryTimeMs: 30000,    // Time before testing recovery
  failureWindowMs: 60000,   // Window for counting failures
  operationTimeoutMs: 30000 // Individual operation timeout
}

Monitoring Circuit Breakers

# Get all circuit breaker states
curl -s localhost:3000/api/admin/circuit-breakers | jq '.'

# Response format:
# {
#   "elevenlabs": { "state": "CLOSED", "failures": 0, "lastFailure": null },
#   "twilio": { "state": "HALF_OPEN", "failures": 3, "lastFailure": "2025-12-22T..." }
# }

Manual Circuit Breaker Control

Force open (emergency block):

curl -X POST localhost:3000/api/admin/circuit-breakers/elevenlabs/force-open \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"reason": "Provider experiencing degradation"}'

Reset to closed:

curl -X POST localhost:3000/api/admin/circuit-breakers/elevenlabs/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

---

Graceful Degradation

Degradation Hierarchy

When systems fail, Petunia degrades gracefully in this order:

1. Voice Synthesis

   • Primary: ElevenLabs
   • Fallback: Twilio
   • Graceful: Pre-recorded fallback messages

2. AI Responses

   • Primary: OpenAI GPT-4
   • Fallback: Anthropic Claude
   • Graceful: Template-based responses

3. Real-time Updates

   • Primary: WebSocket
   • Fallback: Polling (30s interval)
   • Graceful: Manual refresh prompt

4. Payments

   • Primary: Stripe direct
   • Fallback: Queued for retry
   • Graceful: Grace period extension

Enabling Degraded Modes

# Enable polling fallback for WebSocket issues
curl -X POST localhost:3000/api/admin/features/websocket-fallback \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": true}'

# Enable template responses for AI failures
curl -X POST localhost:3000/api/admin/features/ai-fallback-templates \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": true}'

---

Rate Limit Bypass (Emergency Only)

When to Use Rate Limit Bypass

ONLY use in genuine emergencies:

• Critical business operation blocked by rate limits
• Incident recovery requiring rapid API calls
• Data migration or batch processing

DO NOT use for:

• Regular operations
• Avoiding rate limits during normal usage
• Client requests (unless approved by engineering lead)

Bypass Procedures

1. Temporary User Bypass (5 minutes)

# Grant temporary bypass to specific user
curl -X POST localhost:3000/api/admin/rate-limit/bypass \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "userId": "user-uuid",
    "durationMinutes": 5,
    "reason": "Incident recovery - INC-2025-12-22-001"
  }'

2. Temporary IP Bypass

# Whitelist IP for emergency operations
curl -X POST localhost:3000/api/admin/rate-limit/whitelist-ip \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "ip": "192.168.1.100",
    "durationMinutes": 30,
    "reason": "Data migration batch job"
  }'

3. Global Rate Limit Increase (DANGER)

# Temporarily increase global limits (requires CTO approval)
curl -X POST localhost:3000/api/admin/rate-limit/global \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "multiplier": 2.0,
    "durationMinutes": 60,
    "approvedBy": "cto@petunia.ai",
    "reason": "SEV-1 incident recovery"
  }'

Audit Requirements

All rate limit bypasses are logged:

• User who granted bypass
• Target user/IP
• Duration and reason
• Timestamp

Review bypasses weekly in security audit.

---

Rollback Procedures

Vercel Deployment Rollback

Symptoms requiring rollback:

• New deployment causing errors
• Feature regression identified
• Performance degradation after deploy

Steps:

1. Identify the last good deployment:

   # List recent deployments
   vercel list --prod
   
   # Or via Vercel dashboard → Deployments
   

2. Rollback to previous deployment:

   # Instant rollback via CLI
   vercel rollback <deployment-url>
   
   # Example
   vercel rollback petunia-abc123.vercel.app
   

3. Verify rollback:

   # Check health
   curl -s https://app.petunia.gardenpatch.xyz/api/health
   
   # Verify version
   curl -s https://app.petunia.gardenpatch.xyz/api/version
   

Database Migration Rollback

WARNING: Database rollbacks are complex. Only attempt with DBA guidance.

1. Check migration status:

   npx prisma migrate status
   

2. Create rollback script:

   -- Document the rollback SQL before applying any migration
   -- Store in: /prisma/migrations/{migration}/rollback.sql
   

3. Apply rollback:

   # Execute rollback SQL
   psql $DATABASE_URL -f prisma/migrations/{migration}/rollback.sql
   
   # Mark migration as rolled back
   npx prisma migrate resolve --rolled-back {migration-name}
   

Feature Flag Rollback

For feature-specific issues:

# Disable problematic feature instantly
curl -X POST localhost:3000/api/admin/features/{feature-name} \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": false}'

Environment Variable Rollback

# Via Vercel CLI
vercel env rm PROBLEMATIC_VAR production
vercel env add SAFE_VAR production

# Force redeploy to pick up changes
vercel --prod

---

Communication Templates

Status Page Templates

Investigating (Initial Alert)

Title: Investigating [Component] Issues
Status: Investigating

We are currently investigating reports of [brief description].

Our team is actively looking into this issue. We will provide updates as we learn more.

Posted: [TIME] UTC

Identified (Root Cause Found)

Title: [Component] Issues - Root Cause Identified
Status: Identified

We have identified the root cause of [issue description].

Root Cause: [Brief technical explanation]
Impact: [What users experienced]
ETA for fix: [Time estimate]

We are actively working on a resolution.

Posted: [TIME] UTC

Monitoring (Fix Applied)

Title: [Component] Issues - Fix Applied
Status: Monitoring

We have applied a fix for [issue description].

We are monitoring the situation to ensure stability. Some users may need to refresh or re-login to see improvements.

Posted: [TIME] UTC

Resolved

Title: [Component] Issues - Resolved
Status: Resolved

The issue affecting [component] has been resolved.

Duration: [START TIME] to [END TIME] UTC ([X] minutes)
Impact: [Summary of impact]
Root Cause: [Brief explanation]

We apologize for any inconvenience. A detailed post-mortem will follow.

Posted: [TIME] UTC

Internal Communication Templates

Slack Alert (SEV-1)

🚨 **SEV-1 INCIDENT DECLARED** 🚨

**Issue:** [Brief description]
**Impact:** [User impact]
**Incident Commander:** @[name]
**War Room:** [Zoom/Slack channel link]

All hands requested. Please acknowledge if you're joining the response.

Email to Affected Customers

Subject: [Petunia] Service Disruption - [Date]

Dear [Customer Name],

We experienced a service disruption today that may have affected your account.

What happened:
[Brief, non-technical explanation]

Duration:
[Start time] to [End time] (approximately [X] minutes)

What we're doing:
[Brief explanation of fix and prevention]

We sincerely apologize for any inconvenience. If you experienced any issues during this time, please contact support@petunia.ai.

Best regards,
The Petunia Team

---

Recovery Procedures

Full System Recovery

After a major outage, follow this recovery checklist:

1. Verify Infrastructure

   curl -s localhost:3000/api/health/status | jq '.'
   

2. Check Database Integrity

   # Run integrity check
   curl -X POST localhost:3000/api/admin/database/integrity-check \
     -H "Authorization: Bearer $ADMIN_TOKEN"
   

3. Process Backlogged Queues

   # Check queue depths
   curl -s localhost:3000/api/admin/queues/stats
   
   # If backlogged, scale workers
   # (Vercel function concurrency handles this automatically)
   

4. Verify Third-Party Connectivity

   # Run connectivity tests
   curl -X POST localhost:3000/api/admin/connectivity-test \
     -H "Authorization: Bearer $ADMIN_TOKEN"
   

5. Clear Stale Caches

   # Invalidate problematic caches
   curl -X POST localhost:3000/api/admin/cache/invalidate \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -d '{"pattern": "user:*"}'
   

6. Resume Cron Jobs

   # Trigger missed jobs
   for job in dunning webhook-retry analytics-rollup; do
     curl -X POST localhost:3000/api/cron/$job \
       -H "Authorization: Bearer $CRON_SECRET"
   done
   

7. Monitor Error Rates

   • Check Sentry for new errors
   • Monitor /api/health/status every 5 minutes
   • Watch for circuit breaker state changes

Post-Incident Checklist

• Incident documented in /docs/security/incident-log.md
• Root cause identified
• Timeline established
• Customer impact assessed
• Remediation actions identified
• Follow-up tasks created
• Post-mortem scheduled (for SEV-1/SEV-2)

---

Appendix: Quick Commands Reference

Health & Status

curl localhost:3000/api/health                    # Basic health
curl localhost:3000/api/health/status             # Detailed status
curl localhost:3000/api/ready                     # Readiness probe

Feature Control

curl localhost:3000/api/admin/features            # List all features
curl -X POST localhost:3000/api/admin/features/{name} -d '{"enabled": false}'

Queue Management

curl localhost:3000/api/admin/queues/stats        # Queue statistics
curl localhost:3000/api/admin/queues/failed       # Failed jobs
curl -X POST localhost:3000/api/admin/queues/retry-all

Circuit Breakers

curl localhost:3000/api/admin/circuit-breakers    # All states
curl -X POST localhost:3000/api/admin/circuit-breakers/{name}/reset
curl -X POST localhost:3000/api/admin/circuit-breakers/{name}/force-open

Cron Jobs

curl localhost:3000/api/admin/cron/history        # Execution history
curl -X POST localhost:3000/api/cron/{name}       # Manual trigger

Database

curl localhost:3000/api/admin/database/stats      # Connection stats
curl -X POST localhost:3000/api/admin/database/integrity-check

Rate Limits (Emergency)

curl -X POST localhost:3000/api/admin/rate-limit/bypass -d '{"userId":"...","durationMinutes":5}'
curl -X POST localhost:3000/api/admin/rate-limit/whitelist-ip -d '{"ip":"...","durationMinutes":30}'

Rollback

vercel list --prod                                # List deployments
vercel rollback <deployment-url>                  # Instant rollback

Webhooks

curl localhost:3000/api/admin/webhooks/failed     # Failed webhooks
curl -X POST localhost:3000/api/admin/webhooks/retry/{eventId}  # Retry one
curl localhost:3000/api/admin/webhooks/stats      # Webhook statistics

---

Document History

Version	Date	Author	Changes
1.0.0	2025-12-22	Operations Team	Initial comprehensive runbook