Documentation

INTEGRATION_MONITORING

docs/INTEGRATION_MONITORING.md

Integration Connection Health Monitoring

This document provides comprehensive guidance on monitoring the health and status of platform integrations in Petunia. It covers health check patterns, sync status tracking, monitoring dashboards, failure modes, and troubleshooting procedures.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-13

Overview

Petunia integrates with multiple external platforms (Yelp, Google Business, Facebook, etc.) to provide unified communication and review management. The health monitoring system ensures these integrations remain operational and provides visibility into connection status, sync performance, and error tracking.

Key Components

Connection Models: Platform-level connection definitions
Connection Integrations: Client-specific integration instances
Health Checks: Automated health verification with historical tracking
Sync Jobs: Durable queues for data synchronization
Monitoring Dashboard: Admin UI for health visualization

Health Status States

Status	Description	Typical Cause
`healthy`	Connection operating normally	All checks passing
`warning`	Minor issues detected	Rate limit approaching, slow responses
`error`	Connection failing	API authentication failure, service outage
`unknown`	Health status not yet determined	New connection, check pending

Connection Models

Base Connection Model

The Connection model represents platform-level integration definitions:

model Connection {
  id                    String                  @id
  name                  String                  @unique
  description           String
  icon                  String?
  category              ConnectionCategory      // ai, messaging, social, email, voice, chat, review, business
  isEnabled             Boolean                 @default(false)
  isConfigured          Boolean                 @default(false)
  type                  ConnectionType          // platform, client
  managedBy             ConnectionOwner
  health                Json                    // Current health status
  config                Json?
  ConnectionHealthCheck ConnectionHealthCheck[]
  ConnectionIntegration ConnectionIntegration[]
}

Health JSON Structure:

{
  "status": "healthy",
  "uptime": 99.5,
  "responseTime": 120,
  "errorRate": 0.005,
  "lastChecked": "2025-12-13T10:00:00Z",
  "issues": [],
  "metrics": {
    "checkDuration": 150
  }
}

Connection Health Check Model

Historical health check records:

model ConnectionHealthCheck {
  id           String                 @id
  connectionId String
  status       ConnectionHealthStatus
  uptime       Float?
  responseTime Int?                   // milliseconds
  errorRate    Float?
  issues       Json?                  // Array of issue objects
  checkedAt    DateTime               @default(now())
  Connection   Connection             @relation(fields: [connectionId], references: [id])
}

Platform-Specific Connection Models

Each integration has its own connection model with platform-specific fields:

YelpConnection:

model YelpConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

GoogleConnection:

model GoogleConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

FacebookConnection:

model FacebookConnection {
  id                      String                  @id
  pageId                  String                  @unique
  pageName                String
  accessToken             String?                 // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean                 @default(true)
  syncInterval            Int                     @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean                 @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean                 @default(false)
  connectionIntegrationId String                  @unique
}

Health Check System

ConnectionHealthService

The ConnectionHealthService (/lib/services/connectionHealthService.ts) provides centralized health monitoring:

Check Single Connection

import { ConnectionHealthService } from '@/lib/services/connectionHealthService';

const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  {
    timeout: 30000,        // Max wait time
    metrics: true,         // Collect performance metrics
    detailed: true,        // Include diagnostics
    saveResult: true       // Persist to database
  }
);

// Result structure:
{
  connectionId: string;
  status: 'healthy' | 'warning' | 'error' | 'unknown';
  uptime?: number;
  responseTime?: number;
  errorRate?: number;
  issues: ConnectionIssue[];
  checkedAt: Date;
  metrics?: Record<string, any>;
}

Check All Connections

const results = await ConnectionHealthService.checkAllConnections(
  {
    category: 'social',    // Optional filter
    isEnabled: true
  },
  {
    saveResult: true
  }
);

// Returns array of HealthCheckResult objects

Get Health History

const history = await ConnectionHealthService.getHealthHistory(
  connectionId,
  50  // limit
);

// Returns ConnectionHealthCheck[] ordered by checkedAt DESC

System Health Overview

const overview = await ConnectionHealthService.getSystemHealthOverview();

// Returns:
{
  totalConnections: number;
  healthyCount: number;
  warningCount: number;
  errorCount: number;
  unknownCount: number;
  healthyPercentage: number;
  averageResponseTime?: number;
  averageUptime?: number;
  issuesByCategory: Record<string, number>;
}

Health Check Patterns by Category

Messaging Services (Twilio, RingCentral)

Check: Message delivery success rate
Metrics: Queue size, delivery success rate
Issues: Delivery failures, delayed messages

Check: API rate limits, token validity
Metrics: API quota remaining, reset time
Issues: Rate limit exceeded, API degraded performance

Email Services (SendGrid, Mailgun)

Check: Email delivery rate, bounce rate
Metrics: Delivery rate, bounce rate, open rate
Issues: High bounce rate, delivery failures

Voice Services (Retell, ElevenLabs, Cartesia)

Check: Call success rate, audio quality
Metrics: Call success, audio quality, transcription accuracy
Issues: Service outage, quality degradation

Review Platforms (Yelp, Google, Facebook)

Check: Review sync status, API authentication
Metrics: Sync delay, API quota, reviews per hour
Issues: API authentication failure, sync delays

Sync Status Tracking

Sync Job Models

Platform-specific sync jobs track data synchronization:

YelpReviewSyncJob:

model YelpReviewSyncJob {
  id               String   @id
  yelpConnectionId String
  status           String   // pending, running, completed, failed, dead_letter
  priority         Int      @default(0)
  attempts         Int      @default(0)
  scheduledAt      DateTime
  startedAt        DateTime?
  completedAt      DateTime?
  lastError        String?
  lastErrorAt      DateTime?
  payload          Json?
  result           Json?
  createdAt        DateTime @default(now())
  updatedAt        DateTime @updatedAt
}

GoogleReviewSyncJob:

model GoogleReviewSyncJob {
  id                 String   @id
  googleConnectionId String
  status             String   // pending, running, completed, failed, dead_letter
  priority           Int      @default(0)
  attempts           Int      @default(0)
  scheduledAt        DateTime
  startedAt          DateTime?
  completedAt        DateTime?
  lastError          String?
  lastErrorAt        DateTime?
  payload            Json?
  result             Json?
  createdAt          DateTime @default(now())
  updatedAt          DateTime @updatedAt
}

Sync Job Lifecycle

1. PENDING → Job created and scheduled
   ↓
2. RUNNING → Worker claimed job and executing
   ↓
3a. COMPLETED → Sync successful
3b. FAILED → Error occurred, retry scheduled
3c. DEAD_LETTER → Max retries exceeded

Sync Job Service Pattern

// Enqueue sync job
const job = await yelpReviewSyncJobService.enqueueJob(
  yelpConnectionId,
  {
    scheduledAt: new Date(),
    priority: 0,
    payload: { fullSync: false }
  }
);

// Claim next job (worker pattern)
const job = await yelpReviewSyncJobService.claimNextJob();

// Mark job as completed
await yelpReviewSyncJobService.markCompleted(
  job.id,
  {
    total: 25,
    added: 5,
    updated: 2
  }
);

// Mark job as failed
await yelpReviewSyncJobService.markFailed(
  job.id,
  error,
  3  // attempt number
);

Sync Configuration

Each connection tracks sync settings:

Field	Type	Description
`syncEnabled`	Boolean	Whether automatic sync is enabled
`syncInterval`	Int	Minutes between syncs (default: 15)
`lastSyncAt`	DateTime	Timestamp of last successful sync

CallLogSyncState Pattern

For voice integrations (RingCentral):

model CallLogSyncState {
  id                      String                 @id
  connectionIntegrationId String
  phoneNumber             String                 // E.164 format
  status                  CallLogSyncStateStatus // active, paused, error
  cursor                  Json?                  // { lastStartTime, lastCallId, pageToken }
  lastSyncedAt            DateTime?
  lastErrorCode           String?
  lastErrorMessage        String?
  lastErrorAt             DateTime?
}

Monitoring Dashboard

ConnectionHealthMonitor Component

Location: /components/connections/admin/ConnectionHealthMonitor.tsx

Features:

Real-time health status visualization
Search and filter by status
Sort by multiple metrics
Refresh on demand
Detailed issue tracking

Usage:

import { ConnectionHealthMonitor } from '@/components/connections/admin/ConnectionHealthMonitor';

<ConnectionHealthMonitor connections={connections} />

Metrics Displayed:

Total connections count
Healthy percentage
Warning count
Error count
Per-connection uptime
Response time
Error rate
Last checked timestamp
Active issues

Admin Dashboard Integration

The health monitor integrates with the admin dashboard at:

/app/admin/connections - Connection overview
/app/admin/connections/[connectionId] - Detailed connection view

API Endpoints

Health Check Endpoints

GET /api/connections/[connectionId]/health

Retrieve health history for a specific connection.

Query Parameters:

limit - Number of records (default: 10, max: 100)

Response:

{
  "success": true,
  "data": {
    "currentHealth": {
      "status": "healthy",
      "uptime": 99.5,
      "responseTime": 120,
      "errorRate": 0.005,
      "lastChecked": "2025-12-13T10:00:00Z"
    },
    "healthHistory": [
      {
        "id": "health_xxx",
        "connectionId": "conn_xxx",
        "status": "healthy",
        "uptime": 99.5,
        "responseTime": 120,
        "errorRate": 0.005,
        "issues": [],
        "checkedAt": "2025-12-13T10:00:00Z"
      }
    ]
  }
}

POST /api/connections/[connectionId]/health

Trigger an on-demand health check.

Response:

{
  "success": true,
  "message": "Health check triggered successfully",
  "data": {
    "connectionId": "conn_xxx",
    "status": "healthy",
    "uptime": 99.5,
    "responseTime": 120,
    "errorRate": 0.005,
    "issues": [],
    "checkedAt": "2025-12-13T10:00:00Z"
  }
}

Sync Job Endpoints

GET /api/connections/yelp/jobs

Get Yelp sync job metrics.

Response:

{
  "success": true,
  "data": {
    "pending": 5,
    "running": 2,
    "completed": 150,
    "failed": 3,
    "dead_letter": 0,
    "recentJobs": [...]
  }
}

GET /api/connections/google/jobs

Get Google sync job metrics (similar structure).

Failure Modes and Recovery

Common Failure Modes

1. Token Expiration

Symptoms:

401 Unauthorized responses
Health status: error
Issue: "API authentication failure"

Recovery:

// Automatic token refresh (OAuth connections)
if (connection.refreshToken && isTokenExpired(connection.tokenExpiresAt)) {
  await refreshAccessToken(connection);
}

// Manual reconnection required
// Notify client to reconnect via OAuth flow

Prevention:

Monitor tokenExpiresAt field
Trigger refresh 24 hours before expiry
Alert clients when manual reconnection needed

2. Rate Limiting

Symptoms:

429 Too Many Requests
Health status: warning or error
Issue: "API rate limit exceeded"

Recovery:

// Exponential backoff with jitter
const backoffMs = Math.min(
  baseBackoffMs * Math.pow(2, attemptNumber),
  maxBackoffMs
);
await sleep(backoffMs + Math.random() * 1000);

Prevention:

Track API quota usage
Implement request throttling
Monitor metrics.apiRemainingQuota
Alert at 80% quota consumption

3. Webhook Failures

Symptoms:

Missing webhook events
Health status: warning
Issue: "Webhook delivery delayed"

Recovery:

// Fall back to polling
if (webhookEnabled && !recentWebhookActivity) {
  logger.warn('Webhook inactive, triggering poll sync');
  await triggerPollSync(connection);
}

Prevention:

Monitor webhook delivery timestamps
Implement webhook health checks
Set up webhook retry mechanisms
Alert on 3+ consecutive failures

4. Sync Job Deadlocks

Symptoms:

Jobs stuck in running state
Increasing pending backlog
No completedAt timestamp

Recovery:

// Timeout stuck jobs (>30 minutes)
const stuckJobs = await prisma.yelpReviewSyncJob.updateMany({
  where: {
    status: 'running',
    startedAt: { lt: new Date(Date.now() - 30 * 60 * 1000) }
  },
  data: {
    status: 'failed',
    lastError: 'Job timeout - worker may have crashed'
  }
});

Prevention:

Implement job heartbeats
Set maximum job duration
Monitor worker health
Alert on jobs >15 minutes in running

5. Data Sync Conflicts

Symptoms:

Duplicate records
Missing data
Sync result errors

Recovery:

// Idempotency via external ID
await prisma.yelpReview.upsert({
  where: { externalId: review.id },
  create: reviewData,
  update: reviewData
});

Prevention:

Always use externalId for deduplication
Implement optimistic locking
Track sync cursors properly
Alert on high duplicate rates

Alerting Recommendations

Critical Alerts (PagerDuty/On-Call)

Condition	Threshold	Action
Multiple connections down	>3 connections in `error` state	Investigate platform issue
Service-wide outage	All connections for one category failing	Check external platform status
Dead letter queue growth	>5 jobs in `dead_letter`	Manual intervention required
Auth failure spike	>10% connections with auth errors	Verify API credentials

Alert Query:

const criticalIssues = await prisma.connection.count({
  where: {
    health: {
      path: ['status'],
      equals: 'error'
    },
    isEnabled: true
  }
});

if (criticalIssues > 3) {
  await sendPagerDutyAlert('Multiple connections down');
}

Warning Alerts (Slack/Email)

Condition	Threshold	Action
High error rate	>5% error rate for >1 hour	Review logs, check for patterns
Slow response times	Avg response time >2s	Check network/API performance
Sync backlog growing	Pending jobs increasing over time	Scale workers or investigate
Token expiration soon	<24 hours until expiry	Notify client to reconnect

Alert Query:

const healthyPercentage = (healthyCount / totalConnections) * 100;

if (healthyPercentage < 90) {
  await sendSlackAlert(`Connection health at ${healthyPercentage.toFixed(1)}%`);
}

Monitoring Metrics

Set up CloudWatch/Datadog dashboards for:

Health Status Distribution
- Pie chart: healthy vs warning vs error
- Trend over 24 hours
Response Time Trends
- Line graph: avg response time per category
- Alert on sustained increases
Sync Job Metrics
- Bar chart: pending/running/completed/failed
- Dead letter queue size
Error Rate by Category
- Heatmap: error rate per connection category
- Identify problematic integrations
Uptime SLO
- Gauge: Overall platform uptime
- Target: 99.9% uptime

Troubleshooting Guide

Diagnosis Workflow

1. Check Connection Health Status
   └─> GET /api/connections/[id]/health

2. Review Health Check Issues
   └─> Check `issues` array in response

3. Examine Sync Job Status
   └─> GET /api/connections/[platform]/jobs

4. Check Recent Error Logs
   └─> Filter logs by connectionId

5. Verify Platform Credentials
   └─> Check tokenExpiresAt, webhookEnabled

6. Test Platform API Directly
   └─> Use platform-specific test endpoints

Common Issues and Solutions

Issue: "Connection not receiving messages"

Checklist:

Verify isEnabled: true on ConnectionIntegration
Check webhook is configured in platform dashboard
Verify webhook secret matches
Test webhook endpoint with curl
Check for webhook signature validation errors in logs
Confirm portal mapping in database

Resolution:

# Test webhook endpoint
curl -X POST https://your-domain.com/api/webhooks/yelp/messages \
  -H "Content-Type: application/json" \
  -H "X-Yelp-Webhook-Signature: test" \
  -d '{"event_type":"message.created","data":{}}'

# Check webhook configuration
SELECT * FROM "YelpConnection" WHERE "webhookEnabled" = true;

Issue: "Sync jobs stuck in pending"

Checklist:

Verify cron job is running (check logs)
Check for worker process errors
Look for database connection issues
Verify no jobs stuck in running state
Check system resources (CPU/memory)

Resolution:

// Force process pending jobs
import { yelpReviewSyncJobService } from '@/lib/jobs/yelpReviewSyncJobService';

const job = await yelpReviewSyncJobService.claimNextJob();
if (job) {
  await processYelpReviewSync(job);
}

Issue: "High error rate on connection"

Checklist:

Check external platform status page
Review error messages in issues array
Verify API credentials are valid
Check for rate limiting (429 responses)
Look for network connectivity issues

Resolution:

// Trigger manual health check
const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  { detailed: true, metrics: true }
);

console.log('Issues detected:', result.issues);

Issue: "Token expired but no notification sent"

Checklist:

Verify token expiry monitoring is enabled
Check notification service logs
Confirm client email is correct
Test notification endpoint manually

Resolution:

// Manual token refresh for OAuth connections
import { googleConnectionService } from '@/lib/connections/google/googleConnectionService';

await googleConnectionService.refreshToken(connectionId);

Debug Mode

Enable detailed logging for troubleshooting:

// Set environment variable
DEBUG=petunia:connections:*

// Or in code
logger.setLevel('debug');

// Logs will include:
// - API request/response details
// - Health check execution steps
// - Sync job processing details
// - Error stack traces

Best Practices

1. Regular Health Checks

Run automated health checks every 15 minutes
Store results for trend analysis
Alert on consecutive failures (3+)

2. Proactive Token Management

Refresh tokens 24 hours before expiry
Implement graceful token refresh with fallback
Notify clients when manual reconnection needed

3. Sync Job Optimization

Use exponential backoff for retries
Implement job prioritization
Monitor dead letter queue closely
Archive completed jobs after 30 days

4. Error Handling

Always include detailed error context
Categorize errors (transient vs permanent)
Implement circuit breakers for external APIs
Log all API failures with request/response

5. Monitoring and Alerting

Set realistic SLOs (99.5% uptime per connection)
Create tiered alerting (critical vs warning)
Use runbooks for common failure scenarios
Review metrics weekly for trends

6. Client Communication

Send proactive notifications for token expiry
Provide clear reconnection instructions
Display connection health in client dashboard
Offer self-service troubleshooting tools

7. Testing

Write integration tests for health checks
Mock external API failures
Test token refresh flows
Simulate rate limiting scenarios

8. Documentation

Keep platform API documentation up to date
Document known issues and workarounds
Maintain changelog of integration changes
Share postmortems for major outages

Platform Integrations Runbook - Webhook configuration and environment variables
Client Integrations - Integration setup and pricing model
Architecture - System architecture and design patterns
Testing Guide - Integration testing procedures

Changelog

Version 1.0.0 (2025-12-13)

Initial documentation
Comprehensive health monitoring patterns
Sync job tracking guidance
Troubleshooting procedures
Alerting recommendations

Documentation

INTEGRATION_MONITORING

docs/INTEGRATION_MONITORING.md

Docs home Guides Support

Integration Connection Health Monitoring

This document provides comprehensive guidance on monitoring the health and status of platform integrations in Petunia. It covers health check patterns, sync status tracking, monitoring dashboards, failure modes, and troubleshooting procedures.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-13

Overview

Petunia integrates with multiple external platforms (Yelp, Google Business, Facebook, etc.) to provide unified communication and review management. The health monitoring system ensures these integrations remain operational and provides visibility into connection status, sync performance, and error tracking.

Key Components

Connection Models: Platform-level connection definitions
Connection Integrations: Client-specific integration instances
Health Checks: Automated health verification with historical tracking
Sync Jobs: Durable queues for data synchronization
Monitoring Dashboard: Admin UI for health visualization

Health Status States

Status	Description	Typical Cause
`healthy`	Connection operating normally	All checks passing
`warning`	Minor issues detected	Rate limit approaching, slow responses
`error`	Connection failing	API authentication failure, service outage
`unknown`	Health status not yet determined	New connection, check pending

Connection Models

Base Connection Model

The Connection model represents platform-level integration definitions:

model Connection {
  id                    String                  @id
  name                  String                  @unique
  description           String
  icon                  String?
  category              ConnectionCategory      // ai, messaging, social, email, voice, chat, review, business
  isEnabled             Boolean                 @default(false)
  isConfigured          Boolean                 @default(false)
  type                  ConnectionType          // platform, client
  managedBy             ConnectionOwner
  health                Json                    // Current health status
  config                Json?
  ConnectionHealthCheck ConnectionHealthCheck[]
  ConnectionIntegration ConnectionIntegration[]
}

Health JSON Structure:

{
  "status": "healthy",
  "uptime": 99.5,
  "responseTime": 120,
  "errorRate": 0.005,
  "lastChecked": "2025-12-13T10:00:00Z",
  "issues": [],
  "metrics": {
    "checkDuration": 150
  }
}

Connection Health Check Model

Historical health check records:

model ConnectionHealthCheck {
  id           String                 @id
  connectionId String
  status       ConnectionHealthStatus
  uptime       Float?
  responseTime Int?                   // milliseconds
  errorRate    Float?
  issues       Json?                  // Array of issue objects
  checkedAt    DateTime               @default(now())
  Connection   Connection             @relation(fields: [connectionId], references: [id])
}

Platform-Specific Connection Models

Each integration has its own connection model with platform-specific fields:

YelpConnection:

model YelpConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

GoogleConnection:

model GoogleConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

FacebookConnection:

model FacebookConnection {
  id                      String                  @id
  pageId                  String                  @unique
  pageName                String
  accessToken             String?                 // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean                 @default(true)
  syncInterval            Int                     @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean                 @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean                 @default(false)
  connectionIntegrationId String                  @unique
}

Health Check System

ConnectionHealthService

The ConnectionHealthService (/lib/services/connectionHealthService.ts) provides centralized health monitoring:

Check Single Connection

import { ConnectionHealthService } from '@/lib/services/connectionHealthService';

const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  {
    timeout: 30000,        // Max wait time
    metrics: true,         // Collect performance metrics
    detailed: true,        // Include diagnostics
    saveResult: true       // Persist to database
  }
);

// Result structure:
{
  connectionId: string;
  status: 'healthy' | 'warning' | 'error' | 'unknown';
  uptime?: number;
  responseTime?: number;
  errorRate?: number;
  issues: ConnectionIssue[];
  checkedAt: Date;
  metrics?: Record<string, any>;
}

Check All Connections

const results = await ConnectionHealthService.checkAllConnections(
  {
    category: 'social',    // Optional filter
    isEnabled: true
  },
  {
    saveResult: true
  }
);

// Returns array of HealthCheckResult objects

Get Health History

const history = await ConnectionHealthService.getHealthHistory(
  connectionId,
  50  // limit
);

// Returns ConnectionHealthCheck[] ordered by checkedAt DESC

System Health Overview

const overview = await ConnectionHealthService.getSystemHealthOverview();

// Returns:
{
  totalConnections: number;
  healthyCount: number;
  warningCount: number;
  errorCount: number;
  unknownCount: number;
  healthyPercentage: number;
  averageResponseTime?: number;
  averageUptime?: number;
  issuesByCategory: Record<string, number>;
}

Health Check Patterns by Category

Messaging Services (Twilio, RingCentral)

Check: Message delivery success rate
Metrics: Queue size, delivery success rate
Issues: Delivery failures, delayed messages

Check: API rate limits, token validity
Metrics: API quota remaining, reset time
Issues: Rate limit exceeded, API degraded performance

Email Services (SendGrid, Mailgun)

Check: Email delivery rate, bounce rate
Metrics: Delivery rate, bounce rate, open rate
Issues: High bounce rate, delivery failures

Voice Services (Retell, ElevenLabs, Cartesia)

Check: Call success rate, audio quality
Metrics: Call success, audio quality, transcription accuracy
Issues: Service outage, quality degradation

Review Platforms (Yelp, Google, Facebook)

Check: Review sync status, API authentication
Metrics: Sync delay, API quota, reviews per hour
Issues: API authentication failure, sync delays

Sync Status Tracking

Sync Job Models

Platform-specific sync jobs track data synchronization:

YelpReviewSyncJob:

model YelpReviewSyncJob {
  id               String   @id
  yelpConnectionId String
  status           String   // pending, running, completed, failed, dead_letter
  priority         Int      @default(0)
  attempts         Int      @default(0)
  scheduledAt      DateTime
  startedAt        DateTime?
  completedAt      DateTime?
  lastError        String?
  lastErrorAt      DateTime?
  payload          Json?
  result           Json?
  createdAt        DateTime @default(now())
  updatedAt        DateTime @updatedAt
}

GoogleReviewSyncJob:

model GoogleReviewSyncJob {
  id                 String   @id
  googleConnectionId String
  status             String   // pending, running, completed, failed, dead_letter
  priority           Int      @default(0)
  attempts           Int      @default(0)
  scheduledAt        DateTime
  startedAt          DateTime?
  completedAt        DateTime?
  lastError          String?
  lastErrorAt        DateTime?
  payload            Json?
  result             Json?
  createdAt          DateTime @default(now())
  updatedAt          DateTime @updatedAt
}

Sync Job Lifecycle

1. PENDING → Job created and scheduled
   ↓
2. RUNNING → Worker claimed job and executing
   ↓
3a. COMPLETED → Sync successful
3b. FAILED → Error occurred, retry scheduled
3c. DEAD_LETTER → Max retries exceeded

Sync Job Service Pattern

// Enqueue sync job
const job = await yelpReviewSyncJobService.enqueueJob(
  yelpConnectionId,
  {
    scheduledAt: new Date(),
    priority: 0,
    payload: { fullSync: false }
  }
);

// Claim next job (worker pattern)
const job = await yelpReviewSyncJobService.claimNextJob();

// Mark job as completed
await yelpReviewSyncJobService.markCompleted(
  job.id,
  {
    total: 25,
    added: 5,
    updated: 2
  }
);

// Mark job as failed
await yelpReviewSyncJobService.markFailed(
  job.id,
  error,
  3  // attempt number
);

Sync Configuration

Each connection tracks sync settings:

Field	Type	Description
`syncEnabled`	Boolean	Whether automatic sync is enabled
`syncInterval`	Int	Minutes between syncs (default: 15)
`lastSyncAt`	DateTime	Timestamp of last successful sync

CallLogSyncState Pattern

For voice integrations (RingCentral):

model CallLogSyncState {
  id                      String                 @id
  connectionIntegrationId String
  phoneNumber             String                 // E.164 format
  status                  CallLogSyncStateStatus // active, paused, error
  cursor                  Json?                  // { lastStartTime, lastCallId, pageToken }
  lastSyncedAt            DateTime?
  lastErrorCode           String?
  lastErrorMessage        String?
  lastErrorAt             DateTime?
}

Monitoring Dashboard

ConnectionHealthMonitor Component

Location: /components/connections/admin/ConnectionHealthMonitor.tsx

Features:

Real-time health status visualization
Search and filter by status
Sort by multiple metrics
Refresh on demand
Detailed issue tracking

Usage:

import { ConnectionHealthMonitor } from '@/components/connections/admin/ConnectionHealthMonitor';

<ConnectionHealthMonitor connections={connections} />

Metrics Displayed:

Total connections count
Healthy percentage
Warning count
Error count
Per-connection uptime
Response time
Error rate
Last checked timestamp
Active issues

Admin Dashboard Integration

The health monitor integrates with the admin dashboard at:

/app/admin/connections - Connection overview
/app/admin/connections/[connectionId] - Detailed connection view

API Endpoints

Health Check Endpoints

GET /api/connections/[connectionId]/health

Retrieve health history for a specific connection.

Query Parameters:

limit - Number of records (default: 10, max: 100)

Response:

{
  "success": true,
  "data": {
    "currentHealth": {
      "status": "healthy",
      "uptime": 99.5,
      "responseTime": 120,
      "errorRate": 0.005,
      "lastChecked": "2025-12-13T10:00:00Z"
    },
    "healthHistory": [
      {
        "id": "health_xxx",
        "connectionId": "conn_xxx",
        "status": "healthy",
        "uptime": 99.5,
        "responseTime": 120,
        "errorRate": 0.005,
        "issues": [],
        "checkedAt": "2025-12-13T10:00:00Z"
      }
    ]
  }
}

POST /api/connections/[connectionId]/health

Trigger an on-demand health check.

Response:

{
  "success": true,
  "message": "Health check triggered successfully",
  "data": {
    "connectionId": "conn_xxx",
    "status": "healthy",
    "uptime": 99.5,
    "responseTime": 120,
    "errorRate": 0.005,
    "issues": [],
    "checkedAt": "2025-12-13T10:00:00Z"
  }
}

Sync Job Endpoints

GET /api/connections/yelp/jobs

Get Yelp sync job metrics.

Response:

{
  "success": true,
  "data": {
    "pending": 5,
    "running": 2,
    "completed": 150,
    "failed": 3,
    "dead_letter": 0,
    "recentJobs": [...]
  }
}

GET /api/connections/google/jobs

Get Google sync job metrics (similar structure).

Failure Modes and Recovery

Common Failure Modes

1. Token Expiration

Symptoms:

401 Unauthorized responses
Health status: error
Issue: "API authentication failure"

Recovery:

// Automatic token refresh (OAuth connections)
if (connection.refreshToken && isTokenExpired(connection.tokenExpiresAt)) {
  await refreshAccessToken(connection);
}

// Manual reconnection required
// Notify client to reconnect via OAuth flow

Prevention:

Monitor tokenExpiresAt field
Trigger refresh 24 hours before expiry
Alert clients when manual reconnection needed

2. Rate Limiting

Symptoms:

429 Too Many Requests
Health status: warning or error
Issue: "API rate limit exceeded"

Recovery:

// Exponential backoff with jitter
const backoffMs = Math.min(
  baseBackoffMs * Math.pow(2, attemptNumber),
  maxBackoffMs
);
await sleep(backoffMs + Math.random() * 1000);

Prevention:

Track API quota usage
Implement request throttling
Monitor metrics.apiRemainingQuota
Alert at 80% quota consumption

3. Webhook Failures

Symptoms:

Missing webhook events
Health status: warning
Issue: "Webhook delivery delayed"

Recovery:

// Fall back to polling
if (webhookEnabled && !recentWebhookActivity) {
  logger.warn('Webhook inactive, triggering poll sync');
  await triggerPollSync(connection);
}

Prevention:

Monitor webhook delivery timestamps
Implement webhook health checks
Set up webhook retry mechanisms
Alert on 3+ consecutive failures

4. Sync Job Deadlocks

Symptoms:

Jobs stuck in running state
Increasing pending backlog
No completedAt timestamp

Recovery:

// Timeout stuck jobs (>30 minutes)
const stuckJobs = await prisma.yelpReviewSyncJob.updateMany({
  where: {
    status: 'running',
    startedAt: { lt: new Date(Date.now() - 30 * 60 * 1000) }
  },
  data: {
    status: 'failed',
    lastError: 'Job timeout - worker may have crashed'
  }
});

Prevention:

Implement job heartbeats
Set maximum job duration
Monitor worker health
Alert on jobs >15 minutes in running

5. Data Sync Conflicts

Symptoms:

Duplicate records
Missing data
Sync result errors

Recovery:

// Idempotency via external ID
await prisma.yelpReview.upsert({
  where: { externalId: review.id },
  create: reviewData,
  update: reviewData
});

Prevention:

Always use externalId for deduplication
Implement optimistic locking
Track sync cursors properly
Alert on high duplicate rates

Alerting Recommendations

Critical Alerts (PagerDuty/On-Call)

Condition	Threshold	Action
Multiple connections down	>3 connections in `error` state	Investigate platform issue
Service-wide outage	All connections for one category failing	Check external platform status
Dead letter queue growth	>5 jobs in `dead_letter`	Manual intervention required
Auth failure spike	>10% connections with auth errors	Verify API credentials

Alert Query:

const criticalIssues = await prisma.connection.count({
  where: {
    health: {
      path: ['status'],
      equals: 'error'
    },
    isEnabled: true
  }
});

if (criticalIssues > 3) {
  await sendPagerDutyAlert('Multiple connections down');
}

Warning Alerts (Slack/Email)

Condition	Threshold	Action
High error rate	>5% error rate for >1 hour	Review logs, check for patterns
Slow response times	Avg response time >2s	Check network/API performance
Sync backlog growing	Pending jobs increasing over time	Scale workers or investigate
Token expiration soon	<24 hours until expiry	Notify client to reconnect

Alert Query:

const healthyPercentage = (healthyCount / totalConnections) * 100;

if (healthyPercentage < 90) {
  await sendSlackAlert(`Connection health at ${healthyPercentage.toFixed(1)}%`);
}

Monitoring Metrics

Set up CloudWatch/Datadog dashboards for:

Health Status Distribution
- Pie chart: healthy vs warning vs error
- Trend over 24 hours
Response Time Trends
- Line graph: avg response time per category
- Alert on sustained increases
Sync Job Metrics
- Bar chart: pending/running/completed/failed
- Dead letter queue size
Error Rate by Category
- Heatmap: error rate per connection category
- Identify problematic integrations
Uptime SLO
- Gauge: Overall platform uptime
- Target: 99.9% uptime

Troubleshooting Guide

Diagnosis Workflow

1. Check Connection Health Status
   └─> GET /api/connections/[id]/health

2. Review Health Check Issues
   └─> Check `issues` array in response

3. Examine Sync Job Status
   └─> GET /api/connections/[platform]/jobs

4. Check Recent Error Logs
   └─> Filter logs by connectionId

5. Verify Platform Credentials
   └─> Check tokenExpiresAt, webhookEnabled

6. Test Platform API Directly
   └─> Use platform-specific test endpoints

Common Issues and Solutions

Issue: "Connection not receiving messages"

Checklist:

Verify isEnabled: true on ConnectionIntegration
Check webhook is configured in platform dashboard
Verify webhook secret matches
Test webhook endpoint with curl
Check for webhook signature validation errors in logs
Confirm portal mapping in database

Resolution:

# Test webhook endpoint
curl -X POST https://your-domain.com/api/webhooks/yelp/messages \
  -H "Content-Type: application/json" \
  -H "X-Yelp-Webhook-Signature: test" \
  -d '{"event_type":"message.created","data":{}}'

# Check webhook configuration
SELECT * FROM "YelpConnection" WHERE "webhookEnabled" = true;

Issue: "Sync jobs stuck in pending"

Checklist:

Verify cron job is running (check logs)
Check for worker process errors
Look for database connection issues
Verify no jobs stuck in running state
Check system resources (CPU/memory)

Resolution:

// Force process pending jobs
import { yelpReviewSyncJobService } from '@/lib/jobs/yelpReviewSyncJobService';

const job = await yelpReviewSyncJobService.claimNextJob();
if (job) {
  await processYelpReviewSync(job);
}

Issue: "High error rate on connection"

Checklist:

Check external platform status page
Review error messages in issues array
Verify API credentials are valid
Check for rate limiting (429 responses)
Look for network connectivity issues

Resolution:

// Trigger manual health check
const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  { detailed: true, metrics: true }
);

console.log('Issues detected:', result.issues);

Issue: "Token expired but no notification sent"

Checklist:

Verify token expiry monitoring is enabled
Check notification service logs
Confirm client email is correct
Test notification endpoint manually

Resolution:

// Manual token refresh for OAuth connections
import { googleConnectionService } from '@/lib/connections/google/googleConnectionService';

await googleConnectionService.refreshToken(connectionId);

Debug Mode

Enable detailed logging for troubleshooting:

// Set environment variable
DEBUG=petunia:connections:*

// Or in code
logger.setLevel('debug');

// Logs will include:
// - API request/response details
// - Health check execution steps
// - Sync job processing details
// - Error stack traces

Best Practices

1. Regular Health Checks

Run automated health checks every 15 minutes
Store results for trend analysis
Alert on consecutive failures (3+)

2. Proactive Token Management

Refresh tokens 24 hours before expiry
Implement graceful token refresh with fallback
Notify clients when manual reconnection needed

3. Sync Job Optimization

Use exponential backoff for retries
Implement job prioritization
Monitor dead letter queue closely
Archive completed jobs after 30 days

4. Error Handling

Always include detailed error context
Categorize errors (transient vs permanent)
Implement circuit breakers for external APIs
Log all API failures with request/response

5. Monitoring and Alerting

Set realistic SLOs (99.5% uptime per connection)
Create tiered alerting (critical vs warning)
Use runbooks for common failure scenarios
Review metrics weekly for trends

6. Client Communication

Send proactive notifications for token expiry
Provide clear reconnection instructions
Display connection health in client dashboard
Offer self-service troubleshooting tools

7. Testing

Write integration tests for health checks
Mock external API failures
Test token refresh flows
Simulate rate limiting scenarios

8. Documentation

Keep platform API documentation up to date
Document known issues and workarounds
Maintain changelog of integration changes
Share postmortems for major outages

Platform Integrations Runbook - Webhook configuration and environment variables
Client Integrations - Integration setup and pricing model
Architecture - System architecture and design patterns
Testing Guide - Integration testing procedures

Changelog

Version 1.0.0 (2025-12-13)

Initial documentation
Comprehensive health monitoring patterns
Sync job tracking guidance
Troubleshooting procedures
Alerting recommendations

INTEGRATION_MONITORING

Integration Connection Health Monitoring

Table of Contents

Overview

Key Components

Health Status States

Connection Models

Base Connection Model

Connection Health Check Model

Platform-Specific Connection Models

Health Check System

ConnectionHealthService

Check Single Connection

Check All Connections

Get Health History

System Health Overview

Health Check Patterns by Category

Messaging Services (Twilio, RingCentral)

Social Platforms (Yelp, Facebook, Google)

Email Services (SendGrid, Mailgun)

Voice Services (Retell, ElevenLabs, Cartesia)

Review Platforms (Yelp, Google, Facebook)

Sync Status Tracking

Sync Job Models

Sync Job Lifecycle

Sync Job Service Pattern

Sync Configuration

CallLogSyncState Pattern

Monitoring Dashboard

ConnectionHealthMonitor Component

Admin Dashboard Integration

API Endpoints

Health Check Endpoints

GET /api/connections/[connectionId]/health

POST /api/connections/[connectionId]/health

Sync Job Endpoints

GET /api/connections/yelp/jobs

GET /api/connections/google/jobs

Failure Modes and Recovery

Common Failure Modes

1. Token Expiration

2. Rate Limiting

3. Webhook Failures

4. Sync Job Deadlocks

5. Data Sync Conflicts

Alerting Recommendations

Critical Alerts (PagerDuty/On-Call)

Warning Alerts (Slack/Email)

Monitoring Metrics

Troubleshooting Guide

Diagnosis Workflow

Common Issues and Solutions

Issue: "Connection not receiving messages"

Issue: "Sync jobs stuck in pending"

Issue: "High error rate on connection"

Issue: "Token expired but no notification sent"

Debug Mode

Best Practices

1. Regular Health Checks

2. Proactive Token Management

3. Sync Job Optimization

4. Error Handling

5. Monitoring and Alerting

6. Client Communication

7. Testing

8. Documentation

Related Documentation

Changelog

Version 1.0.0 (2025-12-13)

INTEGRATION_MONITORING

Integration Connection Health Monitoring

Table of Contents

Overview

Key Components

Health Status States

Connection Models

Base Connection Model

Connection Health Check Model

Platform-Specific Connection Models

Health Check System