Real-Time WebSocket System

Owner: Theo Ramos Version: 2.2.0

Owner	Last verified	Next verification due	Last CI run ID	Coverage %	Open risks
Theo Ramos	2025-12-28	2026-01-27	ci.yml#2191	Pending (websocket API + inbox Cypress smoke)	Polling fallback untested; Redis adapter scale tests missing

Verification

• CI jobs: ci.yml API + Cypress (last run: ci.yml#2191)
• Commands (seeded demo portal; use demo websocket credentials):
  • pnpm test:api --config jest.config.api.cjs --runTestsByPath tests/app/api/websocket/route.test.ts tests/app/api/socket/route.test.ts tests/app/api/socket/metrics/route.test.ts
  • pnpm cypress:run --spec "cypress/e2e/inbox/inbox-interactions.cy.ts","cypress/e2e/inbox/inbox-error-handling.cy.ts"
  • pnpm db:health
• Test suites + environment:
  • tests/app/api/websocket/route.test.ts, tests/app/api/socket/route.test.ts, tests/app/api/socket/metrics/route.test.ts (CI seeded demo DB; websocket secret from env mock)
  • cypress/e2e/inbox/inbox-interactions.cy.ts, inbox-error-handling.cy.ts (staging demo tenant; exercises realtime message flow)
• Next verification due: 2026-01-04

---

Table of Contents

1. Overview
2. Architecture
3. Authentication
4. Events & Messages
5. Configuration
6. Persistence
7. Client Integration
8. Deployment
9. Troubleshooting

---

Overview

Petunia's real-time WebSocket system provides instant updates for inbox messages, user presence, typing indicators, and conversation events. Built on Socket.IO with Prisma-backed persistence, the system runs as a separate Node.js server on port 3001.

Key Features

• JWT Authentication - Secure token-based authentication
• Room-Based Subscriptions - Portal and conversation-level isolation
• Horizontal Scaling - Redis adapter support for multi-instance deployments
• Persistent State - Prisma models for presence and typing indicators
• Automatic Cleanup - Cron-based expiration of stale data
• Connection Recovery - Automatic reconnection with exponential backoff

Technology Stack

• Server: Socket.IO 4.8+ on Node.js 20
• Database: PostgreSQL via Prisma
• Cache: Redis (optional, for scaling)
• Protocol: WebSocket with fallback to HTTP long-polling

---

Architecture

System Diagram

┌─────────────────┐         JWT Token          ┌──────────────────┐
│   Next.js App   │ ────────────────────────> │  /api/auth/      │
│   (port 3000)   │ <──────────────────────── │   ws-token       │
└────────┬────────┘                            └──────────────────┘
         │
         │ WebSocket Connection
         │ (with JWT in auth handshake)
         ▼
┌─────────────────────────────────────────────────────────────┐
│           Realtime Server (port 3001)                        │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Socket.IO Server                                       │ │
│  │  • JWT Authentication Middleware                        │ │
│  │  • Connection State Recovery (2min window)              │ │
│  │  • CORS Protection                                      │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Presence   │  │    Typing    │  │    Inbox     │      │
│  │   Handler    │  │   Handler    │  │   Handler    │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Cleanup Service (1min interval)                       │ │
│  │  • Expire typing indicators                             │ │
│  │  • Remove stale presence (24h offline)                  │ │
│  └────────────────────────────────────────────────────────┘ │
└───────────────────────┬───────────────────────────────────┘
                        │
                        ▼
           ┌────────────────────────┐
           │   PostgreSQL Database   │
           │  • UserPresence         │
           │  • TypingIndicator      │
           └────────────────────────┘

Redis Pub/Sub Bridge (for webhook → WebSocket):
┌────────────────────────┐     petunia:inbox:events     ┌────────────────────────┐
│  Next.js App           │ ──────────────────────────> │  Realtime Server       │
│  (inbox-realtime.ts)   │                              │  (redis-subscriber.ts) │
│  Publisher             │                              │  Subscriber            │
└────────────────────────┘                              └────────────────────────┘

Optional Redis Adapter (for horizontal scaling):
           ┌────────────────────────┐
           │        Redis            │
           │  (pub/sub for Socket.IO)│
           └────────────────────────┘

Redis Pub/Sub Bridge

The realtime system uses Redis pub/sub to bridge between webhook handlers (running in the Next.js app) and the WebSocket server:

1. Publisher (lib/services/inbox-realtime.ts):

   • Called by webhook handlers when new messages arrive
   • Called by AI/agent response handlers when replies are sent
   • Publishes to petunia:inbox:events channel

2. Subscriber (realtime-server/services/redis-subscriber.ts):

   • Listens to petunia:inbox:events channel
   • Dispatches events to Socket.IO for WebSocket broadcast
   • Handles new_message, conversation_update, message_status events

Event Flow:

Webhook → Message stored → broadcastNewMessage() → Redis pub → Socket.IO emit

Reliability Features:

• Auto-reconnection with exponential backoff (up to 30s)
• In-memory retry queue (100 events max) for failed publishes
• Health status tracking for monitoring endpoints
• Graceful degradation when Redis unavailable

Separate Server Architecture

The realtime-server is a standalone Node.js application in /realtime-server/:

realtime-server/
├── index.ts                    # Main server entry point
├── package.json                # Separate dependencies
├── tsconfig.json               # TypeScript config
├── Dockerfile                  # Production image
├── auth/
│   ├── jwt.ts                  # JWT verification
│   └── permissions.ts          # Event authorization
├── handlers/
│   ├── presence.ts             # Online/offline/away status
│   ├── typing.ts               # Typing indicators
│   ├── inbox.ts                # Message & conversation updates
│   └── analytics.ts            # Usage analytics (future)
└── services/
    ├── prisma.ts               # Database client
    ├── logger.ts               # Structured logging
    ├── metrics.ts              # Metrics collection
    ├── cleanup.ts              # Periodic data cleanup
    └── redis-subscriber.ts     # Redis pub/sub bridge for inbox events

Why Separate?

• Independent scaling (WebSocket server ≠ HTTP server)
• Different resource requirements (persistent connections)
• Isolated dependencies (Socket.IO, Redis adapter)
• Simplified deployment (can deploy realtime-server separately)

Room Structure

Portal Rooms (portal:{portalId})

• All users with access to a portal
• Broadcasts: new conversations, portal-wide events

Conversation Rooms (conversation:{conversationId})

• Users actively viewing a conversation
• Broadcasts: new messages, read receipts, status changes

Typing Rooms (conversation:{conversationId}:typing)

• Users subscribed to typing indicators for a conversation
• Broadcasts: typing start/stop events

---

Authentication

Flow Diagram

Client                    Next.js API                Realtime Server
  │                           │                            │
  │  1. Request WS token      │                            │
  │ ────────────────────────> │                            │
  │                           │                            │
  │                           │  2. Get user from session  │
  │                           │  3. Fetch portal access    │
  │                           │  4. Sign JWT (1h expiry)   │
  │                           │                            │
  │  5. Return JWT            │                            │
  │ <──────────────────────── │                            │
  │                           │                            │
  │  6. Connect with token & portalId                      │
  │ ──────────────────────────────────────────────────────>│
  │                           │                            │
  │                           │  7. Verify JWT signature   │
  │                           │  8. Check portal access    │
  │                           │  9. Join portal room       │
  │                           │                            │
  │  10. Connection confirmed │                            │
  │ <──────────────────────────────────────────────────────│

JWT Token Generation

Endpoint: GET /api/auth/ws-token

Source: /app/api/auth/ws-token/route.ts

Requirements:

• Authenticated user session (NextAuth)
• Valid portal access via PortalUser records

Token Payload:

{
  id: string;              // User ID
  email?: string;          // User email
  name?: string;           // Display name
  isPlatformAdmin: boolean; // Platform admin flag
  portalIds: string[];     // Array of accessible portal IDs
  iat: number;             // Issued at (Unix timestamp)
  exp: number;             // Expires at (iat + 1 hour)
}

Response:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expiresIn": "1h"
}

Error Codes:

• 401 - User not authenticated
• 500 - Server configuration error (missing JWT_SECRET)

Connection Authentication

Client Handshake:

const socket = io('http://localhost:3001', {
  auth: {
    token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...',
    portalId: 'portal_123'
  }
});

Server Validation:

1. Extract token and portalId from handshake
2. Verify JWT signature using NEXTAUTH_SECRET
3. Check token expiration
4. Validate user has access to requested portal
5. Store user data in socket context
6. Join portal:{portalId} room

Rejection Reasons:

• Missing token or portalId
• Invalid JWT signature
• Expired token
• No access to portal
• Malformed token payload

Token Refresh

Tokens expire after 1 hour. The client hook (useWebSocketToken) automatically refreshes 5 minutes before expiry:

// Auto-refresh at 55 minutes (5min buffer)
const REFRESH_BUFFER_MS = 5 * 60 * 1000;
const refreshIn = Math.max(expiryMs - REFRESH_BUFFER_MS, 60000);

On refresh, the WebSocket connection is maintained. New tokens are fetched silently in the background.

---

Events & Messages

Presence Events

Set Presence

// Client → Server
socket.emit('presence:update', {
  status: 'online' | 'away' | 'busy'
});

// Server → Clients (broadcast to portal)
socket.on('presence:update', (data: {
  userId: string;
  userName?: string;
  status: 'online' | 'away' | 'busy' | 'offline';
  lastSeen: string; // ISO 8601
}) => {
  // Update UI with user's new status
});

Subscribe to Presence

// Client → Server
socket.emit('presence:subscribe');

// Server → Client (response)
socket.on('presence:list', (data: {
  users: Array<{
    userId: string;
    userName?: string;
    status: 'online' | 'away' | 'busy';
    connectedAt: string;
  }>;
}) => {
  // Show online users in UI
});

Automatic Events:

• User comes online on connection
• User goes offline on disconnect
• Presence updated on status change

Typing Indicators

Start Typing

// Client → Server
socket.emit('typing:start', {
  conversationId: 'conv_123'
});

// Server → Clients (broadcast to conversation room)
socket.on('typing:update', (data: {
  conversationId: string;
  userId: string;
  userName?: string;
  isTyping: boolean;
}) => {
  // Show "User is typing..." indicator
});

Stop Typing

// Client → Server
socket.emit('typing:stop', {
  conversationId: 'conv_123'
});

// Automatically stops after 5 seconds of inactivity

Subscribe to Typing

// Client → Server
socket.emit('typing:subscribe', {
  conversationId: 'conv_123'
});

// Server → Client (current state)
socket.on('typing:list', (data: {
  conversationId: string;
  users: Array<{
    userId: string;
    userName?: string;
    startedAt: string;
  }>;
}) => {
  // Initialize typing state
});

Auto-Expiration:

• Typing indicators expire after 5 seconds
• Cleanup service removes expired indicators every minute
• Client should re-emit typing:start periodically while user is typing

Inbox Events

Subscribe to Inbox

// Client → Server
socket.emit('inbox:subscribe');

// Server → Client (confirmation)
socket.on('inbox:subscribed', (data: {
  portalId: string;
}) => {
  // Subscription confirmed
});

Subscribe to Conversation

// Client → Server
socket.emit('inbox:conversation:subscribe', {
  conversationId: 'conv_123'
});

// Server → Client (confirmation)
socket.on('inbox:conversation:subscribed', (data: {
  conversationId: string;
}) => {
  // Now receiving real-time updates
});

New Message

// Server → Clients (broadcast to conversation)
socket.on('inbox:message:new', (data: {
  conversationId: string;
  messageId: string;
  content: string;
  channel: string;
  senderId: string;
  senderName?: string;
  timestamp: string;
  attachments?: Array<{
    id: string;
    name: string;
    type: string;
    url: string;
  }>;
}) => {
  // Add message to UI
  // Play notification sound
  // Update unread count
});

Conversation Updated

// Server → Clients (broadcast to portal & conversation)
socket.on('inbox:conversation:updated', (data: {
  conversationId: string;
  update: {
    status?: 'open' | 'closed' | 'snoozed';
    assignedTo?: string;
    assignedToName?: string;
    priority?: 'low' | 'normal' | 'high' | 'urgent';
    labels?: string[];
    unreadCount?: number;
    lastMessageAt?: string;
    lastMessagePreview?: string;
  };
}) => {
  // Update conversation in list
});

Read Receipts

// Client → Server
socket.emit('inbox:messages:read', {
  conversationId: 'conv_123',
  messageIds: ['msg_1', 'msg_2']
});

// Server → Clients (broadcast to conversation)
socket.on('inbox:read_receipt', (data: {
  conversationId: string;
  messageIds: string[];
  readBy: string;
  readByName?: string;
  readAt: string;
}) => {
  // Update message read status
  // Show read indicators
});

Message Status

// Server → Clients (broadcast to conversation)
socket.on('inbox:message:status', (data: {
  conversationId: string;
  messageId: string;
  status: 'sending' | 'sent' | 'delivered' | 'read' | 'failed';
  timestamp: string;
  error?: string;
}) => {
  // Update message status icon
  // Show error if failed
});

Error Handling

All handlers emit error events on permission failures:

socket.on('presence:error', (error: { error: string }) => {
  console.error('Presence error:', error);
});

socket.on('typing:error', (error: { error: string }) => {
  console.error('Typing error:', error);
});

socket.on('inbox:error', (error: { error: string }) => {
  console.error('Inbox error:', error);
});

---

Configuration

Environment Variables

Next.js App (.env.local):

# WebSocket server URL (client-side)
NEXT_PUBLIC_WEBSOCKET_URL=http://localhost:3001

# JWT secret (must match realtime-server)
NEXTAUTH_SECRET=your-secret-here

# Optional: Use demo WebSocket (client-side mock)
NEXT_PUBLIC_WEBSOCKET_TEST_MODE=false

Realtime Server (realtime-server/.env):

# Required: JWT secret (must match NEXTAUTH_SECRET)
JWT_SECRET=your-secret-here
# Or use NEXTAUTH_SECRET for consistency
NEXTAUTH_SECRET=your-secret-here

# Server port
REALTIME_PORT=3001

# CORS allowed origins (comma-separated)
ALLOWED_ORIGINS=http://localhost:3000,https://yourdomain.com

# Optional: Redis for horizontal scaling
REDIS_URL=redis://localhost:6379

# Optional: Log level
LOG_LEVEL=info

Docker Configuration

Dockerfile: /realtime-server/Dockerfile

Multi-stage build:

# Build stage
FROM node:20-alpine AS builder
# ... install deps, build TypeScript

# Production stage
FROM node:20-alpine
# ... copy built files, run as non-root

Key Settings:

• Non-root user (realtime:1001)
• Health check every 30 seconds
• Signal handling via dumb-init
• Exposed port: 3001

Health Check:

// GET http://localhost:3001
{
  "status": "ok",  // or "degraded" if Redis disconnected
  "service": "petunia-realtime-server",
  "timestamp": "2025-12-04T12:00:00.000Z",
  "connections": 42,
  "redis": {
    "status": "connected",  // connected | connecting | reconnecting | disconnected
    "lastConnectedAt": "2025-12-04T12:00:00.000Z",
    "messagesReceived": 1234,
    "messagesProcessed": 1230,
    "messagesFailed": 4
  }
}

Production Deployment

Docker Compose:

services:
  realtime:
    build: ./realtime-server
    ports:
      - "3001:3001"
    environment:
      - JWT_SECRET=${NEXTAUTH_SECRET}
      - REDIS_URL=${REDIS_URL}
      - ALLOWED_ORIGINS=${ALLOWED_ORIGINS}
    depends_on:
      - postgres
      - redis

Environment-Specific URLs:

• Development: http://localhost:3001
• Staging: https://realtime-staging.petunia.gardenpatch.xyz
• Production: https://realtime.petunia.gardenpatch.xyz

Connection Tuning

Socket.IO Settings (in realtime-server/index.ts):

const io = new SocketIOServer(httpServer, {
  cors: {
    origin: ALLOWED_ORIGINS,
    credentials: true,
  },
  connectionStateRecovery: {
    maxDisconnectionDuration: 2 * 60 * 1000, // 2 minutes
    skipMiddlewares: true,
  },
  pingTimeout: 30000,  // 30 seconds
  pingInterval: 25000, // 25 seconds
});

Client Reconnection (in useWebSocket):

const {
  reconnectAttempt,
  startReconnect,
  stopReconnect,
  canReconnect,
} = useWebSocketReconnect({
  maxAttempts: 5,
  delay: 3000, // Start at 3 seconds
});

---

Persistence

Database Models

UserPresence (prisma/schema.prisma):

model UserPresence {
  id        String   @id @default(cuid())
  userId    String
  portalId  String?
  status    String   @default("offline") // online, away, busy, offline
  lastSeen  DateTime @default(now())
  metadata  Json?    // Additional data (device, location, etc.)
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt

  User   User    @relation(fields: [userId], references: [id], onDelete: Cascade)
  Portal Portal? @relation(fields: [portalId], references: [id])

  @@unique([userId, portalId])
  @@index([portalId, status])
  @@index([userId])
  @@index([lastSeen])
}

TypingIndicator (prisma/schema.prisma):

model TypingIndicator {
  id             String   @id @default(cuid())
  conversationId String
  userId         String
  expiresAt      DateTime
  createdAt      DateTime @default(now())

  Conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
  User         User         @relation(fields: [userId], references: [id], onDelete: Cascade)

  @@unique([conversationId, userId])
  @@index([expiresAt])
}

Cleanup Service

Source: /realtime-server/services/cleanup.ts

Schedule: Every 1 minute

Tasks:

1. Expired Typing Indicators

   • Delete records where expiresAt < now()
   • Prevents stale "is typing" indicators

2. Stale Presence Records

   • Delete offline users not seen in 24 hours
   • Keeps presence table lean

Implementation:

// Cleanup interval (1 minute)
const CLEANUP_INTERVAL_MS = 60_000;

async function cleanupExpiredTypingIndicators() {
  await prisma.typingIndicator.deleteMany({
    where: { expiresAt: { lt: new Date() } }
  });
}

async function cleanupStalePresence() {
  const twentyFourHoursAgo = new Date(Date.now() - 24 * 60 * 60 * 1000);
  await prisma.userPresence.deleteMany({
    where: {
      status: 'offline',
      lastSeen: { lt: twentyFourHoursAgo }
    }
  });
}

Startup & Shutdown:

// Start on server boot
cleanupTimer = startCleanupService();

// Stop on graceful shutdown
process.on('SIGTERM', () => {
  if (cleanupTimer) stopCleanupService(cleanupTimer);
});

Data Retention

Data Type	Retention	Cleanup Method
Typing indicators	5 seconds	Auto-expire via expiresAt + cron
Active presence	While connected	Updated on disconnect
Offline presence	24 hours	Cron cleanup
Connection metrics	In-memory only	Cleared on disconnect

---

Client Integration

Hook: useWebSocketToken

Location: /lib/hooks/useWebSocketToken.ts

Purpose: Fetch and auto-refresh JWT tokens for WebSocket authentication

Usage:

const { token, isLoading, error, refresh } = useWebSocketToken();

Features:

• Fetches token on mount (if authenticated)
• Auto-refreshes 5 minutes before expiry
• Returns null if user not authenticated
• Handles refresh failures gracefully

Example:

function MyComponent() {
  const { token, isLoading } = useWebSocketToken();

  if (isLoading) return <div>Connecting...</div>;
  if (!token) return <div>Authentication required</div>;

  return <WebSocketProvider token={token}>...</WebSocketProvider>;
}

Hook: useWebSocket

Location: /lib/hooks/useWebSocket.ts

Purpose: Low-level WebSocket connection management

Usage:

const {
  isConnected,
  connectionState,
  send,
  reconnect,
  disconnect,
  subscribeToInbox,
  subscribeToConversation,
  emitTyping,
  emitReadReceipt,
} = useWebSocket({
  businessId: portalId,
  token: wsToken,
  userId: user.id,
  onMessage: handleMessage,
  onError: handleError,
});

Connection States:

• idle - Not connected
• connecting - Establishing connection
• connected - Active connection
• reconnecting - Attempting to reconnect
• disconnected - Intentionally disconnected
• error - Connection failed

Methods:

// Subscribe to portal inbox
subscribeToInbox(portalId: string): boolean

// Subscribe to specific conversation
subscribeToConversation(conversationId: string): boolean

// Emit typing indicator
emitTyping(conversationId: string, isTyping: boolean): boolean

// Mark messages as read
emitReadReceipt(conversationId: string, messageIds: string[]): boolean

// Send custom message
send(type: string, data: any): boolean

// Manual reconnect
reconnect(): void

// Disconnect
disconnect(): void

Provider: InboxProvider

Location: /lib/providers/InboxProvider.tsx

Purpose: High-level inbox state management with WebSocket integration

Integration:

export function InboxProvider({ children }: { children: ReactNode }) {
  const { currentPortal } = usePortals();
  const { token } = useWebSocketToken();
  const { user } = useAuth();

  // Set up WebSocket connection
  const {
    isConnected,
    subscribeToInbox,
    subscribeToConversation,
    emitTyping,
    emitReadReceipt,
  } = useWebSocket({
    businessId: currentPortal.id,
    token,
    userId: user.id,
    onMessage: handleWebSocketMessage,
  });

  // Subscribe to inbox on connect
  useEffect(() => {
    if (isConnected) {
      subscribeToInbox(currentPortal.id);
    }
  }, [isConnected, currentPortal.id]);

  // Handle incoming messages
  function handleWebSocketMessage(message: WebSocketMessage) {
    if (message.type === 'inbox:message:new') {
      dispatch({ type: 'NEW_MESSAGE', payload: message.data });
    }
    // ... handle other message types
  }

  return (
    <InboxContext.Provider value={{ /* ... */ }}>
      {children}
    </InboxContext.Provider>
  );
}

Features:

• Automatic subscription to portal inbox
• Real-time message updates
• Typing indicator management
• Read receipt handling
• Conversation state synchronization

Context: WebSocketContext

Location: /lib/providers/WebSocketContext.tsx

Purpose: Provide WebSocket connection to entire app

Usage:

export function WebSocketProvider({ children }: { children: ReactNode }) {
  const { currentPortal } = usePortals();
  const { token } = useWebSocketToken();
  const { user } = useAuth();

  const wsConfig = useWebSocket({
    businessId: currentPortal?.id || '',
    token: token || '',
    userId: user?.id || '',
    autoConnect: !!token && !!currentPortal && !!user,
  });

  return (
    <WebSocketContext.Provider value={wsConfig}>
      {children}
    </WebSocketContext.Provider>
  );
}

// In any component:
const { isConnected, connectionState } = useWebSocketContext();

---

Deployment

Development Setup

1. Start PostgreSQL:

   docker-compose up -d postgres
   

2. Run Prisma Migrations:

   pnpm prisma migrate dev
   

3. Start Realtime Server:

   cd realtime-server
   pnpm install
   pnpm dev
   

4. Start Next.js App:

   pnpm dev
   

5. Verify Connection:

   • Open http://localhost:3000/inbox
   • Check browser console for "WebSocket connected"
   • Health check: http://localhost:3001

Docker Deployment

Build Image:

cd realtime-server
docker build -t petunia-realtime:latest .

Run Container:

docker run -d \
  --name petunia-realtime \
  -p 3001:3001 \
  -e JWT_SECRET=$NEXTAUTH_SECRET \
  -e REDIS_URL=$REDIS_URL \
  -e ALLOWED_ORIGINS=$ALLOWED_ORIGINS \
  petunia-realtime:latest

Health Check:

curl http://localhost:3001
# Expected: {"status":"ok","service":"petunia-realtime-server",...}

CI/CD Integration

GitHub Actions (.github/workflows/deploy.yml):

- name: Build realtime-server
  run: |
    cd realtime-server
    docker build -t ${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }} .
    docker push ${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }}

- name: Deploy realtime-server
  run: |
    kubectl set image deployment/realtime-server \
      realtime=${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }}

Kubernetes Deployment

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realtime-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: realtime-server
  template:
    metadata:
      labels:
        app: realtime-server
    spec:
      containers:
      - name: realtime
        image: petunia-realtime:latest
        ports:
        - containerPort: 3001
        env:
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: petunia-secrets
              key: jwt-secret
        - name: REDIS_URL
          value: redis://redis-service:6379
        - name: ALLOWED_ORIGINS
          value: https://app.petunia.gardenpatch.xyz
        livenessProbe:
          httpGet:
            path: /
            port: 3001
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /
            port: 3001
          initialDelaySeconds: 5
          periodSeconds: 10

Service:

apiVersion: v1
kind: Service
metadata:
  name: realtime-service
spec:
  type: LoadBalancer
  ports:
  - port: 3001
    targetPort: 3001
  selector:
    app: realtime-server

Horizontal Scaling

Requirements:

• Redis instance for Socket.IO adapter
• Sticky sessions (or disable connectionStateRecovery)

Enable Redis Adapter:

// realtime-server/index.ts
if (REDIS_URL) {
  const pubClient = new Redis(REDIS_URL);
  const subClient = pubClient.duplicate();
  io.adapter(createAdapter(pubClient, subClient));
}

Verify Scaling:

kubectl scale deployment/realtime-server --replicas=5
kubectl get pods -l app=realtime-server
# Should show 5 pods running

---

Troubleshooting

Connection Issues

Problem: WebSocket fails to connect

Check:

1. Realtime server is running:

   curl http://localhost:3001
   

2. JWT token is valid:

   curl http://localhost:3000/api/auth/ws-token \
     -H "Cookie: next-auth.session-token=..."
   

3. CORS origins configured:

   echo $ALLOWED_ORIGINS
   # Should include client URL
   

4. Browser console errors:

   // Look for Socket.IO errors
   socket.on('connect_error', (error) => {
     console.error('Connection failed:', error);
   });
   

Solution:

• Verify NEXTAUTH_SECRET matches between apps
• Check firewall rules for port 3001
• Ensure WebSocket protocol not blocked by proxy

Authentication Errors

Problem: "Authentication token required" or "Invalid token"

Check:

1. Token generation:

   const { token, error } = useWebSocketToken();
   console.log({ token, error });
   

2. Token payload:

   // Decode JWT (use jwt.io)
   const payload = JSON.parse(atob(token.split('.')[1]));
   console.log(payload);
   

3. Portal access:

   SELECT * FROM PortalUser WHERE userId = '...' AND portalId = '...';
   

Solution:

• Refresh token: const { refresh } = useWebSocketToken(); refresh();
• Check user has PortalUser record for portal
• Verify JWT_SECRET environment variable set

Typing Indicators Not Working

Problem: Typing indicators don't appear or get stuck

Check:

1. Subscription to typing room:

   socket.emit('typing:subscribe', { conversationId });
   

2. Cleanup service running:

   # Check realtime-server logs
   docker logs petunia-realtime | grep cleanup
   

3. Database records:

   SELECT * FROM TypingIndicator WHERE expiresAt > NOW();
   

Solution:

• Re-subscribe to conversation: socket.emit('typing:subscribe', ...)
• Manually stop typing: socket.emit('typing:stop', { conversationId })
• Check cleanup service didn't crash (restart realtime-server)

Presence Not Updating

Problem: Users stuck in "online" or "offline" state

Check:

1. Presence subscription:

   socket.emit('presence:subscribe');
   

2. Database state:

   SELECT * FROM UserPresence
   WHERE userId = '...' AND portalId = '...';
   

3. Disconnect handling:

   socket.on('disconnect', (reason) => {
     console.log('Disconnected:', reason);
   });
   

Solution:

• Manual presence update: socket.emit('presence:update', { status: 'online' })
• Clear stale records:

  DELETE FROM UserPresence
  WHERE lastSeen < NOW() - INTERVAL '24 hours';
  

• Restart realtime-server to reset all presence

High Memory Usage

Problem: Realtime server memory grows over time

Check:

1. Number of connections:

   curl http://localhost:3001 | jq .connections
   

2. Cleanup service metrics:

   docker logs petunia-realtime | grep "Cleaned up"
   

3. Orphaned timers:

   // In typing.ts
   cleanupTimers.size; // Should be small
   

Solution:

• Restart server to clear in-memory state
• Reduce CLEANUP_INTERVAL_MS for more frequent cleanup
• Add memory limit in Docker: --memory=512m
• Enable Redis adapter to distribute load

Debug Logging

Enable Debug Mode:

# Realtime server
LOG_LEVEL=debug pnpm dev

# Client-side
localStorage.debug = 'socket.io-client:*';

Useful Log Filters:

# Connection events
docker logs petunia-realtime | grep "connected\|disconnected"

# Authentication
docker logs petunia-realtime | grep "auth"

# Typing indicators
docker logs petunia-realtime | grep "typing"

# Cleanup service
docker logs petunia-realtime | grep "cleanup"

Performance Monitoring

Metrics Service:

// realtime-server/services/metrics.ts
metricsService.increment('connections.total');
metricsService.gauge('connections.active', io.engine.clientsCount);
metricsService.histogram('auth.latency_ms', latency);

Export to Monitoring:

• Add Prometheus exporter
• Send to DataDog/New Relic
• Stream to CloudWatch

---

Load Testing & Performance Baselines

Overview

WebSocket reconnect storm load tests verify server stability under various stress conditions. Tests are located at realtime-server/tests/load/websocket-load.spec.ts.

Note: These are CI-safe sanity check thresholds, not production benchmarks. Production load testing should use dedicated infrastructure with higher connection counts.

Load Test Scenarios

Scenario	Description	Key Metrics
Thundering Herd	100 clients reconnect simultaneously	Connection P95 < 5s, P99 < 10s
Gradual Storm	100 clients reconnect over 10 seconds	Connection P50 < 3s, P95 < 5s
Sustained Load	50 connections for 30 seconds	95%+ retention, latency P95 < 2s
Message Flood	20 clients × 10 msg/sec	Send P95 < 200ms, 10+ msg/sec throughput
Connection Churn	30 stable + 50 churn cycles	100% stable retention, P95 cycle < 500ms
Partial Outage	50 connections, 50% disconnect/reconnect	90%+ reconnect, P95 < 5s

Performance Baselines (CI-Safe)

Thundering Herd (100 simultaneous):
  - Connection success rate: ≥90%
  - Connection time P95: <5000ms
  - Connection time P99: <10000ms

Gradual Storm (100 over 10s):
  - Connection success rate: ≥95%
  - Connection time P50: <3000ms
  - Connection time P95: <5000ms

Sustained Load (50 for 30s):
  - Connection retention: ≥95%
  - Message latency P95: <2000ms
  - Message delivery rate: ≥90%

Message Flood (20×10 msg/sec):
  - Send latency P95: <200ms
  - Throughput: ≥10 msg/sec delivered

Connection Churn (30+50 cycles):
  - Stable connection retention: 100%
  - Churn cycle success rate: ≥95%
  - Churn cycle time P95: <500ms

Partial Outage (50% reconnect):
  - Reconnection success rate: ≥90%
  - Reconnection time P95: <5000ms

Running Load Tests

# Navigate to realtime-server
cd realtime-server

# Install dependencies
pnpm install

# Run load tests (isolated)
pnpm test -- --testPathPattern="load"

# Run with verbose output
pnpm test -- --testPathPattern="load" --reporter=verbose

Note: Load tests may take 5-10 minutes to complete due to sustained load scenarios.

---

Version History

Version	Date	Changes
2.2.0	2025-12-28	Added WebSocket reconnect storm load tests (MVP 7.18 gap closed)
2.1.1	2025-12-05	Added verification snapshot, CI/test links, and owner mapping
2.1.0	2025-12-04	Added Redis pub/sub bridge with reliability guarantees, auto-reconnect, retry queue
2.0.0	2025-12-04	Replaced in-memory storage with Prisma persistence
1.0.0	2025-12-03	Initial implementation with Socket.IO

---

Related Documentation

• Unified Inbox System
• Authentication & Authorization
• Error Tracking System

---

Maintained by: Petunia Engineering Team Last Review: 2025-12-28