gh-thedotmack-rad-mem-plugin/skills/troubleshoot/operations/worker.md

Worker Service Diagnostics

PM2 worker-specific troubleshooting for claude-mem.

PM2 Worker Overview

The claude-mem worker is a persistent background service managed by PM2. It:

• Runs Express.js server on port 37777 (default)
• Processes observations asynchronously
• Serves the viewer UI
• Provides search API endpoints

Check Worker Status

Basic Status Check

# List all PM2 processes
pm2 list

# JSON format (parseable)
pm2 jlist

# Filter for claude-mem-worker
pm2 status | grep claude-mem-worker

Expected output:

│ claude-mem-worker │ online    │ 12345  │ 0    │ 45m │ 0% │ 85.6mb │

Status meanings:

• online - Worker running correctly
• stopped - Worker stopped (normal shutdown)
• errored - Worker crashed (check logs)
• stopping - Worker shutting down
• Not listed - Worker never started

Detailed Worker Info

# Show detailed information
pm2 show claude-mem-worker

# JSON format
pm2 jlist | grep -A 20 '"name":"claude-mem-worker"'

Worker Health Endpoint

The worker exposes a health endpoint at /health:

# Check health (default port)
curl -s http://127.0.0.1:37777/health

# With custom port
PORT=$(grep CLAUDE_MEM_WORKER_PORT ~/.claude-mem/settings.json | grep -o '[0-9]\+' || echo "37777")
curl -s http://127.0.0.1:$PORT/health

Expected response: {"status":"ok"}

Error responses:

• Connection refused - Worker not running
• Timeout - Worker hung (restart needed)
• Empty response - Worker crashed mid-request

Worker Logs

View Recent Logs

# Last 50 lines
pm2 logs claude-mem-worker --lines 50 --nostream

# Last 200 lines
pm2 logs claude-mem-worker --lines 200 --nostream

# Follow logs in real-time
pm2 logs claude-mem-worker

Search Logs for Errors

# Find errors
pm2 logs claude-mem-worker --lines 500 --nostream | grep -i "error"

# Find exceptions
pm2 logs claude-mem-worker --lines 500 --nostream | grep -i "exception"

# Find failed requests
pm2 logs claude-mem-worker --lines 500 --nostream | grep -i "failed"

# All error patterns
pm2 logs claude-mem-worker --lines 500 --nostream | grep -iE "error|exception|failed|crash"

Common Log Patterns

Good startup:

Worker service started on port 37777
Database initialized
Express server listening

Database errors:

Error: SQLITE_ERROR
Error initializing database
Database locked

Port conflicts:

Error: listen EADDRINUSE
Port 37777 already in use

Crashes:

PM2        | App [claude-mem-worker] exited with code [1]
PM2        | App [claude-mem-worker] will restart in 100ms

Starting the Worker

Basic Start

cd ~/.claude/plugins/marketplaces/thedotmack/
pm2 start ecosystem.config.cjs

Start with Local PM2

If pm2 command not in PATH:

cd ~/.claude/plugins/marketplaces/thedotmack/
node_modules/.bin/pm2 start ecosystem.config.cjs

Force Restart

# Restart if already running
pm2 restart claude-mem-worker

# Delete and start fresh
pm2 delete claude-mem-worker
pm2 start ecosystem.config.cjs

Stopping the Worker

# Graceful stop
pm2 stop claude-mem-worker

# Delete completely (also removes from PM2 list)
pm2 delete claude-mem-worker

Worker Not Starting

Diagnostic Steps

1. Try manual start to see error:

   cd ~/.claude/plugins/marketplaces/thedotmack/
   node plugin/scripts/worker-service.cjs
   

   This runs the worker directly without PM2, showing full error output.

2. Check PM2 itself:

   which pm2
   pm2 --version
   

   If PM2 not found, dependencies not installed.

3. Check dependencies:

   cd ~/.claude/plugins/marketplaces/thedotmack/
   ls node_modules/@anthropic-ai/claude-agent-sdk
   ls node_modules/better-sqlite3
   ls node_modules/express
   ls node_modules/pm2
   

4. Check port availability:

   lsof -i :37777
   

   If port in use, either kill that process or change claude-mem port.

Common Fixes

Dependencies missing:

cd ~/.claude/plugins/marketplaces/thedotmack/
npm install
pm2 start ecosystem.config.cjs

Port conflict:

echo '{"env":{"CLAUDE_MEM_WORKER_PORT":"37778"}}' > ~/.claude-mem/settings.json
pm2 restart claude-mem-worker

Corrupted PM2:

pm2 kill  # Stop PM2 daemon
cd ~/.claude/plugins/marketplaces/thedotmack/
pm2 start ecosystem.config.cjs

Worker Crashing Repeatedly

If worker keeps restarting (check with pm2 status showing high restart count):

Find the Cause

1. Check error logs:

   pm2 logs claude-mem-worker --err --lines 100 --nostream
   

2. Look for crash pattern:

   pm2 logs claude-mem-worker --lines 200 --nostream | grep -A 5 "exited with code"
   

Common Crash Causes

Database corruption:

sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"

If fails, backup and recreate database.

Out of memory: Check if database is too large or memory leak. Restart:

pm2 restart claude-mem-worker

Port conflict race condition: Another process grabbing port intermittently. Change port:

echo '{"env":{"CLAUDE_MEM_WORKER_PORT":"37778"}}' > ~/.claude-mem/settings.json
pm2 restart claude-mem-worker

PM2 Management Commands

# List processes
pm2 list
pm2 jlist  # JSON format

# Show detailed info
pm2 show claude-mem-worker

# Monitor resources
pm2 monit

# Clear logs
pm2 flush claude-mem-worker

# Restart PM2 daemon
pm2 kill
pm2 resurrect  # Restore saved processes

# Save current process list
pm2 save

# Update PM2
npm install -g pm2

Testing Worker Endpoints

Once worker is running, test all endpoints:

# Health check
curl -s http://127.0.0.1:37777/health

# Viewer HTML
curl -s http://127.0.0.1:37777/ | head -20

# Stats API
curl -s http://127.0.0.1:37777/api/stats

# Search API
curl -s "http://127.0.0.1:37777/api/search/observations?q=test&format=index"

# Prompts API
curl -s "http://127.0.0.1:37777/api/prompts?limit=5"

All should return appropriate responses (HTML for viewer, JSON for APIs).