firefrost-operations-manual/docs/skills/arbiter/SKILL.md

---
name: arbiter
description: |
  Understand and work with Arbiter, the core backend for Firefrost Gaming. Use this skill whenever:
  - Making changes to Arbiter code
  - Adding API endpoints
  - Working with Trinity Console
  - Deploying updates to Command Center
  - Debugging webhook or Discord issues
  - Working with the PostgreSQL database
  This skill prevents common mistakes and ensures proper deployment procedures.
---

# Arbiter Skill

Arbiter is the Node.js/Express backend that powers Firefrost Gaming. It handles Stripe webhooks, Discord role management, the Trinity Console admin dashboard, social analytics, and internal APIs.

---

## CRITICAL: Where Is The Code?

**Active code:** `firefrost-services` repo → `services/arbiter-3.0/`

**NOT here:**
- ❌ `services/arbiter/` — This is v2.0, ARCHIVED, do not use
- ❌ `services/_archived/` — Old stuff, historical reference only

**Production location:** Command Center (63.143.34.217) → `/opt/arbiter-3.0/`

---

## Architecture Overview

```
Arbiter 3.x
├── src/
│   ├── index.js              # Entry point
│   ├── routes/
│   │   ├── admin/            # Trinity Console routes
│   │   │   ├── index.js      # Dashboard, main admin routes
│   │   │   ├── subscribers.js
│   │   │   ├── social.js
│   │   │   └── ...
│   │   ├── api.js            # Internal API (/api/internal/*)
│   │   ├── webhook.js        # Stripe webhooks
│   │   └── oauth.js          # Discord OAuth
│   ├── views/
│   │   ├── admin/            # EJS templates for Trinity Console
│   │   └── partials/
│   ├── public/               # Static assets (CSS, JS)
│   └── middleware/
├── package.json
└── .env                      # NOT in git - on server only
```

---

## Key Endpoints

| Route | Purpose |
|-------|---------|
| `/admin` | Trinity Console dashboard |
| `/admin/subscribers` | Subscriber management |
| `/admin/social` | Social analytics |
| `/admin/mcp-logs` | MCP command logs (Task #109) |
| `/api/internal/*` | Internal API (requires INTERNAL_API_TOKEN) |
| `/webhooks/stripe` | Stripe payment webhooks |
| `/oauth/discord/*` | Discord OAuth flow |

---

## Database

**PostgreSQL** on Command Center:
```bash
PGPASSWORD='FireFrost2026!Arbiter' psql -U arbiter -h 127.0.0.1 -d arbiter_db
```

**Key tables:**
- `subscribers` — Subscription records
- `social_posts` — Social media post tracking
- `social_snapshots` — Daily follower/engagement metrics
- `auth_requests` — Trinity Core approval queue (Task #111)

---

## Environment Variables

Located at `/opt/arbiter-3.0/.env` on Command Center (NOT in git):

```
# Core
PORT=3000
NODE_ENV=production

# Database
DATABASE_URL=postgresql://arbiter:xxx@localhost/arbiter_db

# Discord
DISCORD_BOT_TOKEN=xxx
DISCORD_CLIENT_ID=xxx
DISCORD_CLIENT_SECRET=xxx
DISCORD_GUILD_ID=xxx

# Stripe
STRIPE_SECRET_KEY=xxx
STRIPE_WEBHOOK_SECRET=xxx

# Internal API
INTERNAL_API_TOKEN=6fYF1akCRW6pM2F8n3S3RxeIod4YgRniUJNEQurvBP4=

# Session
SESSION_SECRET=xxx
```

---

## Deployment Procedure

**IMPORTANT:** Arbiter is NOT a git repo on the server. Do not `git pull` in `/opt/arbiter-3.0/`.

### Standard Deploy

1. Clone `firefrost-services` to temp location:
```bash
cd /tmp
git clone https://TOKEN@git.firefrostgaming.com/firefrost-gaming/firefrost-services.git
```

2. Copy arbiter-3.0 files (preserve .env and temp):
```bash
rsync -av --exclude='.env' --exclude='node_modules' --exclude='temp' \
  /tmp/firefrost-services/services/arbiter-3.0/ /opt/arbiter-3.0/
```

3. Install dependencies if package.json changed:
```bash
cd /opt/arbiter-3.0
npm install
```

4. Restart service:
```bash
sudo systemctl restart arbiter-3
```

5. Verify:
```bash
sudo systemctl status arbiter-3
curl http://localhost:3000/health
```

6. Cleanup:
```bash
rm -rf /tmp/firefrost-services
```

### Deploy Script

There's a helper script at `/opt/scripts/deploy-arbiter.sh` on Command Center.

---

## Adding New Routes

### Admin Route (Trinity Console page)

1. Create route file: `src/routes/admin/newfeature.js`
2. Register in `src/routes/admin/index.js`:
```javascript
const newfeatureRoutes = require('./newfeature');
router.use('/newfeature', newfeatureRoutes);
```
3. Create view: `src/views/admin/newfeature.ejs`
4. Add nav link in `src/views/partials/header.ejs`

### Internal API Endpoint

Add to `src/routes/api.js`:
```javascript
router.post('/internal/newfeature', authenticateInternal, async (req, res) => {
  // Your logic here
});
```

Call with:
```bash
curl -X POST https://discord-bot.firefrostgaming.com/api/internal/newfeature \
  -H "X-Internal-Token: 6fYF1akCRW6pM2F8n3S3RxeIod4YgRniUJNEQurvBP4=" \
  -H "Content-Type: application/json" \
  -d '{"key": "value"}'
```

---

## Common Mistakes

### ❌ Editing /opt/arbiter-3.0 directly
Changes will be lost on next deploy. Always edit in `firefrost-services` repo.

### ❌ Looking at services/arbiter/ (no -3.0)
That's the archived v2.0 code. It's obsolete.

### ❌ Forgetting to exclude temp/ in rsync
The `temp/` directory on production holds runtime files (exports, uploads, etc.). Always exclude it:
```bash
rsync -av --exclude='.env' --exclude='node_modules' --exclude='temp' ...
```

### ❌ Forgetting to restart after deploy
```bash
sudo systemctl restart arbiter-3
```

### ❌ Committing .env to git
Never. It contains secrets.

### ❌ Using wrong route file
- Dashboard/main pages: `src/routes/admin/index.js`
- NOT `src/routes/admin.js` (that file may not exist or is legacy)

---

## Debugging

### Check logs
```bash
sudo journalctl -u arbiter-3 -f
```

### Check if running
```bash
sudo systemctl status arbiter-3
```

### Test health endpoint
```bash
curl http://localhost:3000/health
```

### Database query
```bash
PGPASSWORD='FireFrost2026!Arbiter' psql -U arbiter -h 127.0.0.1 -d arbiter_db -c "SELECT * FROM subscribers LIMIT 5;"
```

---

## Related Tasks

- Task #87: Lifecycle handlers (cancellation, grace period)
- Task #109: MCP Logging in Trinity Console
- Task #111: Trinity Core approval flow integration

---

## Service Details

| Property | Value |
|----------|-------|
| Service name | `arbiter-3` |
| User | `root` (runs as) |
| Port | 3000 |
| Proxy | nginx → discord-bot.firefrostgaming.com |
| Node version | 20.x |

---

**Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️