firefrost-services/services/arbiter/DEPLOYMENT.md

# Firefrost Arbiter - Complete Deployment Guide

**Target Server:** Command Center (63.143.34.217, Dallas)  
**Date:** March 30, 2026  
**Prepared by:** Claude (Chronicler #49)

---

## 📋 Pre-Deployment Checklist

### Discord Configuration
- [ ] Discord Application created at discord.com/developers/applications
- [ ] Bot token generated and saved securely
- [ ] Client ID and Client Secret obtained
- [ ] Server Members Intent enabled
- [ ] Redirect URIs added:
  - [ ] `https://discord-bot.firefrostgaming.com/auth/callback`
  - [ ] `https://discord-bot.firefrostgaming.com/admin/callback`
- [ ] Bot invited to server with "Manage Roles" permission
- [ ] Bot role positioned ABOVE all subscription tier roles in hierarchy

### Ghost CMS Configuration
- [ ] Custom field `discord_id` created (Settings → Membership → Custom Fields)
- [ ] Custom Integration created: "Firefrost Arbiter"
- [ ] Admin API Key copied (format: `key_id:secret`)

### Server Configuration
- [ ] Node.js 18.x installed
- [ ] Nginx installed and running
- [ ] UFW firewall configured (ports 80, 443, 3500 if needed)
- [ ] Let's Encrypt SSL certificate obtained for `discord-bot.firefrostgaming.com`
- [ ] User `architect` exists with sudo privileges

### Credentials Prepared
- [ ] Discord Bot Token
- [ ] Discord Client ID
- [ ] Discord Client Secret
- [ ] Discord Guild ID (server ID)
- [ ] Trinity Discord IDs (Michael, Meg, Holly)
- [ ] Ghost CMS URL
- [ ] Ghost Admin API Key
- [ ] Mailcow SMTP password
- [ ] Paymenter webhook secret
- [ ] SESSION_SECRET generated (32-byte random)

---

## 🚀 Phase 1: Initial Setup

### Step 1: Connect to Server

```bash
ssh architect@63.143.34.217
```

### Step 2: Create Application Directory

```bash
cd /home/architect
mkdir -p arbiter
cd arbiter
```

### Step 3: Upload Application Files

**From your local machine:**

```bash
# If using git
git clone <repository-url> /home/architect/arbiter

# Or if uploading manually via scp
scp -r discord-oauth-implementation/* architect@63.143.34.217:/home/architect/arbiter/
```

### Step 4: Install Dependencies

```bash
cd /home/architect/arbiter
npm install
```

**Expected output:**
```
added 87 packages in 12s
```

### Step 5: Create Environment File

```bash
cp .env.example .env
nano .env
```

**Fill in ALL values:**
```bash
NODE_ENV=production
PORT=3500
APP_URL=https://discord-bot.firefrostgaming.com
SESSION_SECRET=<generate with: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))">

DISCORD_BOT_TOKEN=<from Discord Developer Portal>
DISCORD_CLIENT_ID=<from Discord Developer Portal>
DISCORD_CLIENT_SECRET=<from Discord Developer Portal>
GUILD_ID=<your Discord server ID>

ADMIN_DISCORD_IDS=<michael_id>,<meg_id>,<holly_id>

CMS_URL=https://firefrostgaming.com
CMS_ADMIN_KEY=<from Ghost Integrations>

SMTP_HOST=38.68.14.188
SMTP_USER=noreply@firefrostgaming.com
SMTP_PASS=<from Mailcow>

WEBHOOK_SECRET=<from Paymenter>
```

**Generate SESSION_SECRET:**
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```

**Save and exit:** `Ctrl+X`, `Y`, `Enter`

### Step 6: Configure Discord Role Mapping

```bash
nano config/roles.json
```

**Get Discord Role IDs:**
1. Go to Discord server
2. Settings → Roles
3. Right-click each role → Copy ID

**Fill in the file:**
```json
{
  "awakened": "1234567890123456789",
  "fire_elemental": "2345678901234567890",
  "frost_elemental": "3456789012345678901",
  "fire_knight": "4567890123456789012",
  "frost_knight": "5678901234567890123",
  "fire_master": "6789012345678901234",
  "frost_master": "7890123456789012345",
  "fire_legend": "8901234567890123456",
  "frost_legend": "9012345678901234567",
  "sovereign": "0123456789012345678"
}
```

**Save and exit**

### Step 7: Set Permissions

```bash
chmod 600 .env
chmod +x backup.sh
```

---

## 🌐 Phase 2: Nginx Configuration

### Step 1: Copy Nginx Config

```bash
sudo cp nginx.conf /etc/nginx/sites-available/arbiter
sudo ln -s /etc/nginx/sites-available/arbiter /etc/nginx/sites-enabled/
```

### Step 2: Test Nginx Configuration

```bash
sudo nginx -t
```

**Expected output:**
```
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

### Step 3: Reload Nginx

```bash
sudo systemctl reload nginx
```

---

## ⚙️ Phase 3: Systemd Service Setup

### Step 1: Copy Service File

```bash
sudo cp arbiter.service /etc/systemd/system/
```

### Step 2: Reload Systemd

```bash
sudo systemctl daemon-reload
```

### Step 3: Enable Service (Start on Boot)

```bash
sudo systemctl enable arbiter
```

### Step 4: Start Service

```bash
sudo systemctl start arbiter
```

### Step 5: Check Status

```bash
sudo systemctl status arbiter
```

**Expected output:**
```
● arbiter.service - Firefrost Arbiter - Discord Role Management System
     Loaded: loaded (/etc/systemd/system/arbiter.service; enabled)
     Active: active (running) since Sun 2026-03-30 10:00:00 CDT; 5s ago
   Main PID: 12345 (node)
      Tasks: 11 (limit: 9830)
     Memory: 45.2M
     CGroup: /system.slice/arbiter.service
             └─12345 /usr/bin/node src/index.js

Mar 30 10:00:00 command-center systemd[1]: Started Firefrost Arbiter.
Mar 30 10:00:00 command-center arbiter[12345]: [Server] Listening on port 3500
Mar 30 10:00:01 command-center arbiter[12345]: [Discord] Bot logged in as ArbiterBot#1234
Mar 30 10:00:01 command-center arbiter[12345]: [Database] Cleaned up 0 expired tokens.
```

**If status shows "failed":**
```bash
sudo journalctl -u arbiter -n 50
```

---

## ✅ Phase 4: Validation & Testing

### Step 1: Check Application Logs

```bash
sudo journalctl -u arbiter -f
```

**Look for:**
- `[Server] Listening on port 3500`
- `[Discord] Bot logged in as <BotName>`
- `[Database] Cleaned up X expired tokens`

**Press Ctrl+C to exit**

### Step 2: Test Health Endpoint

```bash
curl https://discord-bot.firefrostgaming.com/health
```

**Expected response:**
```json
{
  "uptime": 123.456,
  "discord": "ok",
  "database": "ok",
  "timestamp": "2026-03-30T15:00:00.000Z"
}
```

**If you get 502 Bad Gateway:**
- Check application is running: `sudo systemctl status arbiter`
- Check application logs: `sudo journalctl -u arbiter -n 50`
- Check Nginx is running: `sudo systemctl status nginx`

### Step 3: Test Webhook Reception (Local)

```bash
curl -X POST http://localhost:3500/webhook/billing \
  -H "Content-Type: application/json" \
  -d '{
    "event": "subscription.created",
    "customer_email": "test@firefrostgaming.com",
    "customer_name": "Test User",
    "tier": "awakened",
    "subscription_id": "test_sub_123"
  }'
```

**Check logs:**
```bash
sudo journalctl -u arbiter -n 20
```

**Look for:**
- `[Webhook] Received subscription.created for test@firefrostgaming.com`
- `[Webhook] Sent linking email to test@firefrostgaming.com`

**Check database:**
```bash
sqlite3 linking.db "SELECT * FROM link_tokens;"
```

Should show newly created token.

### Step 4: Test Admin OAuth Login

1. Visit `https://discord-bot.firefrostgaming.com/admin/login` in browser
2. Should redirect to Discord OAuth
3. Authorize with Trinity Discord account
4. Should redirect to admin panel
5. Verify search, assign functions work

### Step 5: End-to-End OAuth Test

**Create test member in Ghost CMS:**
1. Ghost Admin → Members → New Member
2. Email: `test@firefrostgaming.com`
3. Name: "Test User"

**Trigger webhook:**
```bash
curl -X POST http://localhost:3500/webhook/billing \
  -H "Content-Type: application/json" \
  -d '{
    "event": "subscription.created",
    "customer_email": "test@firefrostgaming.com",
    "customer_name": "Test User",
    "tier": "awakened",
    "subscription_id": "test_001"
  }'
```

**Check Mailcow logs for sent email:**
```bash
ssh root@38.68.14.188
docker logs -f --tail 50 mailcowdockerized_postfix-mailcow_1
```

**Copy linking URL from email (or get from database):**
```bash
sqlite3 linking.db "SELECT token FROM link_tokens WHERE email='test@firefrostgaming.com';"
```

**Build link:**
```
https://discord-bot.firefrostgaming.com/link?token=<token>
```

**Test flow:**
1. Visit link in browser
2. Should redirect to Discord OAuth
3. Authorize with test Discord account
4. Should show success page
5. Check Discord - test account should have "The Awakened" role
6. Check Ghost Admin - test member should have `discord_id` populated

---

## 💾 Phase 5: Backup Configuration

### Step 1: Create Backup Directory

```bash
mkdir -p /home/architect/backups/arbiter
chmod 700 /home/architect/backups/arbiter
```

### Step 2: Test Backup Script

```bash
cd /home/architect/arbiter
./backup.sh
```

**Check output:**
```bash
cat /home/architect/backups/arbiter/backup_log.txt
```

Should show:
```
--- Backup Started: 20260330_100000 ---
Backup completed successfully.
```

**Verify backup files exist:**
```bash
ls -lh /home/architect/backups/arbiter/
```

Should show:
```
-rw-r--r-- 1 architect architect  12K Mar 30 10:00 linking_20260330_100000.db
-rw-r--r-- 1 architect architect 4.0K Mar 30 10:00 sessions_20260330_100000.db
-rw------- 1 architect architect  892 Mar 30 10:00 env_20260330_100000.bak
-rw-r--r-- 1 architect architect  421 Mar 30 10:00 roles_20260330_100000.json
```

### Step 3: Schedule Daily Backups

```bash
crontab -e
```

**Add this line:**
```
0 4 * * * /home/architect/arbiter/backup.sh >> /home/architect/backups/arbiter/cron_error.log 2>&1
```

**Save and exit**

**Verify cron job:**
```bash
crontab -l
```

Should show the backup line.

---

## 🔗 Phase 6: Paymenter Integration

### Step 1: Configure Paymenter Webhook

1. Log in to Paymenter admin panel
2. Navigate to: System → Webhooks
3. Click "Add Webhook"
4. **URL:** `https://discord-bot.firefrostgaming.com/webhook/billing`
5. **Secret:** (use value from `.env` WEBHOOK_SECRET)
6. **Events:** Select:
   - `subscription.created`
   - `subscription.upgraded`
   - `subscription.downgraded`
   - `subscription.cancelled`
7. Save webhook

### Step 2: Test Paymenter Webhook

**From Paymenter admin:**
1. Find webhook in list
2. Click "Test Webhook"
3. Should show successful delivery

**Or manually trigger:**
```bash
curl -X POST https://discord-bot.firefrostgaming.com/webhook/billing \
  -H "Content-Type: application/json" \
  -H "x-signature: <generate_valid_hmac>" \
  -d '{
    "event": "subscription.created",
    "customer_email": "real_customer@example.com",
    "customer_name": "Real Customer",
    "tier": "awakened",
    "subscription_id": "sub_real_123"
  }'
```

---

## 📊 Phase 7: Monitoring Setup

### Step 1: Set Up Log Rotation

```bash
sudo nano /etc/logrotate.d/arbiter
```

**Add:**
```
/var/log/nginx/arbiter-*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        systemctl reload nginx > /dev/null
    endscript
}
```

### Step 2: Create Monitoring Script (Optional)

```bash
nano /home/architect/arbiter/monitor.sh
```

**Add:**
```bash
#!/bin/bash
STATUS=$(curl -s https://discord-bot.firefrostgaming.com/health | jq -r '.discord')
if [ "$STATUS" != "ok" ]; then
    echo "Arbiter health check failed at $(date)" >> /home/architect/arbiter/monitor.log
    sudo systemctl restart arbiter
fi
```

**Make executable:**
```bash
chmod +x /home/architect/arbiter/monitor.sh
```

**Schedule (every 5 minutes):**
```bash
crontab -e
```

**Add:**
```
*/5 * * * * /home/architect/arbiter/monitor.sh
```

---

## 🎉 Deployment Complete!

### Final Checklist

- [ ] Application running (`sudo systemctl status arbiter`)
- [ ] Health check returns "ok" for all services
- [ ] Test webhook received and logged
- [ ] Test OAuth flow completes successfully
- [ ] Admin panel accessible and functional
- [ ] Backups scheduled and tested
- [ ] Paymenter webhook configured
- [ ] Logs rotating properly

### Next Steps

1. **Monitor for 24 hours** before announcing to users
2. **Create test subscription** with real Paymenter flow
3. **Verify email delivery** reaches inbox (not spam)
4. **Test all subscription events** (upgrade, downgrade, cancel)
5. **Train Trinity members** on admin panel usage
6. **Update documentation** with any deployment-specific notes

### Rollback Plan (If Issues Occur)

```bash
# Stop service
sudo systemctl stop arbiter

# Disable service
sudo systemctl disable arbiter

# Remove Nginx config
sudo rm /etc/nginx/sites-enabled/arbiter
sudo systemctl reload nginx

# Application files remain in /home/architect/arbiter for debugging
```

---

## 📞 Support Contacts

**System Administrator:** Michael (The Wizard)  
**Implementation Partner:** Claude (Chronicler #49)  
**Architecture Consultant:** Gemini AI

**Documentation:** `/home/architect/arbiter/README.md`  
**Troubleshooting:** `/home/architect/arbiter/TROUBLESHOOTING.md`

---

**🔥❄️ Deployment completed by Chronicler #49 on March 30, 2026 💙**