firefrost-services/services/arbiter-3.0/DEPLOYMENT-CHECKLIST.md

# Trinity Console - Deployment Checklist

## Pre-Deployment (Tonight)

- [x] Monorepo directory structure created
- [x] Database migration file created
- [x] Package.json updated with EJS
- [x] README documentation created
- [ ] Receive complete code from Gemini
- [ ] Review all files for completeness
- [ ] Test locally (optional)

## Deployment Day (Tomorrow Morning)

### Step 1: Database Migration (8:00 AM)
```bash
# SSH to Command Center
ssh root@63.143.34.217

# Run migration
sudo -u postgres psql -d arbiter_db -f /opt/arbiter-3.0/migrations/trinity-console.sql

# Verify tables created
sudo -u postgres psql -d arbiter_db -c "\dt"

# Check new columns
sudo -u postgres psql -d arbiter_db -c "\d subscriptions"
```

**Expected output:** 4 new tables, 6 new columns in subscriptions

### Step 2: Code Deployment (8:30 AM)
```bash
# Pull from monorepo
cd /opt/arbiter-3.0
git pull origin main

# Install new dependencies
npm install

# Verify EJS installed
npm list ejs
```

### Step 3: Configuration (9:00 AM)
```bash
# No .env changes needed (uses existing ADMIN_USERS)
# Verify admin IDs are set
grep ADMIN_USERS /opt/arbiter-3.0/.env
```

### Step 4: Service Restart (9:15 AM)
```bash
# Restart Arbiter 3
sudo systemctl restart arbiter-3

# Check logs for errors
sudo journalctl -u arbiter-3 -f
```

**Watch for:**
- "Trinity Console routes mounted" (or similar)
- No errors about missing views
- Port 3500 listening

### Step 5: Access Testing (9:30 AM)
1. Visit: https://discord-bot.firefrostgaming.com/admin
2. Should redirect to Discord OAuth if not logged in
3. After OAuth, should show Trinity Console dashboard
4. Verify Trinity access (Michael, Meg, Holly only)
5. Try non-Trinity user (should get 403)

### Step 6: Feature Testing (10:00 AM - 12:00 PM)

**Dashboard:**
- [ ] Stats cards showing correct counts
- [ ] MRR calculation accurate
- [ ] Servers online count correct
- [ ] Dark mode toggle works

**Players:**
- [ ] Table loads with all whitelisted players
- [ ] Sort by username works
- [ ] Sort by tier works
- [ ] Sort by date works
- [ ] Search filters correctly
- [ ] Pagination works (if >20 players)
- [ ] Edit tier dropdown shows all 10 tiers
- [ ] Changing tier updates database
- [ ] Changing tier triggers Discord role update
- [ ] Whitelist toggle works
- [ ] Bulk select works
- [ ] Bulk actions work

**Servers:**
- [ ] All servers listed (should be 12+)
- [ ] Grouped by TX1 and NC1
- [ ] Shows correct online/offline status
- [ ] Shows last sync timestamp
- [ ] Whitelist status accurate
- [ ] Force sync button works per server
- [ ] Sync all button works
- [ ] Toggle whitelist shows restart warning
- [ ] Real-time updates via SSE (if implemented)

**Financials:**
- [ ] Total MRR calculates correctly
- [ ] Fire vs Frost breakdown accurate
- [ ] Tier counts correct
- [ ] Charts render (if implemented)
- [ ] Lifetime revenue from Sovereign shown

**Grace Period:**
- [ ] Shows users in grace period
- [ ] Countdown timers accurate
- [ ] Payment failure reasons shown
- [ ] Manual override works

**Bans:**
- [ ] Shows banned users
- [ ] Add to ban list works
- [ ] Remove from ban works
- [ ] Ban reasons saved

**Role Audit:**
- [ ] Finds Discord roles without DB records
- [ ] Finds DB records without Discord roles
- [ ] Cleanup tools work
- [ ] Sync all roles works

**Audit Log:**
- [ ] Shows recent admin actions
- [ ] Filter by admin works
- [ ] Filter by action type works
- [ ] Export CSV works

**Additional Features:**
- [ ] Minecraft skins show (if implemented)
- [ ] Discord avatars show (if implemented)
- [ ] Keyboard shortcuts work
- [ ] Mobile responsive
- [ ] Low-bandwidth mode works

### Step 7: Load Testing (2:00 PM)

**Simulate Load:**
- Open dashboard on 3 devices simultaneously
- Force sync all servers
- Edit multiple players in quick succession
- Check for race conditions
- Verify Panel API caching works (shouldn't hit rate limits)

### Step 8: Documentation Update (3:00 PM)

**Update ops manual:**
- Document Trinity Console URL
- Add troubleshooting section
- Screenshot key features
- Update Task #86 (mark as obsolete or complete)
- Add Trinity Console to service inventory

### Step 9: Trinity Training (4:00 PM)

**Walkthrough with Meg and Holly:**
- How to access Trinity Console
- Player management workflow
- Server sync controls
- Grace period monitoring
- What to do if sync fails
- How to read audit log

### Step 10: Go Live (6:00 PM)

**Final checks:**
- [ ] All features tested and working
- [ ] No critical bugs found
- [ ] Trinity trained and comfortable
- [ ] Documentation complete
- [ ] Backup plan if something breaks

**Backup Plan:**
- Original Arbiter 3.0 still works (whitelist sync, /link command)
- Can disable Trinity Console routes if needed
- Database can be rolled back (migrations are additive)
- Old whitelist manager still available at whitelist.firefrostgaming.com

---

## Post-Deployment (Ongoing)

**Week 1:**
- Monitor logs daily for errors
- Watch for sync failures
- Track admin usage patterns
- Gather Trinity feedback

**Week 2:**
- Optimize slow queries
- Add missing features based on feedback
- Polish UI/UX rough edges

**Soft Launch (April 15):**
- Trinity Console battle-tested
- Ready for subscriber management
- Monitoring in place
- Alerts configured

---

## Rollback Plan

**If critical issues found:**

```bash
# Stop service
sudo systemctl stop arbiter-3

# Rollback code
cd /opt/arbiter-3.0
git checkout [previous-commit-hash]

# Rollback database (if needed - migrations are additive, so likely not needed)
# Only do this if new tables cause issues
sudo -u postgres psql -d arbiter_db << 'SQL'
DROP TABLE IF EXISTS player_history CASCADE;
DROP TABLE IF EXISTS admin_audit_log CASCADE;
DROP TABLE IF EXISTS banned_users CASCADE;
ALTER TABLE subscriptions 
  DROP COLUMN IF EXISTS mrr_value,
  DROP COLUMN IF EXISTS referrer_discord_id,
  DROP COLUMN IF EXISTS grace_period_started_at,
  DROP COLUMN IF EXISTS grace_period_ends_at,
  DROP COLUMN IF EXISTS payment_failure_reason,
  DROP COLUMN IF EXISTS last_payment_attempt;
SQL

# Restart service
sudo systemctl start arbiter-3
```

---

## Success Criteria

**Trinity Console is successful if:**
- All 3 Trinity members can access it
- Player management is faster than Discord manually
- Server sync status is visible at a glance
- Grace period tracking prevents accidental cancellations
- Audit log provides accountability
- Works reliably from RV cellular connection
- Zero downtime during deployment

---

## Timeline

**Tonight (11:00 PM CST):** Code ready from Gemini  
**Tomorrow (8:00 AM CST):** Deployment begins  
**Tomorrow (6:00 PM CST):** Go live  
**April 15 (Soft Launch):** First real subscribers

---

**Built with love by The Trinity** 🔥❄️💙