firefrost-services/services/arbiter-3.0/DEPLOYMENT-CHECKLIST.md

Trinity Console - Deployment Checklist

Pre-Deployment (Tonight)

• Monorepo directory structure created
• Database migration file created
• Package.json updated with EJS
• README documentation created
• Receive complete code from Gemini
• Review all files for completeness
• Test locally (optional)

Deployment Day (Tomorrow Morning)

Step 1: Database Migration (8:00 AM)

# SSH to Command Center
ssh root@63.143.34.217

# Run migration
sudo -u postgres psql -d arbiter_db -f /opt/arbiter-3.0/migrations/trinity-console.sql

# Verify tables created
sudo -u postgres psql -d arbiter_db -c "\dt"

# Check new columns
sudo -u postgres psql -d arbiter_db -c "\d subscriptions"

Expected output: 4 new tables, 6 new columns in subscriptions

Step 2: Code Deployment (8:30 AM)

# Pull from monorepo
cd /opt/arbiter-3.0
git pull origin main

# Install new dependencies
npm install

# Verify EJS installed
npm list ejs

Step 3: Configuration (9:00 AM)

# No .env changes needed (uses existing ADMIN_USERS)
# Verify admin IDs are set
grep ADMIN_USERS /opt/arbiter-3.0/.env

Step 4: Service Restart (9:15 AM)

# Restart Arbiter 3
sudo systemctl restart arbiter-3

# Check logs for errors
sudo journalctl -u arbiter-3 -f

Watch for:

• "Trinity Console routes mounted" (or similar)
• No errors about missing views
• Port 3500 listening

Step 5: Access Testing (9:30 AM)

1. Visit: https://discord-bot.firefrostgaming.com/admin
2. Should redirect to Discord OAuth if not logged in
3. After OAuth, should show Trinity Console dashboard
4. Verify Trinity access (Michael, Meg, Holly only)
5. Try non-Trinity user (should get 403)

Step 6: Feature Testing (10:00 AM - 12:00 PM)

Dashboard:

• Stats cards showing correct counts
• MRR calculation accurate
• Servers online count correct
• Dark mode toggle works

Players:

• Table loads with all whitelisted players
• Sort by username works
• Sort by tier works
• Sort by date works
• Search filters correctly
• Pagination works (if >20 players)
• Edit tier dropdown shows all 10 tiers
• Changing tier updates database
• Changing tier triggers Discord role update
• Whitelist toggle works
• Bulk select works
• Bulk actions work

Servers:

• All servers listed (should be 12+)
• Grouped by TX1 and NC1
• Shows correct online/offline status
• Shows last sync timestamp
• Whitelist status accurate
• Force sync button works per server
• Sync all button works
• Toggle whitelist shows restart warning
• Real-time updates via SSE (if implemented)

Financials:

• Total MRR calculates correctly
• Fire vs Frost breakdown accurate
• Tier counts correct
• Charts render (if implemented)
• Lifetime revenue from Sovereign shown

Grace Period:

• Shows users in grace period
• Countdown timers accurate
• Payment failure reasons shown
• Manual override works

Bans:

• Shows banned users
• Add to ban list works
• Remove from ban works
• Ban reasons saved

Role Audit:

• Finds Discord roles without DB records
• Finds DB records without Discord roles
• Cleanup tools work
• Sync all roles works

Audit Log:

• Shows recent admin actions
• Filter by admin works
• Filter by action type works
• Export CSV works

Additional Features:

• Minecraft skins show (if implemented)
• Discord avatars show (if implemented)
• Keyboard shortcuts work
• Mobile responsive
• Low-bandwidth mode works

Step 7: Load Testing (2:00 PM)

Simulate Load:

• Open dashboard on 3 devices simultaneously
• Force sync all servers
• Edit multiple players in quick succession
• Check for race conditions
• Verify Panel API caching works (shouldn't hit rate limits)

Step 8: Documentation Update (3:00 PM)

Update ops manual:

• Document Trinity Console URL
• Add troubleshooting section
• Screenshot key features
• Update Task #86 (mark as obsolete or complete)
• Add Trinity Console to service inventory

Step 9: Trinity Training (4:00 PM)

Walkthrough with Meg and Holly:

• How to access Trinity Console
• Player management workflow
• Server sync controls
• Grace period monitoring
• What to do if sync fails
• How to read audit log

Step 10: Go Live (6:00 PM)

Final checks:

• All features tested and working
• No critical bugs found
• Trinity trained and comfortable
• Documentation complete
• Backup plan if something breaks

Backup Plan:

• Original Arbiter 3.0 still works (whitelist sync, /link command)
• Can disable Trinity Console routes if needed
• Database can be rolled back (migrations are additive)
• Old whitelist manager still available at whitelist.firefrostgaming.com

---

Post-Deployment (Ongoing)

Week 1:

• Monitor logs daily for errors
• Watch for sync failures
• Track admin usage patterns
• Gather Trinity feedback

Week 2:

• Optimize slow queries
• Add missing features based on feedback
• Polish UI/UX rough edges

Soft Launch (April 15):

• Trinity Console battle-tested
• Ready for subscriber management
• Monitoring in place
• Alerts configured

---

Rollback Plan

If critical issues found:

# Stop service
sudo systemctl stop arbiter-3

# Rollback code
cd /opt/arbiter-3.0
git checkout [previous-commit-hash]

# Rollback database (if needed - migrations are additive, so likely not needed)
# Only do this if new tables cause issues
sudo -u postgres psql -d arbiter_db << 'SQL'
DROP TABLE IF EXISTS player_history CASCADE;
DROP TABLE IF EXISTS admin_audit_log CASCADE;
DROP TABLE IF EXISTS banned_users CASCADE;
ALTER TABLE subscriptions 
  DROP COLUMN IF EXISTS mrr_value,
  DROP COLUMN IF EXISTS referrer_discord_id,
  DROP COLUMN IF EXISTS grace_period_started_at,
  DROP COLUMN IF EXISTS grace_period_ends_at,
  DROP COLUMN IF EXISTS payment_failure_reason,
  DROP COLUMN IF EXISTS last_payment_attempt;
SQL

# Restart service
sudo systemctl start arbiter-3

---

Success Criteria

Trinity Console is successful if:

• All 3 Trinity members can access it
• Player management is faster than Discord manually
• Server sync status is visible at a glance
• Grace period tracking prevents accidental cancellations
• Audit log provides accountability
• Works reliably from RV cellular connection
• Zero downtime during deployment

---

Timeline

Tonight (11:00 PM CST): Code ready from Gemini
Tomorrow (8:00 AM CST): Deployment begins
Tomorrow (6:00 PM CST): Go live
April 15 (Soft Launch): First real subscribers

---

Built with love by The Trinity 🔥❄️💙