firefrost-operations-manual/docs/services/trinity-console.md

# Trinity Console - Admin Control Center

**Service:** Trinity Console (Arbiter 3.0 Admin Panel)  
**Purpose:** Unified admin interface for Firefrost Gaming operations  
**Server:** Command Center (63.143.34.217)  
**Status:** ✅ 100% Complete - All 7 modules operational  
**Deployed:** April 1, 2026, 5:00 AM CDT  
**Completed:** April 1, 2026, 11:30 AM CDT  
**Deployed by:** The Rigger (Chronicler #51)  
**Completed by:** Chronicler #52  
**Built by:** Zephyr (Chronicler #50) + Gemini AI

---

## Overview

Trinity Console is the administrative control center for Firefrost Gaming. It provides The Trinity (Michael, Meg, Holly) with a unified web interface to manage players, monitor servers, track revenue, handle grace periods, audit actions, and diagnose role issues.

**URL:** https://discord-bot.firefrostgaming.com/admin

**Built for RV Life:** Designed to run remotely with cellular connection support, enabling full operation from anywhere in the United States.

---

## Architecture

**Technology Stack:**
- **Backend:** Node.js + Express.js
- **Frontend:** EJS templates + htmx + Tailwind CSS
- **Database:** PostgreSQL (arbiter_db)
- **Authentication:** Discord OAuth2 + Passport.js
- **Security:** CSRF protection (csurf middleware)
- **Hosting:** Command Center VPS, Nginx reverse proxy

**Integration Points:**
```
Trinity Console
    ├─ PostgreSQL Database (subscriptions, users, audit logs)
    ├─ Pterodactyl Panel API (server management, whitelists)
    ├─ Discord API (role diagnostics, member data)
    └─ Paymenter (future: billing integration)
```

---

## Access Control

**Authorized Users (The Trinity):**
- **Holly** (unicorn20089): Discord ID `269225344572063754`
- **Michael** (Frostystyle): Discord ID `219309716021444609`
- **Meg** (Gingerfury66): Discord ID `669981568059703316`

**Authentication Flow:**
1. Visit `/admin` → Redirects to Discord OAuth
2. Discord authorization → Callback to Trinity Console
3. Middleware checks Discord ID against ADMIN_USERS whitelist
4. Session created → Access granted

**Security:**
- Discord OAuth2 for identity verification
- Session-based authentication
- CSRF token protection on all POST requests
- Trinity-only access (403 for unauthorized users)

---

## Modules

Trinity Console consists of 7 integrated modules:

### 1. Dashboard
**Purpose:** High-level overview of all operations  
**Features:**
- Active subscriber count
- Total MRR (Monthly Recurring Revenue)
- Server online status (12 game servers)
- Last sync timestamp
- Quick navigation to other modules

### 2. Servers (Server Matrix)
**Purpose:** Real-time game server monitoring and control  
**Features:**
- List all 12 Minecraft servers (TX1 Dallas, NC1 Charlotte)
- Server online/offline status
- Whitelist enabled/disabled status
- Last whitelist sync timestamp
- Force sync whitelist (per server or all servers)
- Toggle whitelist mode
- Automatic polling for status updates

**Key Actions:**
- **Sync Now:** Forces immediate whitelist sync with Pterodactyl Panel
- **Sync All:** Syncs all 12 servers simultaneously
- **Toggle Whitelist:** Enables/disables whitelist mode (requires server restart warning)

### 3. Players (Player Management)
**Purpose:** Subscriber and player database management  
**Features:**
- Search players by Discord ID or Minecraft username
- View player Minecraft skins (rendered live)
- Display subscription tier (Fire/Frost/Admin/Sovereign)
- Show status (active, lifetime, grace_period, cancelled)
- **Change subscription tiers** via dropdown (all tiers including Admin)
- **Toggle staff status** independently of subscription tier
- Pagination for large player lists
- Mobile-responsive design
- All changes logged in Audit Log

**Tier Display:**
- **Fire Path:** Orange badges (Fire Elemental, Fire Knight, Fire Master, Fire Legend)
- **Frost Path:** Cyan badges (Frost Elemental, Frost Knight, Frost Master, Frost Legend)
- **Universal:** Purple badges (The Awakened, Admin, Sovereign)

**Status Indicators:**
- 🟢 Green dot: Active or lifetime subscriptions
- 🟡 Yellow dot: Grace period (payment failed, 3 days to recover)
- 🔴 Red dot: Cancelled or expired

**Actions Column:**
- **Tier Dropdown:** Change any player's subscription tier (including Admin assignment)
- **Staff Checkbox:** Mark players as staff members (independent of subscription)
- Both actions create audit log entries
- Table auto-refreshes after changes
- Example use case: Moderator on Elemental tier = Tier: Elemental, Staff: ✓

**Staff Tracking:**
- Staff status stored in `users.is_staff` column
- Completely separate from subscription tier
- Allows hiring team members who are also subscribers
- Clear separation between employment and subscription

### 4. Financials & Analytics
**Purpose:** Revenue tracking and business intelligence  
**Status:** ✅ Complete (Deployed April 1, 2026)  

**Global Health Metrics:**
- **Active Subscribers:** Total count excluding grace period
- **Monthly Recurring Revenue (MRR):** Sum of active subscription values
- **Annual Run Rate (ARR):** MRR × 12
- **At Risk:** Subscribers in grace period + associated MRR
- **Lifetime Revenue:** Total from Sovereign tier ($499 × count)

**Fire vs Frost Path Intelligence:**
- Side-by-side comparison cards with gradient backgrounds
- Subscriber count per path
- Monthly revenue per path
- Marketing intelligence for path performance decisions

**Tier Performance Table:**
- All tiers with subscriber counts (Active + At Risk)
- MRR breakdown by tier
- Color-coded by path (Fire/Frost/Universal)
- Sortable by tier level (highest to lowest)

**Use Cases:**
- Real-time revenue tracking from RV
- Path popularity analysis for marketing
- At-risk subscriber identification
- Tier upgrade/downgrade trend analysis
- Financial health monitoring


### 5. Grace Period Dashboard
**Purpose:** Recovery mission control for failed payments  
**Features:**
- "We Don't Kick People Out" philosophy
- List all subscribers in grace period
- Total at-risk MRR calculation
- Countdown timers (hours remaining until expiry)
- Payment failure reasons
- Manual override actions:
  - **Extend 24h:** Adds 24 hours to grace period
  - **Manual Payment:** Marks payment as received, restores active status
- Email all at-risk button (Phase 2)

**Grace Period Policy:**
- Payment failure → Automatic 3-day grace period
- After 3 days → Downgrade to permanent Awakened tier ($1)
- Never remove access completely
- Chargeback → Immediate permanent ban (no grace period)

### 6. Audit Log (Accountability Audit)
**Purpose:** Permanent record of all Trinity actions  
**Features:**
- Logs every administrative action
- Filter by admin (Michael, Meg, Holly)
- Filter by action type (force_sync, tier_change, manual_override, etc.)
- Timestamp for each action
- Target identifier (which server or player was affected)
- JSON details for complex actions
- Chronological feed (newest first)

**Logged Actions:**
- Server whitelist syncs
- Grace period extensions
- Manual payment overrides
- Role assignments
- Tier changes
- Server configuration changes

**Purpose:** Accountability, troubleshooting, and compliance tracking

### 7. Role Audit (Discord Role Diagnostics)
**Purpose:** Detect and fix Discord role mismatches  
**Features:**
- **Run Diagnostic Scan:** Compares Discord roles vs database subscriptions
- Finds subscribers without Discord roles
- Finds Discord members with roles but no subscription
- One-click resync for mismatched users
- Perfect sync confirmation message

**Use Cases:**
- New subscriber not getting role → Detected and fixed
- Cancelled subscriber still has role → Detected and removed
- Manual role grants (for testing) → Detected as mismatch
- Post-migration role cleanup

---

## Database Schema

Trinity Console uses 3 new tables plus enhanced `subscriptions`:

### New Tables

**1. player_history**
```sql
CREATE TABLE player_history (
    id SERIAL PRIMARY KEY,
    discord_id VARCHAR(255) REFERENCES users(discord_id),
    previous_tier INT,
    new_tier INT,
    change_reason VARCHAR(255), -- 'upgrade', 'downgrade', 'payment_failed', 'manual'
    changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

**2. admin_audit_log**
```sql
CREATE TABLE admin_audit_log (
    id SERIAL PRIMARY KEY,
    admin_discord_id VARCHAR(255),
    admin_username VARCHAR(255),
    action_type VARCHAR(50), -- 'force_sync', 'manual_role_assign', 'tier_change'
    target_identifier VARCHAR(255), -- Server ID or Player Discord ID
    details JSONB,
    performed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

**3. banned_users**
```sql
CREATE TABLE banned_users (
    id SERIAL PRIMARY KEY,
    discord_id VARCHAR(255) UNIQUE,
    minecraft_username VARCHAR(255),
    minecraft_uuid VARCHAR(255),
    ban_reason VARCHAR(255), -- 'chargeback', 'tos_violation', 'manual'
    banned_by_discord_id VARCHAR(255),
    banned_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    notes TEXT
);
```

### Enhanced subscriptions Table

**New Columns:**
```sql
ALTER TABLE subscriptions ADD COLUMN
    mrr_value DECIMAL(10,2) DEFAULT 0.00,
    referrer_discord_id VARCHAR(255),
    grace_period_started_at TIMESTAMP,
    grace_period_ends_at TIMESTAMP,
    payment_failure_reason TEXT,
    last_payment_attempt TIMESTAMP;
```

### Performance Indexes

```sql
-- Status queries (used in ALL modules)
CREATE INDEX idx_subscriptions_status ON subscriptions(status);

-- Grace period queries
CREATE INDEX idx_subscriptions_grace_period_active 
ON subscriptions(status, grace_period_ends_at) 
WHERE status = 'grace_period';

-- Financials tier breakdown
CREATE INDEX idx_subscriptions_tier_status 
ON subscriptions(tier_level, status);

-- Audit log chronological feed
CREATE INDEX idx_audit_log_performed_at 
ON admin_audit_log(performed_at DESC);
```

---

## Tier System

Trinity Console recognizes 11 subscription tiers:

### Fire Path (🔥)
- **Fire Elemental** (5): $5/month
- **Fire Knight** (10): $10/month
- **Fire Master** (15): $15/month
- **Fire Legend** (20): $20/month

### Frost Path (❄️)
- **Frost Elemental** (105): $5/month
- **Frost Knight** (110): $10/month
- **Frost Master** (115): $15/month
- **Frost Legend** (120): $20/month

### Universal Path (⚡)
- **The Awakened** (1): $1/month (entry tier, permanent after grace expiry)
- **Sovereign** (499): $50 one-time (lifetime access)
- **Admin** (1000): $0 (Trinity members only)

**Configuration File:** `src/routes/admin/constants.js`

---

## Deployment Details

**Server Location:**
- Host: Command Center (63.143.34.217)
- Directory: `/opt/arbiter-3.0`
- Service: `arbiter-3` (systemd)
- Port: 3500 (internal)
- HTTPS: 443 (Nginx reverse proxy)

**Systemd Service:**
```bash
# View status
systemctl status arbiter-3

# View logs
journalctl -u arbiter-3 -f

# Restart service
systemctl restart arbiter-3
```

**Nginx Configuration:**
- SSL Certificate: Let's Encrypt (auto-renews)
- Domain: discord-bot.firefrostgaming.com
- Config: `/etc/nginx/sites-available/discord-bot.firefrostgaming.com`

**Environment Variables:**
```bash
# Database
DB_USER=arbiter
DB_HOST=127.0.0.1
DB_NAME=arbiter_db
DB_PASSWORD=[stored in Vaultwarden]
DB_PORT=5432

# Discord OAuth
DISCORD_CLIENT_ID=1487080166969577502
DISCORD_CLIENT_SECRET=[stored in Vaultwarden]
REDIRECT_URI=https://discord-bot.firefrostgaming.com/auth/discord/callback

# Pterodactyl Panel API
PANEL_URL=https://panel.firefrostgaming.com
PANEL_CLIENT_KEY=[stored in Vaultwarden]

# Access Control
ADMIN_USERS=269225344572063754,219309716021444609,669981568059703316

# Server
PORT=3500
NODE_ENV=production
SESSION_SECRET=[auto-generated]
```

---

## Deployment History

**April 1, 2026 - Initial Deployment**
- **Built by:** Zephyr (Chronicler #50) in 9-hour session
- **Deployed by:** Chronicler #51 in 2-hour deployment
- **Architecture Review:** Gemini AI (Google) partnership
- **Status:** 95% complete (Financials placeholder)
- **Launched at:** 5:00 AM CDT

**Deployment Steps:**
1. Database migration applied (3 tables, 6 columns, 7 indexes)
2. Trinity Console code deployed from Gitea monorepo
3. Dependencies installed (csurf, ejs)
4. Nginx configuration unchanged (already serving Arbiter bot)
5. Service restarted
6. Trinity access verified
7. Mobile responsive fix deployed (Holly's feedback)

**Post-Launch Fixes:**
- Mobile sidebar overlap → Hamburger menu added
- Minecraft skin rendering → Changed to mc-heads.net
- CSRF DOMContentLoaded wrapper → Fixed JavaScript error
- Admin tier added (1000) → Trinity members display correctly

---

## Mobile Support

**Responsive Design:**
- Hamburger menu (☰) on mobile devices
- Slide-out sidebar navigation
- Dark overlay backdrop
- Close button (✕) in sidebar
- Smooth 0.3s transitions
- Desktop layout unchanged (sidebar always visible)

**Breakpoint:** 768px
- **Mobile (<768px):** Sidebar hidden by default, hamburger menu
- **Desktop (≥768px):** Sidebar always visible, no hamburger

**Tested on:** iPhone (Holly's device)

---

## Development Workflow

**Monorepo Structure:**
```
firefrost-services/
└── services/
    └── arbiter-3.0/
        ├── src/
        │   ├── index.js (main Express app)
        │   ├── database.js (PostgreSQL connection)
        │   ├── routes/
        │   │   └── admin/
        │   │       ├── index.js (mounts all modules)
        │   │       ├── middleware.js (Trinity access control)
        │   │       ├── constants.js (tier definitions)
        │   │       ├── players.js
        │   │       ├── servers.js
        │   │       ├── financials.js
        │   │       ├── grace.js
        │   │       ├── audit.js
        │   │       └── roles.js
        │   └── views/
        │       ├── layout.ejs (master template)
        │       └── admin/
        │           ├── dashboard.ejs
        │           ├── players/
        │           ├── servers/
        │           ├── financials/
        │           ├── grace/
        │           ├── audit/
        │           └── roles/
        └── migrations/
            └── trinity-console.sql
```

**Deployment Process:**
1. Develop in monorepo: `git.firefrostgaming.com/firefrost-gaming/firefrost-services`
2. Commit to main branch
3. Clone to `/tmp/` on Command Center
4. Copy files to `/opt/arbiter-3.0/`
5. Install dependencies if needed
6. Restart `arbiter-3` service
7. Clean up `/tmp/`

**Git Repository:** https://git.firefrostgaming.com/firefrost-gaming/firefrost-services

---

## Phase 2 Roadmap

**Priority 1: Financials Module (45-60 min)**
- Real MRR calculations from database
- Fire vs Frost path breakdown
- Tier-by-tier revenue analytics
- At-risk MRR tracking
- Lifetime revenue from Sovereign

**Priority 2: Players Edit Functionality (30 min)**
- Tier change dropdown in Players module
- POST route to update tier_level
- Discord role sync after tier change
- Audit log entry for tier changes

**Priority 3: Email Integration (2-4 hours)**
- Send recovery emails to grace period subscribers
- 48-hour, 24-hour, 12-hour warnings
- Payment recovered confirmation
- Grace period expired notification
- Integration with Mailcow or Paymenter API

**Priority 4: Ban Management UI (45 min)**
- View all banned users
- Ban reason display
- Unban functionality
- Ban notes/history

---

## Troubleshooting

### Service Won't Start
```bash
# Check logs for errors
journalctl -u arbiter-3 -n 50

# Common issues:
# - Database connection failed (check credentials)
# - Port 3500 already in use
# - Missing dependencies (run npm install)
```

### Module Shows "Internal Server Error"
```bash
# Check real-time logs
journalctl -u arbiter-3 -f

# Then refresh the page to see the error
```

### Login Loop (OAuth Fails)
**Symptom:** Redirects to Discord, authorizes, redirects back to login

**Solution:** Verify `app.set('trust proxy', 1);` in `src/index.js`

**Reason:** Nginx does SSL termination, Express needs to trust X-Forwarded-Proto header

### CSRF Token Error (403 Forbidden)
**Symptom:** POST requests fail with 403 Forbidden

**Solutions:**
1. Clear browser cookies
2. Hard refresh (Ctrl+F5)
3. Check browser console for JavaScript errors
4. Verify CSRF middleware is before admin routes in `src/index.js`

### Skin Images Not Loading
**Current Solution:** Using mc-heads.net API (more reliable than Crafatar)

**Fallback:** If UUID invalid, shows Steve skin

---

## Security Considerations

**Authentication:**
- Discord OAuth2 for identity verification
- Whitelist-based access control (3 users only)
- Session-based authentication (7-day cookie expiry)

**CSRF Protection:**
- csurf middleware on all /admin routes
- Session-based tokens (no cookies needed)
- Automatic htmx header injection

**Database:**
- PostgreSQL connection with prepared statements
- Transaction support for multi-step operations
- Audit logging for accountability

**Secrets Management:**
- All credentials in .env file
- .env never committed to Git
- Session secret auto-generated
- Passwords stored in Vaultwarden

**Network Security:**
- HTTPS only (Let's Encrypt SSL)
- Nginx reverse proxy
- Internal port 3500 not exposed
- Command Center firewall (UFW)

---

## Monitoring

**Health Check:**
- URL: `https://discord-bot.firefrostgaming.com/health`
- Method: GET
- Returns: Status, uptime, bot username

**Key Metrics to Watch:**
- Service uptime (systemd status)
- Response time (<2 seconds target)
- Database connection status
- Pterodactyl Panel API availability
- Discord API rate limits

**Logs:**
```bash
# Live logs
journalctl -u arbiter-3 -f

# Last 100 lines
journalctl -u arbiter-3 -n 100

# Errors only
journalctl -u arbiter-3 -p err
```

---

## Related Documentation

- **Deployment Guide:** `TRINITY-CONSOLE-DEPLOYMENT-2026-04-01.md` (in firefrost-services repo)
- **Arbiter Bot (Legacy):** `docs/services/the-arbiter-discord-bot.md`
- **Database Schema:** `services/arbiter-3.0/migrations/trinity-console.sql`
- **Session Handoff:** `docs/sessions/2026-04-01-trinity-console-deployment.md`
- **Zephyr's Memorial:** `docs/relationship/memorials/zephyr-chronicler-50.md`

---

## Support

**Technical Issues:**
- Primary: Michael (Frostystyle)
- Discord: #staff-lounge channel

**Feature Requests:**
- Document in Gitea issues
- Discuss with The Trinity

**Emergency:**
- Service down: Restart via systemctl
- Database issues: Check PostgreSQL status
- Nginx issues: Check nginx -t and restart

---

**Fire + Frost + Foundation = Where Love Builds Legacy** 🔥❄️💙

---

**Last Updated:** April 1, 2026, 7:43 AM CDT  
**Built By:** Zephyr (Chronicler #50) + Gemini AI  
**Deployed By:** Chronicler #51  
**Status:** Production - 95% Complete ✅  
**Next Phase:** Financials implementation (later today)