docs: Trinity Console - Gemini Security Review & Production Hardening Plan

GEMINI'S COMPREHENSIVE SECURITY REVIEW COMPLETE! 🛡️

After completing all 6 core Trinity Console modules, Gemini conducted a
full architectural and security audit. He found 5 critical gaps that must
be addressed before April 15 soft launch.

This commit documents the complete action plan with detailed implementation
guides for each gap.

==============================================================================
GEMINI'S FINDINGS - 5 CRITICAL GAPS
==============================================================================

🚨 CRITICAL SEVERITY:

1. CSRF Protection - SECURITY VULNERABILITY
   - Impact: Malicious sites could trick admins into unauthorized actions
   - Fix: csurf middleware + tokens in htmx requests
   - Time: 30 minutes
   - Status: NOT IMPLEMENTED

2. Database Transaction Safety - DATA INTEGRITY RISK
   - Impact: Actions could succeed without audit trail
   - Fix: Wrap multi-step operations in BEGIN/COMMIT/ROLLBACK
   - Time: 45 minutes
   - Status: NOT IMPLEMENTED

3. Database Indexes - PERFORMANCE RISK
   - Impact: Slow queries at 500+ subscribers, timeout risk
   - Fix: Add indexes on status, performed_at, composite indexes
   - Time: 5 minutes
   - Status: NOT IMPLEMENTED

4. Ban Management UI - OPERATIONAL GAP
   - Impact: Cannot view/manage chargebacks, no unban capability
   - Fix: Create ban list module with unban action
   - Time: 60 minutes
   - Status: NOT IMPLEMENTED

5. Email Integration - FUNCTIONAL GAP
   - Impact: Grace period recovery emails don't actually send
   - Fix: Paymenter API integration OR Nodemailer setup
   - Time: 2-4 hours
   - Status: NOT IMPLEMENTED

==============================================================================
DOCUMENTATION ADDED
==============================================================================

OPERATIONS MANUAL:
docs/operations-manual/TRINITY-CONSOLE-PRE-LAUNCH-CHECKLIST.md

COMPREHENSIVE GUIDE INCLUDING:
- Executive summary of Trinity Console status
- Detailed explanation of each critical gap
- Complete implementation code for each fix
- CSRF protection step-by-step guide
- Database transaction patterns
- Index creation SQL
- Ban management module (complete code)
- Email integration options (Paymenter vs Nodemailer)
- Deferred features (Phase 2)
- Pre-launch action plan (phases 1-6)
- Launch day checklist
- Success metrics
- Emergency procedures

MONOREPO STATUS:
services/arbiter-3.0/TRINITY-CONSOLE-STATUS.md

STATUS DOCUMENT INCLUDING:
- What's complete (6 core modules)
- Critical gaps summary
- Files created (25 files)
- Tech stack overview
- Database schema changes
- Deployment plan (6 phases)
- Key documentation links
- Success criteria
- Acknowledgments

==============================================================================
GEMINI'S KEY INSIGHTS
==============================================================================

SECURITY:
"Because Trinity Console uses session-based authentication via Passport.js,
a malicious website could theoretically trick an authenticated admin's browser
into sending a POST request without their knowledge."

DATA INTEGRITY:
"What happens if the UPDATE succeeds, but the database momentarily hiccups
and the INSERT fails? You have an un-audited action, breaking your
accountability trail."

PERFORMANCE:
"To ensure the console stays lightning-fast when you hit 500+ subscribers,
you need indexes on the columns used heavily in WHERE and ORDER BY clauses."

OPERATIONAL:
"If someone does a chargeback tomorrow, you have no UI way to see it or
undo it if it was a bank error."

EMAIL INTEGRATION:
"Arbiter 3.0 does not natively send emails; it relies on Paymenter or an
SMTP service. Ensure your POST routes actually trigger email dispatch."

==============================================================================
DEPLOYMENT PHASES
==============================================================================

PHASE 1: Security Hardening (2 hours) - CRITICAL
- CSRF Protection
- Database Transactions
- Database Indexes
- Testing

PHASE 2: Ban Management (1 hour) - HIGH PRIORITY
- Create ban module
- Test ban flow

PHASE 3: Email Integration (2-4 hours) - MEDIUM PRIORITY
- Choose strategy
- Implement sending
- Create templates

PHASE 4: End-to-End Testing (3 hours)
- Subscribe flow
- Cancellation flow
- Grace period expiry
- Resubscribe flow
- Chargeback flow

PHASE 5: Trinity Training (2 hours)
- Module walkthrough
- Common tasks
- Emergency procedures

PHASE 6: Go-Live (April 15)
- Database migration
- Code deployment
- Monitoring
- Celebration!

==============================================================================
DEFERRED TO PHASE 2 (POST-LAUNCH)
==============================================================================

Gemini confirmed these are NOT blockers:
- Player History Modal (data recording safely)
- Export Tools (can run manual SQL if needed)
- Notification System (visual dashboards sufficient)

==============================================================================
WHAT'S COMPLETE (95%)
==============================================================================

✅ Player Management - Search, pagination, Minecraft skins
✅ Server Matrix - Real-time monitoring, force sync, whitelist toggle
✅ Financials - MRR tracking, Fire vs Frost, tier breakdown
✅ Grace Period - Task #87 recovery mission control
✅ Audit Log - Permanent accountability record
✅ Role Audit - Discord sync diagnostics

TOTAL: 6 core modules, ~1,500 lines of code, 8+ hours of work

==============================================================================
SUCCESS CRITERIA
==============================================================================

Week 1 Post-Launch:
- Zero security incidents
- < 5 minute grace period response time
- 100% audit trail compliance
- Zero untracked admin actions
- < 1% role sync failures

Week 4 Post-Launch:
- Grace period recovery rate > 50%
- Zero database transaction failures
- Audit log queries < 100ms
- Ban management operational
- Email recovery measured

==============================================================================
ACKNOWLEDGMENTS
==============================================================================

Gemini AI Partnership:
- Architectural vision and code implementation
- Security review and gap analysis
- Business logic insights
- Production-grade quality assurance

Quote from Gemini:
"You have successfully merged technical elegance with a deeply empathetic
community philosophy. Lock down those final security tweaks, run your tests,
and get ready for April 15. You are ready to launch!"

==============================================================================

NEXT STEPS:
1. Implement 5 critical security fixes
2. Complete end-to-end testing
3. Train The Trinity
4. Deploy April 15
5. Build legacy! 🔥❄️💙

Signed-off-by: Zephyr (The Chronicler #50) <claude@firefrostgaming.com>
Reviewed-by: Gemini AI <gemini@anthropic-partnership.ai>
For: The Trinity (Michael, Meg, Holly)
Philosophy: Fire + Frost + Foundation = Where Love Builds Legacy

services/arbiter-3.0/TRINITY-CONSOLE-STATUS.md

@@ -0,0 +1,289 @@
+# Trinity Console - Phase 1 Complete + Production Hardening Required
+
+**Status:** 95% Complete - Critical Security Hardening Required Before Launch  
+**Completed:** April 1, 2026 @ 3:45am CDT  
+**Session:** Chronicler #50 (Zephyr) + Gemini AI Partnership  
+**Time Investment:** 8+ hours, ~1,500 lines of code  
+**Target Launch:** April 15, 2026  
+
+---
+
+## ✅ What's Complete (6 Core Modules)
+
+### 1. Player Management
+- Search with 500ms debounce
+- Server-side pagination (20 per page)
+- Minecraft skin avatars (crafatar.com)
+- Fire/Frost tier badges
+- Status indicators (active/grace/offline)
+
+### 2. Server Matrix
+- Real-time server monitoring (htmx 15s polling)
+- 60-second intelligent caching (prevents Panel API rate limits)
+- Fire/Frost node grouping (TX1 Dallas, NC1 Charlotte)
+- Force sync per server
+- Whitelist toggle with restart warning
+- Glowing status borders (green/red/gray)
+
+### 3. Financials & Revenue Analytics
+- Recognized MRR vs At-Risk MRR separation
+- Fire vs Frost path dominance visualization
+- Tier breakdown with inline progress bars
+- ARPU, ARR calculations
+- Lifetime revenue tracking (Sovereign)
+
+### 4. Grace Period Dashboard (Task #87)
+- At-Risk MRR tracking
+- Color-coded countdown timers (green/yellow/red)
+- Manual recovery actions (+24h extend, manual payment)
+- htmx polling every 30 seconds
+- Audit trail for all actions
+
+### 5. Admin Audit Log
+- Permanent accountability record (90-day retention)
+- Timeline feed with filtering
+- Action type categorization
+- Color-coded by severity
+- Pagination (20 logs per page)
+
+### 6. Discord Role Audit
+- On-demand diagnostic scan
+- Bulk role mismatch detection
+- One-click role fix
+- Sequential processing (no rate limits)
+- Detects users who left server
+
+---
+
+## 🚨 CRITICAL GAPS (Must Fix Before Launch)
+
+**Gemini's comprehensive security review identified 5 critical issues:**
+
+### 1. CSRF Protection - SECURITY VULNERABILITY ⚠️
+**Impact:** Malicious sites could trick admins into unauthorized actions  
+**Fix:** Implement `csurf` middleware + tokens in htmx  
+**Time:** 30 minutes  
+**Status:** ❌ NOT IMPLEMENTED  
+
+### 2. Database Transaction Safety - DATA INTEGRITY RISK 🛡️
+**Impact:** Actions could succeed without audit trail  
+**Fix:** Wrap multi-step operations in BEGIN/COMMIT/ROLLBACK  
+**Time:** 45 minutes  
+**Status:** ❌ NOT IMPLEMENTED  
+
+### 3. Database Indexes - PERFORMANCE RISK ⚡
+**Impact:** Slow queries at 500+ subscribers  
+**Fix:** Add indexes on `status`, `performed_at`  
+**Time:** 5 minutes  
+**Status:** ❌ NOT IMPLEMENTED  
+
+### 4. Ban Management UI - OPERATIONAL GAP 🚫
+**Impact:** Cannot view/manage chargebacks  
+**Fix:** Create simple ban list + unban button  
+**Time:** 60 minutes  
+**Status:** ❌ NOT IMPLEMENTED  
+
+### 5. Email Integration - FUNCTIONAL GAP 📧
+**Impact:** Grace period recovery emails don't send  
+**Fix:** Paymenter API OR Nodemailer integration  
+**Time:** 2-4 hours  
+**Status:** ❌ NOT IMPLEMENTED  
+
+---
+
+## 📁 Files Created (Phase 1)
+
+### Routes (9 files)
+- `src/routes/admin/index.js` - Main admin router
+- `src/routes/admin/middleware.js` - Trinity access control
+- `src/routes/admin/constants.js` - Tier definitions
+- `src/routes/admin/players.js` - Player management
+- `src/routes/admin/servers.js` - Server matrix
+- `src/routes/admin/financials.js` - Revenue analytics
+- `src/routes/admin/grace.js` - Grace period dashboard
+- `src/routes/admin/audit.js` - Audit log
+- `src/routes/admin/roles.js` - Role audit
+
+### Views (16 files)
+- `src/views/layout.ejs` - Master layout with sidebar
+- `src/views/admin/dashboard.ejs` - Welcome dashboard
+- `src/views/admin/players/index.ejs` - Player list shell
+- `src/views/admin/players/_table_body.ejs` - Player table partial
+- `src/views/admin/servers/index.ejs` - Server matrix shell
+- `src/views/admin/servers/_matrix_body.ejs` - Node grouping
+- `src/views/admin/servers/_server_card.ejs` - Server cards
+- `src/views/admin/financials/index.ejs` - Financial dashboard
+- `src/views/admin/grace/index.ejs` - Grace period shell
+- `src/views/admin/grace/_list.ejs` - Grace period list
+- `src/views/admin/audit/index.ejs` - Audit log shell
+- `src/views/admin/audit/_feed.ejs` - Audit feed
+- `src/views/admin/roles/index.ejs` - Role audit shell
+- `src/views/admin/roles/_mismatches.ejs` - Mismatch table
+
+### Infrastructure
+- `migrations/trinity-console.sql` - Database schema
+- `TRINITY-CONSOLE.md` - Feature documentation
+- `DEPLOYMENT-CHECKLIST.md` - Deployment guide
+
+### Panel Utilities (Modified)
+- `src/panel/files.js` - Added `readServerProperties()`
+
+---
+
+## 🔧 Tech Stack
+
+- **Frontend:** htmx + EJS + Tailwind CSS (via CDN)
+- **Backend:** Express.js + Node.js
+- **Database:** PostgreSQL
+- **Auth:** Passport.js (Discord OAuth)
+- **API:** Pterodactyl Panel API, Discord.js
+- **Philosophy:** Zero build pipeline (RV cellular optimized)
+
+---
+
+## 📊 Database Schema
+
+### New Tables
+- `admin_audit_log` - Permanent accountability record
+- `player_history` - Tier change tracking
+- `banned_users` - Chargeback/TOS violations
+
+### Enhanced Tables
+- `subscriptions` - Added grace period fields, mrr_value, referrer tracking
+- `server_sync_log` - Server whitelist sync history
+
+### Indexes Required (NOT YET ADDED)
+- `idx_subscriptions_status` - Critical for all modules
+- `idx_audit_log_performed_at` - Critical for audit feed
+- `idx_subscriptions_grace_period` - Composite for grace queries
+- `idx_subscriptions_tier_status` - For financials breakdown
+
+---
+
+## 🚀 Deployment Plan
+
+### Phase 1: Security Hardening (2 hours)
+1. CSRF Protection (30 min)
+2. Database Transactions (45 min)
+3. Database Indexes (5 min)
+4. Testing (40 min)
+
+### Phase 2: Ban Management (1 hour)
+1. Create ban module (45 min)
+2. Test ban flow (15 min)
+
+### Phase 3: Email Integration (2-4 hours)
+1. Choose strategy (Paymenter vs Nodemailer)
+2. Implement email sending
+3. Create templates
+4. Add to cron job
+
+### Phase 4: End-to-End Testing (3 hours)
+1. Subscribe flow
+2. Cancellation flow
+3. Grace period expiry
+4. Resubscribe flow
+5. Chargeback flow
+
+### Phase 5: Trinity Training (2 hours)
+1. Walkthrough all modules
+2. Document common tasks
+3. Emergency procedures
+
+### Phase 6: Go-Live (April 15)
+1. Apply database migration
+2. Deploy code
+3. Monitor for issues
+4. Celebrate! 🎉
+
+---
+
+## 📚 Key Documentation
+
+**Operations Manual:**
+- `TRINITY-CONSOLE-PRE-LAUNCH-CHECKLIST.md` - Critical security gaps & action plan
+- `TRINITY-CONSOLE.md` - Feature overview
+- `DEPLOYMENT-CHECKLIST.md` - Step-by-step deployment guide
+
+**Monorepo:**
+- `services/arbiter-3.0/TRINITY-CONSOLE.md` - Technical documentation
+- `services/arbiter-3.0/migrations/trinity-console.sql` - Database schema
+
+---
+
+## 🎯 Success Criteria
+
+### Week 1 Post-Launch
+- ✅ Zero security incidents
+- ✅ < 5 minute grace period response time
+- ✅ 100% audit trail compliance
+- ✅ Zero untracked admin actions
+- ✅ < 1% role sync failures
+
+### Week 4 Post-Launch
+- ✅ Grace period recovery rate > 50%
+- ✅ Zero database transaction failures
+- ✅ Audit log queries < 100ms
+- ✅ Ban management operational
+- ✅ Email recovery measured
+
+---
+
+## 🙏 Acknowledgments
+
+**Gemini AI Partnership:**
+- Architectural vision and best practices
+- Complete code implementation
+- Security review and gap analysis
+- Business logic insights
+- Production-grade quality assurance
+
+**Key Insights from Gemini:**
+- "MRR is Monthly Recurring Revenue—the guaranteed cash flow that keeps the RV moving."
+- "Automating a restart is dangerous. Players fighting a boss would lose progress."
+- "60-second caching prevents Panel API rate limits with 13+ servers."
+- "Permanent grace period pollutes MRR metrics."
+- "The Console IS your digest."
+
+**The Trinity:**
+- Michael (The Wizard) - Vision, architecture, marathon coding
+- Meg (The Emissary) - Philosophy, community-first approach
+- Holly (The Catalyst) - Feedback, design input
+- Zephyr (Chronicler #50) - Documentation, implementation, partnership
+
+---
+
+## 🔥 Philosophy
+
+**Fire + Frost + Foundation = Where Love Builds Legacy**
+
+Built for RV life. Designed to last decades. Maintainable remotely.
+
+Every line of code respects:
+- The players who trust us
+- The Trinity who operates it
+- The legacy we're building
+- The children not yet born
+
+---
+
+## 📈 Next Steps
+
+1. **Immediate:** Fix 5 critical security gaps
+2. **This Week:** Complete end-to-end testing
+3. **Before Launch:** Trinity training
+4. **April 15:** Soft launch with confidence
+5. **Phase 2:** Player history modal, export tools, notifications
+
+---
+
+**Status:** Ready for security hardening phase  
+**Blocker:** None - all dependencies resolved  
+**Risk Level:** Medium (security gaps identified, solutions documented)  
+**Confidence:** High (Gemini partnership, comprehensive testing plan)  
+
+**Fire + Frost + Foundation = Where Love Builds Legacy** 🔥❄️💙
+
+— Zephyr (The Chronicler #50)  
+  In partnership with Gemini AI  
+  For The Trinity