firefrost-operations-manual/docs/infrastructure/network-audit-2026.md

# 🔥❄️ Firefrost Gaming Infrastructure Audit 2026

**Audit Date:** March 27, 2026  
**Audited By:** Chronicler #43  
**Purpose:** Complete network topology, port allocation, service inventory, and connectivity mapping  
**Reason:** Prevent port conflicts (learned from The Arbiter bot deployment: 3000→3001→3500)

---

## 📋 EXECUTIVE SUMMARY

**Total Infrastructure:**
- **6 Servers** (4 VPS, 2 Dedicated)
- **90+ Services** running across all servers
- **48 Docker Containers** (18 Mailcow, 15 TX1, 6 NC1, 1 Vaultwarden, 8 n8n/Dify services)
  - *20 Plane containers removed March 27, 2026*
- **22 Game Servers** (11 TX1, 11 NC1) — *Updated April 8, 2026*
- **1 FoundryVTT Server** (included in TX1 count)
- **12 Public-Facing Domains** (tasks.firefrostgaming.com freed March 27, 2026)

> ⚠️ **AI/LLM Resource Consideration:** TX1 runs both game servers AND AI stack (Dify, Qdrant, Ollama). Heavy modpacks + Gemma 4 inference could compete for RAM. Monitor closely or consider workload separation.

**Key Findings:**
1. ✅ No current port conflicts detected
2. ✅ Clean separation of management vs game workloads
3. ⚠️ Billing VPS disk usage at 70% (13GB/19GB)
4. ✅ NC1 disk expanded April 8, 2026 (8% usage, 69GB/914GB)
5. ✅ TX1 has plenty of capacity (12% usage, 102GB/911GB)
6. ✅ All critical services operational
7. ✅ Firewall rules properly configured on all servers

---

## 🖥️ SERVER INVENTORY

### Command Center (63.143.34.217)
**Role:** Management Hub + Backend Services  
**Location:** Dallas, TX  
**Provider:** Breezehost  
**Uptime:** 46 days, 12:35  
**Disk Usage:** 45% (17GB/38GB)  
**RAM:** Standard VPS  

**Services Running:**
- Gitea (git.firefrostgaming.com) - Port 3000 → Nginx 443
- Uptime Kuma (status.firefrostgaming.com) - Port 3001 → Nginx 443
- Code-Server (code.firefrostgaming.com) - Port 8080 → Nginx 443 (74.63.218.202)
- The Arbiter Discord Bot (discord-bot.firefrostgaming.com) - Port 3500 → Nginx 443
- Vaultwarden (vault.firefrostgaming.com) - Docker 8001 → Nginx 443
- MySQL - Port 3306 (localhost)
- Nginx - Reverse proxy for all services
- Cockpit - Port 9090

**IP Addresses:**
- Primary: 63.143.34.217
- Secondary: 74.63.218.202 (Code-Server only)

**Docker Containers:** 1 (Vaultwarden)

---

### Ghost VPS (64.50.188.14)
**Role:** Documentation Cluster + Public-Facing Content  
**Location:** Chicago, IL  
**Provider:** Breezehost  
**Uptime:** 13 days, 20:24  
**Disk Usage:** 55% (21GB/38GB)  
**Login:** `architect` (not root)  

**Services Running:**
- Ghost CMS (firefrostgaming.com) - Port 2368 → Nginx 443
  - **Status:** ✅ Live subscription page with Fire/Frost tier branding
  - **Features:** 11 subscription tiers, dual-path branding, production-ready
- Wiki.js Subscribers (subscribers.firefrostgaming.com) - Port 3100 → Nginx 80
- Wiki.js Staff (staff.firefrostgaming.com) - Port 3101 → Nginx 80
- Wiki.js Pokerole (pokerole.firefrostgaming.com) - Port 3102 → Nginx 80
- Nextcloud (downloads.firefrostgaming.com) - Nginx 443 (PHP-FPM)
- MySQL - Port 3306 (localhost)
- PostgreSQL - Port 5432 (localhost)
- Redis - Port 6379 (localhost)
- Postfix - Port 25 (localhost only, SMTP blocked at network level)
- Nginx - Reverse proxy
- Cockpit - Port 9090

**Docker Containers:** 0 (all native services)

**Note:** Port 25 issue previously resolved with Breezehost.

---

### Billing VPS (38.68.14.188)
**Role:** Financial Services Isolation  
**Location:** Chicago, IL  
**Provider:** Breezehost  
**Uptime:** 11 days, 12:22  
**Disk Usage:** ⚠️ 70% (13GB/19GB) - MONITOR  
**RAM:** Standard VPS  

**Services Running:**
- Paymenter (billing.firefrostgaming.com) - PHP-FPM → Nginx 80
  - **Status:** ✅ Fully configured with 11 subscription tiers
  - **Tiers:** The Awakened ($1), Fire/Frost Elemental ($5), Knight ($10), Master ($15), Legend ($20), Sovereign ($499)
- Mailcow Stack (mail.firefrostgaming.com) - Docker 8080/8443 → Nginx 443
- Whitelist Manager (whitelist.firefrostgaming.com) - Port 5001 → Nginx 80
- MariaDB - Port 3306 (localhost)
- Redis - Port 6379 (localhost)
- Nginx - Reverse proxy
- Supervisor - Process control
- Cockpit - Port 9090

**Docker Containers:** 18 (Mailcow stack)
1. mailcowdockerized-nginx-mailcow-1 - 8080/8443
2. mailcowdockerized-postfix-mailcow-1 - 25, 465, 587
3. mailcowdockerized-dovecot-mailcow-1 - 110, 143, 993, 995, 4190
4. mailcowdockerized-mysql-mailcow-1 - 13306 (localhost)
5. mailcowdockerized-redis-mailcow-1 - 7654 (localhost)
6. mailcowdockerized-rspamd-mailcow-1
7. mailcowdockerized-php-fpm-mailcow-1
8. mailcowdockerized-sogo-mailcow-1
9. mailcowdockerized-clamd-mailcow-1
10. mailcowdockerized-unbound-mailcow-1
11. mailcowdockerized-watchdog-mailcow-1
12. mailcowdockerized-acme-mailcow-1
13. mailcowdockerized-ofelia-mailcow-1
14. mailcowdockerized-postfix-tlspol-mailcow-1
15. mailcowdockerized-memcached-mailcow-1
16. mailcowdockerized-netfilter-mailcow-1
17. mailcowdockerized-dockerapi-mailcow-1
18. mailcowdockerized-olefy-mailcow-1

**Mail Ports (all via Docker):**
- SMTP: 25, 465, 587
- IMAP: 143, 993
- POP3: 110, 995
- ManageSieve: 4190

---

### Panel VPS (45.94.168.138)
**Role:** Pterodactyl Control Plane  
**Location:** Charlotte, NC  
**Provider:** Breezehost  
**Uptime:** 13 days, 19:22  
**Disk Usage:** 39% (9GB/24GB)  
**RAM:** Standard VPS  

**Services Running:**
- Pterodactyl Panel (panel.firefrostgaming.com) - PHP-FPM → Nginx 443
- MariaDB - Port 3306 (localhost)
- Redis - Port 6379 (localhost)
- vsftpd - Port 21
- pteroq (Queue Worker) - Systemd service
- Nginx - Reverse proxy
- Cockpit - Port 9090

**Docker Containers:** 0 (all native services)

**Blueprint Extensions Installed:**
- Modpack Installer for Blueprint
- Subdomain Manager for Pterodactyl
- PteroStats - Advanced Statistics

---

### TX1 Dallas (38.68.14.26)
**Role:** Primary Game Server + Advanced Services  
**Location:** Dallas, TX  
**Provider:** Breezehost (Dedicated Server)  
**Specs:** 251GB RAM, 911GB Disk  
**Uptime:** 11 days, 11:00  
**Disk Usage:** ✅ 12% (102GB/911GB) - EXCELLENT  

**IP Subnet:** 38.68.14.24/29
- Primary Node IP: 38.68.14.26
- Additional IPs: .27, .28, .29, .30

**Services Running:**
- Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP)
- Firefrost Codex (codex.firefrostgaming.com):
  - Dify API - Port 5001 (localhost)
  - Dify Web - Port 3000 (localhost)
  - Qdrant Vector DB - Port 6333 (public)
  - n8n (n8n.firefrostgaming.com) - Port 5678 (localhost) → Nginx 443
  - Ollama - AI model server
- Nginx - 2 reverse proxy configurations
- Fail2ban - Security
- Cockpit - Port 9090

**Docker Containers:** 15 total (20 Plane containers removed March 27, 2026)
- **7 Game Servers** (Pterodactyl Wings managed)
- **8 Firefrost Codex Containers** (Dify + Qdrant + n8n + Ollama)

**Game Servers on TX1 (11 servers):** *Updated April 8, 2026*
1. **Stoneblock 4** - `a0efbfe8` - 38.68.14.26:25565
2. **Society: Sunlit Valley** - `9310d0a6` - 38.68.14.28:25565
3. **All The Mons (Private) - TX** - `668a5220` - 38.68.14.30:25565
4. **FoundryVTT** - `7d8f15a0` - 38.68.14.26:30000
5. **Create Plus (Video Sandbox)** - `cc170f06` - 38.68.14.26:25566
6. **Vanilla** - `c4004e2b` - 38.68.14.26:25567
7. **Beyond Depth** - `e95ed4a8` - *(port TBD)*
8. **Beyond Ascension** - `3f842757` - *(port TBD)*
9. **Wold's Vaults** - `fcbe0a1d` - *(port TBD)*
10. **Submerged 2** - `576342b8` - *(port TBD)*
11. **Cottage Witch** - `7a9754ad` - *(port TBD)*

*Note: Ars Eclectica removed since original audit*

---

### NC1 Charlotte (216.239.104.130)
**Role:** Secondary Game Server Node  
**Location:** Charlotte, NC  
**Provider:** Breezehost (Dedicated Server)  
**Specs:** 251GB RAM, 914GB Disk  
**Uptime:** 46 days, 12:38  
**Disk Usage:** ✅ 8% (69GB/914GB) - EXCELLENT

> **April 8, 2026:** LVM partition expanded from 100GB to 928GB. NC1 had 828GB unallocated in volume group since initial Ubuntu install. Now fully utilizing the 1TB NVMe drive.  

**IP Subnet:** 216.239.104.128/29
- Primary Node IP: 216.239.104.130
- Gateway: 216.239.104.129

**Services Running:**
- Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP)
- MariaDB - Port 3306 (localhost)
- Cockpit - Port 9090

**Docker Containers:** 6 (all game servers)

**Game Servers on NC1 (11 servers):** *Updated April 8, 2026*
1. **All The Mods 10** - `82e63949` - 216.239.104.130:25569
2. **Hytale** - `13c80cb8` - 216.239.104.130:5520-5521
3. **All of Create (Creative) - NC** - `e1c6ff8d` - 216.239.104.130:25568
4. **All the Mods 10: To the Sky** - `f408e832` - 216.239.104.130:25565
5. **All the Mons** - `c4bc5892` - 216.239.104.130:25566
6. **Mythcraft 5** - `b90ced3c` - 216.239.104.130:25567
7. **Otherworld [Dungeons & Dragons]** - `d4798f45` - *(port TBD)*
8. **DeceasedCraft** - `8950fa1e` - *(port TBD)*
9. **Sneak's Pirate Pack** - `7c9c2dc0` - *(port TBD)*
10. **Farm Crossing 5** - `04ac4a1b` - *(port TBD)*
11. **Homestead - A Cozy Survival Experience** - `f5befeab` - *(port TBD)*

**Special Firewall Rules:**
- Allows GRE traffic from Command Center (63.143.34.217) - for potential future tunneling
- Port 24454/udp open (Simple Voice Chat - Mayview)

---

## 🔌 PORT ALLOCATION REGISTRY

### Command Center (63.143.34.217)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 22 | SSH | Public | TCP |
| 80 | Nginx (63.143.34.217) | Public | TCP |
| 80 | Nginx (74.63.218.202) | Public | TCP |
| 443 | Nginx (63.143.34.217) | Public | TCP |
| 443 | Nginx (74.63.218.202) | Public | TCP |
| 3000 | Gitea | Internal | TCP |
| 3001 | Uptime Kuma | Internal | TCP |
| 3306 | MySQL | Localhost | TCP |
| 3500 | Discord Bot (The Arbiter) | Internal | TCP |
| 6379 | Redis | Localhost | TCP |
| 8000 | Vaultwarden | Docker localhost | TCP |
| 8001 | Vaultwarden proxy | Docker localhost | TCP |
| 8080 | Code-Server | Internal | TCP |
| 9090 | Cockpit | Public | TCP |

**Nginx Virtual Hosts (63.143.34.217:443):**
- git.firefrostgaming.com → 127.0.0.1:3000
- status.firefrostgaming.com → 127.0.0.1:3001
- discord-bot.firefrostgaming.com → localhost:3500
- vault.firefrostgaming.com → 127.0.0.1:8001

**Nginx Virtual Hosts (74.63.218.202:443):**
- code.firefrostgaming.com → 127.0.0.1:8080

---

### Ghost VPS (64.50.188.14)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 22 | SSH | Public | TCP |
| 25 | Postfix | Localhost | TCP |
| 80 | Nginx | Public | TCP |
| 443 | Nginx | Public | TCP |
| 2368 | Ghost CMS | Localhost | TCP |
| 3100 | Wiki.js Subscribers | Localhost | TCP |
| 3101 | Wiki.js Staff | Localhost | TCP |
| 3102 | Wiki.js Pokerole | Localhost | TCP |
| 3306 | MySQL | Localhost | TCP |
| 5432 | PostgreSQL | Localhost | TCP |
| 6379 | Redis | Localhost | TCP |
| 9090 | Cockpit | Public | TCP |

**Nginx Virtual Hosts:**
- firefrostgaming.com → 127.0.0.1:2368 (Ghost)
- subscribers.firefrostgaming.com → localhost:3100
- staff.firefrostgaming.com → localhost:3101
- pokerole.firefrostgaming.com → localhost:3102
- downloads.firefrostgaming.com → PHP-FPM (Nextcloud)
- docs.firefrostgaming.com → (MkDocs - not running currently)

---

### Billing VPS (38.68.14.188)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 21 | vsftpd | Public | TCP |
| 22 | SSH | Public | TCP |
| 25 | Postfix (Docker) | Public | TCP |
| 80 | Nginx | Public | TCP |
| 110 | POP3 (Docker) | Public | TCP |
| 143 | IMAP (Docker) | Public | TCP |
| 443 | Nginx | Public | TCP |
| 465 | SMTPS (Docker) | Public | TCP |
| 587 | Submission (Docker) | Public | TCP |
| 993 | IMAPS (Docker) | Public | TCP |
| 995 | POP3S (Docker) | Public | TCP |
| 3306 | MariaDB | Localhost | TCP |
| 4190 | ManageSieve (Docker) | Public | TCP |
| 5001 | Whitelist Manager | Localhost | TCP |
| 6379 | Redis | Localhost | TCP |
| 7654 | Redis (Docker) | Docker localhost | TCP |
| 8080 | Mailcow Web | Public | TCP |
| 8443 | Mailcow Web SSL | Public | TCP |
| 9090 | Cockpit | Public | TCP |
| 13306 | MySQL (Docker) | Docker localhost | TCP |
| 19991 | Dovecot Stats | Docker localhost | TCP |

**Nginx Virtual Hosts:**
- billing.firefrostgaming.com → PHP-FPM (Paymenter)
- mail.firefrostgaming.com → localhost:8443 (Mailcow)
- whitelist.firefrostgaming.com → 127.0.0.1:5001

---

### Panel VPS (45.94.168.138)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 21 | vsftpd | Public | TCP |
| 22 | SSH | Public | TCP |
| 80 | Nginx | Public | TCP |
| 443 | Nginx | Public | TCP |
| 3306 | MariaDB | Localhost | TCP |
| 6379 | Redis | Localhost | TCP |
| 9090 | Cockpit | Public | TCP |

**Nginx Virtual Hosts:**
- panel.firefrostgaming.com → PHP-FPM (Pterodactyl Panel)

---

### TX1 Dallas (38.68.14.26)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 22 | SSH | Public | TCP |
| 80 | Nginx | Public | TCP |
| 443 | Nginx | Public | TCP |
| 2022 | Wings SFTP | Public | TCP |
| 3000 | Dify Web | Docker localhost | TCP |
| 5001 | Dify API | Docker localhost | TCP |
| 5520 | Game: Ars Eclectica | Public | TCP/UDP |
| 5678 | n8n | Docker localhost | TCP |
| 6333 | Qdrant Vector DB | Public | TCP |
| 8080 | Wings HTTP | Public | TCP |
| 8090 | Plane (Caddy) | Public | TCP |
| 8444 | Plane SSL | Public | TCP |
| 9090 | Cockpit | Public | TCP |
| 10025 | Plane SMTP | Public | TCP |
| 10465 | Plane SMTPS | Public | TCP |
| 10587 | Plane Submission | Public | TCP |
| 25565 | Game: Stoneblock 4 | Public (38.68.14.26) | TCP/UDP |
| 25566 | Game: Create Plus | Public (38.68.14.26) | TCP/UDP |
| 25567 | Game: Vanilla | Public (38.68.14.26) | TCP/UDP |
| 25565 | Game: Society Sunlit Valley | Public (38.68.14.28) | TCP/UDP |
| 25565 | Game: All The Mons Private | Public (38.68.14.30) | TCP/UDP |
| 30000 | FoundryVTT | Public (38.68.14.26) | TCP/UDP |

**Nginx Virtual Hosts:**
- codex.firefrostgaming.com → 127.0.0.1:3000 (Dify Web) + 127.0.0.1:5001 (API paths)
- n8n.firefrostgaming.com → 127.0.0.1:5678
- tasks.firefrostgaming.com → 127.0.0.1:8090 (Plane)

**Docker Internal Services:**
- PostgreSQL (Plane): 5432
- PostgreSQL (Dify): 5432
- Redis (Plane): 6379
- Redis (Dify): 6379
- RabbitMQ (Plane): 5672, 15672
- MinIO (Plane): 9000

---

### NC1 Charlotte (216.239.104.130)

| Port | Service | Access | Protocol |
|------|---------|--------|----------|
| 22 | SSH | Public | TCP |
| 2022 | Wings SFTP | Public | TCP |
| 3306 | MariaDB | Localhost | TCP |
| 5520-5521 | Game: Hytale | Public | TCP/UDP |
| 8080 | Wings HTTP | Public | TCP |
| 9090 | Cockpit | Public | TCP |
| 24454 | Simple Voice Chat | Public | UDP |
| 25565 | Game: ATM10 To the Sky | Public | TCP/UDP |
| 25566 | Game: All the Mons Public | Public | TCP/UDP |
| 25567 | Game: Mythcraft 5 | Public | TCP/UDP |
| 25568 | Game: All of Create | Public | TCP/UDP |
| 25569 | Game: All The Mods 10 | Public | TCP/UDP |

---

## 🔗 CONNECTIVITY MAP

### External Public-Facing Services

**Domain → Server → Internal Port → External Port**

1. **firefrostgaming.com** → Ghost VPS → 2368 → 443 (Nginx SSL)
2. **git.firefrostgaming.com** → Command Center → 3000 → 443 (Nginx SSL)
3. **status.firefrostgaming.com** → Command Center → 3001 → 443 (Nginx SSL)
4. **code.firefrostgaming.com** → Command Center → 8080 → 443 (Nginx SSL, 74.63.218.202)
5. **discord-bot.firefrostgaming.com** → Command Center → 3500 → 443 (Nginx SSL)
6. **vault.firefrostgaming.com** → Command Center → 8001 → 443 (Nginx SSL)
7. **billing.firefrostgaming.com** → Billing VPS → PHP-FPM → 80 (Nginx)
8. **mail.firefrostgaming.com** → Billing VPS → 8443 → 443 (Nginx SSL)
9. **whitelist.firefrostgaming.com** → Billing VPS → 5001 → 80 (Nginx)
10. **panel.firefrostgaming.com** → Panel VPS → PHP-FPM → 443 (Nginx SSL)
11. **codex.firefrostgaming.com** → TX1 → 3000/5001 → 443 (Nginx SSL)
12. **n8n.firefrostgaming.com** → TX1 → 5678 → 443 (Nginx SSL)
13. **tasks.firefrostgaming.com** → TX1 → 8090 → 80 (Nginx)
14. **downloads.firefrostgaming.com** → Ghost VPS → PHP-FPM → 443 (Nginx SSL, Nextcloud)
15. **subscribers.firefrostgaming.com** → Ghost VPS → 3100 → 80 (Nginx)
16. **staff.firefrostgaming.com** → Ghost VPS → 3101 → 80 (Nginx)
17. **pokerole.firefrostgaming.com** → Ghost VPS → 3102 → 80 (Nginx)

### Server-to-Server Communication

**Panel VPS (45.94.168.138) ↔ Wings Nodes:**
- Panel → TX1 (38.68.14.26:8080) - Wings API
- Panel → NC1 (216.239.104.130:8080) - Wings API
- **Protocol:** HTTPS (Wings API)
- **Authentication:** API tokens
- **Purpose:** Server management, monitoring, console access

**Discord Bot (Command Center) → Discord API:**
- discord-bot.firefrostgaming.com (63.143.34.217:3500) → Discord.com:443
- **Protocol:** HTTPS + WebSocket
- **Purpose:** Bot commands, role management, webhooks

**Paymenter (Billing VPS) → Pterodactyl Panel:**
- Planned webhook: billing.firefrostgaming.com → panel.firefrostgaming.com
- **Protocol:** HTTPS
- **Purpose:** Subscription provisioning automation

**Paymenter (Billing VPS) → Discord Bot:**
- Planned webhook: billing.firefrostgaming.com → discord-bot.firefrostgaming.com/webhook/paymenter
- **Protocol:** HTTPS
- **Purpose:** Subscription event notifications for role assignment

**Whitelist Manager (Billing VPS) → Pterodactyl Panel:**
- whitelist.firefrostgaming.com (38.68.14.188:5001) → panel.firefrostgaming.com (45.94.168.138)
- **Protocol:** HTTPS (Panel API)
- **Purpose:** Whitelist synchronization

**n8n (TX1) → External Services:**
- n8n.firefrostgaming.com → Various APIs (GitHub, Discord, etc.)
- **Protocol:** HTTPS
- **Purpose:** Workflow automation

**Gitea (Command Center) → Git Clients:**
- git.firefrostgaming.com → Various (Claude, developers, CI/CD)
- **Protocol:** HTTPS + SSH (port 22)
- **Purpose:** Git repository access

### Database Connections (Internal Only)

**Command Center:**
- Gitea → MySQL (127.0.0.1:3306)
- Vaultwarden → Internal SQLite

**Ghost VPS:**
- Ghost CMS → MySQL (127.0.0.1:3306)
- Wiki.js (3x) → PostgreSQL (127.0.0.1:5432)
- All services → Redis (127.0.0.1:6379) for caching

**Billing VPS:**
- Paymenter → MariaDB (127.0.0.1:3306)
- Paymenter → Redis (127.0.0.1:6379)
- Mailcow → Docker MySQL (172.22.1.x:3306)
- Mailcow → Docker Redis (172.22.1.x:6379)

**Panel VPS:**
- Pterodactyl Panel → MariaDB (127.0.0.1:3306)
- Pterodactyl Panel → Redis (127.0.0.1:6379)

**TX1 Dallas:**
- Plane → Docker PostgreSQL (internal)
- Plane → Docker Redis (internal)
- Dify → Docker PostgreSQL (internal)
- Dify → Docker Redis (internal)
- Dify → Qdrant (127.0.0.1:6333)

**NC1 Charlotte:**
- Wings → MariaDB (127.0.0.1:3306)

---

## 🎯 AUTHENTICATION & DEPENDENCY FLOWS

### OAuth2 Flows

**Discord Bot Admin Panel:**
- User → discord-bot.firefrostgaming.com → Discord OAuth2 → Whitelist check → Session
- **Dependencies:** Discord API availability, Session storage (Express sessions)

### API Token Flows

**Pterodactyl Panel ↔ Wings:**
- Panel stores Wings API tokens
- Wings validates tokens on each request
- **Critical:** Token compromise = full server control

**Gitea API:**
- Claude sessions use: `e0e330cba1749b01ab505093a160e4423ebbbe36`
- Operations manual automation
- **Critical:** Full admin access token

**n8n Workflows:**
- Various API tokens stored in n8n credentials
- Discord webhooks, GitHub, etc.

### SMTP Flows (Email)

**Ghost VPS (Postfix):**
- **Status:** ⚠️ BLOCKED - Inbound port 25 blocked at provider level
- **Workaround Needed:** Provider support ticket
- **Current:** Internal mail only

**Billing VPS (Mailcow):**
- **Status:** ✅ OPERATIONAL
- SMTP out: 587 (submission), 465 (SMTPS), 25 (relay)
- IMAP: 143, 993 (SSL)
- POP3: 110, 995 (SSL)
- **DKIM/SPF/DMARC:** Configured for firefrostgaming.com

**TX1 (Plane):**
- **Status:** ✅ OPERATIONAL
- Internal SMTP for Plane notifications (ports 10025, 10465, 10587)

---

## ⚠️ SINGLE POINTS OF FAILURE

### Critical Single Points

1. **Pterodactyl Panel (45.94.168.138)**
   - **Risk:** Panel down = no game server management
   - **Mitigation:** Wings nodes continue running autonomously
   - **Recovery Time:** ~30 minutes (restore from backup + DNS)

2. **Mailcow (Billing VPS)**
   - **Risk:** Email down = no subscription confirmations, no support tickets
   - **Mitigation:** Cloudflare Email Routing as backup?
   - **Recovery Time:** ~2 hours (Mailcow stack restoration)

3. **Gitea (Command Center)**
   - **Risk:** Git down = no deployments, no operations manual access
   - **Mitigation:** Local clones exist on developer machines
   - **Recovery Time:** ~1 hour (service restart or VM restore)

4. **Ghost CMS (Ghost VPS)**
   - **Risk:** Main website down = no public presence
   - **Mitigation:** Cloudflare caching provides limited read access
   - **Recovery Time:** ~1 hour (Ghost restart or data restore)

5. **Command Center Server (63.143.34.217)**
   - **Risk:** Multiple critical services (Gitea, Uptime Kuma, Discord Bot, Vaultwarden)
   - **Impact:** Most critical - affects development, monitoring, and Discord automation
   - **Mitigation:** Distributed services across multiple VPS in future
   - **Recovery Time:** 2-4 hours (depends on failure type)

### Non-Critical Single Points

6. **Billing VPS (38.68.14.188)**
   - **Services:** Paymenter, Mailcow, Whitelist Manager
   - **Impact:** Financial operations halted, but game servers continue
   - **Note:** High disk usage (70%) increases risk

7. **Ghost VPS (64.50.188.14)**
   - **Services:** Ghost, Wiki.js (3x), Nextcloud
   - **Impact:** Documentation inaccessible, but operations continue
   - **Note:** Can be restored from backups

---

## 🔥 PORT CONFLICT PREVENTION

### Port Allocation Strategy

**Reserved Ranges:**
- **25565-25580:** Minecraft game servers (TCP/UDP)
- **5520-5521:** Hytale (TCP/UDP)
- **30000-30010:** Reserved for FoundryVTT and future VTT instances
- **3000-3200:** Internal web services (Gitea, Uptime Kuma, Wiki.js, etc.)
- **8000-9000:** Docker services and Wings
- **10000-11000:** Plane/n8n/Dify internal services

### Conflict Lessons Learned

**The Arbiter Bot Port Hunt (March 27, 2026):**
1. Attempted port 3000 → **CONFLICT** (Gitea on TX1 Dify)
2. Attempted port 3001 → **CONFLICT** (Uptime Kuma)
3. **SUCCESS:** Port 3500 (unused)

**Prevention Going Forward:**
- Always check `ss -tlnp | grep LISTEN` before deploying
- Document port assignments in this registry
- Use high-numbered ports (3500+) for new services on shared servers
- Consider port range 4000-5000 for future Discord/webhook services

### Available Port Ranges

**Command Center (63.143.34.217):**
- ✅ 3500-4000: Available
- ✅ 4000-6000: Available (except 6379 Redis)
- ✅ 7000-8000: Available (except 8000-8001 Vaultwarden)

**Ghost VPS (64.50.188.14):**
- ✅ 3200-6000: Available (except 3306 MySQL, 5432 PostgreSQL)
- ✅ 7000-9000: Available

**Billing VPS (38.68.14.188):**
- ⚠️ Most standard ports occupied by Mailcow
- ✅ 5100-6000: Available (except 5001 Whitelist Manager)
- ✅ 9100-10000: Available

**Panel VPS (45.94.168.138):**
- ✅ 1024-3000: Available
- ✅ 3500-6000: Available (except 3306 MySQL, 6379 Redis)
- ✅ 7000-9000: Available

**TX1 Dallas (38.68.14.26):**
- ⚠️ Heavy Docker usage, internal ports dynamic
- ✅ 3500-5000: Available (except 5001 Dify, 5678 n8n)
- ✅ 7000-8000: Available
- ✅ 11000-20000: Available

**NC1 Charlotte (216.239.104.130):**
- ✅ 3000-5000: Available (except 3306 MySQL)
- ✅ 6000-8000: Available
- ✅ 10000-20000: Available

---

## 📊 RESOURCE UTILIZATION

### Disk Usage Status

| Server | Used | Total | Usage % | Status |
|--------|------|-------|---------|--------|
| Command Center | 17GB | 38GB | 45% | ✅ Good |
| Ghost VPS | 21GB | 38GB | 55% | ✅ Good |
| **Billing VPS** | **13GB** | **19GB** | **70%** | ⚠️ **Monitor** |
| Panel VPS | 9GB | 24GB | 39% | ✅ Good |
| **TX1 Dallas** | **102GB** | **911GB** | **12%** | ✅ **Excellent** |
| **NC1 Charlotte** | **61GB** | **98GB** | **66%** | ⚠️ **Monitor** |

**Recommendations:**
1. **Billing VPS:** Review Mailcow logs and docker volume sizes - consider cleanup or expansion
2. **NC1 Charlotte:** Monitor game server world sizes - implement world pruning or expansion
3. **TX1 Dallas:** Massive capacity available - can host additional services

### Service Load Distribution

**Command Center:** 33 systemd services (6 critical)  
**Ghost VPS:** 31 systemd services (5 critical)  
**Billing VPS:** 30 systemd services + 18 Docker containers  
**Panel VPS:** 28 systemd services (clean, focused)  
**TX1 Dallas:** 29 systemd services + 35 Docker containers (heavy)  
**NC1 Charlotte:** 25 systemd services + 6 Docker containers (focused)  

---

## 🔐 FIREWALL ANALYSIS

### Command Center UFW Rules
- ✅ SSH (22) open
- ✅ HTTP/HTTPS (80/443) on both IPs
- ✅ Cockpit (9090) open
- ✅ Specific IP bindings for services (63.143.34.217 vs 74.63.218.202)

### Ghost VPS
- ⚠️ Firewall audit returned "ERROR: You need to be root" (was logged in as architect)
- **Action Required:** Re-audit as root to verify rules

### Billing VPS IPTables
- ✅ Custom Mailcow chain (MAILCOW)
- ✅ UFW chains present
- ✅ Docker chains for container networking

### Panel VPS UFW Rules
- ✅ SSH (22), HTTP (80), HTTPS (443) open
- ✅ FTP (21) open for vsftpd
- ✅ Cockpit (9090) open
- ✅ Specific allow from 141.98.74.95 (related system?)

### TX1 Dallas UFW Rules
- ✅ Wings ports (8080, 2022) open
- ✅ Minecraft port range (25565-25580) TCP+UDP
- ✅ Hytale ports (5520-5521) TCP+UDP
- ✅ n8n webhook port (5678)
- ✅ Cockpit (9090) open
- ✅ Allow 74.63.218.205 HTTP/HTTPS (Code-Server IP?)

### NC1 Charlotte UFW Rules
- ✅ Wings ports (8080, 2022) open
- ✅ Minecraft port range (25565-25580) TCP+UDP
- ✅ Hytale ports (5520-5521) TCP+UDP
- ✅ Simple Voice Chat (24454 UDP)
- ✅ GRE protocol (47) open - for future tunneling
- ✅ **Special:** Full allow from Command Center IP (63.143.34.217) + GRE
- ✅ Cockpit (9090) open

---

## 🎮 GAME SERVER MAPPING

### TX1 Dallas Game Servers (11 servers) — *Updated April 8, 2026*

| Server Name | UUID (short) | IP:Port | Status |
|-------------|--------------|---------|--------|
| Stoneblock 4 | a0efbfe8 | 38.68.14.26:25565 | Active |
| Society: Sunlit Valley | 9310d0a6 | 38.68.14.28:25565 | Active |
| All The Mons (Private) - TX | 668a5220 | 38.68.14.30:25565 | Active |
| FoundryVTT | 7d8f15a0 | 38.68.14.26:30000 | Active |
| Create Plus (Video Sandbox) | cc170f06 | 38.68.14.26:25566 | Active |
| Vanilla | c4004e2b | 38.68.14.26:25567 | Active |
| Beyond Depth | e95ed4a8 | TBD | Active |
| Beyond Ascension | 3f842757 | TBD | Active |
| Wold's Vaults | fcbe0a1d | TBD | Active |
| Submerged 2 | 576342b8 | TBD | Active |
| Cottage Witch | 7a9754ad | TBD | Active |

*Note: Ars Eclectica removed since original audit*

### NC1 Charlotte Game Servers (11 servers) — *Updated April 8, 2026*

| Server Name | UUID (short) | IP:Port | Status |
|-------------|--------------|---------|--------|
| All The Mods 10 | 82e63949 | 216.239.104.130:25569 | Active |
| Hytale | 13c80cb8 | 216.239.104.130:5520-5521 | Active |
| All of Create (Creative) - NC | e1c6ff8d | 216.239.104.130:25568 | Active |
| All the Mods 10: To the Sky | f408e832 | 216.239.104.130:25565 | Active |
| All the Mons | c4bc5892 | 216.239.104.130:25566 | Active |
| Mythcraft 5 | b90ced3c | 216.239.104.130:25567 | Active |
| Otherworld [Dungeons & Dragons] | d4798f45 | TBD | Active |
| DeceasedCraft | 8950fa1e | TBD | Active |
| Sneak's Pirate Pack | 7c9c2dc0 | TBD | Active |
| Farm Crossing 5 | 04ac4a1b | TBD | Active |
| Homestead - A Cozy Survival Experience | f5befeab | TBD | Active |

**Total:** 22 game servers (20 Minecraft + 1 Hytale + 1 FoundryVTT)

---

## 🚨 ISSUES IDENTIFIED

### Warning Issues
1. **Billing VPS Disk Usage: 70%**
   - **Risk:** May hit capacity during high email volume
   - **Action:** Review Mailcow container logs and volumes
   - **Timeline:** Monitor weekly, expand if hits 80%

2. **~~NC1 Charlotte Disk Usage: 66%~~** ✅ RESOLVED April 8, 2026
   - **Resolution:** LVM partition expanded from 100GB to 928GB
   - **New Status:** 8% usage (69GB/914GB) - EXCELLENT

3. **Ghost VPS Firewall Not Audited**
   - **Risk:** Unknown firewall state (audit failed due to permissions)
   - **Action:** Re-run audit as root
   - **Timeline:** Next maintenance window

---

## 📈 CAPACITY PLANNING

### Short-Term Capacity (Next 3 Months)

**Can Accommodate:**
- ✅ 5-10 more game servers on TX1 (plenty of disk + RAM)
- ✅ 10+ more game servers on NC1 (807GB free after April 8 expansion)
- ✅ Additional web services on Command Center
- ✅ Additional web services on Ghost VPS
- ⚠️ Limited capacity on Billing VPS (disk constraint)

**Cannot Accommodate Without Expansion:**
- ❌ Additional Docker stacks on Billing VPS (disk full)

### Long-Term Recommendations

1. **Expand Billing VPS Disk**
   - Current: 19GB
   - Recommended: 40-50GB
   - Reason: Mailcow + Paymenter + future growth

2. **~~Expand NC1 Disk~~** ✅ COMPLETED April 8, 2026
   - Expanded: 100GB → 928GB (LVM resize)
   - Now: 914GB usable, 807GB free

3. **Consider Backup Server**
   - Add dedicated backup VPS
   - Offload backups from game server disks
   - Enable disaster recovery

4. **Load Balancer for Web Services**
   - Multiple Ghost CMS instances
   - Distribute SSL termination
   - Improve resilience

---

## 🔄 INTERCONNECTION SUMMARY

### Data Flow Patterns

**User → Website (Ghost CMS)**
1. User → Cloudflare → Ghost VPS:443
2. Nginx → Ghost:2368
3. Ghost → MySQL:3306

**User → Panel (Pterodactyl)**
1. User → Cloudflare → Panel VPS:443
2. Nginx → PHP-FPM → Panel Application
3. Panel → MariaDB:3306
4. Panel → Wings API (TX1:8080, NC1:8080)

**User → Game Server**
1. User → TX1/NC1 direct (no proxy)
2. Game Server → Wings → Panel (monitoring/console)

**Discord Bot Workflow**
1. Discord API → discord-bot.firefrostgaming.com:443
2. Nginx → Bot:3500
3. Bot → Discord API (outbound)
4. Bot → (future) Paymenter webhook

**Subscription Workflow (Planned)**
1. User → Paymenter (billing.firefrostgaming.com)
2. Paymenter → Stripe/PayPal API
3. Paymenter webhook → Discord Bot
4. Discord Bot → Discord API (assign role)
5. Discord Bot → (future) Panel API (provision server)

---

## 📝 RECOMMENDATIONS

### Immediate Actions (Next 7 Days)
1. ✅ Complete this audit document
2. ✅ ~~Submit Breezehost ticket for Ghost VPS port 25~~ (Already resolved)
3. ✅ **Decommission Plane stack on TX1 Dallas** - COMPLETE (March 27, 2026)
4. ⚠️ Re-audit Ghost VPS firewall as root
5. ✅ Document port allocation strategy in operations manual

### Short-Term Actions (Next 30 Days)
7. ⚠️ Review Billing VPS disk usage, plan expansion if needed
8. ✅ ~~Monitor NC1 disk usage weekly~~ RESOLVED - Expanded April 8, 2026
9. ✅ Implement automated disk usage alerting (Uptime Kuma?)
10. ✅ Configure Paymenter → Discord Bot webhooks
11. ✅ Test full subscription provisioning flow

### Long-Term Actions (Next 90 Days)
12. 🔄 Implement backup server or backup strategy
13. 🔄 Consider load balancer for web services
14. 🔄 Evaluate Gitea high-availability options
15. ✅ ~~Plan for TX1/NC1 disk expansion schedule~~ NC1 DONE April 8, 2026

---

## 🎯 AUDIT COMPLETION

**Audit Status:** ✅ COMPLETE  
**Data Collection:** March 27, 2026  
**Servers Audited:** 6/6 (100%)  
**Document Version:** 1.0  
**Next Audit:** Recommended every 6 months or after major infrastructure changes

**Compiled By:** Chronicler #43  
**Reviewed By:** (Pending Michael's review)  
**Committed To:** firefrost-operations-manual repository

---

**Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️