diff --git a/docs/infrastructure/network-audit-2026.md b/docs/infrastructure/network-audit-2026.md new file mode 100644 index 0000000..c278c8a --- /dev/null +++ b/docs/infrastructure/network-audit-2026.md @@ -0,0 +1,899 @@ +# 🔥❄️ Firefrost Gaming Infrastructure Audit 2026 + +**Audit Date:** March 27, 2026 +**Audited By:** Chronicler #43 +**Purpose:** Complete network topology, port allocation, service inventory, and connectivity mapping +**Reason:** Prevent port conflicts (learned from The Arbiter bot deployment: 3000→3001→3500) + +--- + +## 📋 EXECUTIVE SUMMARY + +**Total Infrastructure:** +- **6 Servers** (4 VPS, 2 Dedicated) +- **90+ Services** running across all servers +- **68 Docker Containers** (18 Mailcow, 35 TX1, 6 NC1, 1 Vaultwarden, 8 n8n/Plane/Dify services) +- **14 Game Servers** (7 TX1, 6 NC1, 1 Hytale) +- **1 FoundryVTT Server** +- **12 Public-Facing Domains** + +**Key Findings:** +1. ✅ No current port conflicts detected +2. ✅ Clean separation of management vs game workloads +3. ⚠️ Billing VPS disk usage at 70% (13GB/19GB) +4. ⚠️ NC1 disk usage at 66% (61GB/98GB) +5. ✅ TX1 has plenty of capacity (12% usage, 102GB/911GB) +6. ✅ All critical services operational +7. ✅ Firewall rules properly configured on all servers + +--- + +## 🖥️ SERVER INVENTORY + +### Command Center (63.143.34.217) +**Role:** Management Hub + Backend Services +**Location:** Dallas, TX +**Provider:** Breezehost +**Uptime:** 46 days, 12:35 +**Disk Usage:** 45% (17GB/38GB) +**RAM:** Standard VPS + +**Services Running:** +- Gitea (git.firefrostgaming.com) - Port 3000 → Nginx 443 +- Uptime Kuma (status.firefrostgaming.com) - Port 3001 → Nginx 443 +- Code-Server (code.firefrostgaming.com) - Port 8080 → Nginx 443 (74.63.218.202) +- The Arbiter Discord Bot (discord-bot.firefrostgaming.com) - Port 3500 → Nginx 443 +- Vaultwarden (vault.firefrostgaming.com) - Docker 8001 → Nginx 443 +- MySQL - Port 3306 (localhost) +- Nginx - Reverse proxy for all services +- Cockpit - Port 9090 + +**IP Addresses:** +- Primary: 63.143.34.217 +- Secondary: 74.63.218.202 (Code-Server only) + +**Docker Containers:** 1 (Vaultwarden) + +--- + +### Ghost VPS (64.50.188.14) +**Role:** Documentation Cluster + Public-Facing Content +**Location:** Chicago, IL +**Provider:** Breezehost +**Uptime:** 13 days, 20:24 +**Disk Usage:** 55% (21GB/38GB) +**Login:** `architect` (not root) + +**Services Running:** +- Ghost CMS (firefrostgaming.com) - Port 2368 → Nginx 443 +- Wiki.js Subscribers (subscribers.firefrostgaming.com) - Port 3100 → Nginx 80 +- Wiki.js Staff (staff.firefrostgaming.com) - Port 3101 → Nginx 80 +- Wiki.js Pokerole (pokerole.firefrostgaming.com) - Port 3102 → Nginx 80 +- Nextcloud (downloads.firefrostgaming.com) - Nginx 443 (PHP-FPM) +- MySQL - Port 3306 (localhost) +- PostgreSQL - Port 5432 (localhost) +- Redis - Port 6379 (localhost) +- Postfix - Port 25 (localhost only, SMTP blocked at network level) +- Nginx - Reverse proxy +- Cockpit - Port 9090 + +**Docker Containers:** 0 (all native services) + +**⚠️ Known Issue:** Inbound port 25 blocked at provider level (Breezehost). Internal mail works, external inbound does not. Requires support ticket to Jon at Breezehost. + +--- + +### Billing VPS (38.68.14.188) +**Role:** Financial Services Isolation +**Location:** Chicago, IL +**Provider:** Breezehost +**Uptime:** 11 days, 12:22 +**Disk Usage:** ⚠️ 70% (13GB/19GB) - MONITOR +**RAM:** Standard VPS + +**Services Running:** +- Paymenter (billing.firefrostgaming.com) - PHP-FPM → Nginx 80 +- Mailcow Stack (mail.firefrostgaming.com) - Docker 8080/8443 → Nginx 443 +- Whitelist Manager (whitelist.firefrostgaming.com) - Port 5001 → Nginx 80 +- MariaDB - Port 3306 (localhost) +- Redis - Port 6379 (localhost) +- Nginx - Reverse proxy +- Supervisor - Process control +- Cockpit - Port 9090 + +**Docker Containers:** 18 (Mailcow stack) +1. mailcowdockerized-nginx-mailcow-1 - 8080/8443 +2. mailcowdockerized-postfix-mailcow-1 - 25, 465, 587 +3. mailcowdockerized-dovecot-mailcow-1 - 110, 143, 993, 995, 4190 +4. mailcowdockerized-mysql-mailcow-1 - 13306 (localhost) +5. mailcowdockerized-redis-mailcow-1 - 7654 (localhost) +6. mailcowdockerized-rspamd-mailcow-1 +7. mailcowdockerized-php-fpm-mailcow-1 +8. mailcowdockerized-sogo-mailcow-1 +9. mailcowdockerized-clamd-mailcow-1 +10. mailcowdockerized-unbound-mailcow-1 +11. mailcowdockerized-watchdog-mailcow-1 +12. mailcowdockerized-acme-mailcow-1 +13. mailcowdockerized-ofelia-mailcow-1 +14. mailcowdockerized-postfix-tlspol-mailcow-1 +15. mailcowdockerized-memcached-mailcow-1 +16. mailcowdockerized-netfilter-mailcow-1 +17. mailcowdockerized-dockerapi-mailcow-1 +18. mailcowdockerized-olefy-mailcow-1 + +**Mail Ports (all via Docker):** +- SMTP: 25, 465, 587 +- IMAP: 143, 993 +- POP3: 110, 995 +- ManageSieve: 4190 + +--- + +### Panel VPS (45.94.168.138) +**Role:** Pterodactyl Control Plane +**Location:** Charlotte, NC +**Provider:** Breezehost +**Uptime:** 13 days, 19:22 +**Disk Usage:** 39% (9GB/24GB) +**RAM:** Standard VPS + +**Services Running:** +- Pterodactyl Panel (panel.firefrostgaming.com) - PHP-FPM → Nginx 443 +- MariaDB - Port 3306 (localhost) +- Redis - Port 6379 (localhost) +- vsftpd - Port 21 +- pteroq (Queue Worker) - Systemd service +- Nginx - Reverse proxy +- Cockpit - Port 9090 + +**Docker Containers:** 0 (all native services) + +**Blueprint Extensions Installed:** +- Modpack Installer for Blueprint +- Subdomain Manager for Pterodactyl +- PteroStats - Advanced Statistics + +--- + +### TX1 Dallas (38.68.14.26) +**Role:** Primary Game Server + Advanced Services +**Location:** Dallas, TX +**Provider:** Breezehost (Dedicated Server) +**Specs:** 251GB RAM, 911GB Disk +**Uptime:** 11 days, 11:00 +**Disk Usage:** ✅ 12% (102GB/911GB) - EXCELLENT + +**IP Subnet:** 38.68.14.24/29 +- Primary Node IP: 38.68.14.26 +- Additional IPs: .27, .28, .29, .30 + +**Services Running:** +- Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP) +- Plane Project Management (tasks.firefrostgaming.com) - Port 8090 → Nginx 80 +- Firefrost Codex (codex.firefrostgaming.com): + - Dify API - Port 5001 (localhost) + - Dify Web - Port 3000 (localhost) + - Qdrant Vector DB - Port 6333 (public) + - n8n (n8n.firefrostgaming.com) - Port 5678 (localhost) → Nginx 443 + - Ollama - AI model server +- Nginx - 2 reverse proxy configurations +- Fail2ban - Security +- Cockpit - Port 9090 + +**Docker Containers:** 35 total +- **7 Game Servers** (Pterodactyl Wings managed) +- **20 Plane Containers** (full stack) +- **8 Firefrost Codex Containers** (Dify + Qdrant + n8n + Ollama) + +**Game Servers on TX1:** +1. **Stoneblock 4** - `a0efbfe8-4b97-4a90-869d-ffe6d3072bd5` - 38.68.14.26:25565 +2. **Society: Sunlit Valley** - `9310d0a6-62a6-4fe6-82c4-eb483dc68876` - 38.68.14.28:25565 +3. **All The Mons (Private)** - `668a5220-7e72-4379-9165-bdbb84bc9806` - 38.68.14.30:25565 +4. **FoundryVTT** - `7d8f15a0-4ee7-4dd6-85dc-ab42966f733d` - 38.68.14.26:30000 +5. **Ars Eclectica** - `2973589e-1d2d-4896-9da5-f5f6d945ae6b` - 38.68.14.26:5520 +6. **Create Plus** - `cc170f06-5838-4773-a941-677e65e01171` - 38.68.14.26:25566 +7. **Vanilla** - `c4004e2b-04cc-42c4-b25d-f7eadda6f857` - 38.68.14.26:25567 + +--- + +### NC1 Charlotte (216.239.104.130) +**Role:** Secondary Game Server Node +**Location:** Charlotte, NC +**Provider:** Breezehost (Dedicated Server) +**Specs:** 251GB RAM, 98GB Disk +**Uptime:** 46 days, 12:38 +**Disk Usage:** ⚠️ 66% (61GB/98GB) - MONITOR + +**IP Subnet:** 216.239.104.128/29 +- Primary Node IP: 216.239.104.130 +- Gateway: 216.239.104.129 + +**Services Running:** +- Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP) +- MariaDB - Port 3306 (localhost) +- Cockpit - Port 9090 + +**Docker Containers:** 6 (all game servers) + +**Game Servers on NC1:** +1. **All The Mods 10** - `82e63949-8fbf-4a44-b32a-53324e8492bf` - 216.239.104.130:25569 +2. **Hytale** - `13c80cb8-f6f8-4bfe-9cdb-823d7e951584` - 216.239.104.130:5520-5521 +3. **All of Create (Creative)** - `e1c6ff8d-9f75-4a36-9200-598028bd0686` - 216.239.104.130:25568 +4. **All the Mods 10: To the Sky** - `f408e832-5902-4df4-bf94-243f9ceda624` - 216.239.104.130:25565 +5. **All the Mons (Public)** - `c4bc5892-ff9f-4188-905b-d2f0ed611816` - 216.239.104.130:25566 +6. **Mythcraft 5** - `b90ced3c-058c-4c5f-8e92-a2c5d76790b5` - 216.239.104.130:25567 + +**Special Firewall Rules:** +- Allows GRE traffic from Command Center (63.143.34.217) - for potential future tunneling +- Port 24454/udp open (Simple Voice Chat - Mayview) + +--- + +## 🔌 PORT ALLOCATION REGISTRY + +### Command Center (63.143.34.217) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 22 | SSH | Public | TCP | +| 80 | Nginx (63.143.34.217) | Public | TCP | +| 80 | Nginx (74.63.218.202) | Public | TCP | +| 443 | Nginx (63.143.34.217) | Public | TCP | +| 443 | Nginx (74.63.218.202) | Public | TCP | +| 3000 | Gitea | Internal | TCP | +| 3001 | Uptime Kuma | Internal | TCP | +| 3306 | MySQL | Localhost | TCP | +| 3500 | Discord Bot (The Arbiter) | Internal | TCP | +| 6379 | Redis | Localhost | TCP | +| 8000 | Vaultwarden | Docker localhost | TCP | +| 8001 | Vaultwarden proxy | Docker localhost | TCP | +| 8080 | Code-Server | Internal | TCP | +| 9090 | Cockpit | Public | TCP | + +**Nginx Virtual Hosts (63.143.34.217:443):** +- git.firefrostgaming.com → 127.0.0.1:3000 +- status.firefrostgaming.com → 127.0.0.1:3001 +- discord-bot.firefrostgaming.com → localhost:3500 +- vault.firefrostgaming.com → 127.0.0.1:8001 + +**Nginx Virtual Hosts (74.63.218.202:443):** +- code.firefrostgaming.com → 127.0.0.1:8080 + +--- + +### Ghost VPS (64.50.188.14) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 22 | SSH | Public | TCP | +| 25 | Postfix | Localhost | TCP | +| 80 | Nginx | Public | TCP | +| 443 | Nginx | Public | TCP | +| 2368 | Ghost CMS | Localhost | TCP | +| 3100 | Wiki.js Subscribers | Localhost | TCP | +| 3101 | Wiki.js Staff | Localhost | TCP | +| 3102 | Wiki.js Pokerole | Localhost | TCP | +| 3306 | MySQL | Localhost | TCP | +| 5432 | PostgreSQL | Localhost | TCP | +| 6379 | Redis | Localhost | TCP | +| 9090 | Cockpit | Public | TCP | + +**Nginx Virtual Hosts:** +- firefrostgaming.com → 127.0.0.1:2368 (Ghost) +- subscribers.firefrostgaming.com → localhost:3100 +- staff.firefrostgaming.com → localhost:3101 +- pokerole.firefrostgaming.com → localhost:3102 +- downloads.firefrostgaming.com → PHP-FPM (Nextcloud) +- docs.firefrostgaming.com → (MkDocs - not running currently) + +--- + +### Billing VPS (38.68.14.188) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 21 | vsftpd | Public | TCP | +| 22 | SSH | Public | TCP | +| 25 | Postfix (Docker) | Public | TCP | +| 80 | Nginx | Public | TCP | +| 110 | POP3 (Docker) | Public | TCP | +| 143 | IMAP (Docker) | Public | TCP | +| 443 | Nginx | Public | TCP | +| 465 | SMTPS (Docker) | Public | TCP | +| 587 | Submission (Docker) | Public | TCP | +| 993 | IMAPS (Docker) | Public | TCP | +| 995 | POP3S (Docker) | Public | TCP | +| 3306 | MariaDB | Localhost | TCP | +| 4190 | ManageSieve (Docker) | Public | TCP | +| 5001 | Whitelist Manager | Localhost | TCP | +| 6379 | Redis | Localhost | TCP | +| 7654 | Redis (Docker) | Docker localhost | TCP | +| 8080 | Mailcow Web | Public | TCP | +| 8443 | Mailcow Web SSL | Public | TCP | +| 9090 | Cockpit | Public | TCP | +| 13306 | MySQL (Docker) | Docker localhost | TCP | +| 19991 | Dovecot Stats | Docker localhost | TCP | + +**Nginx Virtual Hosts:** +- billing.firefrostgaming.com → PHP-FPM (Paymenter) +- mail.firefrostgaming.com → localhost:8443 (Mailcow) +- whitelist.firefrostgaming.com → 127.0.0.1:5001 + +--- + +### Panel VPS (45.94.168.138) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 21 | vsftpd | Public | TCP | +| 22 | SSH | Public | TCP | +| 80 | Nginx | Public | TCP | +| 443 | Nginx | Public | TCP | +| 3306 | MariaDB | Localhost | TCP | +| 6379 | Redis | Localhost | TCP | +| 9090 | Cockpit | Public | TCP | + +**Nginx Virtual Hosts:** +- panel.firefrostgaming.com → PHP-FPM (Pterodactyl Panel) + +--- + +### TX1 Dallas (38.68.14.26) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 22 | SSH | Public | TCP | +| 80 | Nginx | Public | TCP | +| 443 | Nginx | Public | TCP | +| 2022 | Wings SFTP | Public | TCP | +| 3000 | Dify Web | Docker localhost | TCP | +| 5001 | Dify API | Docker localhost | TCP | +| 5520 | Game: Ars Eclectica | Public | TCP/UDP | +| 5678 | n8n | Docker localhost | TCP | +| 6333 | Qdrant Vector DB | Public | TCP | +| 8080 | Wings HTTP | Public | TCP | +| 8090 | Plane (Caddy) | Public | TCP | +| 8444 | Plane SSL | Public | TCP | +| 9090 | Cockpit | Public | TCP | +| 10025 | Plane SMTP | Public | TCP | +| 10465 | Plane SMTPS | Public | TCP | +| 10587 | Plane Submission | Public | TCP | +| 25565 | Game: Stoneblock 4 | Public (38.68.14.26) | TCP/UDP | +| 25566 | Game: Create Plus | Public (38.68.14.26) | TCP/UDP | +| 25567 | Game: Vanilla | Public (38.68.14.26) | TCP/UDP | +| 25565 | Game: Society Sunlit Valley | Public (38.68.14.28) | TCP/UDP | +| 25565 | Game: All The Mons Private | Public (38.68.14.30) | TCP/UDP | +| 30000 | FoundryVTT | Public (38.68.14.26) | TCP/UDP | + +**Nginx Virtual Hosts:** +- codex.firefrostgaming.com → 127.0.0.1:3000 (Dify Web) + 127.0.0.1:5001 (API paths) +- n8n.firefrostgaming.com → 127.0.0.1:5678 +- tasks.firefrostgaming.com → 127.0.0.1:8090 (Plane) + +**Docker Internal Services:** +- PostgreSQL (Plane): 5432 +- PostgreSQL (Dify): 5432 +- Redis (Plane): 6379 +- Redis (Dify): 6379 +- RabbitMQ (Plane): 5672, 15672 +- MinIO (Plane): 9000 + +--- + +### NC1 Charlotte (216.239.104.130) + +| Port | Service | Access | Protocol | +|------|---------|--------|----------| +| 22 | SSH | Public | TCP | +| 2022 | Wings SFTP | Public | TCP | +| 3306 | MariaDB | Localhost | TCP | +| 5520-5521 | Game: Hytale | Public | TCP/UDP | +| 8080 | Wings HTTP | Public | TCP | +| 9090 | Cockpit | Public | TCP | +| 24454 | Simple Voice Chat | Public | UDP | +| 25565 | Game: ATM10 To the Sky | Public | TCP/UDP | +| 25566 | Game: All the Mons Public | Public | TCP/UDP | +| 25567 | Game: Mythcraft 5 | Public | TCP/UDP | +| 25568 | Game: All of Create | Public | TCP/UDP | +| 25569 | Game: All The Mods 10 | Public | TCP/UDP | + +--- + +## 🔗 CONNECTIVITY MAP + +### External Public-Facing Services + +**Domain → Server → Internal Port → External Port** + +1. **firefrostgaming.com** → Ghost VPS → 2368 → 443 (Nginx SSL) +2. **git.firefrostgaming.com** → Command Center → 3000 → 443 (Nginx SSL) +3. **status.firefrostgaming.com** → Command Center → 3001 → 443 (Nginx SSL) +4. **code.firefrostgaming.com** → Command Center → 8080 → 443 (Nginx SSL, 74.63.218.202) +5. **discord-bot.firefrostgaming.com** → Command Center → 3500 → 443 (Nginx SSL) +6. **vault.firefrostgaming.com** → Command Center → 8001 → 443 (Nginx SSL) +7. **billing.firefrostgaming.com** → Billing VPS → PHP-FPM → 80 (Nginx) +8. **mail.firefrostgaming.com** → Billing VPS → 8443 → 443 (Nginx SSL) +9. **whitelist.firefrostgaming.com** → Billing VPS → 5001 → 80 (Nginx) +10. **panel.firefrostgaming.com** → Panel VPS → PHP-FPM → 443 (Nginx SSL) +11. **codex.firefrostgaming.com** → TX1 → 3000/5001 → 443 (Nginx SSL) +12. **n8n.firefrostgaming.com** → TX1 → 5678 → 443 (Nginx SSL) +13. **tasks.firefrostgaming.com** → TX1 → 8090 → 80 (Nginx) +14. **downloads.firefrostgaming.com** → Ghost VPS → PHP-FPM → 443 (Nginx SSL, Nextcloud) +15. **subscribers.firefrostgaming.com** → Ghost VPS → 3100 → 80 (Nginx) +16. **staff.firefrostgaming.com** → Ghost VPS → 3101 → 80 (Nginx) +17. **pokerole.firefrostgaming.com** → Ghost VPS → 3102 → 80 (Nginx) + +### Server-to-Server Communication + +**Panel VPS (45.94.168.138) ↔ Wings Nodes:** +- Panel → TX1 (38.68.14.26:8080) - Wings API +- Panel → NC1 (216.239.104.130:8080) - Wings API +- **Protocol:** HTTPS (Wings API) +- **Authentication:** API tokens +- **Purpose:** Server management, monitoring, console access + +**Discord Bot (Command Center) → Discord API:** +- discord-bot.firefrostgaming.com (63.143.34.217:3500) → Discord.com:443 +- **Protocol:** HTTPS + WebSocket +- **Purpose:** Bot commands, role management, webhooks + +**Paymenter (Billing VPS) → Pterodactyl Panel:** +- Planned webhook: billing.firefrostgaming.com → panel.firefrostgaming.com +- **Protocol:** HTTPS +- **Purpose:** Subscription provisioning automation + +**Paymenter (Billing VPS) → Discord Bot:** +- Planned webhook: billing.firefrostgaming.com → discord-bot.firefrostgaming.com/webhook/paymenter +- **Protocol:** HTTPS +- **Purpose:** Subscription event notifications for role assignment + +**Whitelist Manager (Billing VPS) → Pterodactyl Panel:** +- whitelist.firefrostgaming.com (38.68.14.188:5001) → panel.firefrostgaming.com (45.94.168.138) +- **Protocol:** HTTPS (Panel API) +- **Purpose:** Whitelist synchronization + +**n8n (TX1) → External Services:** +- n8n.firefrostgaming.com → Various APIs (GitHub, Discord, etc.) +- **Protocol:** HTTPS +- **Purpose:** Workflow automation + +**Gitea (Command Center) → Git Clients:** +- git.firefrostgaming.com → Various (Claude, developers, CI/CD) +- **Protocol:** HTTPS + SSH (port 22) +- **Purpose:** Git repository access + +### Database Connections (Internal Only) + +**Command Center:** +- Gitea → MySQL (127.0.0.1:3306) +- Vaultwarden → Internal SQLite + +**Ghost VPS:** +- Ghost CMS → MySQL (127.0.0.1:3306) +- Wiki.js (3x) → PostgreSQL (127.0.0.1:5432) +- All services → Redis (127.0.0.1:6379) for caching + +**Billing VPS:** +- Paymenter → MariaDB (127.0.0.1:3306) +- Paymenter → Redis (127.0.0.1:6379) +- Mailcow → Docker MySQL (172.22.1.x:3306) +- Mailcow → Docker Redis (172.22.1.x:6379) + +**Panel VPS:** +- Pterodactyl Panel → MariaDB (127.0.0.1:3306) +- Pterodactyl Panel → Redis (127.0.0.1:6379) + +**TX1 Dallas:** +- Plane → Docker PostgreSQL (internal) +- Plane → Docker Redis (internal) +- Dify → Docker PostgreSQL (internal) +- Dify → Docker Redis (internal) +- Dify → Qdrant (127.0.0.1:6333) + +**NC1 Charlotte:** +- Wings → MariaDB (127.0.0.1:3306) + +--- + +## 🎯 AUTHENTICATION & DEPENDENCY FLOWS + +### OAuth2 Flows + +**Discord Bot Admin Panel:** +- User → discord-bot.firefrostgaming.com → Discord OAuth2 → Whitelist check → Session +- **Dependencies:** Discord API availability, Session storage (Express sessions) + +### API Token Flows + +**Pterodactyl Panel ↔ Wings:** +- Panel stores Wings API tokens +- Wings validates tokens on each request +- **Critical:** Token compromise = full server control + +**Gitea API:** +- Claude sessions use: `e0e330cba1749b01ab505093a160e4423ebbbe36` +- Operations manual automation +- **Critical:** Full admin access token + +**n8n Workflows:** +- Various API tokens stored in n8n credentials +- Discord webhooks, GitHub, etc. + +### SMTP Flows (Email) + +**Ghost VPS (Postfix):** +- **Status:** ⚠️ BLOCKED - Inbound port 25 blocked at provider level +- **Workaround Needed:** Provider support ticket +- **Current:** Internal mail only + +**Billing VPS (Mailcow):** +- **Status:** ✅ OPERATIONAL +- SMTP out: 587 (submission), 465 (SMTPS), 25 (relay) +- IMAP: 143, 993 (SSL) +- POP3: 110, 995 (SSL) +- **DKIM/SPF/DMARC:** Configured for firefrostgaming.com + +**TX1 (Plane):** +- **Status:** ✅ OPERATIONAL +- Internal SMTP for Plane notifications (ports 10025, 10465, 10587) + +--- + +## ⚠️ SINGLE POINTS OF FAILURE + +### Critical Single Points + +1. **Pterodactyl Panel (45.94.168.138)** + - **Risk:** Panel down = no game server management + - **Mitigation:** Wings nodes continue running autonomously + - **Recovery Time:** ~30 minutes (restore from backup + DNS) + +2. **Mailcow (Billing VPS)** + - **Risk:** Email down = no subscription confirmations, no support tickets + - **Mitigation:** Cloudflare Email Routing as backup? + - **Recovery Time:** ~2 hours (Mailcow stack restoration) + +3. **Gitea (Command Center)** + - **Risk:** Git down = no deployments, no operations manual access + - **Mitigation:** Local clones exist on developer machines + - **Recovery Time:** ~1 hour (service restart or VM restore) + +4. **Ghost CMS (Ghost VPS)** + - **Risk:** Main website down = no public presence + - **Mitigation:** Cloudflare caching provides limited read access + - **Recovery Time:** ~1 hour (Ghost restart or data restore) + +5. **Command Center Server (63.143.34.217)** + - **Risk:** Multiple critical services (Gitea, Uptime Kuma, Discord Bot, Vaultwarden) + - **Impact:** Most critical - affects development, monitoring, and Discord automation + - **Mitigation:** Distributed services across multiple VPS in future + - **Recovery Time:** 2-4 hours (depends on failure type) + +### Non-Critical Single Points + +6. **Billing VPS (38.68.14.188)** + - **Services:** Paymenter, Mailcow, Whitelist Manager + - **Impact:** Financial operations halted, but game servers continue + - **Note:** High disk usage (70%) increases risk + +7. **Ghost VPS (64.50.188.14)** + - **Services:** Ghost, Wiki.js (3x), Nextcloud + - **Impact:** Documentation inaccessible, but operations continue + - **Note:** Can be restored from backups + +--- + +## 🔥 PORT CONFLICT PREVENTION + +### Port Allocation Strategy + +**Reserved Ranges:** +- **25565-25580:** Minecraft game servers (TCP/UDP) +- **5520-5521:** Hytale (TCP/UDP) +- **30000-30010:** Reserved for FoundryVTT and future VTT instances +- **3000-3200:** Internal web services (Gitea, Uptime Kuma, Wiki.js, etc.) +- **8000-9000:** Docker services and Wings +- **10000-11000:** Plane/n8n/Dify internal services + +### Conflict Lessons Learned + +**The Arbiter Bot Port Hunt (March 27, 2026):** +1. Attempted port 3000 → **CONFLICT** (Gitea on TX1 Dify) +2. Attempted port 3001 → **CONFLICT** (Uptime Kuma) +3. **SUCCESS:** Port 3500 (unused) + +**Prevention Going Forward:** +- Always check `ss -tlnp | grep LISTEN` before deploying +- Document port assignments in this registry +- Use high-numbered ports (3500+) for new services on shared servers +- Consider port range 4000-5000 for future Discord/webhook services + +### Available Port Ranges + +**Command Center (63.143.34.217):** +- ✅ 3500-4000: Available +- ✅ 4000-6000: Available (except 6379 Redis) +- ✅ 7000-8000: Available (except 8000-8001 Vaultwarden) + +**Ghost VPS (64.50.188.14):** +- ✅ 3200-6000: Available (except 3306 MySQL, 5432 PostgreSQL) +- ✅ 7000-9000: Available + +**Billing VPS (38.68.14.188):** +- ⚠️ Most standard ports occupied by Mailcow +- ✅ 5100-6000: Available (except 5001 Whitelist Manager) +- ✅ 9100-10000: Available + +**Panel VPS (45.94.168.138):** +- ✅ 1024-3000: Available +- ✅ 3500-6000: Available (except 3306 MySQL, 6379 Redis) +- ✅ 7000-9000: Available + +**TX1 Dallas (38.68.14.26):** +- ⚠️ Heavy Docker usage, internal ports dynamic +- ✅ 3500-5000: Available (except 5001 Dify, 5678 n8n) +- ✅ 7000-8000: Available +- ✅ 11000-20000: Available + +**NC1 Charlotte (216.239.104.130):** +- ✅ 3000-5000: Available (except 3306 MySQL) +- ✅ 6000-8000: Available +- ✅ 10000-20000: Available + +--- + +## 📊 RESOURCE UTILIZATION + +### Disk Usage Status + +| Server | Used | Total | Usage % | Status | +|--------|------|-------|---------|--------| +| Command Center | 17GB | 38GB | 45% | ✅ Good | +| Ghost VPS | 21GB | 38GB | 55% | ✅ Good | +| **Billing VPS** | **13GB** | **19GB** | **70%** | ⚠️ **Monitor** | +| Panel VPS | 9GB | 24GB | 39% | ✅ Good | +| **TX1 Dallas** | **102GB** | **911GB** | **12%** | ✅ **Excellent** | +| **NC1 Charlotte** | **61GB** | **98GB** | **66%** | ⚠️ **Monitor** | + +**Recommendations:** +1. **Billing VPS:** Review Mailcow logs and docker volume sizes - consider cleanup or expansion +2. **NC1 Charlotte:** Monitor game server world sizes - implement world pruning or expansion +3. **TX1 Dallas:** Massive capacity available - can host additional services + +### Service Load Distribution + +**Command Center:** 33 systemd services (6 critical) +**Ghost VPS:** 31 systemd services (5 critical) +**Billing VPS:** 30 systemd services + 18 Docker containers +**Panel VPS:** 28 systemd services (clean, focused) +**TX1 Dallas:** 29 systemd services + 35 Docker containers (heavy) +**NC1 Charlotte:** 25 systemd services + 6 Docker containers (focused) + +--- + +## 🔐 FIREWALL ANALYSIS + +### Command Center UFW Rules +- ✅ SSH (22) open +- ✅ HTTP/HTTPS (80/443) on both IPs +- ✅ Cockpit (9090) open +- ✅ Specific IP bindings for services (63.143.34.217 vs 74.63.218.202) + +### Ghost VPS +- ⚠️ Firewall audit returned "ERROR: You need to be root" (was logged in as architect) +- **Action Required:** Re-audit as root to verify rules + +### Billing VPS IPTables +- ✅ Custom Mailcow chain (MAILCOW) +- ✅ UFW chains present +- ✅ Docker chains for container networking + +### Panel VPS UFW Rules +- ✅ SSH (22), HTTP (80), HTTPS (443) open +- ✅ FTP (21) open for vsftpd +- ✅ Cockpit (9090) open +- ✅ Specific allow from 141.98.74.95 (related system?) + +### TX1 Dallas UFW Rules +- ✅ Wings ports (8080, 2022) open +- ✅ Minecraft port range (25565-25580) TCP+UDP +- ✅ Hytale ports (5520-5521) TCP+UDP +- ✅ n8n webhook port (5678) +- ✅ Cockpit (9090) open +- ✅ Allow 74.63.218.205 HTTP/HTTPS (Code-Server IP?) + +### NC1 Charlotte UFW Rules +- ✅ Wings ports (8080, 2022) open +- ✅ Minecraft port range (25565-25580) TCP+UDP +- ✅ Hytale ports (5520-5521) TCP+UDP +- ✅ Simple Voice Chat (24454 UDP) +- ✅ GRE protocol (47) open - for future tunneling +- ✅ **Special:** Full allow from Command Center IP (63.143.34.217) + GRE +- ✅ Cockpit (9090) open + +--- + +## 🎮 GAME SERVER MAPPING + +### TX1 Dallas Game Servers (7 servers) + +| Server Name | UUID | IP:Port | Status | +|-------------|------|---------|--------| +| Stoneblock 4 | a0efbfe8-4b97-4a90-869d-ffe6d3072bd5 | 38.68.14.26:25565 | ✅ Up 3 hours | +| Society: Sunlit Valley | 9310d0a6-62a6-4fe6-82c4-eb483dc68876 | 38.68.14.28:25565 | ✅ Up 9 hours | +| All The Mons (Private) | 668a5220-7e72-4379-9165-bdbb84bc9806 | 38.68.14.30:25565 | ✅ Up 9 hours | +| FoundryVTT | 7d8f15a0-4ee7-4dd6-85dc-ab42966f733d | 38.68.14.26:30000 | ✅ Up 9 hours | +| Ars Eclectica | 2973589e-1d2d-4896-9da5-f5f6d945ae6b | 38.68.14.26:5520 | ✅ Up 7 hours | +| Create Plus | cc170f06-5838-4773-a941-677e65e01171 | 38.68.14.26:25566 | ✅ Up 6 days | +| Vanilla | c4004e2b-04cc-42c4-b25d-f7eadda6f857 | 38.68.14.26:25567 | ✅ Up 2 days | + +### NC1 Charlotte Game Servers (6 servers) + +| Server Name | UUID | IP:Port | Status | +|-------------|------|---------|--------| +| All The Mods 10 | 82e63949-8fbf-4a44-b32a-53324e8492bf | 216.239.104.130:25569 | ✅ Up 8 hours | +| Hytale | 13c80cb8-f6f8-4bfe-9cdb-823d7e951584 | 216.239.104.130:5520-5521 | ✅ Up 9 hours | +| All of Create (Creative) | e1c6ff8d-9f75-4a36-9200-598028bd0686 | 216.239.104.130:25568 | ✅ Up 9 hours | +| All the Mods 10: To the Sky | f408e832-5902-4df4-bf94-243f9ceda624 | 216.239.104.130:25565 | ✅ Up 9 hours | +| All the Mons (Public) | c4bc5892-ff9f-4188-905b-d2f0ed611816 | 216.239.104.130:25566 | ✅ Up 8 hours | +| Mythcraft 5 | b90ced3c-058c-4c5f-8e92-a2c5d76790b5 | 216.239.104.130:25567 | ✅ Up 7 hours | + +**Total:** 14 game servers (13 Minecraft + 1 Hytale + 1 FoundryVTT) + +--- + +## 🚨 ISSUES IDENTIFIED + +### Critical Issues +1. **Ghost VPS Port 25 Blocked** + - **Impact:** Cannot receive external email + - **Workaround:** Internal mail only + - **Resolution:** Support ticket to Breezehost (Jon) to unblock 38.68.14.188:25 + - **Ticket Status:** Not yet submitted + +### Warning Issues +2. **Billing VPS Disk Usage: 70%** + - **Risk:** May hit capacity during high email volume + - **Action:** Review Mailcow container logs and volumes + - **Timeline:** Monitor weekly, expand if hits 80% + +3. **NC1 Charlotte Disk Usage: 66%** + - **Risk:** Game worlds growing, may hit capacity + - **Action:** Implement world pruning or disk expansion + - **Timeline:** Monitor weekly, expand if hits 75% + +4. **Ghost VPS Firewall Not Audited** + - **Risk:** Unknown firewall state (audit failed due to permissions) + - **Action:** Re-run audit as root + - **Timeline:** Next maintenance window + +### Monitoring Issues +5. **Plane Monitor Container Restarting** + - **Server:** TX1 Dallas + - **Container:** plane-monitor-1 + - **Status:** Restarting (1) 6 seconds ago + - **Impact:** Unknown - appears to be continuous restart loop + - **Action:** Investigate logs, may need container restart or config fix + +6. **Plane Space Container Unhealthy** + - **Server:** TX1 Dallas + - **Container:** plane-space-1 + - **Status:** Up 11 days (unhealthy) + - **Impact:** Potential feature degradation + - **Action:** Check health endpoint and logs + +--- + +## 📈 CAPACITY PLANNING + +### Short-Term Capacity (Next 3 Months) + +**Can Accommodate:** +- ✅ 5-10 more game servers on TX1 (plenty of disk + RAM) +- ✅ 2-4 more game servers on NC1 (disk space permitting) +- ✅ Additional web services on Command Center +- ✅ Additional web services on Ghost VPS +- ⚠️ Limited capacity on Billing VPS (disk constraint) + +**Cannot Accommodate Without Expansion:** +- ❌ Additional Docker stacks on Billing VPS (disk full) +- ❌ Large-world game servers on NC1 (disk space) + +### Long-Term Recommendations + +1. **Expand Billing VPS Disk** + - Current: 19GB + - Recommended: 40-50GB + - Reason: Mailcow + Paymenter + future growth + +2. **Expand NC1 Disk** + - Current: 98GB + - Recommended: 200GB+ + - Reason: Game world growth over time + +3. **Consider Backup Server** + - Add dedicated backup VPS + - Offload backups from game server disks + - Enable disaster recovery + +4. **Load Balancer for Web Services** + - Multiple Ghost CMS instances + - Distribute SSL termination + - Improve resilience + +--- + +## 🔄 INTERCONNECTION SUMMARY + +### Data Flow Patterns + +**User → Website (Ghost CMS)** +1. User → Cloudflare → Ghost VPS:443 +2. Nginx → Ghost:2368 +3. Ghost → MySQL:3306 + +**User → Panel (Pterodactyl)** +1. User → Cloudflare → Panel VPS:443 +2. Nginx → PHP-FPM → Panel Application +3. Panel → MariaDB:3306 +4. Panel → Wings API (TX1:8080, NC1:8080) + +**User → Game Server** +1. User → TX1/NC1 direct (no proxy) +2. Game Server → Wings → Panel (monitoring/console) + +**Discord Bot Workflow** +1. Discord API → discord-bot.firefrostgaming.com:443 +2. Nginx → Bot:3500 +3. Bot → Discord API (outbound) +4. Bot → (future) Paymenter webhook + +**Subscription Workflow (Planned)** +1. User → Paymenter (billing.firefrostgaming.com) +2. Paymenter → Stripe/PayPal API +3. Paymenter webhook → Discord Bot +4. Discord Bot → Discord API (assign role) +5. Discord Bot → (future) Panel API (provision server) + +--- + +## 📝 RECOMMENDATIONS + +### Immediate Actions (Next 7 Days) +1. ✅ Complete this audit document +2. ⚠️ Submit Breezehost ticket for Ghost VPS port 25 +3. ⚠️ Re-audit Ghost VPS firewall as root +4. ⚠️ Investigate Plane monitor container restart loop +5. ⚠️ Check Plane space container health +6. ✅ Document port allocation strategy in operations manual + +### Short-Term Actions (Next 30 Days) +7. ⚠️ Review Billing VPS disk usage, plan expansion if needed +8. ⚠️ Monitor NC1 disk usage weekly +9. ✅ Implement automated disk usage alerting (Uptime Kuma?) +10. ✅ Configure Paymenter → Discord Bot webhooks +11. ✅ Test full subscription provisioning flow + +### Long-Term Actions (Next 90 Days) +12. 🔄 Implement backup server or backup strategy +13. 🔄 Consider load balancer for web services +14. 🔄 Evaluate Gitea high-availability options +15. 🔄 Plan for TX1/NC1 disk expansion schedule + +--- + +## 🎯 AUDIT COMPLETION + +**Audit Status:** ✅ COMPLETE +**Data Collection:** March 27, 2026 +**Servers Audited:** 6/6 (100%) +**Document Version:** 1.0 +**Next Audit:** Recommended every 6 months or after major infrastructure changes + +**Compiled By:** Chronicler #43 +**Reviewed By:** (Pending Michael's review) +**Committed To:** firefrost-operations-manual repository + +--- + +**Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️ diff --git a/docs/infrastructure/network-diagram-2026.mmd b/docs/infrastructure/network-diagram-2026.mmd new file mode 100644 index 0000000..efed46b --- /dev/null +++ b/docs/infrastructure/network-diagram-2026.mmd @@ -0,0 +1,176 @@ +```mermaid +graph TB + subgraph Internet["🌐 PUBLIC INTERNET"] + Users["Users/Players"] + Discord["Discord API"] + CloudFlare["Cloudflare CDN"] + end + + subgraph CommandCenter["Command Center VPS
63.143.34.217 / 74.63.218.202
Dallas, TX"] + Gitea["Gitea
:3000"] + UptimeKuma["Uptime Kuma
:3001"] + CodeServer["Code-Server
:8080"] + DiscordBot["The Arbiter Bot
:3500"] + Vaultwarden["Vaultwarden
Docker :8001"] + NginxCC["Nginx
:80/:443"] + MySQLCC["MySQL
:3306"] + end + + subgraph GhostVPS["Ghost VPS
64.50.188.14
Chicago, IL"] + Ghost["Ghost CMS
:2368"] + Wiki1["Wiki.js Sub
:3100"] + Wiki2["Wiki.js Staff
:3101"] + Wiki3["Wiki.js Pokerole
:3102"] + Nextcloud["Nextcloud
PHP-FPM"] + NginxGhost["Nginx
:80/:443"] + MySQLGhost["MySQL
:3306"] + PostgresGhost["PostgreSQL
:5432"] + RedisGhost["Redis
:6379"] + end + + subgraph BillingVPS["Billing VPS
38.68.14.188
Chicago, IL
⚠️ 70% Disk"] + Paymenter["Paymenter
PHP-FPM"] + Mailcow["Mailcow Stack
:8080/:8443
18 Containers"] + WhitelistMgr["Whitelist Mgr
:5001"] + NginxBilling["Nginx
:80/:443"] + MariaDBBilling["MariaDB
:3306"] + end + + subgraph PanelVPS["Panel VPS
45.94.168.138
Charlotte, NC"] + PteroPanel["Pterodactyl Panel
PHP-FPM"] + NginxPanel["Nginx
:80/:443"] + MariaDBPanel["MariaDB
:3306"] + RedisPanel["Redis
:6379"] + end + + subgraph TX1["TX1 Dallas Dedicated
38.68.14.26-30
251GB RAM, 911GB Disk
✅ 12% Usage"] + Wings1["Wings
:8080/:2022"] + Plane["Plane Stack
:8090
20 Containers"] + Dify["Dify/Codex
:3000/:5001"] + N8N["n8n
:5678"] + Qdrant["Qdrant
:6333"] + Ollama["Ollama AI"] + NginxTX1["Nginx
:80/:443"] + + subgraph GamesTX1["Game Servers - TX1"] + Game1TX["Stoneblock 4
.26:25565"] + Game2TX["Society Sunlit
.28:25565"] + Game3TX["All The Mons Priv
.30:25565"] + Game4TX["FoundryVTT
.26:30000"] + Game5TX["Ars Eclectica
.26:5520"] + Game6TX["Create Plus
.26:25566"] + Game7TX["Vanilla
.26:25567"] + end + end + + subgraph NC1["NC1 Charlotte Dedicated
216.239.104.130
251GB RAM, 98GB Disk
⚠️ 66% Usage"] + Wings2["Wings
:8080/:2022"] + MariaDBNC["MariaDB
:3306"] + + subgraph GamesNC1["Game Servers - NC1"] + Game1NC["ATM10
:25569"] + Game2NC["Hytale
:5520-5521"] + Game3NC["All of Create
:25568"] + Game4NC["ATM10 Sky
:25565"] + Game5NC["All Mons Pub
:25566"] + Game6NC["Mythcraft 5
:25567"] + end + end + + %% Public Access + Users -->|HTTPS| CloudFlare + CloudFlare -->|HTTPS| NginxCC + CloudFlare -->|HTTPS| NginxGhost + CloudFlare -->|HTTPS| NginxBilling + CloudFlare -->|HTTPS| NginxPanel + CloudFlare -->|HTTPS| NginxTX1 + + Users -->|TCP/UDP| GamesTX1 + Users -->|TCP/UDP| GamesNC1 + + %% Command Center Internal + NginxCC --> Gitea + NginxCC --> UptimeKuma + NginxCC --> CodeServer + NginxCC --> DiscordBot + NginxCC --> Vaultwarden + Gitea --> MySQLCC + + %% Ghost VPS Internal + NginxGhost --> Ghost + NginxGhost --> Wiki1 + NginxGhost --> Wiki2 + NginxGhost --> Wiki3 + NginxGhost --> Nextcloud + Ghost --> MySQLGhost + Wiki1 --> PostgresGhost + Wiki2 --> PostgresGhost + Wiki3 --> PostgresGhost + Ghost --> RedisGhost + Wiki1 --> RedisGhost + Wiki2 --> RedisGhost + Wiki3 --> RedisGhost + + %% Billing VPS Internal + NginxBilling --> Paymenter + NginxBilling --> Mailcow + NginxBilling --> WhitelistMgr + Paymenter --> MariaDBBilling + + %% Panel VPS Internal + NginxPanel --> PteroPanel + PteroPanel --> MariaDBPanel + PteroPanel --> RedisPanel + + %% TX1 Internal + NginxTX1 --> Plane + NginxTX1 --> Dify + NginxTX1 --> N8N + Dify --> Qdrant + Dify --> Ollama + Wings1 --> GamesTX1 + + %% NC1 Internal + Wings2 --> GamesNC1 + Wings2 --> MariaDBNC + + %% Server to Server + PteroPanel -->|Wings API| Wings1 + PteroPanel -->|Wings API| Wings2 + DiscordBot -->|WebSocket/HTTPS| Discord + Paymenter -.->|Webhook Planned| DiscordBot + Paymenter -.->|API Planned| PteroPanel + WhitelistMgr -->|Panel API| PteroPanel + N8N -->|Webhooks| Discord + + %% Styling + classDef vps fill:#4ECDC4,stroke:#333,stroke-width:2px,color:#000 + classDef dedicated fill:#FF6B35,stroke:#333,stroke-width:3px,color:#000 + classDef service fill:#A855F7,stroke:#333,stroke-width:1px,color:#fff + classDef warning fill:#FFD700,stroke:#333,stroke-width:2px,color:#000 + + class CommandCenter,GhostVPS,BillingVPS,PanelVPS vps + class TX1,NC1 dedicated + class Gitea,Ghost,Paymenter,PteroPanel,Wings1,Wings2,Plane,Dify service + class BillingVPS,NC1 warning + + style Users fill:#90EE90,stroke:#333,stroke-width:2px + style Discord fill:#5865F2,stroke:#333,stroke-width:2px + style CloudFlare fill:#F38020,stroke:#333,stroke-width:2px +``` + +**Legend:** +- 🔵 Blue = VPS Servers +- 🔴 Red = Dedicated Servers (251GB RAM each) +- 🟣 Purple = Key Services +- 🟡 Yellow Border = Warning (High Disk Usage) +- ─── Solid Lines = Active Connections +- ─ ─ Dotted Lines = Planned Connections + +**Key Interconnections:** +1. **Panel → Wings**: Pterodactyl management plane +2. **Discord Bot ↔ Discord**: Real-time bot communication +3. **Paymenter → Discord Bot**: Subscription automation (planned) +4. **Paymenter → Panel**: Server provisioning (planned) +5. **Whitelist Manager → Panel**: Player whitelist sync +6. **Cloudflare → All Public Services**: CDN and DDoS protection