firefrost-gaming/firefrost-operations-manual
3
0
Fork 0
Code Issues 146 Pull Requests Packages Projects Releases Wiki Activity
Files
414f12452986dc4033748532df815bb5e3b3cd3d
firefrost-operations-manual/docs/infrastructure/network-audit-2026.md
Claude 414f124529 NC1 disk expanded: 100GB → 928GB (LVM resize) 
Live expansion performed April 8, 2026:
- lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
- resize2fs /dev/ubuntu-vg/ubuntu-lv

NC1 now: 914GB total, 807GB free (8% usage)
Previously: 98GB total, 25GB free (74% usage)

Updated network audit:
- Removed NC1 disk warnings
- Updated capacity planning
- Marked expansion action items complete

Chronicler #69
2026-04-08 08:05:05 +00:00

31 KiB
Raw Blame History

🔥❄️ Firefrost Gaming Infrastructure Audit 2026

Audit Date: March 27, 2026
Audited By: Chronicler #43
Purpose: Complete network topology, port allocation, service inventory, and connectivity mapping
Reason: Prevent port conflicts (learned from The Arbiter bot deployment: 3000→3001→3500)

📋 EXECUTIVE SUMMARY

Total Infrastructure:

  • 6 Servers (4 VPS, 2 Dedicated)
  • 90+ Services running across all servers
  • 48 Docker Containers (18 Mailcow, 15 TX1, 6 NC1, 1 Vaultwarden, 8 n8n/Dify services)
    • 20 Plane containers removed March 27, 2026
  • 22 Game Servers (11 TX1, 11 NC1) — Updated April 8, 2026
  • 1 FoundryVTT Server (included in TX1 count)
  • 12 Public-Facing Domains (tasks.firefrostgaming.com freed March 27, 2026)

⚠️ AI/LLM Resource Consideration: TX1 runs both game servers AND AI stack (Dify, Qdrant, Ollama). Heavy modpacks + Gemma 4 inference could compete for RAM. Monitor closely or consider workload separation.

Key Findings:

  1. ✅ No current port conflicts detected
  2. ✅ Clean separation of management vs game workloads
  3. ⚠️ Billing VPS disk usage at 70% (13GB/19GB)
  4. ✅ NC1 disk expanded April 8, 2026 (8% usage, 69GB/914GB)
  5. ✅ TX1 has plenty of capacity (12% usage, 102GB/911GB)
  6. ✅ All critical services operational
  7. ✅ Firewall rules properly configured on all servers

🖥️ SERVER INVENTORY

Command Center (63.143.34.217)

Role: Management Hub + Backend Services
Location: Dallas, TX
Provider: Breezehost
Uptime: 46 days, 12:35
Disk Usage: 45% (17GB/38GB)
RAM: Standard VPS

Services Running:

  • Gitea (git.firefrostgaming.com) - Port 3000 → Nginx 443
  • Uptime Kuma (status.firefrostgaming.com) - Port 3001 → Nginx 443
  • Code-Server (code.firefrostgaming.com) - Port 8080 → Nginx 443 (74.63.218.202)
  • The Arbiter Discord Bot (discord-bot.firefrostgaming.com) - Port 3500 → Nginx 443
  • Vaultwarden (vault.firefrostgaming.com) - Docker 8001 → Nginx 443
  • MySQL - Port 3306 (localhost)
  • Nginx - Reverse proxy for all services
  • Cockpit - Port 9090

IP Addresses:

  • Primary: 63.143.34.217
  • Secondary: 74.63.218.202 (Code-Server only)

Docker Containers: 1 (Vaultwarden)

Ghost VPS (64.50.188.14)

Role: Documentation Cluster + Public-Facing Content
Location: Chicago, IL
Provider: Breezehost
Uptime: 13 days, 20:24
Disk Usage: 55% (21GB/38GB)
Login: architect (not root)

Services Running:

  • Ghost CMS (firefrostgaming.com) - Port 2368 → Nginx 443
    • Status: ✅ Live subscription page with Fire/Frost tier branding
    • Features: 11 subscription tiers, dual-path branding, production-ready
  • Wiki.js Subscribers (subscribers.firefrostgaming.com) - Port 3100 → Nginx 80
  • Wiki.js Staff (staff.firefrostgaming.com) - Port 3101 → Nginx 80
  • Wiki.js Pokerole (pokerole.firefrostgaming.com) - Port 3102 → Nginx 80
  • Nextcloud (downloads.firefrostgaming.com) - Nginx 443 (PHP-FPM)
  • MySQL - Port 3306 (localhost)
  • PostgreSQL - Port 5432 (localhost)
  • Redis - Port 6379 (localhost)
  • Postfix - Port 25 (localhost only, SMTP blocked at network level)
  • Nginx - Reverse proxy
  • Cockpit - Port 9090

Docker Containers: 0 (all native services)

Note: Port 25 issue previously resolved with Breezehost.

Billing VPS (38.68.14.188)

Role: Financial Services Isolation
Location: Chicago, IL
Provider: Breezehost
Uptime: 11 days, 12:22
Disk Usage: ⚠️ 70% (13GB/19GB) - MONITOR
RAM: Standard VPS

Services Running:

  • Paymenter (billing.firefrostgaming.com) - PHP-FPM → Nginx 80
    • Status: ✅ Fully configured with 11 subscription tiers
    • Tiers: The Awakened ($1), Fire/Frost Elemental ($5), Knight ($10), Master ($15), Legend ($20), Sovereign ($499)
  • Mailcow Stack (mail.firefrostgaming.com) - Docker 8080/8443 → Nginx 443
  • Whitelist Manager (whitelist.firefrostgaming.com) - Port 5001 → Nginx 80
  • MariaDB - Port 3306 (localhost)
  • Redis - Port 6379 (localhost)
  • Nginx - Reverse proxy
  • Supervisor - Process control
  • Cockpit - Port 9090

Docker Containers: 18 (Mailcow stack)

  1. mailcowdockerized-nginx-mailcow-1 - 8080/8443
  2. mailcowdockerized-postfix-mailcow-1 - 25, 465, 587
  3. mailcowdockerized-dovecot-mailcow-1 - 110, 143, 993, 995, 4190
  4. mailcowdockerized-mysql-mailcow-1 - 13306 (localhost)
  5. mailcowdockerized-redis-mailcow-1 - 7654 (localhost)
  6. mailcowdockerized-rspamd-mailcow-1
  7. mailcowdockerized-php-fpm-mailcow-1
  8. mailcowdockerized-sogo-mailcow-1
  9. mailcowdockerized-clamd-mailcow-1
  10. mailcowdockerized-unbound-mailcow-1
  11. mailcowdockerized-watchdog-mailcow-1
  12. mailcowdockerized-acme-mailcow-1
  13. mailcowdockerized-ofelia-mailcow-1
  14. mailcowdockerized-postfix-tlspol-mailcow-1
  15. mailcowdockerized-memcached-mailcow-1
  16. mailcowdockerized-netfilter-mailcow-1
  17. mailcowdockerized-dockerapi-mailcow-1
  18. mailcowdockerized-olefy-mailcow-1

Mail Ports (all via Docker):

  • SMTP: 25, 465, 587
  • IMAP: 143, 993
  • POP3: 110, 995
  • ManageSieve: 4190

Panel VPS (45.94.168.138)

Role: Pterodactyl Control Plane
Location: Charlotte, NC
Provider: Breezehost
Uptime: 13 days, 19:22
Disk Usage: 39% (9GB/24GB)
RAM: Standard VPS

Services Running:

  • Pterodactyl Panel (panel.firefrostgaming.com) - PHP-FPM → Nginx 443
  • MariaDB - Port 3306 (localhost)
  • Redis - Port 6379 (localhost)
  • vsftpd - Port 21
  • pteroq (Queue Worker) - Systemd service
  • Nginx - Reverse proxy
  • Cockpit - Port 9090

Docker Containers: 0 (all native services)

Blueprint Extensions Installed:

  • Modpack Installer for Blueprint
  • Subdomain Manager for Pterodactyl
  • PteroStats - Advanced Statistics

TX1 Dallas (38.68.14.26)

Role: Primary Game Server + Advanced Services
Location: Dallas, TX
Provider: Breezehost (Dedicated Server)
Specs: 251GB RAM, 911GB Disk
Uptime: 11 days, 11:00
Disk Usage: ✅ 12% (102GB/911GB) - EXCELLENT

IP Subnet: 38.68.14.24/29

  • Primary Node IP: 38.68.14.26
  • Additional IPs: .27, .28, .29, .30

Services Running:

  • Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP)
  • Firefrost Codex (codex.firefrostgaming.com):
    • Dify API - Port 5001 (localhost)
    • Dify Web - Port 3000 (localhost)
    • Qdrant Vector DB - Port 6333 (public)
    • n8n (n8n.firefrostgaming.com) - Port 5678 (localhost) → Nginx 443
    • Ollama - AI model server
  • Nginx - 2 reverse proxy configurations
  • Fail2ban - Security
  • Cockpit - Port 9090

Docker Containers: 15 total (20 Plane containers removed March 27, 2026)

  • 7 Game Servers (Pterodactyl Wings managed)
  • 8 Firefrost Codex Containers (Dify + Qdrant + n8n + Ollama)

Game Servers on TX1 (11 servers): Updated April 8, 2026

  1. Stoneblock 4 - a0efbfe8 - 38.68.14.26:25565
  2. Society: Sunlit Valley - 9310d0a6 - 38.68.14.28:25565
  3. All The Mons (Private) - TX - 668a5220 - 38.68.14.30:25565
  4. FoundryVTT - 7d8f15a0 - 38.68.14.26:30000
  5. Create Plus (Video Sandbox) - cc170f06 - 38.68.14.26:25566
  6. Vanilla - c4004e2b - 38.68.14.26:25567
  7. Beyond Depth - e95ed4a8 - (port TBD)
  8. Beyond Ascension - 3f842757 - (port TBD)
  9. Wold's Vaults - fcbe0a1d - (port TBD)
  10. Submerged 2 - 576342b8 - (port TBD)
  11. Cottage Witch - 7a9754ad - (port TBD)

Note: Ars Eclectica removed since original audit

NC1 Charlotte (216.239.104.130)

Role: Secondary Game Server Node
Location: Charlotte, NC
Provider: Breezehost (Dedicated Server)
Specs: 251GB RAM, 914GB Disk
Uptime: 46 days, 12:38
Disk Usage: ✅ 8% (69GB/914GB) - EXCELLENT

April 8, 2026: LVM partition expanded from 100GB to 928GB. NC1 had 828GB unallocated in volume group since initial Ubuntu install. Now fully utilizing the 1TB NVMe drive.

IP Subnet: 216.239.104.128/29

  • Primary Node IP: 216.239.104.130
  • Gateway: 216.239.104.129

Services Running:

  • Pterodactyl Wings - Ports 8080 (HTTP), 2022 (SFTP)
  • MariaDB - Port 3306 (localhost)
  • Cockpit - Port 9090

Docker Containers: 6 (all game servers)

Game Servers on NC1 (11 servers): Updated April 8, 2026

  1. All The Mods 10 - 82e63949 - 216.239.104.130:25569
  2. Hytale - 13c80cb8 - 216.239.104.130:5520-5521
  3. All of Create (Creative) - NC - e1c6ff8d - 216.239.104.130:25568
  4. All the Mods 10: To the Sky - f408e832 - 216.239.104.130:25565
  5. All the Mons - c4bc5892 - 216.239.104.130:25566
  6. Mythcraft 5 - b90ced3c - 216.239.104.130:25567
  7. Otherworld [Dungeons & Dragons] - d4798f45 - (port TBD)
  8. DeceasedCraft - 8950fa1e - (port TBD)
  9. Sneak's Pirate Pack - 7c9c2dc0 - (port TBD)
  10. Farm Crossing 5 - 04ac4a1b - (port TBD)
  11. Homestead - A Cozy Survival Experience - f5befeab - (port TBD)

Special Firewall Rules:

  • Allows GRE traffic from Command Center (63.143.34.217) - for potential future tunneling
  • Port 24454/udp open (Simple Voice Chat - Mayview)

🔌 PORT ALLOCATION REGISTRY

Command Center (63.143.34.217)

Port Service Access Protocol
22 SSH Public TCP
80 Nginx (63.143.34.217) Public TCP
80 Nginx (74.63.218.202) Public TCP
443 Nginx (63.143.34.217) Public TCP
443 Nginx (74.63.218.202) Public TCP
3000 Gitea Internal TCP
3001 Uptime Kuma Internal TCP
3306 MySQL Localhost TCP
3500 Discord Bot (The Arbiter) Internal TCP
6379 Redis Localhost TCP
8000 Vaultwarden Docker localhost TCP
8001 Vaultwarden proxy Docker localhost TCP
8080 Code-Server Internal TCP
9090 Cockpit Public TCP

Nginx Virtual Hosts (63.143.34.217:443):

  • git.firefrostgaming.com → 127.0.0.1:3000
  • status.firefrostgaming.com → 127.0.0.1:3001
  • discord-bot.firefrostgaming.com → localhost:3500
  • vault.firefrostgaming.com → 127.0.0.1:8001

Nginx Virtual Hosts (74.63.218.202:443):

  • code.firefrostgaming.com → 127.0.0.1:8080

Ghost VPS (64.50.188.14)

Port Service Access Protocol
22 SSH Public TCP
25 Postfix Localhost TCP
80 Nginx Public TCP
443 Nginx Public TCP
2368 Ghost CMS Localhost TCP
3100 Wiki.js Subscribers Localhost TCP
3101 Wiki.js Staff Localhost TCP
3102 Wiki.js Pokerole Localhost TCP
3306 MySQL Localhost TCP
5432 PostgreSQL Localhost TCP
6379 Redis Localhost TCP
9090 Cockpit Public TCP

Nginx Virtual Hosts:

  • firefrostgaming.com → 127.0.0.1:2368 (Ghost)
  • subscribers.firefrostgaming.com → localhost:3100
  • staff.firefrostgaming.com → localhost:3101
  • pokerole.firefrostgaming.com → localhost:3102
  • downloads.firefrostgaming.com → PHP-FPM (Nextcloud)
  • docs.firefrostgaming.com → (MkDocs - not running currently)

Billing VPS (38.68.14.188)

Port Service Access Protocol
21 vsftpd Public TCP
22 SSH Public TCP
25 Postfix (Docker) Public TCP
80 Nginx Public TCP
110 POP3 (Docker) Public TCP
143 IMAP (Docker) Public TCP
443 Nginx Public TCP
465 SMTPS (Docker) Public TCP
587 Submission (Docker) Public TCP
993 IMAPS (Docker) Public TCP
995 POP3S (Docker) Public TCP
3306 MariaDB Localhost TCP
4190 ManageSieve (Docker) Public TCP
5001 Whitelist Manager Localhost TCP
6379 Redis Localhost TCP
7654 Redis (Docker) Docker localhost TCP
8080 Mailcow Web Public TCP
8443 Mailcow Web SSL Public TCP
9090 Cockpit Public TCP
13306 MySQL (Docker) Docker localhost TCP
19991 Dovecot Stats Docker localhost TCP

Nginx Virtual Hosts:

  • billing.firefrostgaming.com → PHP-FPM (Paymenter)
  • mail.firefrostgaming.com → localhost:8443 (Mailcow)
  • whitelist.firefrostgaming.com → 127.0.0.1:5001

Panel VPS (45.94.168.138)

Port Service Access Protocol
21 vsftpd Public TCP
22 SSH Public TCP
80 Nginx Public TCP
443 Nginx Public TCP
3306 MariaDB Localhost TCP
6379 Redis Localhost TCP
9090 Cockpit Public TCP

Nginx Virtual Hosts:

  • panel.firefrostgaming.com → PHP-FPM (Pterodactyl Panel)

TX1 Dallas (38.68.14.26)

Port Service Access Protocol
22 SSH Public TCP
80 Nginx Public TCP
443 Nginx Public TCP
2022 Wings SFTP Public TCP
3000 Dify Web Docker localhost TCP
5001 Dify API Docker localhost TCP
5520 Game: Ars Eclectica Public TCP/UDP
5678 n8n Docker localhost TCP
6333 Qdrant Vector DB Public TCP
8080 Wings HTTP Public TCP
8090 Plane (Caddy) Public TCP
8444 Plane SSL Public TCP
9090 Cockpit Public TCP
10025 Plane SMTP Public TCP
10465 Plane SMTPS Public TCP
10587 Plane Submission Public TCP
25565 Game: Stoneblock 4 Public (38.68.14.26) TCP/UDP
25566 Game: Create Plus Public (38.68.14.26) TCP/UDP
25567 Game: Vanilla Public (38.68.14.26) TCP/UDP
25565 Game: Society Sunlit Valley Public (38.68.14.28) TCP/UDP
25565 Game: All The Mons Private Public (38.68.14.30) TCP/UDP
30000 FoundryVTT Public (38.68.14.26) TCP/UDP

Nginx Virtual Hosts:

  • codex.firefrostgaming.com → 127.0.0.1:3000 (Dify Web) + 127.0.0.1:5001 (API paths)
  • n8n.firefrostgaming.com → 127.0.0.1:5678
  • tasks.firefrostgaming.com → 127.0.0.1:8090 (Plane)

Docker Internal Services:

  • PostgreSQL (Plane): 5432
  • PostgreSQL (Dify): 5432
  • Redis (Plane): 6379
  • Redis (Dify): 6379
  • RabbitMQ (Plane): 5672, 15672
  • MinIO (Plane): 9000

NC1 Charlotte (216.239.104.130)

Port Service Access Protocol
22 SSH Public TCP
2022 Wings SFTP Public TCP
3306 MariaDB Localhost TCP
5520-5521 Game: Hytale Public TCP/UDP
8080 Wings HTTP Public TCP
9090 Cockpit Public TCP
24454 Simple Voice Chat Public UDP
25565 Game: ATM10 To the Sky Public TCP/UDP
25566 Game: All the Mons Public Public TCP/UDP
25567 Game: Mythcraft 5 Public TCP/UDP
25568 Game: All of Create Public TCP/UDP
25569 Game: All The Mods 10 Public TCP/UDP

🔗 CONNECTIVITY MAP

External Public-Facing Services

Domain → Server → Internal Port → External Port

  1. firefrostgaming.com → Ghost VPS → 2368 → 443 (Nginx SSL)
  2. git.firefrostgaming.com → Command Center → 3000 → 443 (Nginx SSL)
  3. status.firefrostgaming.com → Command Center → 3001 → 443 (Nginx SSL)
  4. code.firefrostgaming.com → Command Center → 8080 → 443 (Nginx SSL, 74.63.218.202)
  5. discord-bot.firefrostgaming.com → Command Center → 3500 → 443 (Nginx SSL)
  6. vault.firefrostgaming.com → Command Center → 8001 → 443 (Nginx SSL)
  7. billing.firefrostgaming.com → Billing VPS → PHP-FPM → 80 (Nginx)
  8. mail.firefrostgaming.com → Billing VPS → 8443 → 443 (Nginx SSL)
  9. whitelist.firefrostgaming.com → Billing VPS → 5001 → 80 (Nginx)
  10. panel.firefrostgaming.com → Panel VPS → PHP-FPM → 443 (Nginx SSL)
  11. codex.firefrostgaming.com → TX1 → 3000/5001 → 443 (Nginx SSL)
  12. n8n.firefrostgaming.com → TX1 → 5678 → 443 (Nginx SSL)
  13. tasks.firefrostgaming.com → TX1 → 8090 → 80 (Nginx)
  14. downloads.firefrostgaming.com → Ghost VPS → PHP-FPM → 443 (Nginx SSL, Nextcloud)
  15. subscribers.firefrostgaming.com → Ghost VPS → 3100 → 80 (Nginx)
  16. staff.firefrostgaming.com → Ghost VPS → 3101 → 80 (Nginx)
  17. pokerole.firefrostgaming.com → Ghost VPS → 3102 → 80 (Nginx)

Server-to-Server Communication

Panel VPS (45.94.168.138) ↔ Wings Nodes:

  • Panel → TX1 (38.68.14.26:8080) - Wings API
  • Panel → NC1 (216.239.104.130:8080) - Wings API
  • Protocol: HTTPS (Wings API)
  • Authentication: API tokens
  • Purpose: Server management, monitoring, console access

Discord Bot (Command Center) → Discord API:

  • discord-bot.firefrostgaming.com (63.143.34.217:3500) → Discord.com:443
  • Protocol: HTTPS + WebSocket
  • Purpose: Bot commands, role management, webhooks

Paymenter (Billing VPS) → Pterodactyl Panel:

  • Planned webhook: billing.firefrostgaming.com → panel.firefrostgaming.com
  • Protocol: HTTPS
  • Purpose: Subscription provisioning automation

Paymenter (Billing VPS) → Discord Bot:

  • Planned webhook: billing.firefrostgaming.com → discord-bot.firefrostgaming.com/webhook/paymenter
  • Protocol: HTTPS
  • Purpose: Subscription event notifications for role assignment

Whitelist Manager (Billing VPS) → Pterodactyl Panel:

  • whitelist.firefrostgaming.com (38.68.14.188:5001) → panel.firefrostgaming.com (45.94.168.138)
  • Protocol: HTTPS (Panel API)
  • Purpose: Whitelist synchronization

n8n (TX1) → External Services:

  • n8n.firefrostgaming.com → Various APIs (GitHub, Discord, etc.)
  • Protocol: HTTPS
  • Purpose: Workflow automation

Gitea (Command Center) → Git Clients:

  • git.firefrostgaming.com → Various (Claude, developers, CI/CD)
  • Protocol: HTTPS + SSH (port 22)
  • Purpose: Git repository access

Database Connections (Internal Only)

Command Center:

  • Gitea → MySQL (127.0.0.1:3306)
  • Vaultwarden → Internal SQLite

Ghost VPS:

  • Ghost CMS → MySQL (127.0.0.1:3306)
  • Wiki.js (3x) → PostgreSQL (127.0.0.1:5432)
  • All services → Redis (127.0.0.1:6379) for caching

Billing VPS:

  • Paymenter → MariaDB (127.0.0.1:3306)
  • Paymenter → Redis (127.0.0.1:6379)
  • Mailcow → Docker MySQL (172.22.1.x:3306)
  • Mailcow → Docker Redis (172.22.1.x:6379)

Panel VPS:

  • Pterodactyl Panel → MariaDB (127.0.0.1:3306)
  • Pterodactyl Panel → Redis (127.0.0.1:6379)

TX1 Dallas:

  • Plane → Docker PostgreSQL (internal)
  • Plane → Docker Redis (internal)
  • Dify → Docker PostgreSQL (internal)
  • Dify → Docker Redis (internal)
  • Dify → Qdrant (127.0.0.1:6333)

NC1 Charlotte:

  • Wings → MariaDB (127.0.0.1:3306)

🎯 AUTHENTICATION & DEPENDENCY FLOWS

OAuth2 Flows

Discord Bot Admin Panel:

  • User → discord-bot.firefrostgaming.com → Discord OAuth2 → Whitelist check → Session
  • Dependencies: Discord API availability, Session storage (Express sessions)

API Token Flows

Pterodactyl Panel ↔ Wings:

  • Panel stores Wings API tokens
  • Wings validates tokens on each request
  • Critical: Token compromise = full server control

Gitea API:

  • Claude sessions use: e0e330cba1749b01ab505093a160e4423ebbbe36
  • Operations manual automation
  • Critical: Full admin access token

n8n Workflows:

  • Various API tokens stored in n8n credentials
  • Discord webhooks, GitHub, etc.

SMTP Flows (Email)

Ghost VPS (Postfix):

  • Status: ⚠️ BLOCKED - Inbound port 25 blocked at provider level
  • Workaround Needed: Provider support ticket
  • Current: Internal mail only

Billing VPS (Mailcow):

  • Status: ✅ OPERATIONAL
  • SMTP out: 587 (submission), 465 (SMTPS), 25 (relay)
  • IMAP: 143, 993 (SSL)
  • POP3: 110, 995 (SSL)
  • DKIM/SPF/DMARC: Configured for firefrostgaming.com

TX1 (Plane):

  • Status: ✅ OPERATIONAL
  • Internal SMTP for Plane notifications (ports 10025, 10465, 10587)

⚠️ SINGLE POINTS OF FAILURE

Critical Single Points

  1. Pterodactyl Panel (45.94.168.138)

    • Risk: Panel down = no game server management
    • Mitigation: Wings nodes continue running autonomously
    • Recovery Time: ~30 minutes (restore from backup + DNS)

  2. Mailcow (Billing VPS)

    • Risk: Email down = no subscription confirmations, no support tickets
    • Mitigation: Cloudflare Email Routing as backup?
    • Recovery Time: ~2 hours (Mailcow stack restoration)

  3. Gitea (Command Center)

    • Risk: Git down = no deployments, no operations manual access
    • Mitigation: Local clones exist on developer machines
    • Recovery Time: ~1 hour (service restart or VM restore)

  4. Ghost CMS (Ghost VPS)

    • Risk: Main website down = no public presence
    • Mitigation: Cloudflare caching provides limited read access
    • Recovery Time: ~1 hour (Ghost restart or data restore)

  5. Command Center Server (63.143.34.217)

    • Risk: Multiple critical services (Gitea, Uptime Kuma, Discord Bot, Vaultwarden)
    • Impact: Most critical - affects development, monitoring, and Discord automation
    • Mitigation: Distributed services across multiple VPS in future
    • Recovery Time: 2-4 hours (depends on failure type)

Non-Critical Single Points

  1. Billing VPS (38.68.14.188)

    • Services: Paymenter, Mailcow, Whitelist Manager
    • Impact: Financial operations halted, but game servers continue
    • Note: High disk usage (70%) increases risk

  2. Ghost VPS (64.50.188.14)

    • Services: Ghost, Wiki.js (3x), Nextcloud
    • Impact: Documentation inaccessible, but operations continue
    • Note: Can be restored from backups

🔥 PORT CONFLICT PREVENTION

Port Allocation Strategy

Reserved Ranges:

  • 25565-25580: Minecraft game servers (TCP/UDP)
  • 5520-5521: Hytale (TCP/UDP)
  • 30000-30010: Reserved for FoundryVTT and future VTT instances
  • 3000-3200: Internal web services (Gitea, Uptime Kuma, Wiki.js, etc.)
  • 8000-9000: Docker services and Wings
  • 10000-11000: Plane/n8n/Dify internal services

Conflict Lessons Learned

The Arbiter Bot Port Hunt (March 27, 2026):

  1. Attempted port 3000 → CONFLICT (Gitea on TX1 Dify)
  2. Attempted port 3001 → CONFLICT (Uptime Kuma)
  3. SUCCESS: Port 3500 (unused)

Prevention Going Forward:

  • Always check ss -tlnp | grep LISTEN before deploying
  • Document port assignments in this registry
  • Use high-numbered ports (3500+) for new services on shared servers
  • Consider port range 4000-5000 for future Discord/webhook services

Available Port Ranges

Command Center (63.143.34.217):

  • ✅ 3500-4000: Available
  • ✅ 4000-6000: Available (except 6379 Redis)
  • ✅ 7000-8000: Available (except 8000-8001 Vaultwarden)

Ghost VPS (64.50.188.14):

  • ✅ 3200-6000: Available (except 3306 MySQL, 5432 PostgreSQL)
  • ✅ 7000-9000: Available

Billing VPS (38.68.14.188):

  • ⚠️ Most standard ports occupied by Mailcow
  • ✅ 5100-6000: Available (except 5001 Whitelist Manager)
  • ✅ 9100-10000: Available

Panel VPS (45.94.168.138):

  • ✅ 1024-3000: Available
  • ✅ 3500-6000: Available (except 3306 MySQL, 6379 Redis)
  • ✅ 7000-9000: Available

TX1 Dallas (38.68.14.26):

  • ⚠️ Heavy Docker usage, internal ports dynamic
  • ✅ 3500-5000: Available (except 5001 Dify, 5678 n8n)
  • ✅ 7000-8000: Available
  • ✅ 11000-20000: Available

NC1 Charlotte (216.239.104.130):

  • ✅ 3000-5000: Available (except 3306 MySQL)
  • ✅ 6000-8000: Available
  • ✅ 10000-20000: Available

📊 RESOURCE UTILIZATION

Disk Usage Status

Server Used Total Usage % Status
Command Center 17GB 38GB 45% ✅ Good
Ghost VPS 21GB 38GB 55% ✅ Good
Billing VPS 13GB 19GB 70% ⚠️ Monitor
Panel VPS 9GB 24GB 39% ✅ Good
TX1 Dallas 102GB 911GB 12% ✅ Excellent
NC1 Charlotte 61GB 98GB 66% ⚠️ Monitor

Recommendations:

  1. Billing VPS: Review Mailcow logs and docker volume sizes - consider cleanup or expansion
  2. NC1 Charlotte: Monitor game server world sizes - implement world pruning or expansion
  3. TX1 Dallas: Massive capacity available - can host additional services

Service Load Distribution

Command Center: 33 systemd services (6 critical)
Ghost VPS: 31 systemd services (5 critical)
Billing VPS: 30 systemd services + 18 Docker containers
Panel VPS: 28 systemd services (clean, focused)
TX1 Dallas: 29 systemd services + 35 Docker containers (heavy)
NC1 Charlotte: 25 systemd services + 6 Docker containers (focused)

🔐 FIREWALL ANALYSIS

Command Center UFW Rules

  • ✅ SSH (22) open
  • ✅ HTTP/HTTPS (80/443) on both IPs
  • ✅ Cockpit (9090) open
  • ✅ Specific IP bindings for services (63.143.34.217 vs 74.63.218.202)

Ghost VPS

  • ⚠️ Firewall audit returned "ERROR: You need to be root" (was logged in as architect)
  • Action Required: Re-audit as root to verify rules

Billing VPS IPTables

  • ✅ Custom Mailcow chain (MAILCOW)
  • ✅ UFW chains present
  • ✅ Docker chains for container networking

Panel VPS UFW Rules

  • ✅ SSH (22), HTTP (80), HTTPS (443) open
  • ✅ FTP (21) open for vsftpd
  • ✅ Cockpit (9090) open
  • ✅ Specific allow from 141.98.74.95 (related system?)

TX1 Dallas UFW Rules

  • ✅ Wings ports (8080, 2022) open
  • ✅ Minecraft port range (25565-25580) TCP+UDP
  • ✅ Hytale ports (5520-5521) TCP+UDP
  • ✅ n8n webhook port (5678)
  • ✅ Cockpit (9090) open
  • ✅ Allow 74.63.218.205 HTTP/HTTPS (Code-Server IP?)

NC1 Charlotte UFW Rules

  • ✅ Wings ports (8080, 2022) open
  • ✅ Minecraft port range (25565-25580) TCP+UDP
  • ✅ Hytale ports (5520-5521) TCP+UDP
  • ✅ Simple Voice Chat (24454 UDP)
  • ✅ GRE protocol (47) open - for future tunneling
  • ✅ Special: Full allow from Command Center IP (63.143.34.217) + GRE
  • ✅ Cockpit (9090) open

🎮 GAME SERVER MAPPING

TX1 Dallas Game Servers (11 servers) — Updated April 8, 2026

Server Name UUID (short) IP:Port Status
Stoneblock 4 a0efbfe8 38.68.14.26:25565 Active
Society: Sunlit Valley 9310d0a6 38.68.14.28:25565 Active
All The Mons (Private) - TX 668a5220 38.68.14.30:25565 Active
FoundryVTT 7d8f15a0 38.68.14.26:30000 Active
Create Plus (Video Sandbox) cc170f06 38.68.14.26:25566 Active
Vanilla c4004e2b 38.68.14.26:25567 Active
Beyond Depth e95ed4a8 TBD Active
Beyond Ascension 3f842757 TBD Active
Wold's Vaults fcbe0a1d TBD Active
Submerged 2 576342b8 TBD Active
Cottage Witch 7a9754ad TBD Active

Note: Ars Eclectica removed since original audit

NC1 Charlotte Game Servers (11 servers) — Updated April 8, 2026

Server Name UUID (short) IP:Port Status
All The Mods 10 82e63949 216.239.104.130:25569 Active
Hytale 13c80cb8 216.239.104.130:5520-5521 Active
All of Create (Creative) - NC e1c6ff8d 216.239.104.130:25568 Active
All the Mods 10: To the Sky f408e832 216.239.104.130:25565 Active
All the Mons c4bc5892 216.239.104.130:25566 Active
Mythcraft 5 b90ced3c 216.239.104.130:25567 Active
Otherworld [Dungeons & Dragons] d4798f45 TBD Active
DeceasedCraft 8950fa1e TBD Active
Sneak's Pirate Pack 7c9c2dc0 TBD Active
Farm Crossing 5 04ac4a1b TBD Active
Homestead - A Cozy Survival Experience f5befeab TBD Active

Total: 22 game servers (20 Minecraft + 1 Hytale + 1 FoundryVTT)

🚨 ISSUES IDENTIFIED

Warning Issues

  1. Billing VPS Disk Usage: 70%

    • Risk: May hit capacity during high email volume
    • Action: Review Mailcow container logs and volumes
    • Timeline: Monitor weekly, expand if hits 80%

  2. NC1 Charlotte Disk Usage: 66% ✅ RESOLVED April 8, 2026

    • Resolution: LVM partition expanded from 100GB to 928GB
    • New Status: 8% usage (69GB/914GB) - EXCELLENT

  3. Ghost VPS Firewall Not Audited

    • Risk: Unknown firewall state (audit failed due to permissions)
    • Action: Re-run audit as root
    • Timeline: Next maintenance window

📈 CAPACITY PLANNING

Short-Term Capacity (Next 3 Months)

Can Accommodate:

  • ✅ 5-10 more game servers on TX1 (plenty of disk + RAM)
  • ✅ 10+ more game servers on NC1 (807GB free after April 8 expansion)
  • ✅ Additional web services on Command Center
  • ✅ Additional web services on Ghost VPS
  • ⚠️ Limited capacity on Billing VPS (disk constraint)

Cannot Accommodate Without Expansion:

  • ❌ Additional Docker stacks on Billing VPS (disk full)

Long-Term Recommendations

  1. Expand Billing VPS Disk

    • Current: 19GB
    • Recommended: 40-50GB
    • Reason: Mailcow + Paymenter + future growth

  2. Expand NC1 Disk ✅ COMPLETED April 8, 2026

    • Expanded: 100GB → 928GB (LVM resize)
    • Now: 914GB usable, 807GB free

  3. Consider Backup Server

    • Add dedicated backup VPS
    • Offload backups from game server disks
    • Enable disaster recovery

  4. Load Balancer for Web Services

    • Multiple Ghost CMS instances
    • Distribute SSL termination
    • Improve resilience

🔄 INTERCONNECTION SUMMARY

Data Flow Patterns

User → Website (Ghost CMS)

  1. User → Cloudflare → Ghost VPS:443
  2. Nginx → Ghost:2368
  3. Ghost → MySQL:3306

User → Panel (Pterodactyl)

  1. User → Cloudflare → Panel VPS:443
  2. Nginx → PHP-FPM → Panel Application
  3. Panel → MariaDB:3306
  4. Panel → Wings API (TX1:8080, NC1:8080)

User → Game Server

  1. User → TX1/NC1 direct (no proxy)
  2. Game Server → Wings → Panel (monitoring/console)

Discord Bot Workflow

  1. Discord API → discord-bot.firefrostgaming.com:443
  2. Nginx → Bot:3500
  3. Bot → Discord API (outbound)
  4. Bot → (future) Paymenter webhook

Subscription Workflow (Planned)

  1. User → Paymenter (billing.firefrostgaming.com)
  2. Paymenter → Stripe/PayPal API
  3. Paymenter webhook → Discord Bot
  4. Discord Bot → Discord API (assign role)
  5. Discord Bot → (future) Panel API (provision server)

📝 RECOMMENDATIONS

Immediate Actions (Next 7 Days)

  1. ✅ Complete this audit document
  2. ✅ Submit Breezehost ticket for Ghost VPS port 25 (Already resolved)
  3. ✅ Decommission Plane stack on TX1 Dallas - COMPLETE (March 27, 2026)
  4. ⚠️ Re-audit Ghost VPS firewall as root
  5. ✅ Document port allocation strategy in operations manual

Short-Term Actions (Next 30 Days)

  1. ⚠️ Review Billing VPS disk usage, plan expansion if needed
  2. ✅ Monitor NC1 disk usage weekly RESOLVED - Expanded April 8, 2026
  3. ✅ Implement automated disk usage alerting (Uptime Kuma?)
  4. ✅ Configure Paymenter → Discord Bot webhooks
  5. ✅ Test full subscription provisioning flow

Long-Term Actions (Next 90 Days)

  1. 🔄 Implement backup server or backup strategy
  2. 🔄 Consider load balancer for web services
  3. 🔄 Evaluate Gitea high-availability options
  4. ✅ Plan for TX1/NC1 disk expansion schedule NC1 DONE April 8, 2026

🎯 AUDIT COMPLETION

Audit Status: ✅ COMPLETE
Data Collection: March 27, 2026
Servers Audited: 6/6 (100%)
Document Version: 1.0
Next Audit: Recommended every 6 months or after major infrastructure changes

Compiled By: Chronicler #43
Reviewed By: (Pending Michael's review)
Committed To: firefrost-operations-manual repository

Fire + Frost + Foundation = Where Love Builds Legacy 💙🔥❄️

Reference in New Issue View Git Blame Copy Permalink