Files
firefrost-operations-manual/docs/core/infrastructure-manifest.md
Claude (Chronicler #61) 8eff3afa54 docs: Add Arbiter database connection info to infrastructure manifest
Added complete Arbiter 3.0 / Trinity Console section to Command Center:
- Database connection details (host, port, db name, user)
- Migration command pattern (PGPASSWORD approach)
- Deploy pattern for Arbiter updates
- Password reference to Vaultwarden (not plaintext)

This was missing and caused confusion during Task #94 deployment.

Signed-off-by: Claude (Chronicler #61) <claude@firefrostgaming.com>
2026-04-05 10:01:17 +00:00

24 KiB
Raw Blame History

Firefrost Gaming: Infrastructure Manifest

Phase 0.5+: Management Services Complete

Current Version: v2.1

Last Updated: 2026-04-04


1. Management Services (VPS Tier)

Service Location IP Domain Status
Main Website Cloudflare Pages N/A firefrostgaming.com OPERATIONAL
Decap CMS Cloudflare Pages N/A firefrostgaming.com/admin/ OPERATIONAL
Gitea Command Center 63.143.34.217 git.firefrostgaming.com OPERATIONAL
Uptime Kuma Command Center 63.143.34.217 status.firefrostgaming.com OPERATIONAL
Automation Command Center 63.143.34.217 N/A OPERATIONAL
MkDocs Wiki VPS 64.50.188.14 docs.firefrostgaming.com OPERATIONAL
Code-Server Command Center 63.143.34.217 code.firefrostgaming.com OPERATIONAL
Wiki.js (Sub) Wiki VPS 64.50.188.14 subscribers.firefrostgaming.com OPERATIONAL
Wiki.js (Staff) Wiki VPS 64.50.188.14 staff.firefrostgaming.com OPERATIONAL
Wiki.js (Pokerole) Wiki VPS 64.50.188.14 pokerole.firefrostgaming.com OPERATIONAL
NextCloud Wiki VPS 64.50.188.14 downloads.firefrostgaming.com OPERATIONAL
Vaultwarden Command Center 63.143.34.217 vault.firefrostgaming.com DEPLOYED - Needs Config
Whitelist Manager Services VPS 38.68.14.188 whitelist.firefrostgaming.com 🚀 READY TO DEPLOY
Mailcow Services VPS 38.68.14.188 mail.firefrostgaming.com OPERATIONAL
Netdata Multi-server Various N/A (port 19999) 📋 PLANNED
FoundryVTT TX1 Dallas 38.68.14.26 foundry.firefrostgaming.com OPERATIONAL

Website Infrastructure (firefrostgaming.com)

Stack:

  • Frontend: 11ty (Eleventy) static site generator
  • Source Control: Gitea (git.firefrostgaming.com/firefrost-gaming/firefrost-website)
  • Mirror: GitHub (github.com/Frostystyle/firefrost-website)
  • Hosting: Cloudflare Pages (firefrost-website.pages.dev)
  • SSL: Cloudflare managed
  • Auto-Deploy: Gitea → GitHub → Cloudflare Pages (≈60 seconds)

Migration Details:

  • From: Ghost CMS (self-hosted on Wiki VPS)
  • To: 11ty + Cloudflare Pages
  • Date: April 2, 2026 (7:06 PM - 7:57 PM CDT)
  • Duration: 51 minutes
  • Chronicler: #55
  • Documentation: docs/sessions/2026-04-02-ghost-to-11ty-migration-COMPLETE.md
  • Reason: Simpler architecture, edge network performance, eliminate self-hosted CMS

Decap CMS (Trinity Console for Documentation)

URL: https://firefrostgaming.com/admin/

Purpose: Web-based editor for Firefrost Operations Manual documentation. Allows Meg and Holly to edit markdown files without touching code.

Stack:

  • CMS: Decap CMS (static React app)
  • Authentication: Gitea OAuth with PKCE flow
  • Backend Repo: firefrost-gaming/firefrost-operations-manual
  • Hosted On: Cloudflare Pages (same as main website)

OAuth App:

  • Client ID: ad439d72-e724-4f88-ad24-a1187c52b313
  • Redirect URI: https://firefrostgaming.com/admin/

Collections:

  • Core Documents (SESSION-HANDOFF-NEXT.md, tasks.md)
  • Planning Docs (docs/planning/)
  • Milestones (docs/milestones/)
  • Implementation Guides (docs/implementation/)

Deployment: April 3, 2026 by Chronicler #58

Retired Services (April 2026)

Ghost CMS:

  • Status: RETIRED (April 2, 2026)
  • Former Location: Wiki VPS (64.50.188.14)
  • Replaced By: 11ty + Cloudflare Pages
  • Service Stopped: April 2, 2026
  • Data Preserved: All content migrated to 11ty

Paymenter:

  • Status: RETIRED (April 3, 2026)
  • Former Location: Services VPS (38.68.14.188)
  • Replaced By: Direct Stripe integration in Trinity Console + Arbiter 3.0
  • Service Stopped: April 3, 2026
  • Reason: Architectural simplification, eliminate webhook middleman, Discord OAuth linking

1.5 Automation Systems

Deployed on Command Center (63.143.34.217):

System Status Schedule Purpose
Staggered Server Restart 🚀 READY Daily 4:00 AM Automated server restarts with player warnings
World Backup Automation 🚀 READY Daily 3:30 AM Automated world backups to NextCloud
Frostwall Monitor 📋 PLANNED Every 5 min GRE tunnel health monitoring

Notes:

  • Restart system runs AFTER backups (3:30 AM backup, 4:00 AM restart)
  • Both systems use Discord webhooks for notifications
  • Python scripts with systemd/cron scheduling

2. Core Services Hierarchy (VPS Tier)

Friendly Name Service Primary IP Status Role
Panel Pterodactyl Panel 45.94.168.138 Verified Control Plane (Charlotte, NC)
Command Center Management Hub 63.143.34.217 Operational Primary Gateway + Tools + Trinity Console
Services Mailcow/Whitelist 38.68.14.188 Verified Email + Utilities (Chicago, IL)
Wiki Documentation Cluster 64.50.188.14 Verified Wiki.js + Internal Tools (Chicago, IL)
Dev Development/Testing 64.50.188.128 OPERATIONAL Flexible Development Server (Chicago, IL)

Command Center Details

  • Primary IP: 63.143.34.217
  • Services: Gitea, Uptime Kuma, Code-Server, Automation daemon, Arbiter 3.0, Trinity Console
  • Node.js: 20.20.0
  • Nginx: Reverse proxy for all services

Arbiter 3.0 / Trinity Console

  • Path: /opt/arbiter-3.0
  • Service: arbiter-3 (systemctl)
  • Domain: discord-bot.firefrostgaming.com
  • Admin Panel: discord-bot.firefrostgaming.com/admin

Database Connection:

  • Host: 127.0.0.1
  • Port: 5432
  • Database: arbiter_db
  • User: arbiter
  • Password: (in Vaultwarden under "Arbiter PostgreSQL")

Running Migrations:

# From Command Center, run migration files directly:
PGPASSWORD='[password]' psql -U arbiter -d arbiter_db -h 127.0.0.1 -f /path/to/migration.sql

# Or connect interactively:
PGPASSWORD='[password]' psql -U arbiter -d arbiter_db -h 127.0.0.1

Deploy Pattern:

cd /tmp
git clone https://git.firefrostgaming.com/firefrost-gaming/firefrost-services.git
# Copy specific files to /opt/arbiter-3.0
systemctl restart arbiter-3

Panel Details

Dev VPS Details

  • Primary IP: 64.50.188.128
  • Gateway: 64.50.188.1
  • Netmask: 255.255.255.0
  • DNS Resolvers: 1.1.1.1 (Primary), 8.8.8.8 (Secondary)
  • Interface: ens3
  • MAC Address: 00:6C:80:0E:FB:72
  • Network Speed: 1 Gbps (inbound/outbound)
  • IPv6: Disabled (not assigned)
  • Provider: Breezehost
  • Location: Chicago, IL (same as Ghost VPS)
  • Specs: AMD Epyc Cloud-2 (2 CPU, 4GB RAM, 80GB NVMe)
  • OS: Ubuntu Server 24.04 LTS (Noble Numbat) Minimal
  • Swap: 512 MB
  • Cost: $10/month
  • Deployed: April 1, 2026
  • Deployed by: Chronicler #51
  • Purpose: Development, testing, experimentation, CI/CD pipeline testing

Network Configuration:

  • Public network only (no private network)
  • Firewall: UFW (to be configured)
  • VNC: Disabled
  • Cockpit: To be installed (port 9090)

Planned Services:

  • Docker (for containerized testing)
  • Node.js (latest LTS)
  • Git (for development)
  • Cockpit web terminal
  • Development testing environment
  • CI/CD experimentation

Breezehost Locked-In Pricing (Reference for Future Expansion):

  • AMD Epyc Cloud-1: $7/month (1 CPU, 2GB RAM, 40GB NVMe, 2 IPv4)
  • AMD Epyc Cloud-2: $10/month (2 CPU, 4GB RAM, 80GB NVMe, 4 IPv4) ← Current Dev VPS
  • AMD Epyc Cloud-4: $17/month (4 CPU, 8GB RAM, 160GB NVMe, 8 IPv4)
  • AMD Epyc Cloud-12: $33/month (8 CPU, 12GB RAM, 240GB NVMe, 16 IPv4)
  • AMD Epyc Cloud-16: $49/month (12 CPU, 16GB RAM, 320GB NVMe, 24 IPv4)
  • AMD Epyc Cloud-24: $59/month (16 CPU, 24GB RAM, 480GB NVMe, 32 IPv4)
  • AMD Epyc Cloud-32: $89/month (24 CPU, 32GB RAM, 640GB NVMe, 32 IPv4)
  • Cloud GPU RTX4090: $250/month (Maximum CPU, 128GB RAM, 250GB NVMe, GPU)

Note: These are Michael's locked-in rates with Breezehost. Use this pricing when evaluating infrastructure expansion, additional VPS needs, or capacity upgrades. Current market rates may be higher for new customers.

Service: Pterodactyl Panel (Development/Testing - NOT production)

Primary Purpose:

  • Blueprint extension development and testing
  • Panel update validation before production deployment
  • Safe experimentation environment for new configurations
  • Training ground for Trinity members

Flexible Conversion Options: Development infrastructure that adapts to evolving needs. This server can be repurposed as mission requirements change:

  • Alternative Use #1: Additional Wings node (game server capacity expansion)
  • Alternative Use #2: Staging environment for Ghost CMS/service updates
  • Alternative Use #3: CI/CD pipeline server (automated testing)
  • Alternative Use #4: Backup Panel server (failover redundancy)
  • Alternative Use #5: Dedicated hosting for commercial products
  • Alternative Use #6: Additional application server (Wiki.js, etc.)

Key Characteristic: Not locked to single purpose - can be converted based on operational needs without affecting production infrastructure.

Why Not Test on Production Panel:

  • One bad database query = panel lockup
  • Risk to 11 production game servers
  • Potential data corruption in production MySQL
  • Blueprint extensions require isolated testing environment

Current Projects:

  • Modpack Version Checker (commercial Blueprint extension, $1k-6.7k/year projected revenue)
  • Panel v1.13.x upgrade testing (before production deployment)

Setup Standards:

  • SSH access for Trinity (architect user)
  • Cockpit monitoring installed
  • UFW firewall configured
  • Pterodactyl Panel + Blueprint Framework
  • Standard security hardening per FFG infrastructure standards
  • Regular backups (lower priority than production servers)

2.5 Breezehost Dedicated Server Pricing (Locked-In Rates)

Current Dedicated Servers in Production:

  • TX1 Dallas: Dual Silver 4116 (24 cores/48 threads, 128GB RAM, 3.4TB NVMe) - $199/month
  • NC1 Charlotte: AMD 9950x 128GB RAM (16 cores/32 threads, 128GB RAM, 3.6TB NVMe) - $259/month

Available Dedicated Server Options (Reference for Future Expansion):

Server Model Price/Month CPU RAM Storage Network
AMD Mystery Box $99 AMD (varies) Varies Varies Premium
Intel Mystery Box $99 Intel (varies) Varies Varies Premium
AMD 7950x - 128GB RAM $199 16c/32t (7950x) 128GB Varies Premium
Dual Silver 4116 $199 24c/48t (Dual 4116) 128GB 7.4TB NVMe Premium
AMD 9950x - 128GB RAM $259 16c/32t (9950x) 128GB 3.6TB NVMe Premium
Dual Gold 6142 $289 32c/64t (Dual 6142) 256GB 7.4TB NVMe Premium
RTX 4090 Dedicated Server $300 32c/64t 128GB 4TB NVMe GPU
AMD 9950x - 192GB RAM $329 16c/32t (9950x) 192GB 3.6TB NVMe Premium
8x 5000 Bare Metal $3,600 32c/64t 256GB 100TB NVMe Bare Metal

Mystery Box Details:

  • Random hardware assignment (AMD or Intel)
  • Significant discount vs specified hardware
  • Good for development/testing workloads
  • Performance varies by assignment

Note: These are Michael's locked-in rates with Breezehost. Use this pricing when evaluating:

  • Additional game server capacity (3rd dedicated node?)
  • GPU workloads (AI generation, rendering)
  • High-performance computing needs
  • Bare metal requirements
  • Cost comparisons with other providers

Current Utilization:

  • TX1 Dallas: 6 game servers + FoundryVTT
  • NC1 Charlotte: 7 game servers
  • Total: 13 game servers across 2 dedicated nodes

Future Expansion Scenarios:

  • Add TX2 or NC2 for geographic load balancing ($199-259/month)
  • GPU node for AI/rendering workloads ($300/month vs $250 VPS GPU)
  • Mystery Box for additional dev/staging capacity ($99/month)

IMPORTANT: The pricing above reflects RENTAL rates. See Section 2.6 for RTO (Rent-to-Own) analysis and why ownership is preferred for dedicated infrastructure.


2.6 RTO (Rent-to-Own) Philosophy & Financial Analysis

Current Owned Hardware Status

TX1 Dallas:

  • Hardware: Dual Intel Xeon Silver 4116 (24c/48t, 128GB RAM, 3.4TB NVMe)
  • Acquisition: RTO completed June 2024
  • Current cost: $80/month (colocation only)
  • Rental equivalent: $199/month
  • Monthly savings: $119/month
  • Estimated current value: $1,500 (conservative)

NC1 Charlotte:

  • Hardware: AMD Ryzen 9 9950x (16c/32t, 128GB RAM, 3.6TB NVMe)
  • Acquisition: RTO completed June 2024
  • Current cost: $80/month (colocation only)
  • Rental equivalent: $259/month
  • Monthly savings: $179/month
  • Estimated current value: $1,800 (conservative)

Combined Owned Assets:

  • Hardware value: ~$3,300 (conservative) to $4,550 (optimistic)
  • Realistic estimate: $3,800-4,000
  • Monthly colocation: $160/month
  • Rental equivalent: $458/month
  • Monthly savings: $298/month

ROI Analysis (22 Months of Ownership)

Since June 2024 (RTO completion):

  • Months owned: 22 (as of March 2026)
  • Cumulative savings: $6,556 (22 × $298)
  • Hardware asset value: $4,000
  • Total equity position: $10,556

If renting instead (same 22 months):

  • Money spent: $10,076 (22 × $458)
  • Assets owned: $0
  • Equity: $0

Net advantage of ownership: $10,556

Long-Term Projection

Year 3 (next 12 months):

  • Additional savings: $3,576
  • Hardware depreciation: ~$500
  • Equity position: $14,056

Year 5 (48 months owned):

  • Cumulative savings: $14,304
  • Hardware value: ~$2,500
  • Total equity: $16,804

Break-even achieved: Month 14 (savings exceeded hardware depreciation)

Why RTO is Preferred for Dedicated Infrastructure

Rental model:

  • Monthly expense forever
  • No equity building
  • Subject to price increases
  • Nothing to show after 5 years

RTO model:

  • Build equity while paying
  • Massive savings after ownership
  • Hardware asset remains
  • Insulation from price increases
  • Can resell, repurpose, or continue using

Michael's Strategy: "Always keeping an eye out for RTO deals with Breeze" - prioritize ownership over rental for long-term infrastructure.

RTO vs Purchase vs Rental Decision Matrix

When to choose RTO:

  • Long-term capacity need (2+ years)
  • Known workload requirements
  • Dedicated hardware specifications needed
  • Budget allows for slightly higher initial payments
  • Building permanent infrastructure

When to choose Rental:

  • Short-term need (<12 months)
  • Uncertain workload requirements
  • Testing new services/markets
  • Flexibility more important than cost
  • VPS-tier capacity sufficient

When to choose Outright Purchase:

  • Cash flow allows
  • Immediate ownership desired
  • Maximum long-term savings
  • Used hardware market opportunity

Future RTO Opportunities to Monitor

Potential expansion via RTO:

  • 3rd dedicated node (TX2/NC2) for geographic load balancing
  • GPU server for AI workloads (if RTO available)
  • High-core-count server for future scaling
  • Backup/failover dedicated node

Target RTO scenarios:

  • Breezehost promotions/special offers
  • End-of-generation hardware (new gen released, previous gen RTO deals)
  • Used/refurbished dedicated servers (lower RTO payments)
  • Bundle deals (multiple servers, lower per-unit cost)

RTO Deal Evaluation Criteria:

  1. Monthly payment vs rental equivalent
  2. Ownership timeline (24 months preferred max)
  3. Hardware specs vs current/future needs
  4. Post-ownership colocation cost
  5. Total cost of ownership vs 3-year rental

Current Infrastructure Costs (Actual)

Owned Hardware (Colocation):

  • TX1 Dallas: $80/month
  • NC1 Charlotte: $80/month
  • Subtotal: $160/month

Rented VPS:

  • Command Center: $10/month
  • Ghost Machine: $10/month
  • Billing Panel: $7/month
  • Panel Machine: $10/month
  • Subtotal: $37/month

Other:

  • Additional IPs: $10/month
  • thenerdsmiths.com domain: $10/month
  • Subtotal: $20/month

Total Current Infrastructure: $217/month

After Dev VPS added (Wednesday): $227/month

Rental equivalent cost (if not owned): $495+/month
Current savings from ownership: $268+/month


3. NC1 - Charlotte Node Mapping

  • Friendly Name: NC1
  • Primary Node IP: 216.239.104.130
  • Gateway: 216.239.104.129
  • Subnet: 216.239.104.128/29 (Mask: 255.255.255.248)
  • Routing: Reachable from Command Center (resolved Feb 2026)
Server Name IP Address Port UUID
The Ember Project 216.239.104.130 25565 124f9060-58a7-457a-b2cf-b4024fce2951
Minecolonies: Create and Conquer 216.239.104.130 25566 a14201d2-83b2-44e6-ae48-e6c4cbc56f24
All The Mods 10 216.239.104.130 25569 82e63949-8fbf-4a44-b32a-53324e8492bf
EMC Subterra Tech 216.239.104.130 25571 09a95f38-9f8c-404a-9557-3a7c44258223
Homestead 216.239.104.130 25572 2f85d4ef-aa49-4dd6-b448-beb3fca1db12
Hytale 216.239.104.130 5520 13c80cb8-f6f8-4bfe-9cdb-823d7e951584
Mayview 216.239.104.130 25567 29e6d709-771b-43f1-aa89-be5dda0d4c22

Total NC1 Servers: 7


4. TX1 - Dallas Node Mapping

  • Friendly Name: TX1
  • Primary Node IP: 38.68.14.26
  • Gateway: 38.68.14.25
  • Subnet: 38.68.14.24/29 (Mask: 255.255.255.248)
  • Status: Game servers ONLY (management services on VPS tier)
Server Name IP Address Port UUID
Stoneblock 4 38.68.14.26 25565 a0efbfe8-4b97-4a90-869d-ffe6d3072bd5
Reclamation 38.68.14.27 25565 1eb33479-a6bc-4e8f-b64d-d1e4bfa0a8b4
Society: Sunlit Valley 38.68.14.28 25565 9310d0a6-62a6-4fe6-82c4-eb483dc68876
Vanilla 1.21.11 38.68.14.29 25565 3bed1bda-f648-4630-801a-fe9f2e3d3f27
All The Mons 38.68.14.30 25565 668a5220-7e72-4379-9165-bdbb84bc9806
Roguelike Adventures and Dungeons 2 38.68.14.26 25566 5083309f-c76c-4752-a66f-46a81b6cd8aa
FoundryVTT 38.68.14.26 30000 7d8f15a0-4ee7-4dd6-85dc-ab42966f733d

Total TX1 Servers: 7

Note: FoundryVTT domain configured: foundry.firefrostgaming.com (fixed Feb 11/12, 2026 - Holly's ~50% access issue resolved)


5. Global Port Reservations

  • Minecraft: 25565 - 25580
  • Hytale: 5520 - 5521
  • FoundryVTT: 30000
  • Gitea: 3000 (internal, proxied via Nginx)
  • Uptime Kuma: 3001 (internal, proxied via Nginx)
  • Code-Server: 8080 (internal, proxied via Nginx)

6. Infrastructure Architecture

Management Services Strategy (Implemented Feb 9, 2026)

Philosophy: "Backend on Command Center, Frontend on Ghost, Money on Billing, Games on Dedis, Control on Panel"

VPS Tier (Management):

  • Command Center: Gitea, Uptime Kuma, Code-Server, Automation, Trinity Console (backend tools)
  • Wiki VPS: MkDocs, Wiki.js (3 instances), NextCloud (public-facing docs and internal tools)
  • Services VPS: Mailcow, Whitelist Manager (email and utilities)
  • Panel VPS: Pterodactyl Panel only (control plane)

Dedicated Tier (Performance):

  • TX1 Dallas: Game servers ONLY (6 servers)
  • NC1 Charlotte: Game servers ONLY (6 servers)

Rationale:

  • VPS = plug-and-play networking, easy SSL, low maintenance
  • Dedicated = performance for game servers, massive RAM/CPU
  • Clear separation of concerns
  • Optimal use of existing infrastructure ($0 additional cost)

Key Architecture Note (NC1)

All NC1 game servers share a single IP (216.239.104.130) with unique ports. This differs from TX1 where each server has its own IP on the /29 block.


7. Monitoring Status

Uptime Kuma: status.firefrostgaming.com

Infrastructure Monitors:

  • Pterodactyl Panel (100% uptime)
  • Command Center (100% uptime)
  • Main Website - Cloudflare Pages (100% uptime)
  • Gitea (98%+ uptime)
  • Mailcow (100% uptime)

Game Server Monitors (Added Feb 11, 2026):

TX1 Dallas:

  • Stoneblock 4 (38.68.14.26:25565)
  • Reclamation (38.68.14.27:25565)
  • Society: Sunlit Valley (38.68.14.28:25565)
  • Vanilla 1.21.11 (38.68.14.29:25565)
  • All The Mons (38.68.14.30:25565)
  • FoundryVTT (38.68.14.26:30000)

NC1 Charlotte:

  • The Ember Project (216.239.104.130:25565)
  • Minecolonies: Create and Conquer (216.239.104.130:25566)
  • All The Mods 10 (216.239.104.130:25569)
  • EMC Subterra Tech (216.239.104.130:25571)
  • Homestead (216.239.104.130:25572)
  • Hytale (216.239.104.130:5520)

Total Monitors: 17 (5 infrastructure + 12 game servers)


Revision History

Revision Date Author Description of Changes
v1.0 2026-02-07 Michael Initial Phase 0 Audit; Infrastructure mapping for NC1 and TX1.
v1.1 2026-02-07 Michael Added verified Friendly Names for core services and hardware.
v1.2 2026-02-07 Michael Finalized Command Center networking with /29 block (74.63.218.201).
v1.3 2026-02-07 Michael Synced profile for Michael (57) and updated accessibility context.
v1.4 2026-02-09 Michael Management services migrated to VPS tier. Command Center now operational with Gitea + Uptime Kuma. TX1 optimized for game servers only. Added infrastructure architecture section.
v1.5 2026-02-11 Michael + Claude Corrected NC1/TX1 server lists from Panel (actual). NC1 all on single IP wi
v1.6 2026-02-12 The Chronicler Correction
v1.7 2026-02-13 Chronicler the Fourth Update
v1.8 2026-02-13 Chronicler the Fourth Update
v1.9 2026-02-13 Chronicler the Fourth Update
v2.0 2026-03-30 The Versionist (#49) Addition
v2.1 2026-04-04 Chronicler #57 Major Update

End of Document