firefrost-operations-manual/docs/tasks/command-center-security/deployment-guide.md

Command Center Security Hardening - Deployment Guide

Status: Ready to Deploy
Priority: Tier 1 - Security Foundation
Time Estimate: 1 hour
Last Updated: 2026-02-17

---

Overview

Defense-in-depth security hardening for Command Center VPS (Dallas hub). Implements multiple layers of security including Fail2Ban auto-banning, SSH key-only authentication, and firewall optimization.

Command Center Details:

• IP: 63.143.34.217
• Location: Dallas, TX
• Role: Management hub (Gitea, Uptime Kuma, Code-Server, Automation)
• Current Security: UFW enabled, basic SSH

---

Current Security State

✅ Already Implemented:

• UFW firewall enabled (default deny incoming)
• Ports 22, 80, 443 open
• Basic SSH configuration

❌ Missing Protections:

• No Fail2Ban (vulnerable to brute force)
• SSH allows password authentication
• No rate limiting on SSH attempts
• Potential unnecessary open ports

---

Security Hardening Strategy

Layer 1: Auto-Ban (Fail2Ban)

• Monitors log files for suspicious activity
• Auto-bans IPs after failed attempts
• Protects SSH, Nginx, and other services

Layer 2: SSH Hardening

• Key-only authentication (no passwords)
• Reduced MaxAuthTries
• Optional: Non-standard SSH port

Layer 3: Firewall Optimization

• Close unnecessary ports
• Whitelist management IP
• Rate limiting rules

---

Prerequisites

CRITICAL - Before starting:

• SSH key already configured on your local machine
• SSH key added to Command Center (~/.ssh/authorized_keys)
• Test SSH key login works BEFORE disabling password auth
• Management IP known (Michael's static IP)
• Backup access method available (provider console/IPMI)

⚠️ WARNING: If you disable password auth without working SSH keys, you'll be locked out!

---

Phase 1: Test SSH Key Access (10 min)

MANDATORY: Verify SSH keys work before proceeding!

Step 1: Verify SSH Key Exists Locally

# On your local machine
ls -la ~/.ssh/id_rsa ~/.ssh/id_ed25519

# You should see at least one key file

If no keys exist:

# Generate new ED25519 key (recommended)
ssh-keygen -t ed25519 -C "michael@firefrostgaming.com"

# Or RSA key (legacy compatibility)
ssh-keygen -t rsa -b 4096 -C "michael@firefrostgaming.com"

# Press Enter for default location
# Set a strong passphrase (recommended)

---

Step 2: Copy Key to Command Center

# Copy SSH key to server
ssh-copy-id root@63.143.34.217

# Or manually:
cat ~/.ssh/id_ed25519.pub | ssh root@63.143.34.217 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

---

Step 3: Test Key-Based Login

# Try to SSH with key (should NOT ask for password)
ssh root@63.143.34.217

# If it asks for password, your key isn't configured correctly
# DO NOT PROCEED until this works!

If prompted for password: Your SSH key is not properly configured. Debug before continuing.

---

Phase 2: Install and Configure Fail2Ban (20 min)

Step 1: Install Fail2Ban

# SSH to Command Center
ssh root@63.143.34.217

# Update package list
apt update

# Install Fail2Ban
apt install fail2ban -y

# Verify installation
systemctl status fail2ban
# Should show: active (running)

---

Step 2: Configure Fail2Ban

# Create local config (don't edit jail.conf directly)
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

# Edit local config
nano /etc/fail2ban/jail.local

Key settings to configure:

[DEFAULT]
# Ban time (10 minutes = 600 seconds)
bantime = 600

# Find time window (10 minutes)
findtime = 600

# Max retries before ban
maxretry = 5

# Your management IP (never ban this!)
ignoreip = 127.0.0.1/8 ::1 YOUR_MANAGEMENT_IP_HERE

[sshd]
enabled = true
port = 22
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600

Save and exit (Ctrl+X, Y, Enter)

---

Step 3: Enable Additional Jails

# Edit jail.local to enable protection for other services
nano /etc/fail2ban/jail.local

Add at the end:

# Nginx HTTP Auth
[nginx-http-auth]
enabled = true
port = http,https
logpath = /var/log/nginx/error.log

# Nginx Bad Bots
[nginx-badbots]
enabled = true
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 2

---

Step 4: Restart and Verify Fail2Ban

# Restart Fail2Ban
systemctl restart fail2ban

# Check status
systemctl status fail2ban

# View active jails
fail2ban-client status

# Check SSH jail specifically
fail2ban-client status sshd

You should see:

• sshd jail active
• Currently banned IPs: 0 (unless there were recent attacks)
• Total banned: (number)

---

Phase 3: SSH Hardening (20 min)

Step 1: Backup SSH Config

# Create backup
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%Y%m%d)

# Verify backup
ls -la /etc/ssh/sshd_config.backup.*

---

Step 2: Harden SSH Configuration

# Edit SSH config
nano /etc/ssh/sshd_config

Find and modify these lines:

# Disable root password login
PasswordAuthentication no

# Disable challenge-response (another password method)
ChallengeResponseAuthentication no

# Allow key-based auth
PubkeyAuthentication yes

# Disable empty passwords
PermitEmptyPasswords no

# Limit authentication attempts
MaxAuthTries 3

# Limit sessions per connection
MaxSessions 2

# Only allow root (or create separate admin user)
AllowUsers root

# Optional: Change SSH port (if desired)
# Port 2222

# Disable X11 forwarding (not needed)
X11Forwarding no

# Disable TCP forwarding if not needed
# AllowTcpForwarding no

# Enable strict mode (recommended)
StrictModes yes

# Set login grace time (2 minutes)
LoginGraceTime 120

Save and exit (Ctrl+X, Y, Enter)

---

Step 3: Test SSH Config

# Test config for syntax errors
sshd -t

# Should return nothing (no output = success)
# If errors appear, fix them before restarting!

If errors appear:

• Review the error message
• Edit /etc/ssh/sshd_config again
• Run sshd -t again until no errors

---

Step 4: Restart SSH (CAREFULLY!)

⚠️ CRITICAL STEP - Do this carefully!

# Keep current SSH session open!
# Open a SECOND SSH session in another terminal for testing

# In the first session, restart SSH
systemctl restart sshd

# In the second session, try to connect
ssh root@63.143.34.217

# If the second session connects with your key: SUCCESS!
# If not: You still have the first session to fix it

If second session fails:

# In first session (still connected):
nano /etc/ssh/sshd_config
# Fix the issue
sshd -t
systemctl restart sshd
# Try second session again

Only logout of the first session AFTER the second session works!

---

Phase 4: UFW Firewall Review (15 min)

Step 1: Review Current Rules

# List current rules
ufw status verbose

# List rules by number
ufw status numbered

Expected output:

Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere

---

Step 2: Optimize Rules

Add management IP whitelist:

# Allow SSH from your IP only (recommended)
ufw insert 1 allow from YOUR_MANAGEMENT_IP to any port 22 proto tcp

# Remove the general SSH allow rule
ufw delete allow 22/tcp

# Verify
ufw status numbered

Result: SSH now only accessible from your management IP

Add rate limiting (optional):

# Limit SSH connection attempts
ufw limit ssh/tcp

# This limits to 6 connections per 30 seconds per IP

---

Step 3: Close Unnecessary Ports

Check what's listening:

# See what services are listening
ss -tuln

# Or with netstat
netstat -tuln

If you see unexpected open ports:

# Close them via UFW
ufw deny PORT_NUMBER

---

Step 4: Enable UFW Logging

# Enable logging (helps with debugging)
ufw logging on

# Set to medium level
ufw logging medium

# View logs
tail -f /var/log/ufw.log

---

Phase 5: Additional Security Measures (10 min)

Automatic Security Updates

# Install unattended-upgrades
apt install unattended-upgrades -y

# Enable automatic security updates
dpkg-reconfigure -plow unattended-upgrades
# Select "Yes"

# Verify config
cat /etc/apt/apt.conf.d/20auto-upgrades

Should contain:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

---

Install Additional Security Tools

# Install helpful security tools
apt install -y \
  ufw-extras \
  apt-listchanges \
  debsums \
  aide

# Initialize AIDE (file integrity checker)
aide --init
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

---

Verification & Testing

Test 1: SSH Key Access

# From your local machine
ssh root@63.143.34.217

# Should connect WITHOUT asking for password
# If it asks for your key passphrase, that's normal

---

Test 2: Fail2Ban is Working

# On Command Center, check Fail2Ban status
fail2ban-client status sshd

# Try to trigger a ban (from another IP if possible):
# Make 4 failed SSH attempts
# That IP should get banned

# Check banned IPs
fail2ban-client status sshd

---

Test 3: Password Auth is Disabled

# From local machine, try password auth (should fail)
ssh -o PreferredAuthentications=password root@63.143.34.217

# Should refuse: "Permission denied (publickey)"

---

Test 4: UFW Rules Active

# Check firewall status
ufw status verbose

# Verify only necessary ports open
# Verify management IP whitelisted for SSH

---

Monitoring & Maintenance

Daily

• Check Fail2Ban for banned IPs: fail2ban-client status sshd
• Review auth logs: tail -100 /var/log/auth.log | grep Failed

Weekly

• Review UFW logs: grep UFW /var/log/syslog | tail -50
• Check for security updates: apt list --upgradable | grep security

Monthly

• Review all Fail2Ban bans: zgrep 'Ban' /var/log/fail2ban.log*
• Run AIDE file integrity check: aide --check
• Review SSH config for any changes needed

---

Troubleshooting

Locked Out of SSH

Prevention: Always keep one session open while testing!

If locked out:

1. Access via provider console (IPMI/VNC)
2. Edit /etc/ssh/sshd_config
3. Re-enable PasswordAuthentication yes
4. Restart SSH: systemctl restart sshd
5. Fix SSH key issue
6. Re-harden once working

---

Fail2Ban Not Banning

Check logs:

tail -100 /var/log/fail2ban.log

Common issues:

• Log file path incorrect
• Regex pattern doesn't match log format
• MaxRetry set too high

Debug specific jail:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

---

UFW Blocking Legitimate Traffic

Check UFW logs:

grep UFW /var/log/syslog | grep BLOCK

Allow specific IP:

ufw allow from IP_ADDRESS

Temporarily disable UFW (emergency only!):

ufw disable
# Fix issue
ufw enable

---

Management IP Changed

Update Fail2Ban:

nano /etc/fail2ban/jail.local
# Update ignoreip line
systemctl restart fail2ban

Update UFW:

# Remove old rule
ufw delete RULE_NUMBER

# Add new rule
ufw insert 1 allow from NEW_IP to any port 22 proto tcp

---

Security Checklist

After deployment, verify:

• SSH key authentication works
• Password authentication disabled
• Fail2Ban active and monitoring
• UFW firewall optimized
• Management IP whitelisted
• Automatic security updates enabled
• All tests passed
• Backup SSH access method available
• Security config documented
• Team notified of changes

---

Related Tasks

• Frostwall Protocol - Network-level security
• Vaultwarden Setup - Credential management
• Command Center Cleanup - Server housekeeping

---

Fire + Frost + Foundation = Where Love Builds Legacy 💙🔥❄️

---

Document Status: COMPLETE
Ready to Deploy: When SSH access available (1 hour)
Risk Level: MEDIUM (can lock yourself out if not careful)
Backup Required: Console/IPMI access in case of lockout
Test Thoroughly: Always test SSH keys before disabling password auth!