# Command Center Security Hardening - Deployment Guide **Status:** Ready to Deploy **Priority:** Tier 1 - Security Foundation **Time Estimate:** 1 hour **Last Updated:** 2026-02-17 --- ## Overview Defense-in-depth security hardening for Command Center VPS (Dallas hub). Implements multiple layers of security including Fail2Ban auto-banning, SSH key-only authentication, and firewall optimization. **Command Center Details:** - **IP:** 63.143.34.217 - **Location:** Dallas, TX - **Role:** Management hub (Gitea, Uptime Kuma, Code-Server, Automation) - **Current Security:** UFW enabled, basic SSH --- ## Current Security State **✅ Already Implemented:** - UFW firewall enabled (default deny incoming) - Ports 22, 80, 443 open - Basic SSH configuration **❌ Missing Protections:** - No Fail2Ban (vulnerable to brute force) - SSH allows password authentication - No rate limiting on SSH attempts - Potential unnecessary open ports --- ## Security Hardening Strategy ### Layer 1: Auto-Ban (Fail2Ban) - Monitors log files for suspicious activity - Auto-bans IPs after failed attempts - Protects SSH, Nginx, and other services ### Layer 2: SSH Hardening - Key-only authentication (no passwords) - Reduced MaxAuthTries - Optional: Non-standard SSH port ### Layer 3: Firewall Optimization - Close unnecessary ports - Whitelist management IP - Rate limiting rules --- ## Prerequisites **CRITICAL - Before starting:** - [ ] **SSH key already configured** on your local machine - [ ] **SSH key added to Command Center** (`~/.ssh/authorized_keys`) - [ ] **Test SSH key login works** BEFORE disabling password auth - [ ] **Management IP known** (Michael's static IP) - [ ] **Backup access method** available (provider console/IPMI) **⚠️ WARNING:** If you disable password auth without working SSH keys, you'll be locked out! --- ## Phase 1: Test SSH Key Access (10 min) **MANDATORY: Verify SSH keys work before proceeding!** ### Step 1: Verify SSH Key Exists Locally ```bash # On your local machine ls -la ~/.ssh/id_rsa ~/.ssh/id_ed25519 # You should see at least one key file ``` **If no keys exist:** ```bash # Generate new ED25519 key (recommended) ssh-keygen -t ed25519 -C "michael@firefrostgaming.com" # Or RSA key (legacy compatibility) ssh-keygen -t rsa -b 4096 -C "michael@firefrostgaming.com" # Press Enter for default location # Set a strong passphrase (recommended) ``` --- ### Step 2: Copy Key to Command Center ```bash # Copy SSH key to server ssh-copy-id root@63.143.34.217 # Or manually: cat ~/.ssh/id_ed25519.pub | ssh root@63.143.34.217 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys" ``` --- ### Step 3: Test Key-Based Login ```bash # Try to SSH with key (should NOT ask for password) ssh root@63.143.34.217 # If it asks for password, your key isn't configured correctly # DO NOT PROCEED until this works! ``` **If prompted for password:** Your SSH key is not properly configured. Debug before continuing. --- ## Phase 2: Install and Configure Fail2Ban (20 min) ### Step 1: Install Fail2Ban ```bash # SSH to Command Center ssh root@63.143.34.217 # Update package list apt update # Install Fail2Ban apt install fail2ban -y # Verify installation systemctl status fail2ban # Should show: active (running) ``` --- ### Step 2: Configure Fail2Ban ```bash # Create local config (don't edit jail.conf directly) cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local # Edit local config nano /etc/fail2ban/jail.local ``` **Key settings to configure:** ```ini [DEFAULT] # Ban time (10 minutes = 600 seconds) bantime = 600 # Find time window (10 minutes) findtime = 600 # Max retries before ban maxretry = 5 # Your management IP (never ban this!) ignoreip = 127.0.0.1/8 ::1 YOUR_MANAGEMENT_IP_HERE [sshd] enabled = true port = 22 logpath = /var/log/auth.log maxretry = 3 bantime = 3600 ``` **Save and exit** (Ctrl+X, Y, Enter) --- ### Step 3: Enable Additional Jails ```bash # Edit jail.local to enable protection for other services nano /etc/fail2ban/jail.local ``` **Add at the end:** ```ini # Nginx HTTP Auth [nginx-http-auth] enabled = true port = http,https logpath = /var/log/nginx/error.log # Nginx Bad Bots [nginx-badbots] enabled = true port = http,https logpath = /var/log/nginx/access.log maxretry = 2 ``` --- ### Step 4: Restart and Verify Fail2Ban ```bash # Restart Fail2Ban systemctl restart fail2ban # Check status systemctl status fail2ban # View active jails fail2ban-client status # Check SSH jail specifically fail2ban-client status sshd ``` **You should see:** - `sshd` jail active - Currently banned IPs: 0 (unless there were recent attacks) - Total banned: (number) --- ## Phase 3: SSH Hardening (20 min) ### Step 1: Backup SSH Config ```bash # Create backup cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%Y%m%d) # Verify backup ls -la /etc/ssh/sshd_config.backup.* ``` --- ### Step 2: Harden SSH Configuration ```bash # Edit SSH config nano /etc/ssh/sshd_config ``` **Find and modify these lines:** ```bash # Disable root password login PasswordAuthentication no # Disable challenge-response (another password method) ChallengeResponseAuthentication no # Allow key-based auth PubkeyAuthentication yes # Disable empty passwords PermitEmptyPasswords no # Limit authentication attempts MaxAuthTries 3 # Limit sessions per connection MaxSessions 2 # Only allow root (or create separate admin user) AllowUsers root # Optional: Change SSH port (if desired) # Port 2222 # Disable X11 forwarding (not needed) X11Forwarding no # Disable TCP forwarding if not needed # AllowTcpForwarding no # Enable strict mode (recommended) StrictModes yes # Set login grace time (2 minutes) LoginGraceTime 120 ``` **Save and exit** (Ctrl+X, Y, Enter) --- ### Step 3: Test SSH Config ```bash # Test config for syntax errors sshd -t # Should return nothing (no output = success) # If errors appear, fix them before restarting! ``` **If errors appear:** - Review the error message - Edit `/etc/ssh/sshd_config` again - Run `sshd -t` again until no errors --- ### Step 4: Restart SSH (CAREFULLY!) **⚠️ CRITICAL STEP - Do this carefully!** ```bash # Keep current SSH session open! # Open a SECOND SSH session in another terminal for testing # In the first session, restart SSH systemctl restart sshd # In the second session, try to connect ssh root@63.143.34.217 # If the second session connects with your key: SUCCESS! # If not: You still have the first session to fix it ``` **If second session fails:** ```bash # In first session (still connected): nano /etc/ssh/sshd_config # Fix the issue sshd -t systemctl restart sshd # Try second session again ``` **Only logout of the first session AFTER the second session works!** --- ## Phase 4: UFW Firewall Review (15 min) ### Step 1: Review Current Rules ```bash # List current rules ufw status verbose # List rules by number ufw status numbered ``` **Expected output:** ``` Status: active To Action From -- ------ ---- 22/tcp ALLOW Anywhere 80/tcp ALLOW Anywhere 443/tcp ALLOW Anywhere ``` --- ### Step 2: Optimize Rules **Add management IP whitelist:** ```bash # Allow SSH from your IP only (recommended) ufw insert 1 allow from YOUR_MANAGEMENT_IP to any port 22 proto tcp # Remove the general SSH allow rule ufw delete allow 22/tcp # Verify ufw status numbered ``` **Result:** SSH now only accessible from your management IP **Add rate limiting (optional):** ```bash # Limit SSH connection attempts ufw limit ssh/tcp # This limits to 6 connections per 30 seconds per IP ``` --- ### Step 3: Close Unnecessary Ports **Check what's listening:** ```bash # See what services are listening ss -tuln # Or with netstat netstat -tuln ``` **If you see unexpected open ports:** ```bash # Close them via UFW ufw deny PORT_NUMBER ``` --- ### Step 4: Enable UFW Logging ```bash # Enable logging (helps with debugging) ufw logging on # Set to medium level ufw logging medium # View logs tail -f /var/log/ufw.log ``` --- ## Phase 5: Additional Security Measures (10 min) ### Automatic Security Updates ```bash # Install unattended-upgrades apt install unattended-upgrades -y # Enable automatic security updates dpkg-reconfigure -plow unattended-upgrades # Select "Yes" # Verify config cat /etc/apt/apt.conf.d/20auto-upgrades ``` **Should contain:** ``` APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1"; ``` --- ### Install Additional Security Tools ```bash # Install helpful security tools apt install -y \ ufw-extras \ apt-listchanges \ debsums \ aide # Initialize AIDE (file integrity checker) aide --init mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db ``` --- ## Verification & Testing ### Test 1: SSH Key Access ```bash # From your local machine ssh root@63.143.34.217 # Should connect WITHOUT asking for password # If it asks for your key passphrase, that's normal ``` --- ### Test 2: Fail2Ban is Working ```bash # On Command Center, check Fail2Ban status fail2ban-client status sshd # Try to trigger a ban (from another IP if possible): # Make 4 failed SSH attempts # That IP should get banned # Check banned IPs fail2ban-client status sshd ``` --- ### Test 3: Password Auth is Disabled ```bash # From local machine, try password auth (should fail) ssh -o PreferredAuthentications=password root@63.143.34.217 # Should refuse: "Permission denied (publickey)" ``` --- ### Test 4: UFW Rules Active ```bash # Check firewall status ufw status verbose # Verify only necessary ports open # Verify management IP whitelisted for SSH ``` --- ## Monitoring & Maintenance ### Daily - Check Fail2Ban for banned IPs: `fail2ban-client status sshd` - Review auth logs: `tail -100 /var/log/auth.log | grep Failed` ### Weekly - Review UFW logs: `grep UFW /var/log/syslog | tail -50` - Check for security updates: `apt list --upgradable | grep security` ### Monthly - Review all Fail2Ban bans: `zgrep 'Ban' /var/log/fail2ban.log*` - Run AIDE file integrity check: `aide --check` - Review SSH config for any changes needed --- ## Troubleshooting ### Locked Out of SSH **Prevention:** Always keep one session open while testing! **If locked out:** 1. Access via provider console (IPMI/VNC) 2. Edit `/etc/ssh/sshd_config` 3. Re-enable `PasswordAuthentication yes` 4. Restart SSH: `systemctl restart sshd` 5. Fix SSH key issue 6. Re-harden once working --- ### Fail2Ban Not Banning **Check logs:** ```bash tail -100 /var/log/fail2ban.log ``` **Common issues:** - Log file path incorrect - Regex pattern doesn't match log format - MaxRetry set too high **Debug specific jail:** ```bash fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf ``` --- ### UFW Blocking Legitimate Traffic **Check UFW logs:** ```bash grep UFW /var/log/syslog | grep BLOCK ``` **Allow specific IP:** ```bash ufw allow from IP_ADDRESS ``` **Temporarily disable UFW (emergency only!):** ```bash ufw disable # Fix issue ufw enable ``` --- ### Management IP Changed **Update Fail2Ban:** ```bash nano /etc/fail2ban/jail.local # Update ignoreip line systemctl restart fail2ban ``` **Update UFW:** ```bash # Remove old rule ufw delete RULE_NUMBER # Add new rule ufw insert 1 allow from NEW_IP to any port 22 proto tcp ``` --- ## Security Checklist **After deployment, verify:** - [ ] SSH key authentication works - [ ] Password authentication disabled - [ ] Fail2Ban active and monitoring - [ ] UFW firewall optimized - [ ] Management IP whitelisted - [ ] Automatic security updates enabled - [ ] All tests passed - [ ] Backup SSH access method available - [ ] Security config documented - [ ] Team notified of changes --- ## Related Tasks - **Frostwall Protocol** - Network-level security - **Vaultwarden Setup** - Credential management - **Command Center Cleanup** - Server housekeeping --- **Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️ --- **Document Status:** COMPLETE **Ready to Deploy:** When SSH access available (1 hour) **Risk Level:** MEDIUM (can lock yourself out if not careful) **Backup Required:** Console/IPMI access in case of lockout **Test Thoroughly:** Always test SSH keys before disabling password auth!