firefrost-operations-manual/docs/tasks/command-center-security/deployment-guide.md

# Command Center Security Hardening - Deployment Guide

**Status:** Ready to Deploy  
**Priority:** Tier 1 - Security Foundation  
**Time Estimate:** 1 hour  
**Last Updated:** 2026-02-17

---

## Overview

Defense-in-depth security hardening for Command Center VPS (Dallas hub). Implements multiple layers of security including Fail2Ban auto-banning, SSH key-only authentication, and firewall optimization.

**Command Center Details:**
- **IP:** 63.143.34.217
- **Location:** Dallas, TX
- **Role:** Management hub (Gitea, Uptime Kuma, Code-Server, Automation)
- **Current Security:** UFW enabled, basic SSH

---

## Current Security State

**✅ Already Implemented:**
- UFW firewall enabled (default deny incoming)
- Ports 22, 80, 443 open
- Basic SSH configuration

**❌ Missing Protections:**
- No Fail2Ban (vulnerable to brute force)
- SSH allows password authentication
- No rate limiting on SSH attempts
- Potential unnecessary open ports

---

## Security Hardening Strategy

### Layer 1: Auto-Ban (Fail2Ban)
- Monitors log files for suspicious activity
- Auto-bans IPs after failed attempts
- Protects SSH, Nginx, and other services

### Layer 2: SSH Hardening
- Key-only authentication (no passwords)
- Reduced MaxAuthTries
- Optional: Non-standard SSH port

### Layer 3: Firewall Optimization
- Close unnecessary ports
- Whitelist management IP
- Rate limiting rules

---

## Prerequisites

**CRITICAL - Before starting:**

- [ ] **SSH key already configured** on your local machine
- [ ] **SSH key added to Command Center** (`~/.ssh/authorized_keys`)
- [ ] **Test SSH key login works** BEFORE disabling password auth
- [ ] **Management IP known** (Michael's static IP)
- [ ] **Backup access method** available (provider console/IPMI)

**⚠️ WARNING:** If you disable password auth without working SSH keys, you'll be locked out!

---

## Phase 1: Test SSH Key Access (10 min)

**MANDATORY: Verify SSH keys work before proceeding!**

### Step 1: Verify SSH Key Exists Locally

```bash
# On your local machine
ls -la ~/.ssh/id_rsa ~/.ssh/id_ed25519

# You should see at least one key file
```

**If no keys exist:**
```bash
# Generate new ED25519 key (recommended)
ssh-keygen -t ed25519 -C "michael@firefrostgaming.com"

# Or RSA key (legacy compatibility)
ssh-keygen -t rsa -b 4096 -C "michael@firefrostgaming.com"

# Press Enter for default location
# Set a strong passphrase (recommended)
```

---

### Step 2: Copy Key to Command Center

```bash
# Copy SSH key to server
ssh-copy-id root@63.143.34.217

# Or manually:
cat ~/.ssh/id_ed25519.pub | ssh root@63.143.34.217 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
```

---

### Step 3: Test Key-Based Login

```bash
# Try to SSH with key (should NOT ask for password)
ssh root@63.143.34.217

# If it asks for password, your key isn't configured correctly
# DO NOT PROCEED until this works!
```

**If prompted for password:** Your SSH key is not properly configured. Debug before continuing.

---

## Phase 2: Install and Configure Fail2Ban (20 min)

### Step 1: Install Fail2Ban

```bash
# SSH to Command Center
ssh root@63.143.34.217

# Update package list
apt update

# Install Fail2Ban
apt install fail2ban -y

# Verify installation
systemctl status fail2ban
# Should show: active (running)
```

---

### Step 2: Configure Fail2Ban

```bash
# Create local config (don't edit jail.conf directly)
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

# Edit local config
nano /etc/fail2ban/jail.local
```

**Key settings to configure:**

```ini
[DEFAULT]
# Ban time (10 minutes = 600 seconds)
bantime = 600

# Find time window (10 minutes)
findtime = 600

# Max retries before ban
maxretry = 5

# Your management IP (never ban this!)
ignoreip = 127.0.0.1/8 ::1 YOUR_MANAGEMENT_IP_HERE

[sshd]
enabled = true
port = 22
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
```

**Save and exit** (Ctrl+X, Y, Enter)

---

### Step 3: Enable Additional Jails

```bash
# Edit jail.local to enable protection for other services
nano /etc/fail2ban/jail.local
```

**Add at the end:**

```ini
# Nginx HTTP Auth
[nginx-http-auth]
enabled = true
port = http,https
logpath = /var/log/nginx/error.log

# Nginx Bad Bots
[nginx-badbots]
enabled = true
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 2
```

---

### Step 4: Restart and Verify Fail2Ban

```bash
# Restart Fail2Ban
systemctl restart fail2ban

# Check status
systemctl status fail2ban

# View active jails
fail2ban-client status

# Check SSH jail specifically
fail2ban-client status sshd
```

**You should see:**
- `sshd` jail active
- Currently banned IPs: 0 (unless there were recent attacks)
- Total banned: (number)

---

## Phase 3: SSH Hardening (20 min)

### Step 1: Backup SSH Config

```bash
# Create backup
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%Y%m%d)

# Verify backup
ls -la /etc/ssh/sshd_config.backup.*
```

---

### Step 2: Harden SSH Configuration

```bash
# Edit SSH config
nano /etc/ssh/sshd_config
```

**Find and modify these lines:**

```bash
# Disable root password login
PasswordAuthentication no

# Disable challenge-response (another password method)
ChallengeResponseAuthentication no

# Allow key-based auth
PubkeyAuthentication yes

# Disable empty passwords
PermitEmptyPasswords no

# Limit authentication attempts
MaxAuthTries 3

# Limit sessions per connection
MaxSessions 2

# Only allow root (or create separate admin user)
AllowUsers root

# Optional: Change SSH port (if desired)
# Port 2222

# Disable X11 forwarding (not needed)
X11Forwarding no

# Disable TCP forwarding if not needed
# AllowTcpForwarding no

# Enable strict mode (recommended)
StrictModes yes

# Set login grace time (2 minutes)
LoginGraceTime 120
```

**Save and exit** (Ctrl+X, Y, Enter)

---

### Step 3: Test SSH Config

```bash
# Test config for syntax errors
sshd -t

# Should return nothing (no output = success)
# If errors appear, fix them before restarting!
```

**If errors appear:**
- Review the error message
- Edit `/etc/ssh/sshd_config` again
- Run `sshd -t` again until no errors

---

### Step 4: Restart SSH (CAREFULLY!)

**⚠️ CRITICAL STEP - Do this carefully!**

```bash
# Keep current SSH session open!
# Open a SECOND SSH session in another terminal for testing

# In the first session, restart SSH
systemctl restart sshd

# In the second session, try to connect
ssh root@63.143.34.217

# If the second session connects with your key: SUCCESS!
# If not: You still have the first session to fix it
```

**If second session fails:**
```bash
# In first session (still connected):
nano /etc/ssh/sshd_config
# Fix the issue
sshd -t
systemctl restart sshd
# Try second session again
```

**Only logout of the first session AFTER the second session works!**

---

## Phase 4: UFW Firewall Review (15 min)

### Step 1: Review Current Rules

```bash
# List current rules
ufw status verbose

# List rules by number
ufw status numbered
```

**Expected output:**
```
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
```

---

### Step 2: Optimize Rules

**Add management IP whitelist:**

```bash
# Allow SSH from your IP only (recommended)
ufw insert 1 allow from YOUR_MANAGEMENT_IP to any port 22 proto tcp

# Remove the general SSH allow rule
ufw delete allow 22/tcp

# Verify
ufw status numbered
```

**Result:** SSH now only accessible from your management IP

**Add rate limiting (optional):**

```bash
# Limit SSH connection attempts
ufw limit ssh/tcp

# This limits to 6 connections per 30 seconds per IP
```

---

### Step 3: Close Unnecessary Ports

**Check what's listening:**

```bash
# See what services are listening
ss -tuln

# Or with netstat
netstat -tuln
```

**If you see unexpected open ports:**
```bash
# Close them via UFW
ufw deny PORT_NUMBER
```

---

### Step 4: Enable UFW Logging

```bash
# Enable logging (helps with debugging)
ufw logging on

# Set to medium level
ufw logging medium

# View logs
tail -f /var/log/ufw.log
```

---

## Phase 5: Additional Security Measures (10 min)

### Automatic Security Updates

```bash
# Install unattended-upgrades
apt install unattended-upgrades -y

# Enable automatic security updates
dpkg-reconfigure -plow unattended-upgrades
# Select "Yes"

# Verify config
cat /etc/apt/apt.conf.d/20auto-upgrades
```

**Should contain:**
```
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
```

---

### Install Additional Security Tools

```bash
# Install helpful security tools
apt install -y \
  ufw-extras \
  apt-listchanges \
  debsums \
  aide

# Initialize AIDE (file integrity checker)
aide --init
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
```

---

## Verification & Testing

### Test 1: SSH Key Access

```bash
# From your local machine
ssh root@63.143.34.217

# Should connect WITHOUT asking for password
# If it asks for your key passphrase, that's normal
```

---

### Test 2: Fail2Ban is Working

```bash
# On Command Center, check Fail2Ban status
fail2ban-client status sshd

# Try to trigger a ban (from another IP if possible):
# Make 4 failed SSH attempts
# That IP should get banned

# Check banned IPs
fail2ban-client status sshd
```

---

### Test 3: Password Auth is Disabled

```bash
# From local machine, try password auth (should fail)
ssh -o PreferredAuthentications=password root@63.143.34.217

# Should refuse: "Permission denied (publickey)"
```

---

### Test 4: UFW Rules Active

```bash
# Check firewall status
ufw status verbose

# Verify only necessary ports open
# Verify management IP whitelisted for SSH
```

---

## Monitoring & Maintenance

### Daily

- Check Fail2Ban for banned IPs: `fail2ban-client status sshd`
- Review auth logs: `tail -100 /var/log/auth.log | grep Failed`

### Weekly

- Review UFW logs: `grep UFW /var/log/syslog | tail -50`
- Check for security updates: `apt list --upgradable | grep security`

### Monthly

- Review all Fail2Ban bans: `zgrep 'Ban' /var/log/fail2ban.log*`
- Run AIDE file integrity check: `aide --check`
- Review SSH config for any changes needed

---

## Troubleshooting

### Locked Out of SSH

**Prevention:** Always keep one session open while testing!

**If locked out:**
1. Access via provider console (IPMI/VNC)
2. Edit `/etc/ssh/sshd_config`
3. Re-enable `PasswordAuthentication yes`
4. Restart SSH: `systemctl restart sshd`
5. Fix SSH key issue
6. Re-harden once working

---

### Fail2Ban Not Banning

**Check logs:**
```bash
tail -100 /var/log/fail2ban.log
```

**Common issues:**
- Log file path incorrect
- Regex pattern doesn't match log format
- MaxRetry set too high

**Debug specific jail:**
```bash
fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
```

---

### UFW Blocking Legitimate Traffic

**Check UFW logs:**
```bash
grep UFW /var/log/syslog | grep BLOCK
```

**Allow specific IP:**
```bash
ufw allow from IP_ADDRESS
```

**Temporarily disable UFW (emergency only!):**
```bash
ufw disable
# Fix issue
ufw enable
```

---

### Management IP Changed

**Update Fail2Ban:**
```bash
nano /etc/fail2ban/jail.local
# Update ignoreip line
systemctl restart fail2ban
```

**Update UFW:**
```bash
# Remove old rule
ufw delete RULE_NUMBER

# Add new rule
ufw insert 1 allow from NEW_IP to any port 22 proto tcp
```

---

## Security Checklist

**After deployment, verify:**

- [ ] SSH key authentication works
- [ ] Password authentication disabled
- [ ] Fail2Ban active and monitoring
- [ ] UFW firewall optimized
- [ ] Management IP whitelisted
- [ ] Automatic security updates enabled
- [ ] All tests passed
- [ ] Backup SSH access method available
- [ ] Security config documented
- [ ] Team notified of changes

---

## Related Tasks

- **Frostwall Protocol** - Network-level security
- **Vaultwarden Setup** - Credential management
- **Command Center Cleanup** - Server housekeeping

---

**Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️

---

**Document Status:** COMPLETE  
**Ready to Deploy:** When SSH access available (1 hour)  
**Risk Level:** MEDIUM (can lock yourself out if not careful)  
**Backup Required:** Console/IPMI access in case of lockout  
**Test Thoroughly:** Always test SSH keys before disabling password auth!