# RECOVERY AND BACKUP PROCEDURES

**Complete disaster recovery guide for Firefrost Knowledge Engine**

---

## 🎯 BACKUP STRATEGY

**Philosophy:** Git is the source of truth. Vector DB can always be rebuilt.

**What to backup:**
✅ PostgreSQL database (user accounts, settings, chat histories)
✅ n8n volumes (workflows, credentials)
✅ Configuration files (docker-compose.yml, .env, Nginx configs)

**What NOT to backup:**
❌ Qdrant vectors (re-index from Git in minutes)
❌ Redis cache (temporary data only)
❌ Git repositories (Gitea is the master)

---

## 📅 AUTOMATED DAILY BACKUPS

**Schedule:** Daily at 4:00 AM (cron)

**Script location:** `/opt/firefrost_backup.sh`

**What it does:**
1. Dumps PostgreSQL database
2. Copies n8n workflows and credentials
3. Copies configuration files
4. Compresses into tarball
5. Transfers to Command Center (offsite)
6. Removes local backups older than 7 days

**Backup location:**
- **Local:** `/opt/firefrost_codex_YYYYMMDD_HHMM.tar.gz`
- **Offsite:** `root@63.143.34.217:/root/backups/firefrost-codex/`

**Retention:** 7 days local, unlimited offsite

**Monitor backups:**
```bash
# Check recent backups
ls -lh /opt/firefrost_codex_*.tar.gz

# Check backup logs
tail -f /var/log/firefrost-backup.log

# Verify offsite transfer
ssh root@63.143.34.217 "ls -lh /root/backups/firefrost-codex/"
```

---

## 🔄 RECOVERY SCENARIOS

### Scenario A: Qdrant Corrupted (Wrong Answers)

**Symptoms:**
- Codex returns incorrect information
- Searches find wrong documents
- Outdated content returned

**Diagnosis:**
```bash
# Check Qdrant health
curl http://127.0.0.1:6333/
```

**Recovery (5-10 minutes):**
```bash
# Stop Qdrant
docker-compose stop qdrant

# Delete corrupted data
rm -rf /opt/firefrost-codex/volumes/qdrant/storage/*

# Restart Qdrant
docker-compose start qdrant

# Trigger Git sync in n8n to rebuild
# (Open n8n, run "Firefrost Git Sync" workflow manually)
```

**Verification:**
- Test queries return correct current information
- No archived docs in results

---

### Scenario B: n8n Workflows Lost

**Symptoms:**
- Workflows missing or corrupted
- Automation not working
- Can't update docs via Codex

**Recovery (10-15 minutes):**
```bash
# Stop n8n
docker-compose stop n8n

# Extract latest backup
cd /tmp
tar -xzf /opt/firefrost_codex_LATEST_BACKUP.tar.gz
cd codex_backup_*/

# Restore n8n data
rm -rf /opt/firefrost-codex/volumes/n8n/*
cp -r n8n_data/* /opt/firefrost-codex/volumes/n8n/

# Restart n8n
cd /opt/firefrost-codex
docker-compose start n8n
```

**Verification:**
```bash
# Check workflows restored
curl http://127.0.0.1:5678
# Login and verify workflows present
```

---

### Scenario C: Database Corruption

**Symptoms:**
- Can't login to Dify
- User accounts missing
- Settings reset
- Chat history gone

**Recovery (15-20 minutes):**
```bash
# Stop all services
cd /opt/firefrost-codex
docker-compose down

# Extract latest backup
cd /tmp
tar -xzf /opt/firefrost_codex_LATEST_BACKUP.tar.gz
cd codex_backup_*/

# Restore database
docker-compose up -d db
sleep 30  # Wait for database to start
cat dify_postgres.sql | docker exec -i $(docker ps -qf "name=db") psql -U postgres

# Restart all services
cd /opt/firefrost-codex
docker-compose up -d
```

**Verification:**
- Login to Dify works
- User accounts present
- Settings preserved
- Chat history intact

---

### Scenario D: Complete TX1 Server Crash

**Symptoms:**
- TX1 completely down
- Hardware failure
- OS corrupted
- Full rebuild needed

**Recovery (30-60 minutes):**

**Step 1: Provision new server**
- Install Ubuntu 22.04 LTS
- Configure network (same IP if possible)
- Install Docker and Nginx

**Step 2: Retrieve backup from Command Center**
```bash
# On new TX1
mkdir -p /opt
scp root@63.143.34.217:/root/backups/firefrost-codex/firefrost_codex_LATEST.tar.gz /opt/

# Extract
cd /opt
tar -xzf firefrost_codex_LATEST.tar.gz
cd codex_backup_*/
```

**Step 3: Restore configurations**
```bash
# Create deployment directory
mkdir -p /opt/firefrost-codex
cd /opt/firefrost-codex

# Restore files
cp /tmp/codex_backup_*/docker-compose.yml .
cp /tmp/codex_backup_*/.env .
mkdir -p volumes/n8n
cp -r /tmp/codex_backup_*/n8n_data/* volumes/n8n/

# Restore Nginx config
cp /tmp/codex_backup_*/firefrost-codex.conf /etc/nginx/sites-available/
ln -s /etc/nginx/sites-available/firefrost-codex.conf /etc/nginx/sites-enabled/
```

**Step 4: Regenerate SSL certificates**
```bash
systemctl stop nginx
certbot certonly --standalone \
  -d codex.firefrostgaming.com \
  -d n8n.firefrostgaming.com \
  --email codex@firefrostgaming.com \
  --agree-tos
systemctl start nginx
```

**Step 5: Start Docker stack**
```bash
cd /opt/firefrost-codex
docker-compose up -d
sleep 60  # Wait for services
```

**Step 6: Restore database**
```bash
cat /tmp/codex_backup_*/dify_postgres.sql | docker exec -i $(docker ps -qf "name=db") psql -U postgres
docker-compose restart dify-api dify-worker
```

**Step 7: Rebuild Qdrant from Git**
- Access n8n at https://n8n.firefrostgaming.com
- Run "Firefrost Git Sync" workflow manually
- Wait 5-10 minutes for indexing

**Total downtime:** ~45 minutes (assuming new server ready)

---

## 🧪 TESTING BACKUPS

**CRITICAL:** Never trust an untested backup

**Test quarterly (every 3 months):**

**Dry-Run Database Restore:**
```bash
# Create temporary test database
docker run --name test-postgres \
  -e POSTGRES_PASSWORD=test \
  -d postgres:15-alpine

# Restore backup into test database
cat dify_postgres.sql | docker exec -i test-postgres psql -U postgres

# Check for errors
docker logs test-postgres | grep ERROR

# Cleanup
docker rm -f test-postgres
```

**If no errors:** Backup is valid

**Document test results:**
```bash
echo "$(date): Backup test PASSED" >> /var/log/firefrost-backup-tests.log
```

---

## 🚨 EMERGENCY CONTACTS

**If disaster recovery fails:**

1. **Check TROUBLESHOOTING.md** for common issues
2. **Review Docker logs:** `docker-compose logs`
3. **Check Nginx logs:** `/var/log/nginx/error.log`
4. **Verify backups exist:** Both local and Command Center
5. **If stuck:** Document what happened, wait for fresh Chronicler session

**No external support needed - we built this ourselves**

---

## 📊 BACKUP MONITORING

**Verify backups running:**

```bash
# Check cron job
crontab -l | grep firefrost

# Check recent backup log
tail -20 /var/log/firefrost-backup.log

# Check backup exists today
ls -lh /opt/firefrost_codex_$(date +%Y%m%d)*.tar.gz

# Verify transferred to Command Center
ssh root@63.143.34.217 "ls -lh /root/backups/firefrost-codex/ | tail -5"
```

**Add to Uptime Kuma (optional):**
- Monitor backup log file modified date
- Alert if >25 hours since last backup
- Indicates backup failure

---

## 💾 MANUAL BACKUP (Before Major Changes)

**Before any major changes, create manual backup:**

```bash
# Run backup script manually
/opt/firefrost_backup.sh

# Tag the backup
mv /opt/firefrost_codex_$(date +%Y%m%d)*.tar.gz \
   /opt/firefrost_codex_before_major_change_$(date +%Y%m%d).tar.gz
```

**Keep pre-change backups for 30 days**

---

## 🎯 BACKUP SUCCESS CRITERIA

**Daily backups must:**
✅ Complete without errors
✅ Transfer to Command Center successfully
✅ Be restorable (tested quarterly)
✅ Include all critical data (DB, n8n, configs)
✅ Be less than 24 hours old

**If ANY criteria fails:** Investigate immediately

---

**Fire + Frost + Foundation = Where Data Never Dies** 💙🔥❄️

