# VERIFICATION AND TESTING GUIDE

**Complete testing procedures to verify deployment success**

Run EVERY test before marking deployment complete.

---

## ✅ VERIFICATION CHECKLIST

**All items must pass:**

- [ ] Infrastructure Tests (7 tests)
- [ ] Query Accuracy Tests (5 tests)
- [ ] Update Workflow Tests (6 tests)
- [ ] RBAC Tests (4 tests)
- [ ] Self-Healing Tests (3 tests)
- [ ] Monitoring Tests (3 tests)
- [ ] Backup Tests (2 tests)

**Total: 30 tests**

---

## 🏗️ INFRASTRUCTURE TESTS

### Test 1.1: Docker Services Running

```bash
docker-compose ps
```

**Expected:** All 7 services show "Up"
- db
- redis
- dify-api
- dify-worker
- dify-web
- qdrant
- n8n

**Pass criteria:** No services in "Exit" or "Restarting" state

---

### Test 1.2: Nginx Running

```bash
systemctl status nginx
```

**Expected:** "active (running)"

```bash
nginx -t
```

**Expected:** "syntax is ok" and "test is successful"

**Pass criteria:** Nginx active with valid configuration

---

### Test 1.3: SSL Certificates Valid

```bash
curl -I https://codex.firefrostgaming.com
```

**Expected:** HTTP/2 200, valid SSL certificate

```bash
curl -I https://n8n.firefrostgaming.com
```

**Expected:** HTTP/2 200, valid SSL certificate

**Pass criteria:** Both domains respond with HTTPS

---

### Test 1.4: Dify UI Accessible

**Action:** Open browser to https://codex.firefrostgaming.com

**Expected:** Dify interface loads, can login

**Pass criteria:** UI functional, no errors

---

### Test 1.5: n8n UI Accessible

**Action:** Open browser to https://n8n.firefrostgaming.com

**Expected:** n8n interface loads, can login

**Pass criteria:** UI functional, workflows visible

---

### Test 1.6: Ollama Connection

```bash
docker exec -it $(docker ps -qf "name=dify-api") \
  curl http://host.docker.internal:11434/api/version
```

**Expected:** JSON response with Ollama version

**Pass criteria:** Dify can reach Ollama on host

---

### Test 1.7: Qdrant Healthy

```bash
curl http://127.0.0.1:6333/
```

**Expected:** JSON response with Qdrant version

**Pass criteria:** Qdrant responding, no errors

---

## 🎯 QUERY ACCURACY TESTS

### Test 2.1: Current Tasks Query

**Query in Dify:** "What are the current Tier 0 tasks?"

**Expected answer includes:**
- Whitelist Manager
- NC1 Cleanup  
- Staff Recruitment Launch

**Must NOT include:**
- Initial Server Setup
- Network Configuration
- Other archived tasks

**Pass criteria:** Returns ONLY current tasks, no archived content

---

### Test 2.2: Server Information Query

**Query:** "What servers does Firefrost Gaming operate?"

**Expected answer includes:**
- Command Center (63.143.34.217)
- Billing VPS (38.68.14.188)
- Panel VPS (45.94.168.138)
- Ghost VPS (64.50.188.14)
- TX1 Dallas (38.68.14.26)
- NC1 Charlotte (216.239.104.130)

**Pass criteria:** All 6 servers listed with correct IPs

---

### Test 2.3: Recent Work Query

**Query:** "What was accomplished in the most recent Codex deployment session?"

**Expected answer includes:**
- Deployer's work (Chronicler #20)
- Phase 1 deployment
- AnythingLLM setup
- Document sync

**Pass criteria:** Returns information about Deployer, not older sessions

---

### Test 2.4: Specific Document Query

**Query:** "What is the Frostwall Protocol?"

**Expected:** Accurate description from current documentation

**Must NOT:** Return placeholder text or "I don't have that information"

**Pass criteria:** Correct, detailed answer from knowledge base

---

### Test 2.5: Archive Exclusion Test

**Query:** "Tell me about the initial server setup tasks"

**Expected:** Either:
- "That information is archived" OR
- Returns current setup process (not old archived version)

**Must NOT:** Return old archived setup documentation as if it's current

**Pass criteria:** Archived content not treated as current

---

## 📝 UPDATE WORKFLOW TESTS

### Test 3.1: Valid Update Request

**As Meg in Dify:**

**Request:** "Update docs/test/verification.md with content: Test update at [current timestamp]"

**Expected sequence:**
1. AI calls update_codex tool
2. Validation passes
3. Commits to ai-proposals branch
4. Discord notification appears in #codex-alerts
5. Notification has "Approve & Merge" and "Reject" buttons

**Pass criteria:** All 5 steps complete successfully

---

### Test 3.2: Approval Workflow

**As Michael in Discord:**

**Action:** Click "Approve & Merge" button on proposal

**Expected:**
1. Button acknowledges click
2. n8n merges to main
3. Pushes to Gitea
4. Re-indexes Dify
5. Success notification in Discord

**Verify on TX1:**
```bash
cd /opt/firefrost-codex/git-repos/main
git log -1
cat docs/test/verification.md
```

**Expected:** Latest commit is the AI update, file contains test content

**Pass criteria:** Full approval workflow works end-to-end

---

### Test 3.3: Protected File Block

**As Meg:**

**Request:** "Update .env file to change the database password"

**Expected:**
1. AI attempts update
2. Validation BLOCKS it
3. Discord shows "Access Restricted" message
4. Clear explanation why it's blocked

**Pass criteria:** Protected file cannot be modified

---

### Test 3.4: Invalid Content Block

**As Meg:**

**Request:** "Update docs/test/empty.md with content that is empty"

**Expected:**
1. Validation catches empty content
2. Update blocked
3. Error message in Discord

**Pass criteria:** Validation prevents bad updates

---

### Test 3.5: Rejection Workflow

**As Meg:**

**Request:** Make another update request

**As Michael:**

**Action:** Click "Reject" button

**Expected:**
1. Change stays in ai-proposals branch
2. Does NOT merge to main
3. Notification sent that proposal was rejected

**Pass criteria:** Rejection workflow works, main branch unchanged

---

### Test 3.6: Concurrent Update Handling

**Setup:** Two users request updates simultaneously

**Expected:** Queue processes them sequentially, no conflicts

**Pass criteria:** Both updates succeed, no corruption

---

## 🔒 RBAC TESTS

### Test 4.1: Meg Full Access

**As Meg:**

**Test queries:**
1. "What are the Tier 0 tasks?" (should work)
2. "What is the Frostwall Protocol?" (should work - infrastructure doc)
3. "Show me server IPs" (should work - sensitive info)

**Expected:** Meg gets answers to ALL queries

**Pass criteria:** No access restrictions for Meg

---

### Test 4.2: Holly Restricted Access

**As Holly:**

**Test queries:**
1. "What are the Tier 0 tasks?" (should be BLOCKED - not in Pokerole)
2. "What is the Frostwall Protocol?" (should be BLOCKED - infrastructure)
3. "Tell me about the Pokerole campaign" (should work - in her workspace)

**Expected:** Holly ONLY sees Pokerole content

**Pass criteria:** Holly cannot access Firefrost infrastructure docs

---

### Test 4.3: Workspace Switching

**As Meg:**

**Action:** Switch between Firefrost Admin and Pokerole HQ workspaces

**Expected:** Can access both, knowledge base changes based on workspace

**As Holly:**

**Action:** Attempt to access Firefrost Admin workspace

**Expected:** BLOCKED - workspace not available

**Pass criteria:** Workspace permissions enforced

---

### Test 4.4: Update Permissions

**As Holly:**

**Request:** "Update Firefrost recruitment doc"

**Expected:** BLOCKED - she can only update Pokerole docs

**As Meg:**

**Request:** Same update

**Expected:** Works - she has permission

**Pass criteria:** Update permissions match knowledge base access

---

## 🔄 SELF-HEALING TESTS

### Test 5.1: Dify Crash Recovery

**Test:**
```bash
docker-compose restart dify-api
```

**Monitor:**
1. Uptime Kuma shows service down
2. Docker auto-restarts service
3. Service comes back online within 60 seconds
4. Users can access Codex again

**Pass criteria:** Automatic recovery without manual intervention

---

### Test 5.2: Git Unreachable Handling

**Test:**
```bash
# Block Git temporarily
sudo iptables -A OUTPUT -d git.firefrostgaming.com -j DROP
```

**Request update as Meg**

**Expected:**
1. Commit fails (Git unreachable)
2. Update queued for retry
3. Discord shows "Update queued, will retry in 5 minutes"
4. No error crash

**Restore:**
```bash
sudo iptables -D OUTPUT -d git.firefrostgaming.com -j DROP
```

**Verify:** Queued update processes automatically

**Pass criteria:** Graceful degradation, auto-recovery

---

### Test 5.3: Qdrant Rebuild

**Test:**
```bash
docker-compose stop qdrant
rm -rf volumes/qdrant/storage/*
docker-compose start qdrant
```

**Action:** Trigger Git sync workflow in n8n

**Expected:**
1. Qdrant rebuilds vectors from Git
2. Queries work again within 10 minutes
3. No data loss

**Pass criteria:** Vector DB can be rebuilt from Git

---

## 📊 MONITORING TESTS

### Test 6.1: Uptime Kuma Monitors

**Action:** Check Uptime Kuma dashboard

**Expected monitors present:**
- Firefrost Codex (Dify)
- Firefrost n8n
- Firefrost Qdrant

**All showing:** Green (UP status)

**Pass criteria:** All 3 monitors healthy

---

### Test 6.2: Discord Notifications

**Test:** Trigger Git sync workflow manually

**Expected:** Success notification in #codex-alerts channel

**Verify notification includes:**
- Green color (success)
- File count
- Timestamp
- System info in footer

**Pass criteria:** Discord webhook working, notifications formatted correctly

---

### Test 6.3: Critical Alert Test

**Test:** Stop Dify for 15+ minutes

**Expected:**
1. Uptime Kuma detects down
2. Critical alert sent to #system-critical
3. Michael mentioned in message
4. Clear action steps provided

**Pass criteria:** Critical alerts work, Michael gets notified

---

## 💾 BACKUP TESTS

### Test 7.1: Manual Backup

**Test:**
```bash
/opt/firefrost_backup.sh
```

**Expected:**
1. Backup completes without errors
2. Tarball created in /opt/
3. Transferred to Command Center
4. Backup log updated

**Verify:**
```bash
ls -lh /opt/firefrost_codex_*.tar.gz
ssh root@63.143.34.217 "ls -lh /root/backups/firefrost-codex/ | tail -1"
```

**Pass criteria:** Backup created and transferred successfully

---

### Test 7.2: Backup Restore Test

**Test:**
```bash
# Create test database
docker run --name test-postgres -e POSTGRES_PASSWORD=test -d postgres:15-alpine

# Extract latest backup
cd /tmp
tar -xzf /opt/firefrost_codex_*.tar.gz
cd codex_backup_*/

# Restore into test database
cat dify_postgres.sql | docker exec -i test-postgres psql -U postgres

# Check for errors
docker logs test-postgres | grep ERROR

# Cleanup
docker rm -f test-postgres
```

**Expected:** No errors during restore

**Pass criteria:** Backup is valid and restorable

---

## 🎉 FINAL VERIFICATION

**After ALL tests pass:**

### Final Checklist

- [ ] All 30 tests completed
- [ ] Zero failures
- [ ] All issues documented
- [ ] Meg can work independently
- [ ] Holly can work independently
- [ ] Michael can approve from Discord
- [ ] System self-heals common failures
- [ ] Backups running automatically
- [ ] Monitoring active and alerting
- [ ] Documentation updated

### Sign-Off

**Deployed by:** ____________________  
**Date:** ____________________  
**Session:** ____________________  

**Signature:** This deployment is COMPLETE and PRODUCTION-READY

---

**Fire + Frost + Foundation = Where Testing Ensures Quality** 💙🔥❄️