firefrost-operations-manual/docs/tasks/firefrost-codex-migration-to-open-webui/VERIFICATION.md

VERIFICATION AND TESTING GUIDE

Complete testing procedures to verify deployment success

Run EVERY test before marking deployment complete.

---

✅ VERIFICATION CHECKLIST

All items must pass:

• Infrastructure Tests (7 tests)
• Query Accuracy Tests (5 tests)
• Update Workflow Tests (6 tests)
• RBAC Tests (4 tests)
• Self-Healing Tests (3 tests)
• Monitoring Tests (3 tests)
• Backup Tests (2 tests)

Total: 30 tests

---

🏗️ INFRASTRUCTURE TESTS

Test 1.1: Docker Services Running

docker-compose ps

Expected: All 7 services show "Up"

• db
• redis
• dify-api
• dify-worker
• dify-web
• qdrant
• n8n

Pass criteria: No services in "Exit" or "Restarting" state

---

Test 1.2: Nginx Running

systemctl status nginx

Expected: "active (running)"

nginx -t

Expected: "syntax is ok" and "test is successful"

Pass criteria: Nginx active with valid configuration

---

Test 1.3: SSL Certificates Valid

curl -I https://codex.firefrostgaming.com

Expected: HTTP/2 200, valid SSL certificate

curl -I https://n8n.firefrostgaming.com

Expected: HTTP/2 200, valid SSL certificate

Pass criteria: Both domains respond with HTTPS

---

Test 1.4: Dify UI Accessible

Action: Open browser to https://codex.firefrostgaming.com

Expected: Dify interface loads, can login

Pass criteria: UI functional, no errors

---

Test 1.5: n8n UI Accessible

Action: Open browser to https://n8n.firefrostgaming.com

Expected: n8n interface loads, can login

Pass criteria: UI functional, workflows visible

---

Test 1.6: Ollama Connection

docker exec -it $(docker ps -qf "name=dify-api") \
  curl http://host.docker.internal:11434/api/version

Expected: JSON response with Ollama version

Pass criteria: Dify can reach Ollama on host

---

Test 1.7: Qdrant Healthy

curl http://127.0.0.1:6333/

Expected: JSON response with Qdrant version

Pass criteria: Qdrant responding, no errors

---

🎯 QUERY ACCURACY TESTS

Test 2.1: Current Tasks Query

Query in Dify: "What are the current Tier 0 tasks?"

Expected answer includes:

• Whitelist Manager
• NC1 Cleanup
• Staff Recruitment Launch

Must NOT include:

• Initial Server Setup
• Network Configuration
• Other archived tasks

Pass criteria: Returns ONLY current tasks, no archived content

---

Test 2.2: Server Information Query

Query: "What servers does Firefrost Gaming operate?"

Expected answer includes:

• Command Center (63.143.34.217)
• Billing VPS (38.68.14.188)
• Panel VPS (45.94.168.138)
• Ghost VPS (64.50.188.14)
• TX1 Dallas (38.68.14.26)
• NC1 Charlotte (216.239.104.130)

Pass criteria: All 6 servers listed with correct IPs

---

Test 2.3: Recent Work Query

Query: "What was accomplished in the most recent Codex deployment session?"

Expected answer includes:

• Deployer's work (Chronicler #20)
• Phase 1 deployment
• AnythingLLM setup
• Document sync

Pass criteria: Returns information about Deployer, not older sessions

---

Test 2.4: Specific Document Query

Query: "What is the Frostwall Protocol?"

Expected: Accurate description from current documentation

Must NOT: Return placeholder text or "I don't have that information"

Pass criteria: Correct, detailed answer from knowledge base

---

Test 2.5: Archive Exclusion Test

Query: "Tell me about the initial server setup tasks"

Expected: Either:

• "That information is archived" OR
• Returns current setup process (not old archived version)

Must NOT: Return old archived setup documentation as if it's current

Pass criteria: Archived content not treated as current

---

📝 UPDATE WORKFLOW TESTS

Test 3.1: Valid Update Request

As Meg in Dify:

Request: "Update docs/test/verification.md with content: Test update at [current timestamp]"

Expected sequence:

1. AI calls update_codex tool
2. Validation passes
3. Commits to ai-proposals branch
4. Discord notification appears in #codex-alerts
5. Notification has "Approve & Merge" and "Reject" buttons

Pass criteria: All 5 steps complete successfully

---

Test 3.2: Approval Workflow

As Michael in Discord:

Action: Click "Approve & Merge" button on proposal

Expected:

1. Button acknowledges click
2. n8n merges to main
3. Pushes to Gitea
4. Re-indexes Dify
5. Success notification in Discord

Verify on TX1:

cd /opt/firefrost-codex/git-repos/main
git log -1
cat docs/test/verification.md

Expected: Latest commit is the AI update, file contains test content

Pass criteria: Full approval workflow works end-to-end

---

Test 3.3: Protected File Block

As Meg:

Request: "Update .env file to change the database password"

Expected:

1. AI attempts update
2. Validation BLOCKS it
3. Discord shows "Access Restricted" message
4. Clear explanation why it's blocked

Pass criteria: Protected file cannot be modified

---

Test 3.4: Invalid Content Block

As Meg:

Request: "Update docs/test/empty.md with content that is empty"

Expected:

1. Validation catches empty content
2. Update blocked
3. Error message in Discord

Pass criteria: Validation prevents bad updates

---

Test 3.5: Rejection Workflow

As Meg:

Request: Make another update request

As Michael:

Action: Click "Reject" button

Expected:

1. Change stays in ai-proposals branch
2. Does NOT merge to main
3. Notification sent that proposal was rejected

Pass criteria: Rejection workflow works, main branch unchanged

---

Test 3.6: Concurrent Update Handling

Setup: Two users request updates simultaneously

Expected: Queue processes them sequentially, no conflicts

Pass criteria: Both updates succeed, no corruption

---

🔒 RBAC TESTS

Test 4.1: Meg Full Access

As Meg:

Test queries:

1. "What are the Tier 0 tasks?" (should work)
2. "What is the Frostwall Protocol?" (should work - infrastructure doc)
3. "Show me server IPs" (should work - sensitive info)

Expected: Meg gets answers to ALL queries

Pass criteria: No access restrictions for Meg

---

Test 4.2: Holly Restricted Access

As Holly:

Test queries:

1. "What are the Tier 0 tasks?" (should be BLOCKED - not in Pokerole)
2. "What is the Frostwall Protocol?" (should be BLOCKED - infrastructure)
3. "Tell me about the Pokerole campaign" (should work - in her workspace)

Expected: Holly ONLY sees Pokerole content

Pass criteria: Holly cannot access Firefrost infrastructure docs

---

Test 4.3: Workspace Switching

As Meg:

Action: Switch between Firefrost Admin and Pokerole HQ workspaces

Expected: Can access both, knowledge base changes based on workspace

As Holly:

Action: Attempt to access Firefrost Admin workspace

Expected: BLOCKED - workspace not available

Pass criteria: Workspace permissions enforced

---

Test 4.4: Update Permissions

As Holly:

Request: "Update Firefrost recruitment doc"

Expected: BLOCKED - she can only update Pokerole docs

As Meg:

Request: Same update

Expected: Works - she has permission

Pass criteria: Update permissions match knowledge base access

---

🔄 SELF-HEALING TESTS

Test 5.1: Dify Crash Recovery

Test:

docker-compose restart dify-api

Monitor:

1. Uptime Kuma shows service down
2. Docker auto-restarts service
3. Service comes back online within 60 seconds
4. Users can access Codex again

Pass criteria: Automatic recovery without manual intervention

---

Test 5.2: Git Unreachable Handling

Test:

# Block Git temporarily
sudo iptables -A OUTPUT -d git.firefrostgaming.com -j DROP

Request update as Meg

Expected:

1. Commit fails (Git unreachable)
2. Update queued for retry
3. Discord shows "Update queued, will retry in 5 minutes"
4. No error crash

Restore:

sudo iptables -D OUTPUT -d git.firefrostgaming.com -j DROP

Verify: Queued update processes automatically

Pass criteria: Graceful degradation, auto-recovery

---

Test 5.3: Qdrant Rebuild

Test:

docker-compose stop qdrant
rm -rf volumes/qdrant/storage/*
docker-compose start qdrant

Action: Trigger Git sync workflow in n8n

Expected:

1. Qdrant rebuilds vectors from Git
2. Queries work again within 10 minutes
3. No data loss

Pass criteria: Vector DB can be rebuilt from Git

---

📊 MONITORING TESTS

Test 6.1: Uptime Kuma Monitors

Action: Check Uptime Kuma dashboard

Expected monitors present:

• Firefrost Codex (Dify)
• Firefrost n8n
• Firefrost Qdrant

All showing: Green (UP status)

Pass criteria: All 3 monitors healthy

---

Test 6.2: Discord Notifications

Test: Trigger Git sync workflow manually

Expected: Success notification in #codex-alerts channel

Verify notification includes:

• Green color (success)
• File count
• Timestamp
• System info in footer

Pass criteria: Discord webhook working, notifications formatted correctly

---

Test 6.3: Critical Alert Test

Test: Stop Dify for 15+ minutes

Expected:

1. Uptime Kuma detects down
2. Critical alert sent to #system-critical
3. Michael mentioned in message
4. Clear action steps provided

Pass criteria: Critical alerts work, Michael gets notified

---

💾 BACKUP TESTS

Test 7.1: Manual Backup

Test:

/opt/firefrost_backup.sh

Expected:

1. Backup completes without errors
2. Tarball created in /opt/
3. Transferred to Command Center
4. Backup log updated

Verify:

ls -lh /opt/firefrost_codex_*.tar.gz
ssh root@63.143.34.217 "ls -lh /root/backups/firefrost-codex/ | tail -1"

Pass criteria: Backup created and transferred successfully

---

Test 7.2: Backup Restore Test

Test:

# Create test database
docker run --name test-postgres -e POSTGRES_PASSWORD=test -d postgres:15-alpine

# Extract latest backup
cd /tmp
tar -xzf /opt/firefrost_codex_*.tar.gz
cd codex_backup_*/

# Restore into test database
cat dify_postgres.sql | docker exec -i test-postgres psql -U postgres

# Check for errors
docker logs test-postgres | grep ERROR

# Cleanup
docker rm -f test-postgres

Expected: No errors during restore

Pass criteria: Backup is valid and restorable

---

🎉 FINAL VERIFICATION

After ALL tests pass:

Final Checklist

• All 30 tests completed
• Zero failures
• All issues documented
• Meg can work independently
• Holly can work independently
• Michael can approve from Discord
• System self-heals common failures
• Backups running automatically
• Monitoring active and alerting
• Documentation updated

Sign-Off

Deployed by: ____________________
Date: ____________________
Session: ____________________

Signature: This deployment is COMPLETE and PRODUCTION-READY

---

Fire + Frost + Foundation = Where Testing Ensures Quality 💙🔥❄️