2e953ce312
- Comprehensive task documentation for migrating from AnythingLLM to Dify+n8n+Qdrant - 8 detailed documents covering every aspect of deployment - Complete step-by-step commands (zero assumptions) - Prerequisites checklist (20 items) - Deployment plan in 2 parts (11 phases, every command) - Configuration files (all configs with exact content) - Recovery procedures (4 disaster scenarios) - Verification guide (30 tests, complete checklist) - Troubleshooting guide (common issues + solutions) Built by: The Chronicler #21 For: Meg, Holly, and children not yet born Time investment: 10-15 hours execution time Purpose: Enable Meg/Holly autonomous work with Git write-back This deployment enables: - RBAC (Meg sees all, Holly sees Pokerole only) - Git write-back via ai-proposals branch - Discord approval workflow (one-click merge) - Self-healing (80% of failures) - Automated daily backups - Complete monitoring Documentation is so detailed that any future Chronicler can execute this deployment with zero prior knowledge and complete confidence. Fire + Frost + Foundation = Where Love Builds Legacy
571 lines
11 KiB
Markdown
571 lines
11 KiB
Markdown
# VERIFICATION AND TESTING GUIDE
|
|
|
|
**Complete testing procedures to verify deployment success**
|
|
|
|
Run EVERY test before marking deployment complete.
|
|
|
|
---
|
|
|
|
## ✅ VERIFICATION CHECKLIST
|
|
|
|
**All items must pass:**
|
|
|
|
- [ ] Infrastructure Tests (7 tests)
|
|
- [ ] Query Accuracy Tests (5 tests)
|
|
- [ ] Update Workflow Tests (6 tests)
|
|
- [ ] RBAC Tests (4 tests)
|
|
- [ ] Self-Healing Tests (3 tests)
|
|
- [ ] Monitoring Tests (3 tests)
|
|
- [ ] Backup Tests (2 tests)
|
|
|
|
**Total: 30 tests**
|
|
|
|
---
|
|
|
|
## 🏗️ INFRASTRUCTURE TESTS
|
|
|
|
### Test 1.1: Docker Services Running
|
|
|
|
```bash
|
|
docker-compose ps
|
|
```
|
|
|
|
**Expected:** All 7 services show "Up"
|
|
- db
|
|
- redis
|
|
- dify-api
|
|
- dify-worker
|
|
- dify-web
|
|
- qdrant
|
|
- n8n
|
|
|
|
**Pass criteria:** No services in "Exit" or "Restarting" state
|
|
|
|
---
|
|
|
|
### Test 1.2: Nginx Running
|
|
|
|
```bash
|
|
systemctl status nginx
|
|
```
|
|
|
|
**Expected:** "active (running)"
|
|
|
|
```bash
|
|
nginx -t
|
|
```
|
|
|
|
**Expected:** "syntax is ok" and "test is successful"
|
|
|
|
**Pass criteria:** Nginx active with valid configuration
|
|
|
|
---
|
|
|
|
### Test 1.3: SSL Certificates Valid
|
|
|
|
```bash
|
|
curl -I https://codex.firefrostgaming.com
|
|
```
|
|
|
|
**Expected:** HTTP/2 200, valid SSL certificate
|
|
|
|
```bash
|
|
curl -I https://n8n.firefrostgaming.com
|
|
```
|
|
|
|
**Expected:** HTTP/2 200, valid SSL certificate
|
|
|
|
**Pass criteria:** Both domains respond with HTTPS
|
|
|
|
---
|
|
|
|
### Test 1.4: Dify UI Accessible
|
|
|
|
**Action:** Open browser to https://codex.firefrostgaming.com
|
|
|
|
**Expected:** Dify interface loads, can login
|
|
|
|
**Pass criteria:** UI functional, no errors
|
|
|
|
---
|
|
|
|
### Test 1.5: n8n UI Accessible
|
|
|
|
**Action:** Open browser to https://n8n.firefrostgaming.com
|
|
|
|
**Expected:** n8n interface loads, can login
|
|
|
|
**Pass criteria:** UI functional, workflows visible
|
|
|
|
---
|
|
|
|
### Test 1.6: Ollama Connection
|
|
|
|
```bash
|
|
docker exec -it $(docker ps -qf "name=dify-api") \
|
|
curl http://host.docker.internal:11434/api/version
|
|
```
|
|
|
|
**Expected:** JSON response with Ollama version
|
|
|
|
**Pass criteria:** Dify can reach Ollama on host
|
|
|
|
---
|
|
|
|
### Test 1.7: Qdrant Healthy
|
|
|
|
```bash
|
|
curl http://127.0.0.1:6333/
|
|
```
|
|
|
|
**Expected:** JSON response with Qdrant version
|
|
|
|
**Pass criteria:** Qdrant responding, no errors
|
|
|
|
---
|
|
|
|
## 🎯 QUERY ACCURACY TESTS
|
|
|
|
### Test 2.1: Current Tasks Query
|
|
|
|
**Query in Dify:** "What are the current Tier 0 tasks?"
|
|
|
|
**Expected answer includes:**
|
|
- Whitelist Manager
|
|
- NC1 Cleanup
|
|
- Staff Recruitment Launch
|
|
|
|
**Must NOT include:**
|
|
- Initial Server Setup
|
|
- Network Configuration
|
|
- Other archived tasks
|
|
|
|
**Pass criteria:** Returns ONLY current tasks, no archived content
|
|
|
|
---
|
|
|
|
### Test 2.2: Server Information Query
|
|
|
|
**Query:** "What servers does Firefrost Gaming operate?"
|
|
|
|
**Expected answer includes:**
|
|
- Command Center (63.143.34.217)
|
|
- Billing VPS (38.68.14.188)
|
|
- Panel VPS (45.94.168.138)
|
|
- Ghost VPS (64.50.188.14)
|
|
- TX1 Dallas (38.68.14.26)
|
|
- NC1 Charlotte (216.239.104.130)
|
|
|
|
**Pass criteria:** All 6 servers listed with correct IPs
|
|
|
|
---
|
|
|
|
### Test 2.3: Recent Work Query
|
|
|
|
**Query:** "What was accomplished in the most recent Codex deployment session?"
|
|
|
|
**Expected answer includes:**
|
|
- Deployer's work (Chronicler #20)
|
|
- Phase 1 deployment
|
|
- AnythingLLM setup
|
|
- Document sync
|
|
|
|
**Pass criteria:** Returns information about Deployer, not older sessions
|
|
|
|
---
|
|
|
|
### Test 2.4: Specific Document Query
|
|
|
|
**Query:** "What is the Frostwall Protocol?"
|
|
|
|
**Expected:** Accurate description from current documentation
|
|
|
|
**Must NOT:** Return placeholder text or "I don't have that information"
|
|
|
|
**Pass criteria:** Correct, detailed answer from knowledge base
|
|
|
|
---
|
|
|
|
### Test 2.5: Archive Exclusion Test
|
|
|
|
**Query:** "Tell me about the initial server setup tasks"
|
|
|
|
**Expected:** Either:
|
|
- "That information is archived" OR
|
|
- Returns current setup process (not old archived version)
|
|
|
|
**Must NOT:** Return old archived setup documentation as if it's current
|
|
|
|
**Pass criteria:** Archived content not treated as current
|
|
|
|
---
|
|
|
|
## 📝 UPDATE WORKFLOW TESTS
|
|
|
|
### Test 3.1: Valid Update Request
|
|
|
|
**As Meg in Dify:**
|
|
|
|
**Request:** "Update docs/test/verification.md with content: Test update at [current timestamp]"
|
|
|
|
**Expected sequence:**
|
|
1. AI calls update_codex tool
|
|
2. Validation passes
|
|
3. Commits to ai-proposals branch
|
|
4. Discord notification appears in #codex-alerts
|
|
5. Notification has "Approve & Merge" and "Reject" buttons
|
|
|
|
**Pass criteria:** All 5 steps complete successfully
|
|
|
|
---
|
|
|
|
### Test 3.2: Approval Workflow
|
|
|
|
**As Michael in Discord:**
|
|
|
|
**Action:** Click "Approve & Merge" button on proposal
|
|
|
|
**Expected:**
|
|
1. Button acknowledges click
|
|
2. n8n merges to main
|
|
3. Pushes to Gitea
|
|
4. Re-indexes Dify
|
|
5. Success notification in Discord
|
|
|
|
**Verify on TX1:**
|
|
```bash
|
|
cd /opt/firefrost-codex/git-repos/main
|
|
git log -1
|
|
cat docs/test/verification.md
|
|
```
|
|
|
|
**Expected:** Latest commit is the AI update, file contains test content
|
|
|
|
**Pass criteria:** Full approval workflow works end-to-end
|
|
|
|
---
|
|
|
|
### Test 3.3: Protected File Block
|
|
|
|
**As Meg:**
|
|
|
|
**Request:** "Update .env file to change the database password"
|
|
|
|
**Expected:**
|
|
1. AI attempts update
|
|
2. Validation BLOCKS it
|
|
3. Discord shows "Access Restricted" message
|
|
4. Clear explanation why it's blocked
|
|
|
|
**Pass criteria:** Protected file cannot be modified
|
|
|
|
---
|
|
|
|
### Test 3.4: Invalid Content Block
|
|
|
|
**As Meg:**
|
|
|
|
**Request:** "Update docs/test/empty.md with content that is empty"
|
|
|
|
**Expected:**
|
|
1. Validation catches empty content
|
|
2. Update blocked
|
|
3. Error message in Discord
|
|
|
|
**Pass criteria:** Validation prevents bad updates
|
|
|
|
---
|
|
|
|
### Test 3.5: Rejection Workflow
|
|
|
|
**As Meg:**
|
|
|
|
**Request:** Make another update request
|
|
|
|
**As Michael:**
|
|
|
|
**Action:** Click "Reject" button
|
|
|
|
**Expected:**
|
|
1. Change stays in ai-proposals branch
|
|
2. Does NOT merge to main
|
|
3. Notification sent that proposal was rejected
|
|
|
|
**Pass criteria:** Rejection workflow works, main branch unchanged
|
|
|
|
---
|
|
|
|
### Test 3.6: Concurrent Update Handling
|
|
|
|
**Setup:** Two users request updates simultaneously
|
|
|
|
**Expected:** Queue processes them sequentially, no conflicts
|
|
|
|
**Pass criteria:** Both updates succeed, no corruption
|
|
|
|
---
|
|
|
|
## 🔒 RBAC TESTS
|
|
|
|
### Test 4.1: Meg Full Access
|
|
|
|
**As Meg:**
|
|
|
|
**Test queries:**
|
|
1. "What are the Tier 0 tasks?" (should work)
|
|
2. "What is the Frostwall Protocol?" (should work - infrastructure doc)
|
|
3. "Show me server IPs" (should work - sensitive info)
|
|
|
|
**Expected:** Meg gets answers to ALL queries
|
|
|
|
**Pass criteria:** No access restrictions for Meg
|
|
|
|
---
|
|
|
|
### Test 4.2: Holly Restricted Access
|
|
|
|
**As Holly:**
|
|
|
|
**Test queries:**
|
|
1. "What are the Tier 0 tasks?" (should be BLOCKED - not in Pokerole)
|
|
2. "What is the Frostwall Protocol?" (should be BLOCKED - infrastructure)
|
|
3. "Tell me about the Pokerole campaign" (should work - in her workspace)
|
|
|
|
**Expected:** Holly ONLY sees Pokerole content
|
|
|
|
**Pass criteria:** Holly cannot access Firefrost infrastructure docs
|
|
|
|
---
|
|
|
|
### Test 4.3: Workspace Switching
|
|
|
|
**As Meg:**
|
|
|
|
**Action:** Switch between Firefrost Admin and Pokerole HQ workspaces
|
|
|
|
**Expected:** Can access both, knowledge base changes based on workspace
|
|
|
|
**As Holly:**
|
|
|
|
**Action:** Attempt to access Firefrost Admin workspace
|
|
|
|
**Expected:** BLOCKED - workspace not available
|
|
|
|
**Pass criteria:** Workspace permissions enforced
|
|
|
|
---
|
|
|
|
### Test 4.4: Update Permissions
|
|
|
|
**As Holly:**
|
|
|
|
**Request:** "Update Firefrost recruitment doc"
|
|
|
|
**Expected:** BLOCKED - she can only update Pokerole docs
|
|
|
|
**As Meg:**
|
|
|
|
**Request:** Same update
|
|
|
|
**Expected:** Works - she has permission
|
|
|
|
**Pass criteria:** Update permissions match knowledge base access
|
|
|
|
---
|
|
|
|
## 🔄 SELF-HEALING TESTS
|
|
|
|
### Test 5.1: Dify Crash Recovery
|
|
|
|
**Test:**
|
|
```bash
|
|
docker-compose restart dify-api
|
|
```
|
|
|
|
**Monitor:**
|
|
1. Uptime Kuma shows service down
|
|
2. Docker auto-restarts service
|
|
3. Service comes back online within 60 seconds
|
|
4. Users can access Codex again
|
|
|
|
**Pass criteria:** Automatic recovery without manual intervention
|
|
|
|
---
|
|
|
|
### Test 5.2: Git Unreachable Handling
|
|
|
|
**Test:**
|
|
```bash
|
|
# Block Git temporarily
|
|
sudo iptables -A OUTPUT -d git.firefrostgaming.com -j DROP
|
|
```
|
|
|
|
**Request update as Meg**
|
|
|
|
**Expected:**
|
|
1. Commit fails (Git unreachable)
|
|
2. Update queued for retry
|
|
3. Discord shows "Update queued, will retry in 5 minutes"
|
|
4. No error crash
|
|
|
|
**Restore:**
|
|
```bash
|
|
sudo iptables -D OUTPUT -d git.firefrostgaming.com -j DROP
|
|
```
|
|
|
|
**Verify:** Queued update processes automatically
|
|
|
|
**Pass criteria:** Graceful degradation, auto-recovery
|
|
|
|
---
|
|
|
|
### Test 5.3: Qdrant Rebuild
|
|
|
|
**Test:**
|
|
```bash
|
|
docker-compose stop qdrant
|
|
rm -rf volumes/qdrant/storage/*
|
|
docker-compose start qdrant
|
|
```
|
|
|
|
**Action:** Trigger Git sync workflow in n8n
|
|
|
|
**Expected:**
|
|
1. Qdrant rebuilds vectors from Git
|
|
2. Queries work again within 10 minutes
|
|
3. No data loss
|
|
|
|
**Pass criteria:** Vector DB can be rebuilt from Git
|
|
|
|
---
|
|
|
|
## 📊 MONITORING TESTS
|
|
|
|
### Test 6.1: Uptime Kuma Monitors
|
|
|
|
**Action:** Check Uptime Kuma dashboard
|
|
|
|
**Expected monitors present:**
|
|
- Firefrost Codex (Dify)
|
|
- Firefrost n8n
|
|
- Firefrost Qdrant
|
|
|
|
**All showing:** Green (UP status)
|
|
|
|
**Pass criteria:** All 3 monitors healthy
|
|
|
|
---
|
|
|
|
### Test 6.2: Discord Notifications
|
|
|
|
**Test:** Trigger Git sync workflow manually
|
|
|
|
**Expected:** Success notification in #codex-alerts channel
|
|
|
|
**Verify notification includes:**
|
|
- Green color (success)
|
|
- File count
|
|
- Timestamp
|
|
- System info in footer
|
|
|
|
**Pass criteria:** Discord webhook working, notifications formatted correctly
|
|
|
|
---
|
|
|
|
### Test 6.3: Critical Alert Test
|
|
|
|
**Test:** Stop Dify for 15+ minutes
|
|
|
|
**Expected:**
|
|
1. Uptime Kuma detects down
|
|
2. Critical alert sent to #system-critical
|
|
3. Michael mentioned in message
|
|
4. Clear action steps provided
|
|
|
|
**Pass criteria:** Critical alerts work, Michael gets notified
|
|
|
|
---
|
|
|
|
## 💾 BACKUP TESTS
|
|
|
|
### Test 7.1: Manual Backup
|
|
|
|
**Test:**
|
|
```bash
|
|
/opt/firefrost_backup.sh
|
|
```
|
|
|
|
**Expected:**
|
|
1. Backup completes without errors
|
|
2. Tarball created in /opt/
|
|
3. Transferred to Command Center
|
|
4. Backup log updated
|
|
|
|
**Verify:**
|
|
```bash
|
|
ls -lh /opt/firefrost_codex_*.tar.gz
|
|
ssh root@63.143.34.217 "ls -lh /root/backups/firefrost-codex/ | tail -1"
|
|
```
|
|
|
|
**Pass criteria:** Backup created and transferred successfully
|
|
|
|
---
|
|
|
|
### Test 7.2: Backup Restore Test
|
|
|
|
**Test:**
|
|
```bash
|
|
# Create test database
|
|
docker run --name test-postgres -e POSTGRES_PASSWORD=test -d postgres:15-alpine
|
|
|
|
# Extract latest backup
|
|
cd /tmp
|
|
tar -xzf /opt/firefrost_codex_*.tar.gz
|
|
cd codex_backup_*/
|
|
|
|
# Restore into test database
|
|
cat dify_postgres.sql | docker exec -i test-postgres psql -U postgres
|
|
|
|
# Check for errors
|
|
docker logs test-postgres | grep ERROR
|
|
|
|
# Cleanup
|
|
docker rm -f test-postgres
|
|
```
|
|
|
|
**Expected:** No errors during restore
|
|
|
|
**Pass criteria:** Backup is valid and restorable
|
|
|
|
---
|
|
|
|
## 🎉 FINAL VERIFICATION
|
|
|
|
**After ALL tests pass:**
|
|
|
|
### Final Checklist
|
|
|
|
- [ ] All 30 tests completed
|
|
- [ ] Zero failures
|
|
- [ ] All issues documented
|
|
- [ ] Meg can work independently
|
|
- [ ] Holly can work independently
|
|
- [ ] Michael can approve from Discord
|
|
- [ ] System self-heals common failures
|
|
- [ ] Backups running automatically
|
|
- [ ] Monitoring active and alerting
|
|
- [ ] Documentation updated
|
|
|
|
### Sign-Off
|
|
|
|
**Deployed by:** ____________________
|
|
**Date:** ____________________
|
|
**Session:** ____________________
|
|
|
|
**Signature:** This deployment is COMPLETE and PRODUCTION-READY
|
|
|
|
---
|
|
|
|
**Fire + Frost + Foundation = Where Testing Ensures Quality** 💙🔥❄️
|
|
|