firefrost-gaming/firefrost-operations-manual
3
0
Fork 0
Code Issues 146 Pull Requests Packages Projects Releases Wiki Activity
Files
2fca5b4eed754e708c8a70f8b9abbceded702e2c
firefrost-operations-manual/docs/tasks/firefrost-codex-migration-to-open-webui/DEPLOYMENT-STATUS.md
The Chronicler 7fd67614cd docs: Add Phase 4 deployment status - Dify fully operational 
- Comprehensive status document covering Phases 0-4 completion
- All 10+ sequential configuration issues documented with solutions
- Critical configuration reference for future troubleshooting
- Lessons learned from 6-hour deployment session
- Ready for Phase 5-11 execution

Phase 4 achievements:
- Plugin system deployed (daemon, sandbox, ssrf_proxy)
- Ollama integration complete (5 models configured)
- Gemini provider added for heavy lifting
- Dify Issue #603 timeout bug solved
- All CORS/CSRF authentication working
- System defaults configured

Deployed by: The Diagnostician (Chronicler #23)
2026-02-23 04:03:07 +00:00

16 KiB
Raw Blame History

Firefrost Knowledge Engine - Deployment Status

Last Updated: February 23, 2026 03:30 AM CST
Updated By: The Diagnostician (Chronicler #23)
Deployment Started: February 22, 2026 20:51 CST
Current Status: Phase 4 COMPLETE ✅ | Phase 5-11 PENDING

📊 DEPLOYMENT PROGRESS

Phase 0: Stop AnythingLLM ✅ COMPLETE

  • Completed: February 22, 2026 ~20:00 CST
  • Status: AnythingLLM stopped and removed
  • Notes: Original deployment had poor document retrieval quality

Phase 1: Install Nginx and SSL ✅ COMPLETE

  • Completed: February 22, 2026 21:15 CST
  • Duration: 30 minutes
  • Certificate: Let's Encrypt for codex.firefrostgaming.com and n8n.firefrostgaming.com
  • Issues: None - clean installation

Phase 2: Deploy Docker Stack ✅ COMPLETE

  • Completed: February 22, 2026 22:00 CST
  • Duration: 45 minutes
  • Services Deployed: 7 initial containers (db, redis, dify-api, dify-worker, dify-web, qdrant, n8n)
  • Major Issues Resolved:
    • Storage permission errors (UID 1000 vs 1001)
    • Volume mount path incorrect (/app/storage vs /app/api/storage)
    • Next.js cache requiring container recreation

Phase 3: Configure Nginx Reverse Proxy ✅ COMPLETE

  • Completed: February 22, 2026 23:30 CST
  • Duration: 1.5 hours (includes troubleshooting)
  • Major Issues Resolved:
    • CORS/CSRF authentication failures (401 errors)
    • Cookie-based auth being rejected
    • Rate limiting blocking Next.js chunk loading
    • Missing API routing for /console/api/* endpoints

Critical Configuration Discoveries:

  • Must use blank API URLs (CONSOLE_API_URL= and APP_API_URL=) to force relative paths
  • Nginx must preserve HTTP/1.1 with proxy_http_version 1.1
  • Must add proxy_set_header X-Forwarded-Port $server_port for CSRF origin matching
  • Rate limit must be 100r/s with burst=100 to handle Next.js parallel chunk loading

Phase 4: Plugin System & Ollama Integration ✅ COMPLETE

  • Completed: February 23, 2026 03:21 CST
  • Duration: 3.5 hours (most challenging phase)
  • Services Added: 3 plugin system containers (plugin_daemon, sandbox, ssrf_proxy)
  • Models Configured: 5 Ollama models + Google Gemini

This phase required solving 10+ sequential configuration issues:

Issue 1: Plugin Daemon Not Found

  • Error: "Failed to request plugin daemon"
  • Cause: Dify v1.13.0 requires new plugin architecture not in original docker-compose
  • Solution: Added 3 containers: plugin_daemon, sandbox, ssrf_proxy

Issue 2: Plugin Daemon Missing .env File

  • Error: failed to load .env file: open .env: no such file or directory
  • Solution: Added volume mount: ./.env:/app/.env:ro

Issue 3: Missing DifyInnerApiURL

  • Error: Key: 'Config.DifyInnerApiURL' Error:Field validation for 'DifyInnerApiURL' failed on the 'required' tag
  • Solution: Added environment variables: 
    DIFY_INNER_API_URL: http://dify-api:5001
DIFY_INNER_API_KEY: ${DIFY_SECRET_KEY}

Issue 4: Missing Remote Installing Host

  • Error: plugin remote installing host is empty
  • Solution: Added: 
    PLUGIN_REMOTE_INSTALLING_HOST: 0.0.0.0
PLUGIN_REMOTE_INSTALLING_PORT: 5003

Issue 5: Missing Storage Paths

  • Error: Plugin daemon started but installation failed silently
  • Solution: Added complete plugin storage configuration: 
    PLUGIN_WORKING_PATH: /app/storage/cwd
PLUGIN_STORAGE_TYPE: local
PLUGIN_STORAGE_LOCAL_ROOT: /app/storage
PLUGIN_INSTALLED_PATH: plugin
PLUGIN_PACKAGE_CACHE_PATH: plugin_packages
PLUGIN_MEDIA_CACHE_PATH: assets
  • Volume: ./volumes/plugin_daemon/storage:/app/storage

Issue 6: Sandbox Config Missing

  • Error: failed to init config: open conf/config.yaml: no such file or directory
  • Solution: Created ./volumes/sandbox/conf/config.yaml: 
    worker:
  timeout: 15
server:
  port: 8194
enable_network: true

Issue 7: Plugin Installation Timeout (Dify Issue #603)

  • Error: failed to install dependencies: failed to start command: context canceled
  • Cause: dify-api drops HTTP connection before plugin installation completes
  • This was the hardest bug - required consulting Gemini conversation history
  • Solutions:
    • Added PLUGIN_DAEMON_TIMEOUT: 600 to dify-api and dify-worker
    • Added PYTHON_ENV_INIT_TIMEOUT: 300 to plugin_daemon
    • Added PLUGIN_MAX_EXECUTION_TIMEOUT: 600 to plugin_daemon
    • Added UV_HTTP_TIMEOUT: 300 (integer, NOT "300s")

Issue 8: Wrong Plugin Daemon Image

  • Error: Continued timeout issues even with timeout fixes
  • Cause: Using unstable main-local-linux-amd64 image with known bugs
  • Solution: Switched to stable langgenius/dify-plugin-daemon:0.5.3-local

Issue 9: Sandbox Not Connected to API

  • Error: Code execution features wouldn't work
  • Solution: Added to dify-api environment: 
    CODE_EXECUTION_ENDPOINT: http://sandbox:8194
CODE_EXECUTION_API_KEY: dify-sandbox

Issue 10: Ollama DNS Resolution

  • Error: Plugin couldn't resolve host.docker.internal
  • Solution: Used direct IP address http://38.68.14.26:11434 instead

Final Result:

  • ✅ Ollama plugin installed successfully
  • ✅ 5 models configured: qwen2.5-coder:7b/32b, llama3.3:70b, llama3.2-vision:11b, nomic-embed-text
  • ✅ Google Gemini provider added
  • ✅ System defaults set (llama3.3:70b for reasoning, nomic-embed-text for embeddings)

🎯 WHAT'S WORKING NOW

Infrastructure (10 Containers)

All containers healthy and communicating:

  1. PostgreSQL 15 - Database (users, workspaces, settings)
  2. Redis 6 - Cache and sessions
  3. dify-api - Backend API (127.0.0.1:5001)
  4. dify-worker - Background task processor
  5. dify-web - Next.js frontend (127.0.0.1:3000)
  6. Qdrant - Vector database (127.0.0.1:6333)
  7. plugin_daemon - Plugin marketplace manager (v0.5.3-local)
  8. sandbox - Code execution environment
  9. ssrf_proxy - Security proxy
  10. n8n - Workflow automation (127.0.0.1:5678)

Access & Authentication

AI Models

  • ✅ Local Models (Ollama):
    • qwen2.5-coder:7b (fast coding - 4.7GB)
    • qwen2.5-coder:32b (advanced coding - 19GB)
    • llama3.3:70b (reasoning - system default - 42GB)
    • llama3.2-vision:11b (image analysis - 7.8GB)
    • nomic-embed-text (embeddings - system default - 274MB)
  • ✅ Cloud Models: Google Gemini (for heavy lifting)

System Configuration

  • ✅ Nginx: Reverse proxy with SSL, rate limiting, security headers
  • ✅ CORS: Properly configured for https://codex.firefrostgaming.com
  • ✅ CSRF: Headers preserved through proxy
  • ✅ Plugin System: Fully operational with timeouts configured
  • ✅ Storage: Permissions correct (UID 1001)

⏳ PHASES REMAINING

Phase 5: Configure Discord Integration ⏳ PENDING

Estimated Time: 1 hour
Dependencies: Phase 4 complete ✅
Tasks:

  • Create Discord webhooks (#codex-alerts, #system-critical)
  • Configure n8n webhook nodes for notifications
  • Test notification delivery
  • Set up error alert templates

Phase 6: Setup Git Integration ⏳ PENDING

Estimated Time: 2-3 hours
Dependencies: Phase 5 complete
Tasks:

  • Configure SSH keys for Gitea access
  • Create n8n Git Sync workflow (pull + filter + index)
  • Create n8n Git Write-Back workflow (validate + commit)
  • Test ai-proposals branch workflow
  • Implement Discord approval buttons

Phase 7: Configure Monitoring ⏳ PENDING

Estimated Time: 1 hour
Dependencies: Phase 6 complete
Tasks:

  • Set up Uptime Kuma monitors
  • Configure Docker restart triggers
  • Test self-healing workflows
  • Document failure modes

Phase 8: User Onboarding ⏳ PENDING

Estimated Time: 30 minutes
Dependencies: Phase 7 complete
Tasks:

  • Create Meg's admin account (gingerfury)
  • Create Holly's user account (Unicorn20089)
  • Configure workspace permissions
  • Test access control

Phase 9: Testing and Verification ⏳ PENDING

Estimated Time: 2 hours
Dependencies: Phase 8 complete
Tasks:

  • Upload operations manual documents
  • Test RAG queries
  • Test Git write-back
  • Test Discord notifications
  • Test tier-based access control

Phase 10: Backup Automation ⏳ PENDING

Estimated Time: 1 hour
Dependencies: Phase 9 complete
Tasks:

  • Create backup script
  • Set up cron job
  • Configure offsite rsync to Command Center
  • Test restore procedure

Phase 11: Final Cleanup ⏳ PENDING

Estimated Time: 30 minutes
Dependencies: Phase 10 complete
Tasks:

  • Remove AnythingLLM completely
  • Clean up unused Docker images
  • Document final configuration
  • Update operations manual

📝 CRITICAL CONFIGURATION REFERENCE

Environment Variables (.env)

CRITICAL - Must be blank for CSRF to work:

CONSOLE_API_URL=
APP_API_URL=

Public URLs:

CONSOLE_WEB_URL=https://codex.firefrostgaming.com
APP_WEB_URL=https://codex.firefrostgaming.com

CORS (must match domain exactly):

CONSOLE_CORS_ALLOW_ORIGINS=https://codex.firefrostgaming.com
WEB_API_CORS_ALLOW_ORIGINS=https://codex.firefrostgaming.com

Plugin System (critical timeouts for Issue #603):

PLUGIN_DAEMON_URL=http://plugin_daemon:5002
PLUGIN_DAEMON_KEY=${DIFY_SECRET_KEY}
PLUGIN_DAEMON_TIMEOUT=600  # dify-api must hold connection
PYTHON_ENV_INIT_TIMEOUT=300  # plugin_daemon
PLUGIN_MAX_EXECUTION_TIMEOUT=600  # plugin_daemon
UV_HTTP_TIMEOUT=300  # INTEGER not "300s"

Sandbox:

CODE_EXECUTION_ENDPOINT=http://sandbox:8194
CODE_EXECUTION_API_KEY=dify-sandbox

Ollama:

OLLAMA_API_BASE_URL=http://host.docker.internal:11434  # Not used in v1.13.0
# Actual connection: http://38.68.14.26:11434 (configured via plugin UI)

Nginx Critical Headers

For /console/api/* endpoints:

proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Forwarded-Port $server_port;

Rate Limiting:

limit_req_zone $binary_remote_addr zone=codex_limit:10m rate=100r/s;
limit_req zone=codex_limit burst=100 nodelay;

Docker Image Tags

MUST use stable tags, NOT bleeding-edge:

  • ✅ langgenius/dify-plugin-daemon:0.5.3-local
  • ❌ langgenius/dify-plugin-daemon:main-local-linux-amd64 (has Issue #603 bug)

Storage Permissions

Dify storage must be owned by UID 1001:

chown -R 1001:1001 ./volumes/dify/storage

Volume mount path must be:

- ./volumes/dify/storage:/app/api/storage  # NOT /app/storage

🔧 TROUBLESHOOTING QUICK REFERENCE

Issue: Blank Dashboard with 401 Errors

Cause: Absolute API URLs breaking CSRF
Solution: Set CONSOLE_API_URL= and APP_API_URL= to blank

Issue: Plugin Installation Fails with "context canceled"

Cause: Dify Issue #603 - HTTP timeout
Solutions:

  1. Use stable image 0.5.3-local not main
  2. Set PLUGIN_DAEMON_TIMEOUT: 600 in dify-api
  3. Set UV_HTTP_TIMEOUT: 300 (integer) in plugin_daemon

Issue: Config Changes Not Applied

Cause: Docker caches broken container state
Solution: Force recreation:

docker-compose stop <service>
docker-compose rm -f <service>
docker-compose up -d <service>

Issue: "Permission Denied" Writing to Storage

Cause: UID mismatch
Solution:

chown -R 1001:1001 ./volumes/dify/storage

Issue: Nginx Returns 502 Bad Gateway

Causes:

  1. Containers not running: docker-compose ps
  2. Wrong ports: Check 127.0.0.1:3000 and 127.0.0.1:5001
  3. CORS issues: Check browser console

📊 RESOURCE USAGE (TX1 Dallas)

Current Consumption

  • RAM (Idle): ~10GB (all services, no models loaded)
  • RAM (Active): ~92GB (with llama3.3:70b loaded)
    • Model: ~80GB
    • Services: ~12GB
  • Disk: ~85GB total
    • Docker images: ~8GB
    • Ollama models: 73.5GB
    • Dify volumes: ~3GB

Available Headroom

  • RAM: 251GB - 92GB = 159GB free
  • Disk: 1TB - 85GB = 915GB free
  • Plenty of room for game servers

🎓 LESSONS LEARNED

What Worked Well

  1. Incremental debugging - Solve one error at a time
  2. Gemini consultation - Provided critical Issue #603 diagnosis
  3. Complete container recreation - Use rm -f not just restart
  4. Reading error logs immediately - Caught issues fast

What Didn't Work

  1. Using bleeding-edge images - Stick to stable releases
  2. Assuming defaults - Plugin system needs 15+ env variables
  3. Simple restart after config changes - Must recreate containers

Critical Discoveries

  1. Blank API URLs are correct for reverse proxy setups
  2. CSRF requires specific nginx headers preserved
  3. Plugin system is brand new (v1.13.0) - poorly documented
  4. Timeout hierarchy matters - API, daemon, and UV all need configs
  5. UID mismatches common - Always check container user

👥 TEAM CREDITS

The Blueprint (Chronicler #21):

  • Designed complete architecture with Gemini
  • Created deployment plan
  • Identified Dify as superior choice over AnythingLLM/Open WebUI

The Diagnostician (Chronicler #23):

  • Executed Phases 0-4 deployment
  • Debugged 10+ sequential configuration issues
  • Solved Dify Issue #603 timeout bug
  • Documented all solutions

Google Gemini:

  • Provided architectural recommendations
  • Diagnosed root causes of complex errors
  • Suggested complete plugin daemon configuration
  • Identified Issue #603 in Dify GitHub

🚀 NEXT SESSION GOALS

Primary Objective: Complete Phase 5-6 (Discord + Git Integration)

Specific Tasks:

  1. Configure Discord webhooks
  2. Build n8n Git Sync workflow
  3. Build n8n Git Write-Back workflow
  4. Test ai-proposals branch workflow
  5. Upload first batch of operations manual documents
  6. Test RAG queries

Time Estimate: 3-4 hours

Prerequisites:

  • All of Phase 4 working ✅
  • SSH keys for Gitea access
  • Discord webhook URLs
  • Clear head (not 3 AM!)

📚 RELATED DOCUMENTATION

In This Directory:

  • README.md - Overall project description
  • DEPLOYMENT-PLAN-PART-1.md - Phases 0-3 (✅ COMPLETE)
  • DEPLOYMENT-PLAN-PART-2.md - Phases 4-11 (Phase 4 ✅, rest pending)
  • CONFIGURATION-FILES.md - All config file templates
  • TROUBLESHOOTING.md - Common issues and solutions
  • VERIFICATION.md - Testing procedures
  • RECOVERY.md - Backup and disaster recovery

External Documentation Created:

  • /home/claude/DIFY-ARCHITECTURE-COMPLETE.md - Complete technical overview
  • /home/claude/FIREFROST-CODEX-TROUBLESHOOTING-GUIDE.md - Comprehensive troubleshooting
  • /home/claude/FIREFROST-CODEX-DEPLOYMENT-GUIDE.md - Step-by-step deployment
  • /home/claude/PHASE-4-COMPLETION-SUMMARY.md - Tonight's session summary

✅ SUCCESS CRITERIA - PHASE 4

All Phase 4 success criteria met:

  • ✅ Dify accessible via https://codex.firefrostgaming.com
  • ✅ SSL certificate valid and working
  • ✅ Authentication working (cookie-based with CSRF)
  • ✅ Dashboard loading correctly
  • ✅ All 10 containers running and healthy
  • ✅ Plugin system operational
  • ✅ Ollama provider installed
  • ✅ 5 local models configured
  • ✅ Google Gemini provider added
  • ✅ System defaults set (llama3.3:70b, nomic-embed-text)
  • ✅ Admin account created and working
  • ✅ Zero additional monthly cost (self-hosted)
  • ✅ Response time under 15 seconds

PHASE 4 STATUS: COMPLETE ✅

Fire + Frost + Foundation + Codex = Where Love Builds Legacy 💙🔥❄️

Version: 1.0
Status: Phase 4 Complete, Ready for Phase 5
Last Updated: February 23, 2026 03:30 AM CST
Updated By: The Diagnostician (Chronicler #23)

Reference in New Issue View Git Blame Copy Permalink