firefrost-operations-manual/docs/emergency-protocols/YELLOW-ALERT-partial-degradation.md

⚠️ YELLOW ALERT - Partial Service Degradation Protocol

Status: Elevated Response Procedure
Alert Level: YELLOW ALERT
Priority: HIGH
Last Updated: 2026-02-17

---

⚠️ YELLOW ALERT DEFINITION

Partial service degradation or single critical system failure:

• One or more game servers down (but not all)
• Single management service unavailable
• Performance degradation (high latency, low TPS)
• Single node failure (TX1 or NC1 affected)
• Non-critical but user-impacting issues

This requires prompt attention but is not business-critical.

---

📊 YELLOW ALERT TRIGGERS

Automatic triggers:

• Any game server offline for >15 minutes
• TPS below 15 on any server for >30 minutes
• Panel/billing system inaccessible for >10 minutes
• More than 5 player complaints in 15 minutes
• Uptime Kuma shows red status for any service
• Memory usage >90% for >20 minutes

---

📞 RESPONSE PROCEDURE (15-30 minutes)

Step 1: ASSESS SITUATION (5 minutes)

Determine scope:

• Which services are affected?
• How many players impacted?
• Is degradation worsening?
• Any revenue impact?
• Can it wait or needs immediate action?

Quick checks:

# Check server status
ssh root@63.143.34.217 "systemctl status"

# Check game servers in Pterodactyl
curl https://panel.firefrostgaming.com/api/client

# Check resource usage
ssh root@38.68.14.26 "htop"

---

Step 2: COMMUNICATE (3 minutes)

If user-facing impact:

Discord #server-status:

⚠️ SERVICE NOTICE

We're experiencing issues with [specific service/server].

Affected: [Server name(s)]
Status: Investigating
ETA: [Estimate]

Players on unaffected servers: No action needed
Players on affected server: Please standby

Updates will be posted here.

If internal only:

• Post in #staff-lounge
• No public announcement needed

---

Step 3: DIAGNOSE & FIX (10-20 minutes)

See scenario-specific procedures below.

---

🔧 COMMON YELLOW ALERT SCENARIOS

Scenario 1: Single Game Server Down

Quick diagnostics:

# Via Pterodactyl panel
1. Check server status in panel
2. View console for errors
3. Check resource usage graphs

# Common causes:
- Out of memory (OOM)
- Crash from mod conflict
- World corruption
- Java process died

Resolution:

# Restart server via panel
1. Stop server
2. Wait 30 seconds
3. Start server
4. Monitor console for successful startup
5. Test player connection

If restart fails:

• Check logs for error messages
• Restore from backup if world corrupted
• Rollback recent mod changes
• Allocate more RAM if OOM

Recovery time: 5-15 minutes

---

Scenario 2: Low TPS / Server Lag

Diagnostics:

# In-game
/tps
/forge tps

# Via SSH
top -u minecraft
htop
iostat

Common causes:

• Chunk loading lag
• Redstone contraptions
• Mob farms
• Memory pressure
• Disk I/O bottleneck

Quick fixes:

# Clear entities
/kill @e[type=!player]

# Reduce view distance temporarily
# (via server.properties or Pterodactyl)

# Restart server during low-traffic time

Long-term solutions:

• Optimize JVM flags (see optimization guide)
• Add more RAM
• Limit chunk loading
• Remove lag-causing builds

Recovery time: 10-30 minutes

---

Scenario 3: Pterodactyl Panel Inaccessible

Quick checks:

# Panel server (45.94.168.138)
ssh root@45.94.168.138

# Check panel service
systemctl status pteroq
systemctl status wings

# Check Nginx
systemctl status nginx

# Check database
systemctl status mariadb

Common fixes:

# Restart panel services
systemctl restart pteroq wings nginx

# Check disk space (common cause)
df -h

# If database issue
systemctl restart mariadb

Recovery time: 5-10 minutes

---

Scenario 4: Billing/Whitelist Manager Down

Impact: Players cannot subscribe or whitelist

Diagnostics:

# Billing VPS (38.68.14.188)
ssh root@38.68.14.188

# Check services
systemctl status paymenter
systemctl status whitelist-manager
systemctl status nginx

Quick fix:

systemctl restart [affected-service]

Recovery time: 2-5 minutes

---

Scenario 5: Frostwall Tunnel Degraded

Symptoms:

• High latency on specific node
• Packet loss
• Intermittent disconnections

Diagnostics:

# On Command Center
ping 10.0.1.2  # TX1 tunnel
ping 10.0.2.2  # NC1 tunnel

# Check tunnel interface
ip link show gre-tx1
ip link show gre-nc1

# Check routing
ip route show

Quick fix:

# Restart specific tunnel
ip link set gre-tx1 down
ip link set gre-tx1 up

# Or restart all networking
systemctl restart networking

Recovery time: 5-10 minutes

---

Scenario 6: High Memory Usage (Pre-OOM)

Warning signs:

• Memory >90% on any server
• Swap usage increasing
• JVM GC warnings in logs

Immediate action:

# Identify memory hog
htop
ps aux --sort=-%mem | head

# If game server:
# Schedule restart during low-traffic

# If other service:
systemctl restart [service]

Prevention:

• Enable swap if not present
• Right-size RAM allocation
• Schedule regular restarts

Recovery time: 5-20 minutes

---

Scenario 7: Discord Bot Offline

Impact: Automated features unavailable

Quick fix:

# Restart bot container/service
docker restart [bot-name]
# or
systemctl restart [bot-service]

# Check bot token hasn't expired

Recovery time: 2-5 minutes

---

✅ RESOLUTION VERIFICATION

Before downgrading from Yellow Alert:

• Affected service operational
• Players can connect/use service
• No error messages in logs
• Performance metrics normal
• Root cause identified
• Temporary or permanent fix applied
• Monitoring in place for recurrence

---

📢 RESOLUTION COMMUNICATION

Public (if announced):

✅ RESOLVED

[Service/Server] is now operational.

Cause: [Brief explanation]
Duration: [X minutes]

Thank you for your patience!

Staff-only:

Yellow Alert cleared: [Service]
Cause: [Details]
Fix: [What was done]
Prevention: [Next steps]

---

📊 ESCALATION TO RED ALERT

Escalate if:

• Multiple services failing simultaneously
• Fix attempts unsuccessful after 30 minutes
• Issue worsening despite interventions
• Provider reports hardware failure
• Security breach suspected

When escalating:

• Follow RED ALERT protocol immediately
• Document what was tried
• Preserve logs/state for diagnosis

---

🔄 POST-INCIDENT TASKS

For significant Yellow Alerts:

1. Document incident (brief summary)
2. Update monitoring (prevent recurrence)
3. Review capacity (if resource-related)
4. Schedule preventive maintenance (if needed)

---

Fire + Frost + Foundation = Where Love Builds Legacy 💙🔥❄️

---

Protocol Status: ACTIVE
Version: 1.0