# Staggered Server Restart System - Deployment Plan

**Status:** Planning Complete, Ready to Implement  
**Priority:** Tier 3 - Automation & Optimization  
**Time Estimate:** 2 hours implementation  
**Last Updated:** 2026-02-17

---

## Overview

Automated staggered restart system for all 11 Minecraft servers. Prevents simultaneous restarts, maintains continuous player access, and reduces server load by restarting servers one at a time with configurable spacing.

**The Problem:**
- Manual server restarts are time-consuming
- Simultaneous restarts = all servers down at once
- Memory leaks require regular restarts (especially ATM10)
- Players need warning before restarts

**The Solution:**
- Automated Python script using Pterodactyl API
- Staggered restarts (one server at a time)
- Configurable spacing between restarts (default: 5 minutes)
- Player warnings before restart
- Skip servers with active players (optional)
- Discord notifications
- Scheduled via cron

---

## Architecture

```
Command Center (Cron Job)
    ↓
Python Script
    ↓
Pterodactyl API
    ↓
Game Servers (restart one at a time)
    ↓
Discord Webhook (notifications)
```

**Flow:**
1. Cron triggers script at scheduled time (e.g., 4 AM daily)
2. Script fetches list of all 11 Minecraft servers from Pterodactyl
3. For each server:
   - Check if players online (optional skip logic)
   - Send in-game warning (5 min, 3 min, 1 min, 30 sec)
   - Execute restart via API
   - Wait for server to come back online
   - Wait configured delay (5 min) before next server
4. Post summary to Discord when complete

---

## Features

### Core Features

**✅ Staggered Restarts**
- One server at a time
- Configurable delay between restarts (default: 5 min)
- Prevents infrastructure overload
- Maintains player access to other servers

**✅ Player Warnings**
- 5 minute warning
- 3 minute warning
- 1 minute warning
- 30 second warning
- Messages sent via Pterodactyl console commands

**✅ Active Player Detection (Optional)**
- Check player count before restart
- Skip servers with active players
- Retry later or on next cycle
- Configurable threshold (skip if > X players)

**✅ Discord Notifications**
- Start notification (restart cycle beginning)
- Per-server notifications (Server X restarted)
- Completion notification (all servers done)
- Error notifications (if restart fails)

**✅ Error Handling**
- Retry failed restarts (3 attempts)
- Continue to next server if one fails
- Log all errors
- Discord alert on failures

---

## Server List

**11 Minecraft Servers (order matters for restart sequence):**

**TX1 Dallas (5 servers):**
1. Vanilla 1.21.11 - `3bed1bda-f648-4630-801a-fe9f2e3d3f27` (lowest priority - simple)
2. All The Mons - `668a5220-7e72-4379-9165-bdbb84bc9806`
3. Stoneblock 4 - `a0efbfe8-4b97-4a90-869d-ffe6d3072bd5`
4. Society: Sunlit Valley - `9310d0a6-62a6-4fe6-82c4-eb483dc68876`
5. Reclamation - `1eb33479-a6bc-4e8f-b64d-d1e4bfa0a8b4`

**NC1 Charlotte (6 servers):**
6. Homestead - `2f85d4ef-aa49-4dd6-b448-beb3fca1db12`
7. EMC Subterra Tech - `09a95f38-9f8c-404a-9557-3a7c44258223`
8. The Ember Project - `124f9060-58a7-457a-b2cf-b4024fce2951`
9. Minecolonies: Create and Conquer - `a14201d2-83b2-44e6-ae48-e6c4cbc56f24`
10. All The Mods 10 - `82e63949-8fbf-4a44-b32a-53324e8492bf` (highest priority - memory leak)

**Note:** FoundryVTT and Hytale excluded (not Minecraft)

**Restart Order Logic:**
- Start with simple/low-population servers
- End with complex/high-population servers
- ATM10 last (most likely to have players, most need of restart)

---

## Implementation

### Script Location

**File:** `/opt/automation/staggered-restart.py`  
**Config:** `/opt/automation/restart-config.json`  
**Logs:** `/var/log/staggered-restart.log`

### Configuration File (restart-config.json)

```json
{
  "pterodactyl": {
    "url": "https://panel.firefrostgaming.com",
    "api_key": "PTERODACTYL_API_KEY_HERE"
  },
  "discord": {
    "webhook_url": "DISCORD_WEBHOOK_URL_HERE",
    "notifications_enabled": true
  },
  "restart_settings": {
    "delay_between_restarts": 300,
    "warning_intervals": [300, 180, 60, 30],
    "skip_if_players": false,
    "player_threshold": 5,
    "max_retries": 3,
    "retry_delay": 60
  },
  "servers": [
    {
      "name": "Vanilla 1.21.11",
      "uuid": "3bed1bda-f648-4630-801a-fe9f2e3d3f27",
      "priority": 1
    },
    {
      "name": "All The Mons",
      "uuid": "668a5220-7e72-4379-9165-bdbb84bc9806",
      "priority": 2
    },
    {
      "name": "Stoneblock 4",
      "uuid": "a0efbfe8-4b97-4a90-869d-ffe6d3072bd5",
      "priority": 3
    },
    {
      "name": "Society: Sunlit Valley",
      "uuid": "9310d0a6-62a6-4fe6-82c4-eb483dc68876",
      "priority": 4
    },
    {
      "name": "Reclamation",
      "uuid": "1eb33479-a6bc-4e8f-b64d-d1e4bfa0a8b4",
      "priority": 5
    },
    {
      "name": "Homestead",
      "uuid": "2f85d4ef-aa49-4dd6-b448-beb3fca1db12",
      "priority": 6
    },
    {
      "name": "EMC Subterra Tech",
      "uuid": "09a95f38-9f8c-404a-9557-3a7c44258223",
      "priority": 7
    },
    {
      "name": "The Ember Project",
      "uuid": "124f9060-58a7-457a-b2cf-b4024fce2951",
      "priority": 8
    },
    {
      "name": "Minecolonies: Create and Conquer",
      "uuid": "a14201d2-83b2-44e6-ae48-e6c4cbc56f24",
      "priority": 9
    },
    {
      "name": "All The Mods 10",
      "uuid": "82e63949-8fbf-4a44-b32a-53324e8492bf",
      "priority": 10
    }
  ]
}
```

### Main Script (staggered-restart.py)

**See artifact for complete Python script** (will create after this overview)

**Key functions:**
- `load_config()` - Load JSON configuration
- `get_server_status(uuid)` - Check if server is online
- `get_player_count(uuid)` - Count online players
- `send_warning(uuid, message)` - Send in-game message
- `restart_server(uuid)` - Trigger restart via API
- `wait_for_online(uuid)` - Poll until server is back
- `discord_notify(message)` - Send to Discord
- `main()` - Orchestrate the restart cycle

---

## Deployment Steps

### Phase 1: Prerequisites (10 min)

- [ ] Pterodactyl API key with server control permissions
- [ ] Discord webhook URL for notifications channel
- [ ] Command Center SSH access
- [ ] Python 3.9+ installed on Command Center
- [ ] Required Python packages: `requests`, `json`, `time`, `logging`

### Phase 2: Install Script (15 min)

```bash
# On Command Center
mkdir -p /opt/automation
cd /opt/automation

# Create config file
nano restart-config.json
# Paste config, update API key and webhook URL

# Create Python script
nano staggered-restart.py
# Paste script contents

# Make executable
chmod +x staggered-restart.py

# Install dependencies
pip3 install requests --break-system-packages

# Create log directory
mkdir -p /var/log
touch /var/log/staggered-restart.log
chmod 644 /var/log/staggered-restart.log
```

### Phase 3: Test Run (30 min)

```bash
# Test with ONE server first (Vanilla - least impactful)
# Edit config to include only Vanilla server
python3 /opt/automation/staggered-restart.py

# Watch logs
tail -f /var/log/staggered-restart.log

# Verify:
# - Warning messages sent to server
# - Server restarted successfully
# - Discord notification received
# - Logs written properly
```

### Phase 4: Schedule with Cron (15 min)

```bash
# Edit crontab
crontab -e

# Add daily restart at 4 AM CST
0 4 * * * /usr/bin/python3 /opt/automation/staggered-restart.py >> /var/log/staggered-restart.log 2>&1

# Or weekly on Sundays at 4 AM
0 4 * * 0 /usr/bin/python3 /opt/automation/staggered-restart.py >> /var/log/staggered-restart.log 2>&1
```

**Restart Schedule Options:**
- **Daily:** `0 4 * * *` (4 AM every day)
- **Weekly:** `0 4 * * 0` (4 AM every Sunday)
- **Twice Weekly:** `0 4 * * 0,3` (4 AM Sunday and Wednesday)
- **Monthly:** `0 4 1 * *` (4 AM first day of month)

**Choose based on:**
- ATM10 memory leak severity (daily if bad)
- Player activity patterns
- Server performance needs

### Phase 5: Monitor & Adjust (Ongoing)

- [ ] Monitor first 3 restart cycles
- [ ] Adjust timing if players complain
- [ ] Fine-tune delay between restarts
- [ ] Adjust skip logic if needed
- [ ] Add/remove servers as needed

---

## Discord Integration

### Webhook Setup

1. In Discord, go to server settings
2. Integrations → Webhooks
3. Create new webhook for #server-status channel
4. Copy webhook URL
5. Add to restart-config.json

### Notification Examples

**Start:**
```
🔄 **Staggered Server Restart Started**
Time: 4:00 AM CST
Servers: 11 total
Estimated completion: ~60 minutes
```

**Per-Server:**
```
✅ **Reclamation** restarted successfully
Online: Yes | Players: 0 | Duration: 45 seconds
```

**Completion:**
```
✅ **All Servers Restarted**
Total: 11 servers
Successful: 11
Failed: 0
Duration: 57 minutes
```

**Error:**
```
❌ **Restart Failed: ATM10**
Reason: API timeout
Attempts: 3/3
Action: Manual restart required
```

---

## Advanced Features (Phase 2)

**Future enhancements:**

**Smart Scheduling:**
- Different schedules per server
- Priority-based restart order
- Peak hour avoidance

**Enhanced Player Detection:**
- Check for admin override (restart even with players)
- Gentle kick players with warning
- Transfer players to similar server before restart

**Performance Monitoring:**
- Track restart duration per server
- Memory usage before/after restart
- TPS monitoring integration

**Advanced Notifications:**
- Per-server Discord channels
- SMS/email alerts on failures
- Grafana dashboard integration

---

## Troubleshooting

### Script doesn't run via cron

**Check:**
```bash
# Verify cron is running
systemctl status cron

# Check cron logs
grep CRON /var/log/syslog

# Test script manually
/usr/bin/python3 /opt/automation/staggered-restart.py
```

### API key doesn't work

**Verify:**
```bash
# Test API key
curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://panel.firefrostgaming.com/api/client
```

### Server won't restart

**Check:**
- Server status in Pterodactyl panel
- API permissions (needs server control)
- Server resource limits
- Pterodactyl Wings status on node

### Warnings not sending

**Verify:**
- Server console access in Pterodactyl
- Console command format correct
- Server is actually online when warnings sent

---

## Maintenance

### Daily

- Check Discord for restart notifications
- Verify all servers came back online

### Weekly

- Review logs for errors
- Check restart duration trends
- Adjust schedule if needed

### Monthly

- Review player feedback
- Optimize restart order
- Update server list (add/remove servers)
- Test manual run to verify functionality

---

## Related Tasks

- **World Backup Automation** - Run backups before restarts
- **Netdata Deployment** - Monitor restart impact on resources
- **Discord Reorganization** - #server-status channel for notifications

---

**Fire + Frost + Foundation = Where Love Builds Legacy** 💙🔥❄️

---

**Document Status:** COMPLETE  
**Ready for Implementation:** When SSH access available (2 hours)  
**Dependencies:** Pterodactyl API key, Discord webhook, Command Center access  
**Owner:** The Wizard (Michael) + The Chronicler (automation)
