Add troubleshooting guides from 2026-02-16 crash diagnosis session
- Hytale troubleshooting guide covering world corruption, missing Assets.zip, and recovery procedures - NC1 cleanup guidelines documenting safe cleanup practices and lessons learned - Both guides created from real incident: ATM 10 entity crash + Hytale Assets.zip recovery - Includes command examples, file structure references, and emergency recovery steps
This commit is contained in:
The Tenth Chronicler
248
docs/troubleshooting/hytale-troubleshooting-guide.md
Normal file
248
docs/troubleshooting/hytale-troubleshooting-guide.md
Normal file
@@ -0,0 +1,248 @@
|
||||
# Hytale Server Troubleshooting Guide
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
### 1. "World default already exists on disk!" Error
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
java.lang.IllegalArgumentException: World default already exists on disk!
|
||||
at com.hypixel.hytale.server.core.universe.Universe.addWorld
|
||||
```
|
||||
|
||||
**Cause:** Corrupted world state or plugin databases referencing old world data.
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# SSH to server
|
||||
ssh root@<server-ip>
|
||||
cd /var/lib/pterodactyl/volumes/<hytale-uuid>
|
||||
|
||||
# Backup universe folder first
|
||||
tar -czf universe-backup-$(date +%Y%m%d-%H%M%S).tar.gz universe/
|
||||
|
||||
# Remove corrupted state and plugin data
|
||||
rm universe/memories.json
|
||||
rm -rf universe/SimpleClaims/
|
||||
rm -rf universe/BarterShop/
|
||||
rm -rf universe/SimpleStorage/
|
||||
rm -rf universe/mods/
|
||||
rm -rf universe/players/
|
||||
rm -f universe/warps.json
|
||||
rm -rf universe/worlds/default
|
||||
|
||||
# Restart server - Hytale will regenerate everything
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. "Failed to load any asset packs" / Missing Assets.zip
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
WARN [AssetModule|P] Failed to load manifest for pack at Assets.zip
|
||||
INFO [HytaleServer] Shutting down... 'Failed to load any asset packs'
|
||||
```
|
||||
|
||||
**Cause:** Missing or corrupted `Assets.zip` file (often deleted during cleanup).
|
||||
|
||||
**Solution A - Re-download using hytale-downloader:**
|
||||
```bash
|
||||
cd /var/lib/pterodactyl/volumes/<hytale-uuid>
|
||||
|
||||
# Download full server package
|
||||
./hytale-downloader-linux-amd64 -download-path Server-Package.zip
|
||||
|
||||
# Extract the nested Assets.zip
|
||||
unzip -j Server-Package.zip Assets.zip
|
||||
|
||||
# Verify file size (should be ~3.3GB)
|
||||
ls -lh Assets.zip
|
||||
|
||||
# Clean up
|
||||
rm Server-Package.zip
|
||||
|
||||
# Restart server
|
||||
```
|
||||
|
||||
**Solution B - Upload from local files:**
|
||||
- Upload your local Hytale `Assets.zip` (3.3GB) via SFTP/Pterodactyl file manager
|
||||
- Place in server root directory
|
||||
|
||||
---
|
||||
|
||||
### 3. Invalid/Corrupted Assets.zip
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
WARN [AssetModule|P] Skipping pack at Assets.zip: missing or invalid manifest.json
|
||||
```
|
||||
|
||||
**Cause:** The downloaded file is the server package (1.4GB), not the actual Assets.zip (3.3GB).
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# The hytale-downloader gives you a package containing Assets.zip
|
||||
# You need to extract the real Assets.zip from inside it
|
||||
|
||||
# Rename the downloaded package
|
||||
mv Assets.zip Server-Package.zip
|
||||
|
||||
# Extract the actual Assets.zip
|
||||
unzip -j Server-Package.zip Assets.zip
|
||||
|
||||
# Verify size - should be ~3.3GB, not 1.4GB
|
||||
ls -lh Assets.zip
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Structure Reference
|
||||
|
||||
### Critical Files (DO NOT DELETE)
|
||||
```
|
||||
/var/lib/pterodactyl/volumes/<uuid>/
|
||||
├── Assets.zip # 3.3GB - Core game assets (CRITICAL)
|
||||
├── Server/ # Server binaries
|
||||
├── hytale-downloader # Downloader tool
|
||||
├── config.json # Server configuration
|
||||
└── universe/ # World data
|
||||
├── memories.json # World state tracking
|
||||
├── worlds/ # World folders
|
||||
│ └── default/ # Default world
|
||||
├── SimpleClaims/ # Plugin: Claims system
|
||||
├── BarterShop/ # Plugin: Economy
|
||||
├── SimpleStorage/ # Plugin: Storage
|
||||
├── mods/ # Plugin: Mod data
|
||||
└── players/ # Player data
|
||||
```
|
||||
|
||||
### Safe to Delete (for cleanup)
|
||||
```
|
||||
/backups/ # Old world backups
|
||||
*.log # Old log files
|
||||
universe-backup-*.tar.gz # Manual backups you created
|
||||
Server-Package.zip # After extracting Assets.zip
|
||||
```
|
||||
|
||||
### NEVER Delete These
|
||||
```
|
||||
Assets.zip # Core game assets
|
||||
Server/ # Server binaries
|
||||
hytale-downloader* # Needed for updates
|
||||
config.json # Server settings
|
||||
.hytale-downloader-credentials.json # Auth tokens
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## NC1 Cleanup Best Practices
|
||||
|
||||
### ❌ DANGEROUS Commands
|
||||
```bash
|
||||
# DO NOT USE THESE - They delete critical files:
|
||||
rm -f /var/lib/pterodactyl/volumes/<uuid>/*.zip
|
||||
rm -rf /var/lib/pterodactyl/volumes/<uuid>/backups
|
||||
```
|
||||
|
||||
### ✅ SAFE Cleanup Commands
|
||||
```bash
|
||||
# Instead, be specific:
|
||||
rm -f /var/lib/pterodactyl/volumes/<uuid>/backup*.zip
|
||||
rm -f /var/lib/pterodactyl/volumes/<uuid>/Server-Package.zip
|
||||
rm -rf /var/lib/pterodactyl/volumes/<uuid>/backups/
|
||||
|
||||
# Always verify before deleting:
|
||||
ls -lah /var/lib/pterodactyl/volumes/<uuid>/*.zip
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: Hytale Downloader
|
||||
|
||||
```bash
|
||||
# Check available version
|
||||
./hytale-downloader-linux-amd64 -print-version
|
||||
|
||||
# Download latest release
|
||||
./hytale-downloader-linux-amd64 -download-path Assets.zip
|
||||
|
||||
# Download to specific path
|
||||
./hytale-downloader-linux-amd64 -download-path /path/to/save.zip
|
||||
|
||||
# Use different patchline (beta/experimental)
|
||||
./hytale-downloader-linux-amd64 -patchline beta -download-path Assets.zip
|
||||
|
||||
# Check for downloader updates
|
||||
./hytale-downloader-linux-amd64 -check-update
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recovery Checklist
|
||||
|
||||
If Hytale won't start, follow this checklist:
|
||||
|
||||
1. **Check Assets.zip exists and is 3.3GB:**
|
||||
```bash
|
||||
ls -lh Assets.zip
|
||||
```
|
||||
|
||||
2. **Check for error in latest log:**
|
||||
```bash
|
||||
tail -100 logs/latest.log
|
||||
```
|
||||
|
||||
3. **Verify config.json is valid:**
|
||||
```bash
|
||||
cat config.json
|
||||
```
|
||||
|
||||
4. **Check universe/worlds/default exists:**
|
||||
```bash
|
||||
ls -la universe/worlds/
|
||||
```
|
||||
|
||||
5. **If world corruption suspected:**
|
||||
- Back up universe folder
|
||||
- Delete memories.json and plugin databases
|
||||
- Delete universe/worlds/default
|
||||
- Restart server
|
||||
|
||||
6. **If Assets.zip missing:**
|
||||
- Use hytale-downloader to re-download
|
||||
- Extract nested Assets.zip from package
|
||||
- Verify 3.3GB file size
|
||||
|
||||
---
|
||||
|
||||
## When to Wipe vs. Repair
|
||||
|
||||
### Repair (Keep World Data):
|
||||
- World won't load: Delete memories.json
|
||||
- Plugin errors: Delete plugin databases
|
||||
- Player issues: Delete specific player file
|
||||
|
||||
### Wipe (Fresh Start):
|
||||
- Unrecoverable corruption
|
||||
- Want clean vanilla server
|
||||
- Testing new configurations
|
||||
|
||||
```bash
|
||||
# Full wipe (DESTRUCTIVE):
|
||||
rm -rf universe/
|
||||
# Server will regenerate fresh world on next start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- Hytale Server Documentation: Check official docs
|
||||
- Firefrost Gaming Gitea: `git.firefrostgaming.com`
|
||||
- This guide stored in: `docs/troubleshooting/hytale-troubleshooting-guide.md`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-16
|
||||
**Tested On:** Hytale Server Version 2026.02.06-aa1b071c2
|
||||
318
docs/troubleshooting/nc1-cleanup-guidelines.md
Normal file
318
docs/troubleshooting/nc1-cleanup-guidelines.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# NC1 Server Cleanup Guidelines
|
||||
|
||||
## Overview
|
||||
This document provides safe cleanup procedures for the NC1 Charlotte server to prevent accidental deletion of critical game files.
|
||||
|
||||
---
|
||||
|
||||
## Critical Warning: Know What You're Deleting
|
||||
|
||||
**THE GOLDEN RULE:** Always list files before deleting them with `ls -lah` to verify what will be removed.
|
||||
|
||||
---
|
||||
|
||||
## Safe Cleanup Procedures
|
||||
|
||||
### 1. Journal/Log Cleanup (SAFE)
|
||||
```bash
|
||||
# Clean systemd journals
|
||||
journalctl --vacuum-time=7d
|
||||
journalctl --vacuum-size=1G
|
||||
|
||||
# Clean old logs
|
||||
find /var/log -type f -name "*.log.*" -mtime +30 -delete
|
||||
find /var/log -type f -name "*.gz" -mtime +30 -delete
|
||||
```
|
||||
|
||||
### 2. Docker Cleanup (SAFE)
|
||||
```bash
|
||||
# Remove unused containers, networks, images
|
||||
docker system prune -af --volumes
|
||||
|
||||
# Remove old images only
|
||||
docker image prune -a
|
||||
```
|
||||
|
||||
### 3. APT Cache Cleanup (SAFE)
|
||||
```bash
|
||||
apt-get clean
|
||||
apt-get autoclean
|
||||
apt-get autoremove -y
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Game Server Specific Cleanup
|
||||
|
||||
### ❌ DANGEROUS - DO NOT USE
|
||||
```bash
|
||||
# THESE WILL DELETE CRITICAL FILES:
|
||||
rm -f /var/lib/pterodactyl/volumes/<uuid>/*.zip
|
||||
rm -rf /var/lib/pterodactyl/volumes/<uuid>/backups
|
||||
```
|
||||
|
||||
### ✅ SAFE - Server-Specific Cleanup
|
||||
|
||||
#### Hytale Server
|
||||
**UUID:** `13c80cb8-f6f8-4bfe-9cdb-823d7e951584`
|
||||
|
||||
**Critical Files - NEVER DELETE:**
|
||||
- `Assets.zip` (3.3GB) - Core game assets
|
||||
- `Server/` directory - Server binaries
|
||||
- `hytale-downloader*` - Update tool
|
||||
- `config.json` - Server configuration
|
||||
- `.hytale-downloader-credentials.json` - Auth tokens
|
||||
- `universe/` - World data (unless intentionally wiping)
|
||||
|
||||
**Safe to Delete:**
|
||||
```bash
|
||||
cd /var/lib/pterodactyl/volumes/13c80cb8-f6f8-4bfe-9cdb-823d7e951584
|
||||
|
||||
# Old manual backups (verify first!)
|
||||
ls -lah universe-backup-*.tar.gz
|
||||
rm -f universe-backup-*.tar.gz # Only after verifying dates
|
||||
|
||||
# Server package after extraction
|
||||
rm -f Server-Package.zip
|
||||
|
||||
# Old backup directory (ONLY if you have external backups)
|
||||
ls -lah backups/ # Verify first!
|
||||
rm -rf backups/ # ONLY if confirmed backed up elsewhere
|
||||
|
||||
# Old logs
|
||||
rm -f logs/*.log.*.gz
|
||||
```
|
||||
|
||||
#### Minecraft Servers (ATM 10, etc.)
|
||||
|
||||
**Critical Files - NEVER DELETE:**
|
||||
- `world/` - Main world
|
||||
- `world_nether/` - Nether dimension
|
||||
- `world_the_end/` - End dimension
|
||||
- Server JAR files
|
||||
- `server.properties`
|
||||
- `ops.json`, `whitelist.json`
|
||||
|
||||
**Safe to Delete:**
|
||||
```bash
|
||||
cd /var/lib/pterodactyl/volumes/<minecraft-uuid>
|
||||
|
||||
# Old crash reports
|
||||
rm -rf crash-reports/*.txt
|
||||
|
||||
# Old logs
|
||||
rm -f logs/*.log.gz
|
||||
|
||||
# Old backups (verify externally backed up first!)
|
||||
ls -lah *backup*.tar.gz
|
||||
rm -f *backup*.tar.gz # Only after external verification
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pre-Cleanup Checklist
|
||||
|
||||
Before ANY cleanup operation:
|
||||
|
||||
1. **Stop affected servers** (if deleting from server directories)
|
||||
2. **List files to be deleted:**
|
||||
```bash
|
||||
ls -lah <target-directory>
|
||||
```
|
||||
3. **Verify file sizes and dates** - ensure you're not deleting recent/large critical files
|
||||
4. **Have external backups** - if deleting backups, ensure they exist elsewhere
|
||||
5. **Document what you're deleting** - take notes for recovery if needed
|
||||
|
||||
---
|
||||
|
||||
## Emergency Recovery
|
||||
|
||||
### If You Accidentally Deleted Critical Files
|
||||
|
||||
#### Hytale Assets.zip Recovery:
|
||||
```bash
|
||||
cd /var/lib/pterodactyl/volumes/13c80cb8-f6f8-4bfe-9cdb-823d7e951584
|
||||
|
||||
# Re-download using hytale-downloader
|
||||
./hytale-downloader-linux-amd64 -download-path Server-Package.zip
|
||||
|
||||
# Extract nested Assets.zip
|
||||
unzip -j Server-Package.zip Assets.zip
|
||||
|
||||
# Verify size (should be 3.3GB)
|
||||
ls -lh Assets.zip
|
||||
|
||||
# Clean up package
|
||||
rm Server-Package.zip
|
||||
```
|
||||
|
||||
#### Hytale World Data Recovery:
|
||||
If you deleted `universe/` or its contents:
|
||||
```bash
|
||||
# Check for backup tarballs
|
||||
ls -lah universe-backup-*.tar.gz
|
||||
|
||||
# Restore from most recent backup
|
||||
tar -xzf universe-backup-YYYYMMDD-HHMMSS.tar.gz
|
||||
|
||||
# If no backup exists, server will generate fresh world on restart
|
||||
```
|
||||
|
||||
#### Minecraft World Recovery:
|
||||
```bash
|
||||
# Check for world backups
|
||||
ls -lah world-backup-*.tar.gz
|
||||
|
||||
# Restore
|
||||
tar -xzf world-backup-YYYYMMDD-HHMMSS.tar.gz
|
||||
|
||||
# If no backup, you may have lost the world permanently
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Disk Space Analysis Before Cleanup
|
||||
|
||||
**Always analyze first to target the actual space hogs:**
|
||||
|
||||
```bash
|
||||
# Check overall disk usage
|
||||
df -h
|
||||
|
||||
# Find largest directories
|
||||
du -h --max-depth=1 / | sort -hr | head -20
|
||||
|
||||
# Find largest files
|
||||
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -hr | head -20
|
||||
|
||||
# Check Pterodactyl volumes specifically
|
||||
du -h --max-depth=2 /var/lib/pterodactyl/volumes/ | sort -hr | head -20
|
||||
|
||||
# Check Docker space usage
|
||||
docker system df
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Cleanup Schedule
|
||||
|
||||
### Weekly:
|
||||
- Docker system prune
|
||||
- Journal vacuum (keep 7 days)
|
||||
- Old log rotation
|
||||
|
||||
### Monthly:
|
||||
- Review game server backups
|
||||
- Clean old crash reports
|
||||
- APT cache cleanup
|
||||
|
||||
### Quarterly:
|
||||
- Full disk analysis
|
||||
- Archive old game server backups to external storage
|
||||
- Review Pterodactyl volume usage
|
||||
|
||||
---
|
||||
|
||||
## Space Reclamation Examples
|
||||
|
||||
**From our 2026-02-16 cleanup session:**
|
||||
|
||||
| Action | Space Reclaimed |
|
||||
|--------|----------------|
|
||||
| Delete Hytale backups | ~10GB |
|
||||
| Journal vacuum (7d) | 3.7GB |
|
||||
| Docker system prune | 3.5GB |
|
||||
| **Total** | **17.42GB** |
|
||||
|
||||
**Lessons learned:**
|
||||
- ✅ Cleaned journals safely
|
||||
- ✅ Removed Docker cruft
|
||||
- ❌ Accidentally deleted `Assets.zip` with wildcard `*.zip`
|
||||
- ❌ Accidentally wiped Hytale world (was in `/backups/`)
|
||||
|
||||
---
|
||||
|
||||
## Command Safety Tips
|
||||
|
||||
### Use Specific Patterns, Not Wildcards
|
||||
```bash
|
||||
# ❌ DANGEROUS - Deletes ALL zip files:
|
||||
rm -f *.zip
|
||||
|
||||
# ✅ SAFE - Only deletes backup zips:
|
||||
rm -f backup-*.zip
|
||||
rm -f *-backup.zip
|
||||
|
||||
# ✅ SAFER - List first, then delete:
|
||||
ls -lah *.zip
|
||||
# Review output, then delete specific files
|
||||
rm -f specific-file.zip
|
||||
```
|
||||
|
||||
### Always Use `-i` for Interactive Deletion
|
||||
```bash
|
||||
# Prompts before each deletion:
|
||||
rm -i *.zip
|
||||
|
||||
# Prompts before recursive deletion:
|
||||
rm -ri backups/
|
||||
```
|
||||
|
||||
### Use `find` with Confirmation
|
||||
```bash
|
||||
# Find and list candidates:
|
||||
find /var/log -name "*.gz" -mtime +30 -ls
|
||||
|
||||
# Review, then delete:
|
||||
find /var/log -name "*.gz" -mtime +30 -delete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pterodactyl Panel Considerations
|
||||
|
||||
**Prefer Panel-based operations when possible:**
|
||||
- Backups: Use Pterodactyl's built-in backup system
|
||||
- File deletion: Use file manager (provides trash/restore)
|
||||
- Server reinstall: Use panel reinstall feature
|
||||
|
||||
**Only use SSH/direct access when:**
|
||||
- Bulk operations needed
|
||||
- Panel is unavailable
|
||||
- Advanced troubleshooting required
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: What NOT to Delete
|
||||
|
||||
### Hytale (`13c80cb8-f6f8-4bfe-9cdb-823d7e951584`)
|
||||
```
|
||||
❌ Assets.zip
|
||||
❌ Server/
|
||||
❌ hytale-downloader*
|
||||
❌ config.json
|
||||
❌ .hytale-downloader-credentials.json
|
||||
❌ universe/ (unless wiping intentionally)
|
||||
```
|
||||
|
||||
### Minecraft (All servers)
|
||||
```
|
||||
❌ world*/
|
||||
❌ *.jar (server files)
|
||||
❌ server.properties
|
||||
❌ ops.json, whitelist.json
|
||||
❌ Recent backups (check dates!)
|
||||
```
|
||||
|
||||
### System-Wide
|
||||
```
|
||||
❌ /var/lib/pterodactyl/volumes/ (entire directory)
|
||||
❌ Critical Docker images (in use)
|
||||
❌ Recent journals (< 7 days)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-16
|
||||
**NC1 Server:** 216.239.104.130 (Charlotte, NC datacenter)
|
||||
**Maintained by:** Firefrost Gaming
|
||||
Reference in New Issue
Block a user