From 4ef855cfa7da42b141c45efc7bf543be92f486d5 Mon Sep 17 00:00:00 2001 From: The Tenth Chronicler Date: Mon, 16 Feb 2026 02:29:00 +0000 Subject: [PATCH] Add troubleshooting guides from 2026-02-16 crash diagnosis session - Hytale troubleshooting guide covering world corruption, missing Assets.zip, and recovery procedures - NC1 cleanup guidelines documenting safe cleanup practices and lessons learned - Both guides created from real incident: ATM 10 entity crash + Hytale Assets.zip recovery - Includes command examples, file structure references, and emergency recovery steps --- .../hytale-troubleshooting-guide.md | 248 ++++++++++++++ .../troubleshooting/nc1-cleanup-guidelines.md | 318 ++++++++++++++++++ 2 files changed, 566 insertions(+) create mode 100644 docs/troubleshooting/hytale-troubleshooting-guide.md create mode 100644 docs/troubleshooting/nc1-cleanup-guidelines.md diff --git a/docs/troubleshooting/hytale-troubleshooting-guide.md b/docs/troubleshooting/hytale-troubleshooting-guide.md new file mode 100644 index 0000000..79fb97c --- /dev/null +++ b/docs/troubleshooting/hytale-troubleshooting-guide.md @@ -0,0 +1,248 @@ +# Hytale Server Troubleshooting Guide + +## Common Issues & Solutions + +### 1. "World default already exists on disk!" Error + +**Symptoms:** +``` +java.lang.IllegalArgumentException: World default already exists on disk! +at com.hypixel.hytale.server.core.universe.Universe.addWorld +``` + +**Cause:** Corrupted world state or plugin databases referencing old world data. + +**Solution:** +```bash +# SSH to server +ssh root@ +cd /var/lib/pterodactyl/volumes/ + +# Backup universe folder first +tar -czf universe-backup-$(date +%Y%m%d-%H%M%S).tar.gz universe/ + +# Remove corrupted state and plugin data +rm universe/memories.json +rm -rf universe/SimpleClaims/ +rm -rf universe/BarterShop/ +rm -rf universe/SimpleStorage/ +rm -rf universe/mods/ +rm -rf universe/players/ +rm -f universe/warps.json +rm -rf universe/worlds/default + +# Restart server - Hytale will regenerate everything +``` + +--- + +### 2. "Failed to load any asset packs" / Missing Assets.zip + +**Symptoms:** +``` +WARN [AssetModule|P] Failed to load manifest for pack at Assets.zip +INFO [HytaleServer] Shutting down... 'Failed to load any asset packs' +``` + +**Cause:** Missing or corrupted `Assets.zip` file (often deleted during cleanup). + +**Solution A - Re-download using hytale-downloader:** +```bash +cd /var/lib/pterodactyl/volumes/ + +# Download full server package +./hytale-downloader-linux-amd64 -download-path Server-Package.zip + +# Extract the nested Assets.zip +unzip -j Server-Package.zip Assets.zip + +# Verify file size (should be ~3.3GB) +ls -lh Assets.zip + +# Clean up +rm Server-Package.zip + +# Restart server +``` + +**Solution B - Upload from local files:** +- Upload your local Hytale `Assets.zip` (3.3GB) via SFTP/Pterodactyl file manager +- Place in server root directory + +--- + +### 3. Invalid/Corrupted Assets.zip + +**Symptoms:** +``` +WARN [AssetModule|P] Skipping pack at Assets.zip: missing or invalid manifest.json +``` + +**Cause:** The downloaded file is the server package (1.4GB), not the actual Assets.zip (3.3GB). + +**Solution:** +```bash +# The hytale-downloader gives you a package containing Assets.zip +# You need to extract the real Assets.zip from inside it + +# Rename the downloaded package +mv Assets.zip Server-Package.zip + +# Extract the actual Assets.zip +unzip -j Server-Package.zip Assets.zip + +# Verify size - should be ~3.3GB, not 1.4GB +ls -lh Assets.zip +``` + +--- + +## File Structure Reference + +### Critical Files (DO NOT DELETE) +``` +/var/lib/pterodactyl/volumes// +├── Assets.zip # 3.3GB - Core game assets (CRITICAL) +├── Server/ # Server binaries +├── hytale-downloader # Downloader tool +├── config.json # Server configuration +└── universe/ # World data + ├── memories.json # World state tracking + ├── worlds/ # World folders + │ └── default/ # Default world + ├── SimpleClaims/ # Plugin: Claims system + ├── BarterShop/ # Plugin: Economy + ├── SimpleStorage/ # Plugin: Storage + ├── mods/ # Plugin: Mod data + └── players/ # Player data +``` + +### Safe to Delete (for cleanup) +``` +/backups/ # Old world backups +*.log # Old log files +universe-backup-*.tar.gz # Manual backups you created +Server-Package.zip # After extracting Assets.zip +``` + +### NEVER Delete These +``` +Assets.zip # Core game assets +Server/ # Server binaries +hytale-downloader* # Needed for updates +config.json # Server settings +.hytale-downloader-credentials.json # Auth tokens +``` + +--- + +## NC1 Cleanup Best Practices + +### ❌ DANGEROUS Commands +```bash +# DO NOT USE THESE - They delete critical files: +rm -f /var/lib/pterodactyl/volumes//*.zip +rm -rf /var/lib/pterodactyl/volumes//backups +``` + +### ✅ SAFE Cleanup Commands +```bash +# Instead, be specific: +rm -f /var/lib/pterodactyl/volumes//backup*.zip +rm -f /var/lib/pterodactyl/volumes//Server-Package.zip +rm -rf /var/lib/pterodactyl/volumes//backups/ + +# Always verify before deleting: +ls -lah /var/lib/pterodactyl/volumes//*.zip +``` + +--- + +## Quick Reference: Hytale Downloader + +```bash +# Check available version +./hytale-downloader-linux-amd64 -print-version + +# Download latest release +./hytale-downloader-linux-amd64 -download-path Assets.zip + +# Download to specific path +./hytale-downloader-linux-amd64 -download-path /path/to/save.zip + +# Use different patchline (beta/experimental) +./hytale-downloader-linux-amd64 -patchline beta -download-path Assets.zip + +# Check for downloader updates +./hytale-downloader-linux-amd64 -check-update +``` + +--- + +## Recovery Checklist + +If Hytale won't start, follow this checklist: + +1. **Check Assets.zip exists and is 3.3GB:** + ```bash + ls -lh Assets.zip + ``` + +2. **Check for error in latest log:** + ```bash + tail -100 logs/latest.log + ``` + +3. **Verify config.json is valid:** + ```bash + cat config.json + ``` + +4. **Check universe/worlds/default exists:** + ```bash + ls -la universe/worlds/ + ``` + +5. **If world corruption suspected:** + - Back up universe folder + - Delete memories.json and plugin databases + - Delete universe/worlds/default + - Restart server + +6. **If Assets.zip missing:** + - Use hytale-downloader to re-download + - Extract nested Assets.zip from package + - Verify 3.3GB file size + +--- + +## When to Wipe vs. Repair + +### Repair (Keep World Data): +- World won't load: Delete memories.json +- Plugin errors: Delete plugin databases +- Player issues: Delete specific player file + +### Wipe (Fresh Start): +- Unrecoverable corruption +- Want clean vanilla server +- Testing new configurations + +```bash +# Full wipe (DESTRUCTIVE): +rm -rf universe/ +# Server will regenerate fresh world on next start +``` + +--- + +## Support & Resources + +- Hytale Server Documentation: Check official docs +- Firefrost Gaming Gitea: `git.firefrostgaming.com` +- This guide stored in: `docs/troubleshooting/hytale-troubleshooting-guide.md` + +--- + +**Last Updated:** 2026-02-16 +**Tested On:** Hytale Server Version 2026.02.06-aa1b071c2 diff --git a/docs/troubleshooting/nc1-cleanup-guidelines.md b/docs/troubleshooting/nc1-cleanup-guidelines.md new file mode 100644 index 0000000..66b6180 --- /dev/null +++ b/docs/troubleshooting/nc1-cleanup-guidelines.md @@ -0,0 +1,318 @@ +# NC1 Server Cleanup Guidelines + +## Overview +This document provides safe cleanup procedures for the NC1 Charlotte server to prevent accidental deletion of critical game files. + +--- + +## Critical Warning: Know What You're Deleting + +**THE GOLDEN RULE:** Always list files before deleting them with `ls -lah` to verify what will be removed. + +--- + +## Safe Cleanup Procedures + +### 1. Journal/Log Cleanup (SAFE) +```bash +# Clean systemd journals +journalctl --vacuum-time=7d +journalctl --vacuum-size=1G + +# Clean old logs +find /var/log -type f -name "*.log.*" -mtime +30 -delete +find /var/log -type f -name "*.gz" -mtime +30 -delete +``` + +### 2. Docker Cleanup (SAFE) +```bash +# Remove unused containers, networks, images +docker system prune -af --volumes + +# Remove old images only +docker image prune -a +``` + +### 3. APT Cache Cleanup (SAFE) +```bash +apt-get clean +apt-get autoclean +apt-get autoremove -y +``` + +--- + +## Game Server Specific Cleanup + +### ❌ DANGEROUS - DO NOT USE +```bash +# THESE WILL DELETE CRITICAL FILES: +rm -f /var/lib/pterodactyl/volumes//*.zip +rm -rf /var/lib/pterodactyl/volumes//backups +``` + +### ✅ SAFE - Server-Specific Cleanup + +#### Hytale Server +**UUID:** `13c80cb8-f6f8-4bfe-9cdb-823d7e951584` + +**Critical Files - NEVER DELETE:** +- `Assets.zip` (3.3GB) - Core game assets +- `Server/` directory - Server binaries +- `hytale-downloader*` - Update tool +- `config.json` - Server configuration +- `.hytale-downloader-credentials.json` - Auth tokens +- `universe/` - World data (unless intentionally wiping) + +**Safe to Delete:** +```bash +cd /var/lib/pterodactyl/volumes/13c80cb8-f6f8-4bfe-9cdb-823d7e951584 + +# Old manual backups (verify first!) +ls -lah universe-backup-*.tar.gz +rm -f universe-backup-*.tar.gz # Only after verifying dates + +# Server package after extraction +rm -f Server-Package.zip + +# Old backup directory (ONLY if you have external backups) +ls -lah backups/ # Verify first! +rm -rf backups/ # ONLY if confirmed backed up elsewhere + +# Old logs +rm -f logs/*.log.*.gz +``` + +#### Minecraft Servers (ATM 10, etc.) + +**Critical Files - NEVER DELETE:** +- `world/` - Main world +- `world_nether/` - Nether dimension +- `world_the_end/` - End dimension +- Server JAR files +- `server.properties` +- `ops.json`, `whitelist.json` + +**Safe to Delete:** +```bash +cd /var/lib/pterodactyl/volumes/ + +# Old crash reports +rm -rf crash-reports/*.txt + +# Old logs +rm -f logs/*.log.gz + +# Old backups (verify externally backed up first!) +ls -lah *backup*.tar.gz +rm -f *backup*.tar.gz # Only after external verification +``` + +--- + +## Pre-Cleanup Checklist + +Before ANY cleanup operation: + +1. **Stop affected servers** (if deleting from server directories) +2. **List files to be deleted:** + ```bash + ls -lah + ``` +3. **Verify file sizes and dates** - ensure you're not deleting recent/large critical files +4. **Have external backups** - if deleting backups, ensure they exist elsewhere +5. **Document what you're deleting** - take notes for recovery if needed + +--- + +## Emergency Recovery + +### If You Accidentally Deleted Critical Files + +#### Hytale Assets.zip Recovery: +```bash +cd /var/lib/pterodactyl/volumes/13c80cb8-f6f8-4bfe-9cdb-823d7e951584 + +# Re-download using hytale-downloader +./hytale-downloader-linux-amd64 -download-path Server-Package.zip + +# Extract nested Assets.zip +unzip -j Server-Package.zip Assets.zip + +# Verify size (should be 3.3GB) +ls -lh Assets.zip + +# Clean up package +rm Server-Package.zip +``` + +#### Hytale World Data Recovery: +If you deleted `universe/` or its contents: +```bash +# Check for backup tarballs +ls -lah universe-backup-*.tar.gz + +# Restore from most recent backup +tar -xzf universe-backup-YYYYMMDD-HHMMSS.tar.gz + +# If no backup exists, server will generate fresh world on restart +``` + +#### Minecraft World Recovery: +```bash +# Check for world backups +ls -lah world-backup-*.tar.gz + +# Restore +tar -xzf world-backup-YYYYMMDD-HHMMSS.tar.gz + +# If no backup, you may have lost the world permanently +``` + +--- + +## Disk Space Analysis Before Cleanup + +**Always analyze first to target the actual space hogs:** + +```bash +# Check overall disk usage +df -h + +# Find largest directories +du -h --max-depth=1 / | sort -hr | head -20 + +# Find largest files +find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -hr | head -20 + +# Check Pterodactyl volumes specifically +du -h --max-depth=2 /var/lib/pterodactyl/volumes/ | sort -hr | head -20 + +# Check Docker space usage +docker system df +``` + +--- + +## Recommended Cleanup Schedule + +### Weekly: +- Docker system prune +- Journal vacuum (keep 7 days) +- Old log rotation + +### Monthly: +- Review game server backups +- Clean old crash reports +- APT cache cleanup + +### Quarterly: +- Full disk analysis +- Archive old game server backups to external storage +- Review Pterodactyl volume usage + +--- + +## Space Reclamation Examples + +**From our 2026-02-16 cleanup session:** + +| Action | Space Reclaimed | +|--------|----------------| +| Delete Hytale backups | ~10GB | +| Journal vacuum (7d) | 3.7GB | +| Docker system prune | 3.5GB | +| **Total** | **17.42GB** | + +**Lessons learned:** +- ✅ Cleaned journals safely +- ✅ Removed Docker cruft +- ❌ Accidentally deleted `Assets.zip` with wildcard `*.zip` +- ❌ Accidentally wiped Hytale world (was in `/backups/`) + +--- + +## Command Safety Tips + +### Use Specific Patterns, Not Wildcards +```bash +# ❌ DANGEROUS - Deletes ALL zip files: +rm -f *.zip + +# ✅ SAFE - Only deletes backup zips: +rm -f backup-*.zip +rm -f *-backup.zip + +# ✅ SAFER - List first, then delete: +ls -lah *.zip +# Review output, then delete specific files +rm -f specific-file.zip +``` + +### Always Use `-i` for Interactive Deletion +```bash +# Prompts before each deletion: +rm -i *.zip + +# Prompts before recursive deletion: +rm -ri backups/ +``` + +### Use `find` with Confirmation +```bash +# Find and list candidates: +find /var/log -name "*.gz" -mtime +30 -ls + +# Review, then delete: +find /var/log -name "*.gz" -mtime +30 -delete +``` + +--- + +## Pterodactyl Panel Considerations + +**Prefer Panel-based operations when possible:** +- Backups: Use Pterodactyl's built-in backup system +- File deletion: Use file manager (provides trash/restore) +- Server reinstall: Use panel reinstall feature + +**Only use SSH/direct access when:** +- Bulk operations needed +- Panel is unavailable +- Advanced troubleshooting required + +--- + +## Quick Reference: What NOT to Delete + +### Hytale (`13c80cb8-f6f8-4bfe-9cdb-823d7e951584`) +``` +❌ Assets.zip +❌ Server/ +❌ hytale-downloader* +❌ config.json +❌ .hytale-downloader-credentials.json +❌ universe/ (unless wiping intentionally) +``` + +### Minecraft (All servers) +``` +❌ world*/ +❌ *.jar (server files) +❌ server.properties +❌ ops.json, whitelist.json +❌ Recent backups (check dates!) +``` + +### System-Wide +``` +❌ /var/lib/pterodactyl/volumes/ (entire directory) +❌ Critical Docker images (in use) +❌ Recent journals (< 7 days) +``` + +--- + +**Last Updated:** 2026-02-16 +**NC1 Server:** 216.239.104.130 (Charlotte, NC datacenter) +**Maintained by:** Firefrost Gaming