fix: Add UTF-8 encoding to all file operations for Windows compatibility

Fixes #209 - UnicodeDecodeError on Windows with non-ASCII characters

**Problem:**
Windows users with non-English locales (Chinese, Japanese, Korean, etc.)
experienced GBK/SHIFT-JIS codec errors when the system default encoding
is not UTF-8.

Error: 'gbk' codec can't decode byte 0xac in position 206: illegal
multibyte sequence

**Root Cause:**
File operations using open() without explicit encoding parameter use
the system default encoding, which on Windows Chinese edition is GBK.
JSON files contain UTF-8 encoded characters that fail to decode with GBK.

**Solution:**
Added encoding='utf-8' to ALL file operations across:
- doc_scraper.py (4 instances):
  * load_config() - line 1310
  * check_existing_data() - line 1416
  * save_checkpoint() - line 173
  * load_checkpoint() - line 186

- github_scraper.py (1 instance):
  * main() config loading - line 922

- unified_scraper.py (10 instances):
  * All JSON read/write operations - lines 134, 153, 205, 239, 275,
    278, 325, 328, 342, 364

**Test Results:**
-  All 612 tests passing (100% pass rate)
-  Backward compatible (UTF-8 is standard on Linux/macOS)
-  Fixes Windows locale issues

**Impact:**
-  Works on ALL Windows locales (Chinese, Japanese, Korean, etc.)
-  Maintains compatibility with Linux/macOS
-  Prevents future encoding issues

**Thanks to:** @my5icol for the detailed bug report and fix suggestion!
This commit is contained in:
yusyus
2025-12-28 18:27:50 +03:00
parent eb3b9d9175
commit c411eb24ec
3 changed files with 15 additions and 15 deletions

View File

@@ -919,7 +919,7 @@ Examples:
# Build config from args or file
if args.config:
with open(args.config, 'r') as f:
with open(args.config, 'r', encoding='utf-8') as f:
config = json.load(f)
elif args.repo:
config = {