📝 docs: add skip_llms_txt to config parameters documentation

2025-11-20 14:00:55 -08:00
parent a0b1c2f42f
commit 91692db87c
1 changed files with 1 additions and 0 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -434,6 +434,7 @@ Config files (`configs/*.json`) define scraping behavior:
 - `categories`: Keyword mapping for categorization
 - `rate_limit`: Delay between requests (seconds)
 - `max_pages`: Maximum pages to scrape
+- `skip_llms_txt`: Skip llms.txt detection, force HTML scraping (default: false)

 ## Key Features & Implementation