diff --git a/CLAUDE.md b/CLAUDE.md index a15cfcc..2177795 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -434,6 +434,7 @@ Config files (`configs/*.json`) define scraping behavior: - `categories`: Keyword mapping for categorization - `rate_limit`: Delay between requests (seconds) - `max_pages`: Maximum pages to scrape +- `skip_llms_txt`: Skip llms.txt detection, force HTML scraping (default: false) ## Key Features & Implementation