- Add skip_llms_txt config option (default: False) - Validate value is boolean, warn and default to False if not - Support in both sync and async scraping modes - Add 17 tests for config, behavior, and edge cases