yusyus ea289cebe1 feat: Make EXCLUDED_DIRS configurable for local repository analysis ...

Closes #203

Adds configuration options to customize directory exclusions during local
repository analysis, while maintaining backward compatibility with smart
defaults.

**New Config Options:**

1. `exclude_dirs_additional` - Extend defaults (most common)
   - Adds custom directories to default exclusions
   - Example: ["proprietary", "legacy", "third_party"]
   - Total exclusions = defaults + additional

2. `exclude_dirs` - Replace defaults (advanced users)
   - Completely overrides default exclusions
   - Example: ["node_modules", ".git", "custom_vendor"]
   - Gives full control over exclusions

**Implementation:**

- Modified GitHubScraper.__init__() to parse exclude_dirs config
- Changed should_exclude_dir() to use instance variable instead of global
- Added logging for custom exclusions (INFO for extend, WARNING for replace)
- Maintains backward compatibility (no config = use defaults)

**Testing:**

- Added 12 comprehensive tests in test_excluded_dirs_config.py
  - 3 tests for defaults (backward compatibility)
  - 3 tests for extend mode
  - 3 tests for replace mode
  - 1 test for precedence
  - 2 tests for edge cases
- All 12 new tests passing ✅
- All 22 existing github_scraper tests passing ✅

**Documentation:**

- Updated CLAUDE.md config parameters section
- Added detailed "Configurable Directory Exclusions" feature section
- Included examples for both modes
- Listed common use cases (monorepos, enterprise, legacy codebases)

**Use Cases:**

- Monorepos with custom directory structures
- Enterprise projects with non-standard naming conventions
- Including unusual directories for analysis
- Minimal exclusions for small/simple projects

**Backward Compatibility:**

✅ Fully backward compatible - existing configs work unchanged
✅ Smart defaults maintained when no config provided
✅ All existing tests pass

Co-authored-by: jimmy058910 <jimmy058910@users.noreply.github.com>

2025-11-29 23:53:27 +03:00

..

fixtures

Add unified scraping tests and example conflict data

2025-10-29 23:19:32 +03:00

__init__.py

Add comprehensive test system with 71 tests (100% pass rate)

2025-10-19 02:08:58 +03:00

conftest.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

mcp_integration_test.md

Add comprehensive MCP setup guide and integration test template

2025-10-19 17:01:37 +03:00

test_async_scraping.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_cli_paths.py

test: Fix tests for modern Python packaging structure

2025-11-10 21:35:44 +03:00

test_config_validation.py

✨ feat: add skip_llms_txt config option to bypass llms.txt detection

2025-11-20 13:55:46 -08:00

test_constants.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

test_estimate_pages.py

test: Fix tests for modern Python packaging structure

2025-11-10 21:35:44 +03:00

test_excluded_dirs_config.py

feat: Make EXCLUDED_DIRS configurable for local repository analysis

2025-11-29 23:53:27 +03:00

test_github_scraper.py

fix: Update test file imports to use proper package paths

2025-11-29 21:55:46 +03:00

test_integration.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_llms_txt_detector.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

test_llms_txt_downloader.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_llms_txt_parser.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_mcp_server.py

fix: Update test imports and merge_sources for v2.0.0 release

2025-11-11 22:26:52 +03:00

test_package_skill.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_package_structure.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_parallel_scraping.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_pdf_advanced_features.py

Add PDF Advanced Features (v1.2.0)

2025-10-23 21:43:05 +03:00

test_pdf_extractor.py

Add PDF Advanced Features (v1.2.0)

2025-10-23 21:43:05 +03:00

test_pdf_scraper.py

fix: Update test file imports to use proper package paths

2025-11-29 21:55:46 +03:00

test_pr144_concerns.py

feat: Complete refactoring with async support, type safety, and package structure

2025-10-26 13:05:39 +03:00

test_quality_checker.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_scraper_features.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_setup_scripts.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_skip_llms_txt.py

fix: Update test imports to use proper package names

2025-11-29 22:56:37 +03:00

test_terminal_detection.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_unified_mcp_integration.py

fix: Update test imports and merge_sources for v2.0.0 release

2025-11-11 22:26:52 +03:00

test_unified.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_upload_skill.py

test: Fix tests for modern Python packaging structure

2025-11-10 21:35:44 +03:00

test_utilities.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00