yusyus ae96526d4b feat(C2.7): Add standalone codebase-scraper CLI tool ...

- Created src/skill_seekers/cli/codebase_scraper.py (450 lines)
- Standalone tool for analyzing local codebases without GitHub API
- Full .gitignore support using pathspec library

Features:
- Directory tree walking with .gitignore respect
- Multi-language code analysis (Python, JavaScript, TypeScript, C++)
- Language filtering (--languages Python,JavaScript)
- File pattern matching (--file-patterns "*.py,src/**/*.js")
- API reference generation (--build-api-reference)
- Comment extraction (enabled by default)
- Configurable analysis depth (surface/deep/full)
- Smart directory exclusion (node_modules, venv, .git, etc.)

CLI Usage:
    skill-seekers-codebase --directory /path/to/repo --output output/codebase/
    skill-seekers-codebase --directory . --depth deep --build-api-reference
    skill-seekers-codebase --directory . --languages Python,JavaScript

Output:
- code_analysis.json - Complete analysis results
- api_reference/*.md - Generated API documentation (optional)

Tests:
- Created tests/test_codebase_scraper.py with 15 tests
- All tests passing ✅
- Test coverage: Language detection (5 tests), directory exclusion (4 tests),
  directory walking (4 tests), .gitignore loading (2 tests)

Dependencies Added:
- pathspec>=0.12.1 - For .gitignore parsing

Entry Point:
- Added skill-seekers-codebase to pyproject.toml

Related Issues:
- Closes #69 (C2.7 Create codebase_scraper.py CLI tool)
- Part of C2 Local Codebase Scraping roadmap (TIER 3)

Files Modified:
- src/skill_seekers/cli/codebase_scraper.py (CREATE - 450 lines)
- tests/test_codebase_scraper.py (CREATE - 160 lines)
- pyproject.toml (+2 lines - pathspec dependency + entry point)

2026-01-01 23:10:55 +03:00

..

fixtures

Add unified scraping tests and example conflict data

2025-10-29 23:19:32 +03:00

test_adaptors

test: Update version assertions to 2.5.0

2025-12-28 22:29:44 +03:00

__init__.py

Add comprehensive test system with 71 tests (100% pass rate)

2025-10-19 02:08:58 +03:00

conftest.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

mcp_integration_test.md

Add comprehensive MCP setup guide and integration test template

2025-10-19 17:01:37 +03:00

test_api_reference_builder.py

feat(C2.4): Add API reference generator from code signatures

2026-01-01 23:00:36 +03:00

test_async_scraping.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_cli_paths.py

test: Update version assertions to 2.5.1

2025-12-30 23:27:18 +03:00

test_code_analyzer.py

feat(C2.5): Add inline comment extraction for Python/JS/C++

2026-01-01 23:02:34 +03:00

test_codebase_scraper.py

feat(C2.7): Add standalone codebase-scraper CLI tool

2026-01-01 23:10:55 +03:00

test_config_validation.py

✨ feat: add skip_llms_txt config option to bypass llms.txt detection

2025-11-20 13:55:46 -08:00

test_constants.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

test_estimate_pages.py

test: Fix tests for modern Python packaging structure

2025-11-10 21:35:44 +03:00

test_excluded_dirs_config.py

test: Add comprehensive test coverage for exclude_dirs feature

2025-11-30 00:13:49 +03:00

test_git_repo.py

feat(A1.9): Add multi-source git repository support for config fetching

2025-12-21 19:28:22 +03:00

test_git_sources_e2e.py

test(A1.9): Add comprehensive E2E tests for git source features

2025-12-21 19:45:06 +03:00

test_github_scraper.py

fix: Complete fix for Issue #219 - All three problems resolved

2026-01-01 20:57:03 +03:00

test_install_agent.py

fix: Fix GitHub Actions CI failures (agent count + anthropic dependency)

2026-01-01 21:15:09 +03:00

test_install_multiplatform.py

feat: Complete multi-platform feature parity implementation

2025-12-28 21:35:21 +03:00

test_install_skill_e2e.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_install_skill.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_integration.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_issue_219_e2e.py

test: Increase timeout for E2E test to fix macOS CI flakiness

2026-01-01 21:18:24 +03:00

test_language_detector.py

feat: Add unified language detector for code analysis

2025-12-21 22:53:05 +03:00

test_llms_txt_detector.py

fix: Add package installation check and fix test imports (Task 2.1)

2025-11-29 22:13:13 +03:00

test_llms_txt_downloader.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_llms_txt_parser.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_mcp_fastmcp.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_mcp_git_sources.py

test(A1.9): Add comprehensive E2E tests for git source features

2025-12-21 19:45:06 +03:00

test_mcp_server.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_package_skill.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_package_structure.py

chore: Bump version to v2.5.2 - Package Configuration Improvement

2026-01-01 18:57:21 +03:00

test_parallel_scraping.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_pdf_advanced_features.py

Add PDF Advanced Features (v1.2.0)

2025-10-23 21:43:05 +03:00

test_pdf_extractor.py

feat: Add unified language detector for code analysis

2025-12-21 22:53:05 +03:00

test_pdf_scraper.py

fix: Update test file imports to use proper package paths

2025-11-29 21:55:46 +03:00

test_pr144_concerns.py

feat: Complete refactoring with async support, type safety, and package structure

2025-10-26 13:05:39 +03:00

test_quality_checker.py

feat(quality): add skill completeness checks (#207)

2026-01-01 19:54:48 +03:00

test_scraper_features.py

feat: Add modern Python packaging - Phase 1 (Foundation)

2025-11-07 01:14:24 +03:00

test_server_fastmcp_http.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_setup_scripts.py

feat: v2.4.0 - MCP 2025 upgrade with multi-agent support (#217)

2025-12-26 00:45:48 +03:00

test_skip_llms_txt.py

fix: Update test imports to use proper package names

2025-11-29 22:56:37 +03:00

test_smart_summarization.py

feat: Add smart summarization for large skills in local enhancement

2025-12-28 18:06:50 +03:00

test_source_manager.py

feat(A1.9): Add multi-source git repository support for config fetching

2025-12-21 19:28:22 +03:00

test_swift_detection.py

feat: Add comprehensive Swift language detection support (#223)

2026-01-01 19:25:53 +03:00

test_terminal_detection.py

fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)

2025-11-29 21:34:51 +03:00

test_unified_mcp_integration.py

fix: Update test imports and merge_sources for v2.0.0 release

2025-11-11 22:26:52 +03:00

test_unified.py

fix: Update test imports for new package structure

2025-11-07 01:21:29 +03:00

test_upload_skill.py

test: Fix tests for modern Python packaging structure

2025-11-10 21:35:44 +03:00

test_utilities.py

fix: Make retry timing test more robust for CI

2025-12-21 22:57:27 +03:00