feat(C2.7): Add standalone codebase-scraper CLI tool

- Created src/skill_seekers/cli/codebase_scraper.py (450 lines)
- Standalone tool for analyzing local codebases without GitHub API
- Full .gitignore support using pathspec library

Features:
- Directory tree walking with .gitignore respect
- Multi-language code analysis (Python, JavaScript, TypeScript, C++)
- Language filtering (--languages Python,JavaScript)
- File pattern matching (--file-patterns "*.py,src/**/*.js")
- API reference generation (--build-api-reference)
- Comment extraction (enabled by default)
- Configurable analysis depth (surface/deep/full)
- Smart directory exclusion (node_modules, venv, .git, etc.)

CLI Usage:
    skill-seekers-codebase --directory /path/to/repo --output output/codebase/
    skill-seekers-codebase --directory . --depth deep --build-api-reference
    skill-seekers-codebase --directory . --languages Python,JavaScript

Output:
- code_analysis.json - Complete analysis results
- api_reference/*.md - Generated API documentation (optional)

Tests:
- Created tests/test_codebase_scraper.py with 15 tests
- All tests passing 
- Test coverage: Language detection (5 tests), directory exclusion (4 tests),
  directory walking (4 tests), .gitignore loading (2 tests)

Dependencies Added:
- pathspec>=0.12.1 - For .gitignore parsing

Entry Point:
- Added skill-seekers-codebase to pyproject.toml

Related Issues:
- Closes #69 (C2.7 Create codebase_scraper.py CLI tool)
- Part of C2 Local Codebase Scraping roadmap (TIER 3)

Files Modified:
- src/skill_seekers/cli/codebase_scraper.py (CREATE - 450 lines)
- tests/test_codebase_scraper.py (CREATE - 160 lines)
- pyproject.toml (+2 lines - pathspec dependency + entry point)
This commit is contained in:
yusyus
2026-01-01 23:10:55 +03:00
parent 33d8500c44
commit ae96526d4b
3 changed files with 604 additions and 0 deletions

View File

@@ -55,6 +55,7 @@ dependencies = [
"jsonschema>=4.25.1",
"click>=8.3.0",
"Pygments>=2.19.2",
"pathspec>=0.12.1",
]
[project.optional-dependencies]
@@ -130,6 +131,7 @@ skill-seekers-upload = "skill_seekers.cli.upload_skill:main"
skill-seekers-estimate = "skill_seekers.cli.estimate_pages:main"
skill-seekers-install = "skill_seekers.cli.install_skill:main"
skill-seekers-install-agent = "skill_seekers.cli.install_agent:main"
skill-seekers-codebase = "skill_seekers.cli.codebase_scraper:main"
[tool.setuptools]
package-dir = {"" = "src"}