feat(C2.7): Add standalone codebase-scraper CLI tool
- Created src/skill_seekers/cli/codebase_scraper.py (450 lines)
- Standalone tool for analyzing local codebases without GitHub API
- Full .gitignore support using pathspec library
Features:
- Directory tree walking with .gitignore respect
- Multi-language code analysis (Python, JavaScript, TypeScript, C++)
- Language filtering (--languages Python,JavaScript)
- File pattern matching (--file-patterns "*.py,src/**/*.js")
- API reference generation (--build-api-reference)
- Comment extraction (enabled by default)
- Configurable analysis depth (surface/deep/full)
- Smart directory exclusion (node_modules, venv, .git, etc.)
CLI Usage:
skill-seekers-codebase --directory /path/to/repo --output output/codebase/
skill-seekers-codebase --directory . --depth deep --build-api-reference
skill-seekers-codebase --directory . --languages Python,JavaScript
Output:
- code_analysis.json - Complete analysis results
- api_reference/*.md - Generated API documentation (optional)
Tests:
- Created tests/test_codebase_scraper.py with 15 tests
- All tests passing ✅
- Test coverage: Language detection (5 tests), directory exclusion (4 tests),
directory walking (4 tests), .gitignore loading (2 tests)
Dependencies Added:
- pathspec>=0.12.1 - For .gitignore parsing
Entry Point:
- Added skill-seekers-codebase to pyproject.toml
Related Issues:
- Closes #69 (C2.7 Create codebase_scraper.py CLI tool)
- Part of C2 Local Codebase Scraping roadmap (TIER 3)
Files Modified:
- src/skill_seekers/cli/codebase_scraper.py (CREATE - 450 lines)
- tests/test_codebase_scraper.py (CREATE - 160 lines)
- pyproject.toml (+2 lines - pathspec dependency + entry point)
This commit is contained in:
@@ -55,6 +55,7 @@ dependencies = [
|
||||
"jsonschema>=4.25.1",
|
||||
"click>=8.3.0",
|
||||
"Pygments>=2.19.2",
|
||||
"pathspec>=0.12.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -130,6 +131,7 @@ skill-seekers-upload = "skill_seekers.cli.upload_skill:main"
|
||||
skill-seekers-estimate = "skill_seekers.cli.estimate_pages:main"
|
||||
skill-seekers-install = "skill_seekers.cli.install_skill:main"
|
||||
skill-seekers-install-agent = "skill_seekers.cli.install_agent:main"
|
||||
skill-seekers-codebase = "skill_seekers.cli.codebase_scraper:main"
|
||||
|
||||
[tool.setuptools]
|
||||
package-dir = {"" = "src"}
|
||||
|
||||
Reference in New Issue
Block a user