Remove unnecessary files:
- configs/.DS_Store (macOS system file, should not be tracked)
This ensures only relevant project files are version controlled
and improves repository hygiene.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Comprehensive documentation updates reflecting MCP integration:
README.md:
- Add MCP Integration and Tests Passing badges
- Enhance MCP section with "Tested and Working" status
- Add links to both setup and testing guides
docs/MCP_SETUP.md:
- Update status to reflect production testing
- Add integration testing verification notes
- Confirm all 6 tools working with natural language
CLAUDE.md:
- Add prominent MCP Integration section at top
- List all 6 available MCP tools with descriptions
- Add setup instructions and production status
docs/TEST_MCP_IN_CLAUDE_CODE.md (moved from root):
- Relocate testing guide to docs/ for better organization
- Provides step-by-step MCP integration testing workflow
- Documents complete test suite for all 6 tools
All documentation now accurately reflects the fully tested and
working MCP integration verified in production Claude Code environment.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add 4 test configuration files used for validating MCP functionality:
- astro.json: Astro framework documentation (15 pages, production test)
- python-tutorial-test.json: Python tutorial (minimal test case)
- tailwind.json: Tailwind CSS documentation (test case)
- test-manual.json: Manual testing configuration
These configs were used to verify:
- Config generation via generate_config tool
- Config validation via validate_config tool
- Page estimation via estimate_pages tool
- Full scraping workflow via scrape_docs tool
- Skill packaging via package_skill tool
All tests passed successfully in production Claude Code environment.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement complete Model Context Protocol server providing 6 tools for
documentation skill generation:
- list_configs: List all available preset configurations
- generate_config: Create new config files for any documentation site
- validate_config: Validate config file structure and parameters
- estimate_pages: Fast page count estimation before scraping
- scrape_docs: Full documentation scraping and skill building
- package_skill: Package skill directory into uploadable .zip
Features:
- Async/await architecture for efficient I/O operations
- Full MCP protocol compliance
- Comprehensive error handling and user-friendly messages
- Integration with existing CLI tools (doc_scraper.py, etc.)
- 25 unit tests with 100% pass rate
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- MCP_TEST_SCRIPT.md: Complete 10-test script with verification
- QUICK_MCP_TEST.md: Quick 6-test version for fast testing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
**TODO.md Updates:**
- Mark current 4 tasks as STARTED
- Add "In Progress" and "Completed Today" sections
- Document current branch: MCP_refactor
- Clear tracking of sprint progress
**GitHub Issues Created (templates):**
1. Fix 3 test failures (warnings vs errors)
2. Create MCP setup guide for Claude Code
3. Test MCP server with actual Claude Code
4. Update documentation for monorepo structure
**Issue Templates Include:**
- Detailed problem descriptions
- Step-by-step solutions
- Acceptance criteria
- Files to modify
- Test plans
**Next Steps:**
User can create issues via:
- GitHub web UI (copy from ISSUES_TO_CREATE.md)
- GitHub CLI (gh issue create)
- Or work directly from TODO.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major restructure to support both CLI usage and MCP integration:
**Repository Structure:**
- cli/ - All CLI tools (doc_scraper, estimate_pages, enhance_skill, etc.)
- mcp/ - New MCP server for Claude Code integration
- configs/ - Shared configuration files
- tests/ - Updated to import from cli/
- docs/ - Shared documentation
**MCP Server (NEW):**
- mcp/server.py - Full MCP server implementation
- 6 tools available:
* generate_config - Create config from URL
* estimate_pages - Fast page count estimation
* scrape_docs - Full documentation scraping
* package_skill - Package to .zip
* list_configs - Show available presets
* validate_config - Validate config files
- mcp/README.md - Complete MCP documentation
- mcp/requirements.txt - MCP dependencies
**CLI Tools (Moved to cli/):**
- All existing functionality preserved
- Same commands, same behavior
- Tests updated to import from cli.doc_scraper
**Tests:**
- 68/71 passing (95.8%)
- Updated imports from doc_scraper to cli.doc_scraper
- Fixed validate_config() tuple unpacking (errors, warnings)
- 3 minor test failures (checking warnings instead of errors)
**Benefits:**
- Use as CLI tool: python3 cli/doc_scraper.py
- Use via MCP: Integrated with Claude Code
- Shared code and configs
- Single source of truth
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove max_pages upper limit (was 10,000, now unlimited)
- Remove rate_limit upper limit (was 10s, now unlimited)
- Convert missing selector checks from errors to warnings
- Add warnings system (non-blocking) vs errors (blocking)
- Allow users to scrape large documentation sites (45k+ pages)
- Allow flexible rate limiting for different site requirements
All reasonable validations remain (required fields, valid URLs,
correct data types, no negative values).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add estimate_pages.py script (~270 lines)
- Fast estimation without downloading content (HEAD requests only)
- Shows estimated total pages and recommended max_pages
- Validates URL patterns work correctly
- Estimates scraping time based on rate_limit
- Update CLAUDE.md with estimator workflow and commands
- Update README.md features section with estimation benefits
- Usage: python3 estimate_pages.py configs/react.json
- Time: 1-2 minutes vs 20-40 minutes for full scrape
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Test Framework:
- Created tests/ directory structure
- Added __init__.py for test package
- Implemented 71 comprehensive tests across 3 test suites
Test Suites:
1. test_config_validation.py (25 tests)
- Valid/invalid config structure
- Required fields validation
- Name format validation
- URL format validation
- Selectors validation
- URL patterns validation
- Categories validation
- Rate limit validation (0-10 range)
- Max pages validation (1-10000 range)
- Start URLs validation
2. test_scraper_features.py (28 tests)
- URL validation (include/exclude patterns)
- Language detection (Python, JavaScript, GDScript, C++, etc.)
- Pattern extraction from documentation
- Smart categorization (by URL, title, content)
- Text cleaning utilities
3. test_integration.py (18 tests)
- Dry-run mode functionality
- Config loading and validation
- Real config files validation (godot, react, vue, django, fastapi, steam)
- URL processing and normalization
- Content extraction
Test Runner (run_tests.py):
- Custom colored test runner with ANSI colors
- Detailed test summary with breakdown by category
- Success rate calculation
- Command-line options:
--suite: Run specific test suite
--verbose: Show each test name
--quiet: Minimal output
--failfast: Stop on first failure
--list: List all available tests
- Execution time: ~1 second for full suite
Documentation:
- Added comprehensive TESTING.md guide
- Test writing templates
- Best practices
- Coverage information
- Troubleshooting guide
.gitignore:
- Added Python cache files
- Added output directory
- Added IDE and OS files
Test Results:
✅ 71/71 tests passing (100% pass rate)
✅ All existing configs validated
✅ Fast execution (<1 second)
✅ Ready for CI/CD integration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
High Priority:
- Fix hardcoded package_skill.py path (line 778)
Changed from: /mnt/skills/examples/skill-creator/scripts/package_skill.py
Changed to: package_skill.py (local repository path)
Medium Priority:
- Add comprehensive config validation
* Validates required fields (name, base_url)
* Validates name format (alphanumeric, hyphens, underscores)
* Validates base_url format (http/https)
* Validates selectors structure and recommends standard selectors
* Validates url_patterns (include/exclude lists)
* Validates categories structure
* Validates rate_limit range (0-10 seconds)
* Validates max_pages range (1-10000)
* Validates start_urls format if present
* Provides clear error messages for invalid configs
- Add --dry-run flag for preview mode
* Previews first 20 URLs without saving data
* Shows what would be scraped without creating files
* Discovers links to estimate total pages
* Displays configuration summary
* No directories created in dry-run mode
* Useful for testing configs before full scrape
All changes tested and working correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
