yusyus
80382551b1
Fix Issue #7 : Fix all broken configs and add Laravel support
...
Tested and fixed all 11 production configs - now 100% working!
Fixed Configs:
1. Django (configs/django.json)
- ❌ Was using: div.document (selector doesn't exist)
- ✅ Now using: article (1,688 chars of content)
- Verified on: https://docs.djangoproject.com/en/stable/
2. Astro (configs/astro.json)
- ❌ Was using: homepage URL (no article element)
- ✅ Now using: /en/getting-started/ with article selector
- Added: start_urls, categories, improved URL patterns
- Increased max_pages from 15 to 100
3. Tailwind (configs/tailwind.json)
- ❌ Was using: article (selector doesn't exist)
- ✅ Now using: div.prose (195 chars of content)
- Verified on: https://tailwindcss.com/docs
New Config:
4. Laravel (configs/laravel.json) - NEW!
- Created complete Laravel 9.x config
- Selector: #main-content (16,131 chars of content)
- Base URL: https://laravel.com/docs/9.x/
- Includes: 8 start_urls covering installation, routing,
controllers, views, Blade, Eloquent, migrations, auth
- Categories: getting_started, routing, views, models,
authentication, api
- max_pages: 500
Test Results:
✅ 11/11 configs tested and verified (100%)
✅ All selectors extract content properly
✅ All base URLs accessible
Working Configs:
- ✅ astro.json
- ✅ django.json
- ✅ fastapi.json
- ✅ godot.json
- ✅ godot-large-example.json
- ✅ kubernetes.json
- ✅ laravel.json (NEW)
- ✅ react.json
- ✅ steam-economy-complete.json
- ✅ tailwind.json
- ✅ vue.json
How I Tested:
1. Created test_selectors.py to find correct CSS selectors
2. Tested each config's base_url + selector combination
3. Verified content extraction (not just "found" but actual text)
4. Ensured meaningful content length (50+ chars minimum)
Fixes Issue #7 - Laravel scraping not working
Fixes #7
2025-10-21 00:16:39 +03:00
yusyus
bddb57f5ef
Add large documentation handling (40K+ pages support)
...
Implement comprehensive system for handling very large documentation sites
with intelligent splitting strategies and router/hub architecture.
**New CLI Tools:**
- cli/split_config.py: Split large configs into focused sub-skills
* Strategies: auto, category, router, size
* Configurable target pages per skill (default: 5000)
* Dry-run mode for preview
- cli/generate_router.py: Create intelligent router/hub skills
* Auto-generates routing logic based on keywords
* Creates SKILL.md with topic-to-skill mapping
* Infers router name from sub-skills
- cli/package_multi.py: Batch package multiple skills
* Package router + all sub-skills in one command
* Progress tracking for each skill
**MCP Integration:**
- Added split_config tool (8 total MCP tools now)
- Added generate_router tool
- Supports 40K+ page documentation via MCP
**Configuration:**
- New split_strategy parameter in configs
- split_config section for fine-tuned control
- checkpoint section for resume capability (ready for Phase 4)
- Example: configs/godot-large-example.json
**Documentation:**
- docs/LARGE_DOCUMENTATION.md (500+ lines)
* Complete guide for 10K+ page documentation
* All splitting strategies explained
* Detailed workflows with examples
* Best practices and troubleshooting
* Real-world examples (AWS, Microsoft, Godot)
**Features:**
✅ Handle 40K+ page documentation efficiently
✅ Parallel scraping support (5x-10x faster)
✅ Router + sub-skills architecture
✅ Intelligent keyword-based routing
✅ Multiple splitting strategies
✅ Full MCP integration
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-19 20:48:03 +03:00
yusyus
59c2f9126d
Optimize all framework configs with start_urls for better coverage
...
All configs now follow the steam-economy-complete.json pattern with:
- Multiple start_urls for comprehensive entry points
- Improved include patterns for better targeting
- Enhanced exclude patterns to skip irrelevant pages
Godot Config:
- Added 7 start_urls covering getting started, scripting, 2D, 3D, physics, animation, and classes
- Added include patterns: /getting_started/, /tutorials/, /classes/
- More focused scraping of core documentation
React Config:
- Added 6 start_urls covering learn, quick-start, reference, and hooks
- Existing patterns maintained (already well-optimized)
Vue Config:
- Added 6 start_urls covering introduction, essentials, components, composables, and API
- Fixed base_url from https://vuejs.org/guide/ to https://vuejs.org/
- Added /partners/ to exclude list
Django Config:
- Added 7 start_urls covering intro, models, views, templates, forms, auth, and reference
- Added /intro/ to include patterns
- Added /releases/ to exclude list (changelog not needed)
FastAPI Config:
- Added 7 start_urls covering tutorial, first-steps, path-params, body, dependencies, advanced, and reference
- Added /deployment/ to exclude list
Benefits:
- Better initial page discovery
- More comprehensive documentation coverage
- Faster scraping (direct entry to important sections)
- Reduced unnecessary page crawling
- Consistent pattern across all configs
All configs tested and validated:
✅ 71/71 tests passing
✅ All 6 configs validated successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-19 02:24:56 +03:00
