80382551b1
Tested and fixed all 11 production configs - now 100% working! Fixed Configs: 1. Django (configs/django.json) - ❌ Was using: div.document (selector doesn't exist) - ✅ Now using: article (1,688 chars of content) - Verified on: https://docs.djangoproject.com/en/stable/ 2. Astro (configs/astro.json) - ❌ Was using: homepage URL (no article element) - ✅ Now using: /en/getting-started/ with article selector - Added: start_urls, categories, improved URL patterns - Increased max_pages from 15 to 100 3. Tailwind (configs/tailwind.json) - ❌ Was using: article (selector doesn't exist) - ✅ Now using: div.prose (195 chars of content) - Verified on: https://tailwindcss.com/docs New Config: 4. Laravel (configs/laravel.json) - NEW! - Created complete Laravel 9.x config - Selector: #main-content (16,131 chars of content) - Base URL: https://laravel.com/docs/9.x/ - Includes: 8 start_urls covering installation, routing, controllers, views, Blade, Eloquent, migrations, auth - Categories: getting_started, routing, views, models, authentication, api - max_pages: 500 Test Results: ✅ 11/11 configs tested and verified (100%) ✅ All selectors extract content properly ✅ All base URLs accessible Working Configs: - ✅ astro.json - ✅ django.json - ✅ fastapi.json - ✅ godot.json - ✅ godot-large-example.json - ✅ kubernetes.json - ✅ laravel.json (NEW) - ✅ react.json - ✅ steam-economy-complete.json - ✅ tailwind.json - ✅ vue.json How I Tested: 1. Created test_selectors.py to find correct CSS selectors 2. Tested each config's base_url + selector combination 3. Verified content extraction (not just "found" but actual text) 4. Ensured meaningful content length (50+ chars minimum) Fixes Issue #7 - Laravel scraping not working Fixes #7
35 lines
1.3 KiB
JSON
35 lines
1.3 KiB
JSON
{
|
|
"name": "django",
|
|
"description": "Django web framework for Python. Use for Django models, views, templates, ORM, authentication, and web development.",
|
|
"base_url": "https://docs.djangoproject.com/en/stable/",
|
|
"start_urls": [
|
|
"https://docs.djangoproject.com/en/stable/intro/",
|
|
"https://docs.djangoproject.com/en/stable/topics/db/models/",
|
|
"https://docs.djangoproject.com/en/stable/topics/http/views/",
|
|
"https://docs.djangoproject.com/en/stable/topics/templates/",
|
|
"https://docs.djangoproject.com/en/stable/topics/forms/",
|
|
"https://docs.djangoproject.com/en/stable/topics/auth/",
|
|
"https://docs.djangoproject.com/en/stable/ref/models/"
|
|
],
|
|
"selectors": {
|
|
"main_content": "article",
|
|
"title": "h1",
|
|
"code_blocks": "pre"
|
|
},
|
|
"url_patterns": {
|
|
"include": ["/intro/", "/topics/", "/ref/", "/howto/"],
|
|
"exclude": ["/faq/", "/misc/", "/releases/"]
|
|
},
|
|
"categories": {
|
|
"getting_started": ["intro", "tutorial", "install"],
|
|
"models": ["models", "database", "orm", "queries"],
|
|
"views": ["views", "urlconf", "routing"],
|
|
"templates": ["templates", "template"],
|
|
"forms": ["forms", "form"],
|
|
"authentication": ["auth", "authentication", "user"],
|
|
"api": ["ref", "reference"]
|
|
},
|
|
"rate_limit": 0.3,
|
|
"max_pages": 500
|
|
}
|