Update documentation for unified multi-source scraping (v2.0.0)
Major documentation update explaining the new unified scraping system that combines documentation + GitHub + PDF sources in a single skill with automatic conflict detection.
## Changes:
**README.md:**
- Update version badge to v2.0.0
- Add "Unified Multi-Source Scraping" to Key Features section
- Add comprehensive Option 5 section showing:
- Problem statement (documentation drift)
- Solution with code example
- Conflict detection types and severity levels
- Transparent reporting with side-by-side comparison
- List of advantages (identifies gaps, catches changes, single source of truth)
- Available unified configs
- Link to full guide (docs/UNIFIED_SCRAPING.md)
**CLAUDE.md:**
- Update Current Status to v2.0.0
- Add "Major Release: Unified Multi-Source Scraping" in Recent Updates
- Update configs count from 11/11 to 15/15 (added 4 unified configs)
- Add new "Unified Multi-Source Scraping" section under Core Commands
- Include command examples and feature highlights
- Explain what makes unified scraping special
**QUICKSTART.md:**
- Add Option D: Unified Multi-Source to Step 2
- Add unified configs to Available Presets section
- Show react_unified, django_unified, fastapi_unified, godot_unified examples
## Value:
This documentation update explains how unified scraping helps developers:
- Mix documentation + code in one skill
- Automatically detect conflicts (missing_in_docs, missing_in_code, signature_mismatch)
- Get transparent side-by-side comparisons with ⚠️ warnings
- Identify documentation gaps and outdated docs
- Create a single source of truth combining both sources
Related to: Phase 7-11 unified scraper implementation (commit 5d8c7e3)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
yusyus
43
CLAUDE.md
43
CLAUDE.md
@@ -2,13 +2,23 @@
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## 🎯 Current Status (October 21, 2025)
|
||||
## 🎯 Current Status (October 26, 2025)
|
||||
|
||||
**Version:** v1.0.0 (Production Ready)
|
||||
**Version:** v2.0.0 (Production Ready - Major Feature Release)
|
||||
**Active Development:** Flexible, incremental task-based approach
|
||||
|
||||
### Recent Updates (This Week):
|
||||
|
||||
**🚀 Major Release: Unified Multi-Source Scraping (v2.0.0)**
|
||||
- **NEW**: Combine documentation + GitHub + PDF in one skill
|
||||
- **NEW**: Automatic conflict detection between docs and code
|
||||
- **NEW**: Rule-based and AI-powered merging
|
||||
- **NEW**: Transparent conflict reporting with side-by-side comparison
|
||||
- **NEW**: 4 example unified configs (React, Django, FastAPI, Godot)
|
||||
- **NEW**: Complete documentation in docs/UNIFIED_SCRAPING.md
|
||||
- **NEW**: Integration tests (6/6 passing)
|
||||
- **Status**: ✅ Production ready and fully tested
|
||||
|
||||
**✅ Community Response (H1 Group):**
|
||||
- **Issue #8 Fixed** - Added BULLETPROOF_QUICKSTART.md and TROUBLESHOOTING.md for beginners
|
||||
- **Issue #7 Fixed** - Fixed all 11 configs (Django, Laravel, Astro, Tailwind) - 100% working
|
||||
@@ -17,8 +27,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
- **MCP Setup Fixed** - Path expansion bug resolved in setup_mcp.sh
|
||||
|
||||
**📦 Configs Status:**
|
||||
- ✅ **11/11 production configs verified working** (100% success rate)
|
||||
- ✅ New Laravel config added
|
||||
- ✅ **15/15 production configs verified working** (100% success rate)
|
||||
- ✅ 4 new unified configs added (React, Django, FastAPI, Godot)
|
||||
- ✅ All selectors tested and validated
|
||||
|
||||
**📋 Next Up:**
|
||||
@@ -95,7 +105,7 @@ export ANTHROPIC_API_KEY=sk-ant-...
|
||||
### Quick Start - Use a Preset
|
||||
|
||||
```bash
|
||||
# Scrape and build with a preset configuration
|
||||
# Single-source scraping (documentation only)
|
||||
python3 cli/doc_scraper.py --config configs/godot.json
|
||||
python3 cli/doc_scraper.py --config configs/react.json
|
||||
python3 cli/doc_scraper.py --config configs/vue.json
|
||||
@@ -104,6 +114,29 @@ python3 cli/doc_scraper.py --config configs/laravel.json
|
||||
python3 cli/doc_scraper.py --config configs/fastapi.json
|
||||
```
|
||||
|
||||
### Unified Multi-Source Scraping (**NEW - v2.0.0**)
|
||||
|
||||
```bash
|
||||
# Combine documentation + GitHub + PDF in one skill
|
||||
python3 cli/unified_scraper.py --config configs/react_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/django_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/fastapi_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/godot_unified.json
|
||||
|
||||
# Override merge mode
|
||||
python3 cli/unified_scraper.py --config configs/react_unified.json --merge-mode claude-enhanced
|
||||
|
||||
# Result: One comprehensive skill with conflict detection
|
||||
```
|
||||
|
||||
**What makes it special:**
|
||||
- ✅ Detects discrepancies between documentation and code
|
||||
- ✅ Shows both versions side-by-side with ⚠️ warnings
|
||||
- ✅ Identifies outdated docs and undocumented features
|
||||
- ✅ Single source of truth showing intent (docs) AND reality (code)
|
||||
|
||||
**See full guide:** [docs/UNIFIED_SCRAPING.md](docs/UNIFIED_SCRAPING.md)
|
||||
|
||||
### First-Time User Workflow (Recommended)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -27,6 +27,13 @@ python3 cli/doc_scraper.py --interactive
|
||||
python3 cli/doc_scraper.py --name react --url https://react.dev/
|
||||
```
|
||||
|
||||
**Option D: Unified Multi-Source (NEW - v2.0.0)**
|
||||
```bash
|
||||
# Combine documentation + GitHub code in one skill
|
||||
python3 cli/unified_scraper.py --config configs/react_unified.json
|
||||
```
|
||||
*Detects conflicts between docs and code automatically!*
|
||||
|
||||
### Step 3: Enhance SKILL.md (Recommended)
|
||||
|
||||
```bash
|
||||
@@ -63,6 +70,12 @@ python3 cli/doc_scraper.py --config configs/django.json
|
||||
|
||||
# FastAPI
|
||||
python3 cli/doc_scraper.py --config configs/fastapi.json
|
||||
|
||||
# Unified Multi-Source (NEW!)
|
||||
python3 cli/unified_scraper.py --config configs/react_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/django_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/fastapi_unified.json
|
||||
python3 cli/unified_scraper.py --config configs/godot_unified.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
90
README.md
90
README.md
@@ -2,7 +2,7 @@
|
||||
|
||||
# Skill Seeker
|
||||
|
||||
[](https://github.com/yusufkaraaslan/Skill_Seekers/releases/tag/v1.3.0)
|
||||
[](https://github.com/yusufkaraaslan/Skill_Seekers/releases/tag/v2.0.0)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://modelcontextprotocol.io)
|
||||
@@ -48,7 +48,7 @@ Skill Seeker is an automated tool that transforms any documentation website into
|
||||
- ✅ **Parallel Processing** - 3x faster for large PDFs
|
||||
- ✅ **Intelligent Caching** - 50% faster on re-runs
|
||||
|
||||
### 🐙 GitHub Repository Scraping (**NEW - v1.4.0**)
|
||||
### 🐙 GitHub Repository Scraping (**v1.4.0**)
|
||||
- ✅ **Repository Structure** - Extract README, file tree, and language breakdown
|
||||
- ✅ **GitHub Issues** - Fetch open/closed issues with labels and milestones
|
||||
- ✅ **CHANGELOG Extraction** - Automatically find and extract version history
|
||||
@@ -56,6 +56,15 @@ Skill Seeker is an automated tool that transforms any documentation website into
|
||||
- ✅ **Surface Layer Approach** - API signatures and docs (no implementation dumps)
|
||||
- ✅ **MCP Integration** - Natural language: "Scrape GitHub repo facebook/react"
|
||||
|
||||
### 🔄 Unified Multi-Source Scraping (**NEW - v2.0.0**)
|
||||
- ✅ **Combine Multiple Sources** - Mix documentation + GitHub + PDF in one skill
|
||||
- ✅ **Conflict Detection** - Automatically finds discrepancies between docs and code
|
||||
- ✅ **Intelligent Merging** - Rule-based or AI-powered conflict resolution
|
||||
- ✅ **Transparent Reporting** - Side-by-side comparison with ⚠️ warnings
|
||||
- ✅ **Documentation Gap Analysis** - Identifies outdated docs and undocumented features
|
||||
- ✅ **Single Source of Truth** - One skill showing both intent (docs) and reality (code)
|
||||
- ✅ **Backward Compatible** - Legacy single-source configs still work
|
||||
|
||||
### 🤖 AI & Enhancement
|
||||
- ✅ **AI-Powered Enhancement** - Transforms basic templates into comprehensive guides
|
||||
- ✅ **No API Costs** - FREE local enhancement using Claude Code Max
|
||||
@@ -173,6 +182,83 @@ python3 cli/github_scraper.py --repo django/django \
|
||||
- ✅ Repository metadata (stars, language, topics)
|
||||
- ✅ File structure and language breakdown
|
||||
|
||||
### Option 5: Unified Multi-Source Scraping (**NEW - v2.0.0**)
|
||||
|
||||
**The Problem:** Documentation and code often drift apart. Docs might be outdated, missing features that exist in code, or documenting features that were removed.
|
||||
|
||||
**The Solution:** Combine documentation + GitHub + PDF into one unified skill that shows BOTH what's documented AND what actually exists, with clear warnings about discrepancies.
|
||||
|
||||
```bash
|
||||
# Create unified config (mix documentation + GitHub)
|
||||
cat > configs/myframework_unified.json << 'EOF'
|
||||
{
|
||||
"name": "myframework",
|
||||
"description": "Complete framework knowledge from docs + code",
|
||||
"merge_mode": "rule-based",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.myframework.com/",
|
||||
"extract_api": true,
|
||||
"max_pages": 200
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "owner/myframework",
|
||||
"include_code": true,
|
||||
"code_analysis_depth": "surface"
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Run unified scraper
|
||||
python3 cli/unified_scraper.py --config configs/myframework_unified.json
|
||||
|
||||
# Upload output/myframework.zip to Claude - Done!
|
||||
```
|
||||
|
||||
**Time:** ~30-45 minutes | **Quality:** Production-ready with conflict detection | **Cost:** Free
|
||||
|
||||
**What Makes It Special:**
|
||||
|
||||
✅ **Conflict Detection** - Automatically finds 4 types of discrepancies:
|
||||
- 🔴 **Missing in code** (high): Documented but not implemented
|
||||
- 🟡 **Missing in docs** (medium): Implemented but not documented
|
||||
- ⚠️ **Signature mismatch**: Different parameters/types
|
||||
- ℹ️ **Description mismatch**: Different explanations
|
||||
|
||||
✅ **Transparent Reporting** - Shows both versions side-by-side:
|
||||
```markdown
|
||||
#### `move_local_x(delta: float)`
|
||||
|
||||
⚠️ **Conflict**: Documentation signature differs from implementation
|
||||
|
||||
**Documentation says:**
|
||||
```
|
||||
def move_local_x(delta: float)
|
||||
```
|
||||
|
||||
**Code implementation:**
|
||||
```python
|
||||
def move_local_x(delta: float, snap: bool = False) -> None
|
||||
```
|
||||
```
|
||||
|
||||
✅ **Advantages:**
|
||||
- **Identifies documentation gaps** - Find outdated or missing docs automatically
|
||||
- **Catches code changes** - Know when APIs change without docs being updated
|
||||
- **Single source of truth** - One skill showing intent (docs) AND reality (code)
|
||||
- **Actionable insights** - Get suggestions for fixing each conflict
|
||||
- **Development aid** - See what's actually in the codebase vs what's documented
|
||||
|
||||
**Example Unified Configs:**
|
||||
- `configs/react_unified.json` - React docs + GitHub repo
|
||||
- `configs/django_unified.json` - Django docs + GitHub repo
|
||||
- `configs/fastapi_unified.json` - FastAPI docs + GitHub repo
|
||||
|
||||
**Full Guide:** See [docs/UNIFIED_SCRAPING.md](docs/UNIFIED_SCRAPING.md) for complete documentation.
|
||||
|
||||
## How It Works
|
||||
|
||||
```mermaid
|
||||
|
||||
Reference in New Issue
Block a user