docs: complete documentation overhaul with v3.1.0 release notes and zh-CN translations
Documentation restructure: - New docs/getting-started/ guide (4 files: install, quick-start, first-skill, next-steps) - New docs/user-guide/ section (6 files: core concepts through troubleshooting) - New docs/reference/ section (CLI_REFERENCE, CONFIG_FORMAT, ENVIRONMENT_VARIABLES, MCP_REFERENCE) - New docs/advanced/ section (custom-workflows, mcp-server, multi-source) - New docs/ARCHITECTURE.md - system architecture overview - Archived legacy files (QUICKSTART.md, QUICK_REFERENCE.md, docs/guides/USAGE.md) to docs/archive/legacy/ Chinese (zh-CN) translations: - Full zh-CN mirror of all user-facing docs (getting-started, user-guide, reference, advanced) - GitHub Actions workflow for translation sync (.github/workflows/translate-docs.yml) - Translation sync checker script (scripts/check_translation_sync.sh) - Translation helper script (scripts/translate_doc.py) Content updates: - CHANGELOG.md: [Unreleased] → [3.1.0] - 2026-02-22 - README.md: updated with new doc structure links - AGENTS.md: updated agent documentation - docs/features/UNIFIED_SCRAPING.md: updated for unified scraper workflow JSON config Analysis/planning artifacts (kept for reference): - DOCUMENTATION_OVERHAUL_PLAN.md, DOCUMENTATION_OVERHAUL_SUMMARY.md - FEATURE_GAP_ANALYSIS.md, IMPLEMENTATION_GAPS_ANALYSIS.md, CREATE_COMMAND_COVERAGE_ANALYSIS.md - CHINESE_TRANSLATION_IMPLEMENTATION_SUMMARY.md, ISSUE_260_UPDATE.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
yusyus
207
docs/archive/legacy/QUICKSTART.md
Normal file
207
docs/archive/legacy/QUICKSTART.md
Normal file
@@ -0,0 +1,207 @@
|
||||
> ⚠️ **DEPRECATED**: This document is outdated and uses old CLI patterns.
|
||||
>
|
||||
> For up-to-date documentation, please see:
|
||||
> - [Quick Start Guide](docs/getting-started/02-quick-start.md) - 3 commands to first skill
|
||||
> - [Installation Guide](docs/getting-started/01-installation.md) - Complete installation
|
||||
> - [Documentation Hub](docs/README.md) - All documentation
|
||||
>
|
||||
> *This file is kept for historical reference only.*
|
||||
|
||||
---
|
||||
|
||||
# Quick Start Guide
|
||||
|
||||
## 🚀 3 Steps to Create a Skill
|
||||
|
||||
### Step 1: Install Dependencies
|
||||
|
||||
```bash
|
||||
pip3 install requests beautifulsoup4
|
||||
```
|
||||
|
||||
> **Note:** Skill_Seekers automatically checks for llms.txt files first, which is 10x faster when available.
|
||||
|
||||
### Step 2: Run the Tool
|
||||
|
||||
**Option A: Use a Preset (Easiest)**
|
||||
```bash
|
||||
skill-seekers scrape --config configs/godot.json
|
||||
```
|
||||
|
||||
**Option B: Interactive Mode**
|
||||
```bash
|
||||
skill-seekers scrape --interactive
|
||||
```
|
||||
|
||||
**Option C: Quick Command**
|
||||
```bash
|
||||
skill-seekers scrape --name react --url https://react.dev/
|
||||
```
|
||||
|
||||
**Option D: Unified Multi-Source (NEW - v2.0.0)**
|
||||
```bash
|
||||
# Combine documentation + GitHub code in one skill
|
||||
skill-seekers unified --config configs/react_unified.json
|
||||
```
|
||||
*Detects conflicts between docs and code automatically!*
|
||||
|
||||
### Step 3: Enhance SKILL.md (Recommended)
|
||||
|
||||
```bash
|
||||
# LOCAL enhancement (no API key, uses Claude Code Max)
|
||||
skill-seekers enhance output/godot/
|
||||
```
|
||||
|
||||
**This takes 60 seconds and dramatically improves the SKILL.md quality!**
|
||||
|
||||
### Step 4: Package the Skill
|
||||
|
||||
```bash
|
||||
skill-seekers package output/godot/
|
||||
```
|
||||
|
||||
**Done!** You now have `godot.zip` ready to use.
|
||||
|
||||
---
|
||||
|
||||
## 📋 Available Presets
|
||||
|
||||
```bash
|
||||
# Godot Engine
|
||||
skill-seekers scrape --config configs/godot.json
|
||||
|
||||
# React
|
||||
skill-seekers scrape --config configs/react.json
|
||||
|
||||
# Vue.js
|
||||
skill-seekers scrape --config configs/vue.json
|
||||
|
||||
# Django
|
||||
skill-seekers scrape --config configs/django.json
|
||||
|
||||
# FastAPI
|
||||
skill-seekers scrape --config configs/fastapi.json
|
||||
|
||||
# Unified Multi-Source (NEW!)
|
||||
skill-seekers unified --config configs/react_unified.json
|
||||
skill-seekers unified --config configs/django_unified.json
|
||||
skill-seekers unified --config configs/fastapi_unified.json
|
||||
skill-seekers unified --config configs/godot_unified.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Using Existing Data (Fast!)
|
||||
|
||||
If you already scraped once:
|
||||
|
||||
```bash
|
||||
skill-seekers scrape --config configs/godot.json
|
||||
|
||||
# When prompted:
|
||||
✓ Found existing data: 245 pages
|
||||
Use existing data? (y/n): y
|
||||
|
||||
# Builds in seconds!
|
||||
```
|
||||
|
||||
Or use `--skip-scrape`:
|
||||
```bash
|
||||
skill-seekers scrape --config configs/godot.json --skip-scrape
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Complete Example (Recommended Workflow)
|
||||
|
||||
```bash
|
||||
# 1. Install (once)
|
||||
pip3 install requests beautifulsoup4
|
||||
|
||||
# 2. Scrape React docs with LOCAL enhancement
|
||||
skill-seekers scrape --config configs/react.json --enhance-local
|
||||
# Wait 15-30 minutes (scraping) + 60 seconds (enhancement)
|
||||
|
||||
# 3. Package
|
||||
skill-seekers package output/react/
|
||||
|
||||
# 4. Use react.zip in Claude!
|
||||
```
|
||||
|
||||
**Alternative: Enhancement after scraping**
|
||||
```bash
|
||||
# 2a. Scrape only (no enhancement)
|
||||
skill-seekers scrape --config configs/react.json
|
||||
|
||||
# 2b. Enhance later
|
||||
skill-seekers enhance output/react/
|
||||
|
||||
# 3. Package
|
||||
skill-seekers package output/react/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
### Test with Small Pages First
|
||||
Edit config file:
|
||||
```json
|
||||
{
|
||||
"max_pages": 20 // Test with just 20 pages
|
||||
}
|
||||
```
|
||||
|
||||
### Rebuild Instantly
|
||||
```bash
|
||||
# After first scrape, you can rebuild instantly:
|
||||
skill-seekers scrape --config configs/react.json --skip-scrape
|
||||
```
|
||||
|
||||
### Create Custom Config
|
||||
```bash
|
||||
# Copy a preset
|
||||
cp configs/react.json configs/myframework.json
|
||||
|
||||
# Edit it
|
||||
nano configs/myframework.json
|
||||
|
||||
# Use it
|
||||
skill-seekers scrape --config configs/myframework.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 What You Get
|
||||
|
||||
```
|
||||
output/
|
||||
├── godot_data/ # Raw scraped data (reusable!)
|
||||
└── godot/ # The skill
|
||||
├── SKILL.md # With real code examples!
|
||||
└── references/ # Organized docs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ❓ Need Help?
|
||||
|
||||
See **README.md** for:
|
||||
- Complete documentation
|
||||
- Config file structure
|
||||
- Troubleshooting
|
||||
- Advanced usage
|
||||
|
||||
---
|
||||
|
||||
## 🎮 Let's Go!
|
||||
|
||||
```bash
|
||||
# Godot
|
||||
skill-seekers scrape --config configs/godot.json
|
||||
|
||||
# Or interactive
|
||||
skill-seekers scrape --interactive
|
||||
```
|
||||
|
||||
That's it! 🚀
|
||||
477
docs/archive/legacy/QUICK_REFERENCE.md
Normal file
477
docs/archive/legacy/QUICK_REFERENCE.md
Normal file
@@ -0,0 +1,477 @@
|
||||
> ⚠️ **DEPRECATED**: This document contains phantom commands and outdated patterns.
|
||||
>
|
||||
> For up-to-date documentation, please see:
|
||||
> - [Quick Start Guide](getting-started/02-quick-start.md) - 3 commands to first skill
|
||||
> - [CLI Reference](reference/CLI_REFERENCE.md) - Complete command reference
|
||||
> - [Documentation Hub](README.md) - All documentation
|
||||
>
|
||||
> *This file is kept for historical reference only.*
|
||||
|
||||
---
|
||||
|
||||
# Quick Reference - Skill Seekers Cheat Sheet
|
||||
|
||||
**Version:** 3.1.0-dev | **Quick Commands** | **One-Page Reference**
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Basic installation
|
||||
pip install skill-seekers
|
||||
|
||||
# With all platforms
|
||||
pip install skill-seekers[all-llms]
|
||||
|
||||
# Development mode
|
||||
pip install -e ".[all-llms,dev]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### Documentation Scraping
|
||||
|
||||
```bash
|
||||
# Scrape with preset config
|
||||
skill-seekers scrape --config react
|
||||
|
||||
# Scrape custom site
|
||||
skill-seekers scrape --base-url https://docs.example.com --name my-framework
|
||||
|
||||
# Rebuild without re-scraping
|
||||
skill-seekers scrape --config react --skip-scrape
|
||||
|
||||
# Async scraping (2-3x faster)
|
||||
skill-seekers scrape --config react --async
|
||||
```
|
||||
|
||||
### GitHub Repository Analysis
|
||||
|
||||
```bash
|
||||
# Basic analysis
|
||||
skill-seekers github https://github.com/facebook/react
|
||||
|
||||
# Deep C3.x analysis (patterns, tests, guides)
|
||||
skill-seekers github https://github.com/vercel/next.js --analysis-depth c3x
|
||||
|
||||
# With GitHub token (higher rate limits)
|
||||
GITHUB_TOKEN=ghp_... skill-seekers github https://github.com/org/repo
|
||||
```
|
||||
|
||||
### PDF Extraction
|
||||
|
||||
```bash
|
||||
# Extract from PDF
|
||||
skill-seekers pdf manual.pdf --name product-manual
|
||||
|
||||
# With OCR (scanned PDFs)
|
||||
skill-seekers pdf scanned.pdf --enable-ocr
|
||||
|
||||
# Large PDF (chunked processing)
|
||||
skill-seekers pdf large.pdf --chunk-size 50
|
||||
```
|
||||
|
||||
### Multi-Source Scraping
|
||||
|
||||
```bash
|
||||
# Unified scraping (docs + GitHub + PDF)
|
||||
skill-seekers unified --config configs/unified/react-unified.json
|
||||
|
||||
# Merge separate sources
|
||||
skill-seekers merge-sources \
|
||||
--docs output/react-docs \
|
||||
--github output/react-github \
|
||||
--output output/react-complete
|
||||
```
|
||||
|
||||
### AI Enhancement
|
||||
|
||||
```bash
|
||||
# API mode (fast, costs ~$0.15-0.30)
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skill-seekers enhance output/react/
|
||||
|
||||
# LOCAL mode (free, uses Claude Code Max)
|
||||
skill-seekers enhance output/react/ --mode LOCAL
|
||||
|
||||
# Background enhancement
|
||||
skill-seekers enhance output/react/ --background
|
||||
|
||||
# Monitor background enhancement
|
||||
skill-seekers enhance-status output/react/ --watch
|
||||
|
||||
# Apply a workflow preset during create
|
||||
skill-seekers create ./my-project --enhance-workflow security-focus
|
||||
|
||||
# Chain multiple workflow presets
|
||||
skill-seekers create ./my-project \
|
||||
--enhance-workflow security-focus \
|
||||
--enhance-workflow minimal
|
||||
```
|
||||
|
||||
### Enhancement Workflow Presets
|
||||
|
||||
```bash
|
||||
# List all available workflows (bundled + user)
|
||||
skill-seekers workflows list
|
||||
|
||||
# Show the YAML content of a workflow
|
||||
skill-seekers workflows show security-focus
|
||||
|
||||
# Copy a bundled workflow to user dir for editing
|
||||
skill-seekers workflows copy security-focus
|
||||
|
||||
# Copy multiple bundled workflows at once
|
||||
skill-seekers workflows copy security-focus minimal api-documentation
|
||||
|
||||
# Install a custom YAML file as a user workflow
|
||||
skill-seekers workflows add ./my-workflow.yaml
|
||||
|
||||
# Install multiple YAML files at once
|
||||
skill-seekers workflows add ./wf-a.yaml ./wf-b.yaml
|
||||
|
||||
# Install with a custom name (single file only)
|
||||
skill-seekers workflows add ./my-workflow.yaml --name my-custom-name
|
||||
|
||||
# Remove a user workflow (bundled presets cannot be removed)
|
||||
skill-seekers workflows remove my-workflow
|
||||
|
||||
# Remove multiple user workflows at once
|
||||
skill-seekers workflows remove wf-a wf-b
|
||||
|
||||
# Validate a workflow by name or file path
|
||||
skill-seekers workflows validate security-focus
|
||||
skill-seekers workflows validate ./my-workflow.yaml
|
||||
```
|
||||
|
||||
**Bundled presets:** `default`, `minimal`, `security-focus`, `architecture-comprehensive`, `api-documentation`
|
||||
**User presets dir:** `~/.config/skill-seekers/workflows/`
|
||||
|
||||
### Packaging & Upload
|
||||
|
||||
```bash
|
||||
# Package for Claude AI
|
||||
skill-seekers package output/react/ --target claude
|
||||
|
||||
# Package for all platforms
|
||||
for platform in claude gemini openai markdown; do
|
||||
skill-seekers package output/react/ --target $platform
|
||||
done
|
||||
|
||||
# Upload to Claude AI
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skill-seekers upload output/react-claude.zip --target claude
|
||||
|
||||
# Upload to Google Gemini
|
||||
export GOOGLE_API_KEY=AIza...
|
||||
skill-seekers upload output/react-gemini.tar.gz --target gemini
|
||||
```
|
||||
|
||||
### Complete Workflow
|
||||
|
||||
```bash
|
||||
# One command: fetch → scrape → enhance → package → upload
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skill-seekers install react --target claude --enhance --upload
|
||||
|
||||
# Multi-platform install
|
||||
skill-seekers install react --target claude,gemini,openai --enhance --upload
|
||||
|
||||
# Without enhancement or upload
|
||||
skill-seekers install vue --target markdown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Workflow 1: Quick Skill from Docs
|
||||
|
||||
```bash
|
||||
# 1. Scrape documentation
|
||||
skill-seekers scrape --config react
|
||||
|
||||
# 2. Package for Claude
|
||||
skill-seekers package output/react/ --target claude
|
||||
|
||||
# 3. Upload to Claude
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skill-seekers upload output/react-claude.zip --target claude
|
||||
```
|
||||
|
||||
### Workflow 2: GitHub Repo to Skill
|
||||
|
||||
```bash
|
||||
# 1. Analyze repository with C3.x features
|
||||
skill-seekers github https://github.com/facebook/react --analysis-depth c3x
|
||||
|
||||
# 2. Package for multiple platforms
|
||||
skill-seekers package output/react/ --target claude,gemini,openai
|
||||
```
|
||||
|
||||
### Workflow 3: Complete Multi-Source Skill
|
||||
|
||||
```bash
|
||||
# 1. Create unified config (configs/unified/my-framework.json)
|
||||
{
|
||||
"name": "my-framework",
|
||||
"sources": {
|
||||
"documentation": {"type": "docs", "base_url": "https://docs..."},
|
||||
"github": {"type": "github", "repo_url": "https://github..."},
|
||||
"pdf": {"type": "pdf", "pdf_path": "manual.pdf"}
|
||||
}
|
||||
}
|
||||
|
||||
# 2. Run unified scraping
|
||||
skill-seekers unified --config configs/unified/my-framework.json
|
||||
|
||||
# 3. Enhance with AI
|
||||
skill-seekers enhance output/my-framework/
|
||||
|
||||
# 4. Package and upload
|
||||
skill-seekers package output/my-framework/ --target claude
|
||||
skill-seekers upload output/my-framework-claude.zip --target claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server
|
||||
|
||||
### Starting MCP Server
|
||||
|
||||
```bash
|
||||
# stdio mode (Claude Code, VS Code + Cline)
|
||||
skill-seekers-mcp
|
||||
|
||||
# HTTP mode (Cursor, Windsurf, IntelliJ)
|
||||
skill-seekers-mcp --transport http --port 8765
|
||||
```
|
||||
|
||||
### MCP Tools (26 total)
|
||||
|
||||
**Core Tools:**
|
||||
1. `list_configs` - List preset configurations
|
||||
2. `generate_config` - Generate config from docs URL
|
||||
3. `validate_config` - Validate config structure
|
||||
4. `estimate_pages` - Estimate page count
|
||||
5. `scrape_docs` - Scrape documentation
|
||||
6. `package_skill` - Package to .zip
|
||||
7. `upload_skill` - Upload to platform
|
||||
8. `enhance_skill` - AI enhancement
|
||||
9. `install_skill` - Complete workflow
|
||||
|
||||
**Extended Tools:**
|
||||
10. `scrape_github` - GitHub analysis
|
||||
11. `scrape_pdf` - PDF extraction
|
||||
12. `unified_scrape` - Multi-source scraping
|
||||
13. `merge_sources` - Merge docs + code
|
||||
14. `detect_conflicts` - Find discrepancies
|
||||
15. `split_config` - Split large configs
|
||||
16. `generate_router` - Generate router skills
|
||||
17. `add_config_source` - Register git repos
|
||||
18. `fetch_config` - Fetch configs from git
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Claude AI (default platform)
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Google Gemini
|
||||
export GOOGLE_API_KEY=AIza...
|
||||
|
||||
# OpenAI ChatGPT
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
# GitHub (higher rate limits)
|
||||
export GITHUB_TOKEN=ghp_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests (1,880+)
|
||||
pytest tests/ -v
|
||||
|
||||
# Run with coverage
|
||||
pytest tests/ --cov=src/skill_seekers --cov-report=html
|
||||
|
||||
# Fast tests only (skip slow tests)
|
||||
pytest tests/ -m "not slow"
|
||||
|
||||
# Specific test category
|
||||
pytest tests/test_mcp*.py -v # MCP tests
|
||||
pytest tests/test_*_integration.py -v # Integration tests
|
||||
pytest tests/test_*_e2e.py -v # E2E tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Quality
|
||||
|
||||
```bash
|
||||
# Linting with Ruff
|
||||
ruff check . # Check for issues
|
||||
ruff check --fix . # Auto-fix issues
|
||||
ruff format . # Format code
|
||||
|
||||
# Run before commit
|
||||
ruff check . && ruff format --check . && pytest tests/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Preset Configurations (24)
|
||||
|
||||
**Web Frameworks:**
|
||||
- `react`, `vue`, `angular`, `svelte`, `nextjs`
|
||||
|
||||
**Python:**
|
||||
- `django`, `flask`, `fastapi`, `sqlalchemy`, `pytest`
|
||||
|
||||
**Game Development:**
|
||||
- `godot`, `pygame`, `unity`
|
||||
|
||||
**Tools & Libraries:**
|
||||
- `docker`, `kubernetes`, `terraform`, `ansible`
|
||||
|
||||
**Unified (Docs + GitHub):**
|
||||
- `react-unified`, `vue-unified`, `nextjs-unified`, etc.
|
||||
|
||||
**List all configs:**
|
||||
```bash
|
||||
skill-seekers list-configs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tips & Tricks
|
||||
|
||||
### Speed Up Scraping
|
||||
|
||||
```bash
|
||||
# Use async mode (2-3x faster)
|
||||
skill-seekers scrape --config react --async
|
||||
|
||||
# Rebuild without re-scraping
|
||||
skill-seekers scrape --config react --skip-scrape
|
||||
```
|
||||
|
||||
### Save API Costs
|
||||
|
||||
```bash
|
||||
# Use LOCAL mode for free AI enhancement
|
||||
skill-seekers enhance output/react/ --mode LOCAL
|
||||
|
||||
# Or skip enhancement entirely
|
||||
skill-seekers install react --target claude --no-enhance
|
||||
```
|
||||
|
||||
### Large Documentation
|
||||
|
||||
```bash
|
||||
# Generate router skill (>500 pages)
|
||||
skill-seekers generate-router output/large-docs/
|
||||
|
||||
# Split configuration
|
||||
skill-seekers split-config configs/large.json --output configs/split/
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
```bash
|
||||
# Verbose output
|
||||
skill-seekers scrape --config react --verbose
|
||||
|
||||
# Dry run (no actual scraping)
|
||||
skill-seekers scrape --config react --dry-run
|
||||
|
||||
# Show config without scraping
|
||||
skill-seekers validate-config configs/react.json
|
||||
```
|
||||
|
||||
### Batch Processing
|
||||
|
||||
```bash
|
||||
# Process multiple configs
|
||||
for config in react vue angular svelte; do
|
||||
skill-seekers install $config --target claude
|
||||
done
|
||||
|
||||
# Parallel processing
|
||||
skill-seekers install react --target claude &
|
||||
skill-seekers install vue --target claude &
|
||||
wait
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
**Configurations:**
|
||||
- Preset configs: `skill-seekers-configs/official/*.json`
|
||||
- Custom configs: `configs/*.json`
|
||||
|
||||
**Output:**
|
||||
- Scraped data: `output/{name}_data/`
|
||||
- Built skills: `output/{name}/`
|
||||
- Packages: `output/{name}-{platform}.{zip|tar.gz}`
|
||||
|
||||
**MCP:**
|
||||
- Server: `src/skill_seekers/mcp/server_fastmcp.py`
|
||||
- Tools: `src/skill_seekers/mcp/tools/*.py`
|
||||
|
||||
**Tests:**
|
||||
- All tests: `tests/test_*.py`
|
||||
- Fixtures: `tests/fixtures/`
|
||||
|
||||
---
|
||||
|
||||
## Error Messages
|
||||
|
||||
| Error | Meaning | Solution |
|
||||
|-------|---------|----------|
|
||||
| `NetworkError` | Connection failed | Check URL, internet connection |
|
||||
| `InvalidConfigError` | Bad config | Validate with `validate-config` |
|
||||
| `RateLimitError` | Too many requests | Increase `rate_limit` in config |
|
||||
| `ScrapingError` | Scraping failed | Check selectors, URL patterns |
|
||||
| `APIError` | Platform API failed | Check API key, quota |
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
```bash
|
||||
# Command help
|
||||
skill-seekers --help
|
||||
skill-seekers scrape --help
|
||||
skill-seekers install --help
|
||||
|
||||
# Version info
|
||||
skill-seekers --version
|
||||
|
||||
# Check configuration
|
||||
skill-seekers validate-config configs/my-config.json
|
||||
```
|
||||
|
||||
**Documentation:**
|
||||
- [Full README](../README.md)
|
||||
- [Usage Guide](guides/USAGE.md)
|
||||
- [API Reference](reference/API_REFERENCE.md)
|
||||
- [Troubleshooting](../TROUBLESHOOTING.md)
|
||||
|
||||
**Links:**
|
||||
- GitHub: https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
- PyPI: https://pypi.org/project/skill-seekers/
|
||||
- Issues: https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
---
|
||||
|
||||
**Version:** 3.1.0-dev | **Test Count:** 1,880+ | **MCP Tools:** 26 | **Platforms:** 16+ (Claude, Gemini, OpenAI, LangChain, LlamaIndex, ChromaDB, FAISS, Cursor, Windsurf, and more)
|
||||
66
docs/archive/legacy/README.md
Normal file
66
docs/archive/legacy/README.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Legacy Documentation Archive
|
||||
|
||||
> **Status:** Archived
|
||||
> **Reason:** Outdated patterns, phantom commands, or superseded by new docs
|
||||
|
||||
---
|
||||
|
||||
## Archived Files
|
||||
|
||||
| File | Reason | Replaced By |
|
||||
|------|--------|-------------|
|
||||
| `QUICKSTART.md` | Old CLI patterns | `docs/getting-started/02-quick-start.md` |
|
||||
| `USAGE.md` | `python3 cli/X.py` pattern | `docs/user-guide/` + `docs/reference/CLI_REFERENCE.md` |
|
||||
| `QUICK_REFERENCE.md` | Phantom commands | `docs/reference/CLI_REFERENCE.md` |
|
||||
|
||||
---
|
||||
|
||||
## Why These Were Archived
|
||||
|
||||
### QUICKSTART.md
|
||||
|
||||
**Issues:**
|
||||
- Referenced `pip3 install requests beautifulsoup4` instead of `pip install skill-seekers`
|
||||
- Missing modern commands like `create`
|
||||
|
||||
**Use Instead:** [docs/getting-started/02-quick-start.md](../../getting-started/02-quick-start.md)
|
||||
|
||||
---
|
||||
|
||||
### USAGE.md
|
||||
|
||||
**Issues:**
|
||||
- Used `python3 cli/doc_scraper.py` pattern (removed in v3.x)
|
||||
- Referenced `python3 cli/enhance_skill_local.py` (now `skill-seekers enhance`)
|
||||
- Referenced `python3 cli/estimate_pages.py` (now `skill-seekers estimate`)
|
||||
|
||||
**Use Instead:**
|
||||
- [docs/reference/CLI_REFERENCE.md](../../reference/CLI_REFERENCE.md) - Complete command reference
|
||||
- [docs/user-guide/](../../user-guide/) - Common tasks
|
||||
|
||||
---
|
||||
|
||||
### QUICK_REFERENCE.md
|
||||
|
||||
**Issues:**
|
||||
- Documented phantom commands like `skill-seekers merge-sources`
|
||||
- Documented phantom commands like `skill-seekers split-config`
|
||||
- Documented phantom commands like `skill-seekers generate-router`
|
||||
|
||||
**Use Instead:** [docs/reference/CLI_REFERENCE.md](../../reference/CLI_REFERENCE.md)
|
||||
|
||||
---
|
||||
|
||||
## Current Documentation
|
||||
|
||||
For up-to-date documentation, see:
|
||||
|
||||
- [docs/README.md](../../README.md) - Documentation hub
|
||||
- [docs/getting-started/](../../getting-started/) - New user guides
|
||||
- [docs/user-guide/](../../user-guide/) - Common tasks
|
||||
- [docs/reference/](../../reference/) - Technical reference
|
||||
- [docs/advanced/](../../advanced/) - Power user topics
|
||||
|
||||
---
|
||||
|
||||
*Last archived: 2026-02-16*
|
||||
822
docs/archive/legacy/USAGE.md
Normal file
822
docs/archive/legacy/USAGE.md
Normal file
@@ -0,0 +1,822 @@
|
||||
> ⚠️ **DEPRECATED**: This document uses outdated CLI patterns (`python3 cli/X.py`).
|
||||
>
|
||||
> For up-to-date documentation, please see:
|
||||
> - [CLI Reference](../reference/CLI_REFERENCE.md) - Complete command reference
|
||||
> - [User Guides](../user-guide/) - Common tasks and workflows
|
||||
> - [Documentation Hub](../README.md) - All documentation
|
||||
>
|
||||
> *This file is kept for historical reference only.*
|
||||
|
||||
---
|
||||
|
||||
# Complete Usage Guide for Skill Seeker
|
||||
|
||||
Comprehensive reference for all commands, options, and workflows.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Quick Reference](#quick-reference)
|
||||
- [Main Tool: doc_scraper.py](#main-tool-doc_scraperpy)
|
||||
- [Estimator: estimate_pages.py](#estimator-estimate_pagespy)
|
||||
- [Enhancement Tools](#enhancement-tools)
|
||||
- [Packaging Tool](#packaging-tool)
|
||||
- [Testing Tools](#testing-tools)
|
||||
- [Available Configs](#available-configs)
|
||||
- [Common Workflows](#common-workflows)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# 1. Estimate pages (fast, 1-2 min)
|
||||
python3 cli/estimate_pages.py configs/react.json
|
||||
|
||||
# 2. Scrape documentation (20-40 min)
|
||||
python3 cli/doc_scraper.py --config configs/react.json
|
||||
|
||||
# 3. Enhance with Claude Code (60 sec)
|
||||
python3 cli/enhance_skill_local.py output/react/
|
||||
|
||||
# 4. Package to .zip (instant)
|
||||
python3 cli/package_skill.py output/react/
|
||||
|
||||
# 5. Test everything (1 sec)
|
||||
python3 cli/run_tests.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Main Tool: doc_scraper.py
|
||||
|
||||
### Full Help
|
||||
|
||||
```
|
||||
usage: doc_scraper.py [-h] [--interactive] [--config CONFIG] [--name NAME]
|
||||
[--url URL] [--description DESCRIPTION] [--skip-scrape]
|
||||
[--dry-run] [--enhance] [--enhance-local]
|
||||
[--api-key API_KEY]
|
||||
|
||||
Convert documentation websites to Claude skills
|
||||
|
||||
options:
|
||||
-h, --help Show this help message and exit
|
||||
--interactive, -i Interactive configuration mode
|
||||
--config, -c CONFIG Load configuration from file (e.g., configs/godot.json)
|
||||
--name NAME Skill name
|
||||
--url URL Base documentation URL
|
||||
--description, -d DESCRIPTION
|
||||
Skill description
|
||||
--skip-scrape Skip scraping, use existing data
|
||||
--dry-run Preview what will be scraped without actually scraping
|
||||
--enhance Enhance SKILL.md using Claude API after building
|
||||
(requires API key)
|
||||
--enhance-local Enhance SKILL.md using Claude Code in new terminal
|
||||
(no API key needed)
|
||||
--api-key API_KEY Anthropic API key for --enhance (or set ANTHROPIC_API_KEY)
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
**1. Use Preset Config (Recommended)**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --config configs/godot.json
|
||||
python3 cli/doc_scraper.py --config configs/react.json
|
||||
python3 cli/doc_scraper.py --config configs/vue.json
|
||||
python3 cli/doc_scraper.py --config configs/django.json
|
||||
python3 cli/doc_scraper.py --config configs/fastapi.json
|
||||
```
|
||||
|
||||
**2. Interactive Mode**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --interactive
|
||||
# Wizard walks you through:
|
||||
# - Skill name
|
||||
# - Base URL
|
||||
# - Description
|
||||
# - Selectors (optional)
|
||||
# - URL patterns (optional)
|
||||
# - Rate limit
|
||||
# - Max pages
|
||||
```
|
||||
|
||||
**3. Quick Mode (Minimal)**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py \
|
||||
--name react \
|
||||
--url https://react.dev/ \
|
||||
--description "React framework for building UIs"
|
||||
```
|
||||
|
||||
**4. Dry-Run (Preview)**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --config configs/react.json --dry-run
|
||||
# Shows what will be scraped without downloading data
|
||||
# No directories created
|
||||
# Fast validation
|
||||
```
|
||||
|
||||
**5. Skip Scraping (Use Cached Data)**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
|
||||
# Uses existing output/godot_data/
|
||||
# Fast rebuild (1-3 minutes)
|
||||
# Useful for testing changes
|
||||
```
|
||||
|
||||
**6. With Local Enhancement**
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
|
||||
# Scrapes + enhances in one command
|
||||
# Opens new terminal for Claude Code
|
||||
# No API key needed
|
||||
```
|
||||
|
||||
**7. With API Enhancement**
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
python3 cli/doc_scraper.py --config configs/react.json --enhance
|
||||
|
||||
# Or with inline API key:
|
||||
python3 cli/doc_scraper.py --config configs/react.json --enhance --api-key sk-ant-...
|
||||
```
|
||||
|
||||
### Output Structure
|
||||
|
||||
```
|
||||
output/
|
||||
├── {name}_data/ # Scraped raw data (cached)
|
||||
│ ├── pages/
|
||||
│ │ ├── page_0.json
|
||||
│ │ ├── page_1.json
|
||||
│ │ └── ...
|
||||
│ └── summary.json # Scraping stats
|
||||
│
|
||||
└── {name}/ # Built skill directory
|
||||
├── SKILL.md # Main skill file
|
||||
├── SKILL.md.backup # Backup (if enhanced)
|
||||
├── references/ # Categorized docs
|
||||
│ ├── index.md
|
||||
│ ├── getting_started.md
|
||||
│ ├── api.md
|
||||
│ └── ...
|
||||
├── scripts/ # Empty (user scripts)
|
||||
└── assets/ # Empty (user assets)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Estimator: estimate_pages.py
|
||||
|
||||
### Full Help
|
||||
|
||||
```
|
||||
usage: estimate_pages.py [-h] [--max-discovery MAX_DISCOVERY]
|
||||
[--timeout TIMEOUT]
|
||||
config
|
||||
|
||||
Estimate page count for Skill Seeker configs
|
||||
|
||||
positional arguments:
|
||||
config Path to config JSON file
|
||||
|
||||
options:
|
||||
-h, --help Show this help message and exit
|
||||
--max-discovery, -m MAX_DISCOVERY
|
||||
Maximum pages to discover (default: 1000)
|
||||
--timeout, -t TIMEOUT
|
||||
HTTP request timeout in seconds (default: 30)
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
**1. Quick Estimate (100 pages)**
|
||||
```bash
|
||||
python3 cli/estimate_pages.py configs/react.json --max-discovery 100
|
||||
# Time: ~30-60 seconds
|
||||
# Good for: Quick validation
|
||||
```
|
||||
|
||||
**2. Standard Estimate (1000 pages - default)**
|
||||
```bash
|
||||
python3 cli/estimate_pages.py configs/godot.json
|
||||
# Time: ~1-2 minutes
|
||||
# Good for: Most use cases
|
||||
```
|
||||
|
||||
**3. Deep Estimate (2000 pages)**
|
||||
```bash
|
||||
python3 cli/estimate_pages.py configs/vue.json --max-discovery 2000
|
||||
# Time: ~3-5 minutes
|
||||
# Good for: Large documentation sites
|
||||
```
|
||||
|
||||
**4. Custom Timeout**
|
||||
```bash
|
||||
python3 cli/estimate_pages.py configs/django.json --timeout 60
|
||||
# Useful for slow servers
|
||||
```
|
||||
|
||||
### Output Example
|
||||
|
||||
```
|
||||
🔍 Estimating pages for: react
|
||||
📍 Base URL: https://react.dev/
|
||||
🎯 Start URLs: 6
|
||||
⏱️ Rate limit: 0.5s
|
||||
🔢 Max discovery: 1000
|
||||
|
||||
⏳ Discovered: 180 pages (1.3 pages/sec)
|
||||
|
||||
======================================================================
|
||||
📊 ESTIMATION RESULTS
|
||||
======================================================================
|
||||
|
||||
Config: react
|
||||
Base URL: https://react.dev/
|
||||
|
||||
✅ Pages Discovered: 180
|
||||
⏳ Pages Pending: 50
|
||||
📈 Estimated Total: 230
|
||||
|
||||
⏱️ Time Elapsed: 140.5s
|
||||
⚡ Discovery Rate: 1.28 pages/sec
|
||||
|
||||
======================================================================
|
||||
💡 RECOMMENDATIONS
|
||||
======================================================================
|
||||
|
||||
✅ Current max_pages (300) is sufficient
|
||||
|
||||
⏱️ Estimated full scrape time: 1.9 minutes
|
||||
(Based on rate_limit: 0.5s)
|
||||
```
|
||||
|
||||
**What It Shows:**
|
||||
- Estimated total pages to scrape
|
||||
- Whether current `max_pages` is sufficient
|
||||
- Recommended `max_pages` value
|
||||
- Estimated scraping time
|
||||
- Discovery rate (pages/sec)
|
||||
|
||||
---
|
||||
|
||||
## Enhancement Tools
|
||||
|
||||
### enhance_skill_local.py (Recommended)
|
||||
|
||||
**No API key needed - uses Claude Code Max plan**
|
||||
|
||||
```bash
|
||||
# Usage
|
||||
python3 cli/enhance_skill_local.py output/react/
|
||||
python3 cli/enhance_skill_local.py output/godot/
|
||||
|
||||
# What it does:
|
||||
# 1. Reads SKILL.md and references/
|
||||
# 2. Opens new terminal with Claude Code
|
||||
# 3. Claude enhances SKILL.md
|
||||
# 4. Backs up original to SKILL.md.backup
|
||||
# 5. Saves enhanced version
|
||||
|
||||
# Time: ~60 seconds
|
||||
# Cost: Free (uses your Claude Code Max plan)
|
||||
```
|
||||
|
||||
### enhance_skill.py (Alternative)
|
||||
|
||||
**Requires Anthropic API key**
|
||||
|
||||
```bash
|
||||
# Install dependency first
|
||||
pip3 install anthropic
|
||||
|
||||
# Usage with environment variable
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
python3 cli/enhance_skill.py output/react/
|
||||
|
||||
# Usage with inline API key
|
||||
python3 cli/enhance_skill.py output/godot/ --api-key sk-ant-...
|
||||
|
||||
# What it does:
|
||||
# 1. Reads SKILL.md and references/
|
||||
# 2. Calls Claude API (Sonnet 4)
|
||||
# 3. Enhances SKILL.md
|
||||
# 4. Backs up original to SKILL.md.backup
|
||||
# 5. Saves enhanced version
|
||||
|
||||
# Time: ~30-60 seconds
|
||||
# Cost: ~$0.01-0.10 per skill (depending on size)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Packaging Tool
|
||||
|
||||
### package_skill.py
|
||||
|
||||
```bash
|
||||
# Usage
|
||||
python3 cli/package_skill.py output/react/
|
||||
python3 cli/package_skill.py output/godot/
|
||||
|
||||
# What it does:
|
||||
# 1. Validates SKILL.md exists
|
||||
# 2. Creates .zip with all skill files
|
||||
# 3. Saves to output/{name}.zip
|
||||
|
||||
# Output:
|
||||
# output/react.zip
|
||||
# output/godot.zip
|
||||
|
||||
# Time: Instant
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Tools
|
||||
|
||||
### run_tests.py
|
||||
|
||||
```bash
|
||||
# Run all tests (default)
|
||||
python3 cli/run_tests.py
|
||||
# 71 tests, ~1 second
|
||||
|
||||
# Verbose output
|
||||
python3 cli/run_tests.py -v
|
||||
python3 cli/run_tests.py --verbose
|
||||
|
||||
# Quiet output
|
||||
python3 cli/run_tests.py -q
|
||||
python3 cli/run_tests.py --quiet
|
||||
|
||||
# Stop on first failure
|
||||
python3 cli/run_tests.py -f
|
||||
python3 cli/run_tests.py --failfast
|
||||
|
||||
# Run specific test suite
|
||||
python3 cli/run_tests.py --suite config
|
||||
python3 cli/run_tests.py --suite features
|
||||
python3 cli/run_tests.py --suite integration
|
||||
|
||||
# List all tests
|
||||
python3 cli/run_tests.py --list
|
||||
```
|
||||
|
||||
### Individual Tests
|
||||
|
||||
```bash
|
||||
# Run single test file
|
||||
python3 -m unittest tests.test_config_validation
|
||||
python3 -m unittest tests.test_scraper_features
|
||||
python3 -m unittest tests.test_integration
|
||||
|
||||
# Run single test class
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation
|
||||
|
||||
# Run single test method
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_valid_complete_config
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Available Configs
|
||||
|
||||
### Preset Configs (Ready to Use)
|
||||
|
||||
| Config | Framework | Pages | Description |
|
||||
|--------|-----------|-------|-------------|
|
||||
| `godot.json` | Godot Engine | ~500 | Game engine documentation |
|
||||
| `react.json` | React | ~300 | React framework docs |
|
||||
| `vue.json` | Vue.js | ~250 | Vue.js framework docs |
|
||||
| `django.json` | Django | ~400 | Django web framework |
|
||||
| `fastapi.json` | FastAPI | ~200 | FastAPI Python framework |
|
||||
| `steam-economy-complete.json` | Steam | ~100 | Steam Economy API docs |
|
||||
|
||||
### View Config Details
|
||||
|
||||
```bash
|
||||
# List all configs
|
||||
ls configs/
|
||||
|
||||
# View config content
|
||||
cat configs/react.json
|
||||
python3 -m json.tool configs/godot.json
|
||||
```
|
||||
|
||||
### Config Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "react",
|
||||
"base_url": "https://react.dev/",
|
||||
"description": "React - JavaScript library for building UIs",
|
||||
"start_urls": [
|
||||
"https://react.dev/learn",
|
||||
"https://react.dev/reference/react",
|
||||
"https://react.dev/reference/react-dom"
|
||||
],
|
||||
"selectors": {
|
||||
"main_content": "article",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": ["/learn/", "/reference/"],
|
||||
"exclude": ["/blog/", "/community/"]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": ["learn", "tutorial", "intro"],
|
||||
"api": ["reference", "api", "hooks"],
|
||||
"guides": ["guide"]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 300
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Workflow 1: Use Preset (Fastest)
|
||||
|
||||
```bash
|
||||
# 1. Estimate (optional, 1-2 min)
|
||||
python3 cli/estimate_pages.py configs/react.json
|
||||
|
||||
# 2. Scrape with local enhancement (25 min)
|
||||
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
|
||||
|
||||
# 3. Package (instant)
|
||||
python3 cli/package_skill.py output/react/
|
||||
|
||||
# Result: output/react.zip
|
||||
# Upload to Claude!
|
||||
```
|
||||
|
||||
### Workflow 2: Custom Documentation
|
||||
|
||||
```bash
|
||||
# 1. Create config
|
||||
cat > configs/my-docs.json << 'EOF'
|
||||
{
|
||||
"name": "my-docs",
|
||||
"base_url": "https://docs.example.com/",
|
||||
"description": "My documentation site",
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 200
|
||||
}
|
||||
EOF
|
||||
|
||||
# 2. Estimate
|
||||
python3 cli/estimate_pages.py configs/my-docs.json
|
||||
|
||||
# 3. Dry-run test
|
||||
python3 cli/doc_scraper.py --config configs/my-docs.json --dry-run
|
||||
|
||||
# 4. Full scrape
|
||||
python3 cli/doc_scraper.py --config configs/my-docs.json
|
||||
|
||||
# 5. Enhance
|
||||
python3 cli/enhance_skill_local.py output/my-docs/
|
||||
|
||||
# 6. Package
|
||||
python3 cli/package_skill.py output/my-docs/
|
||||
```
|
||||
|
||||
### Workflow 3: Interactive Mode
|
||||
|
||||
```bash
|
||||
# 1. Start interactive wizard
|
||||
python3 cli/doc_scraper.py --interactive
|
||||
|
||||
# 2. Answer prompts:
|
||||
# - Name: my-framework
|
||||
# - URL: https://framework.dev/
|
||||
# - Description: My favorite framework
|
||||
# - Selectors: (uses defaults)
|
||||
# - Rate limit: 0.5
|
||||
# - Max pages: 100
|
||||
|
||||
# 3. Enhance
|
||||
python3 cli/enhance_skill_local.py output/my-framework/
|
||||
|
||||
# 4. Package
|
||||
python3 cli/package_skill.py output/my-framework/
|
||||
```
|
||||
|
||||
### Workflow 4: Quick Mode
|
||||
|
||||
```bash
|
||||
python3 cli/doc_scraper.py \
|
||||
--name vue \
|
||||
--url https://vuejs.org/ \
|
||||
--description "Vue.js framework" \
|
||||
--enhance-local
|
||||
```
|
||||
|
||||
### Workflow 5: Rebuild from Cache
|
||||
|
||||
```bash
|
||||
# Already scraped once?
|
||||
# Skip re-scraping, just rebuild
|
||||
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
|
||||
|
||||
# Try new enhancement
|
||||
python3 cli/enhance_skill_local.py output/godot/
|
||||
|
||||
# Re-package
|
||||
python3 cli/package_skill.py output/godot/
|
||||
```
|
||||
|
||||
### Workflow 6: Testing New Config
|
||||
|
||||
```bash
|
||||
# 1. Create test config with low max_pages
|
||||
cat > configs/test.json << 'EOF'
|
||||
{
|
||||
"name": "test-site",
|
||||
"base_url": "https://docs.test.com/",
|
||||
"max_pages": 20,
|
||||
"rate_limit": 0.1
|
||||
}
|
||||
EOF
|
||||
|
||||
# 2. Estimate
|
||||
python3 cli/estimate_pages.py configs/test.json --max-discovery 50
|
||||
|
||||
# 3. Dry-run
|
||||
python3 cli/doc_scraper.py --config configs/test.json --dry-run
|
||||
|
||||
# 4. Small scrape
|
||||
python3 cli/doc_scraper.py --config configs/test.json
|
||||
|
||||
# 5. Validate output
|
||||
ls output/test-site/
|
||||
ls output/test-site/references/
|
||||
|
||||
# 6. If good, increase max_pages and re-run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "Rate limit exceeded"
|
||||
|
||||
```bash
|
||||
# Increase rate_limit in config
|
||||
# Default: 0.5 seconds
|
||||
# Conservative: 1.0 seconds
|
||||
# Very conservative: 2.0 seconds
|
||||
|
||||
# Edit config:
|
||||
{
|
||||
"rate_limit": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
### Issue: "Too many pages"
|
||||
|
||||
```bash
|
||||
# Estimate first
|
||||
python3 cli/estimate_pages.py configs/my-config.json
|
||||
|
||||
# Set max_pages based on estimate
|
||||
# Add buffer: estimated + 50
|
||||
|
||||
# Edit config:
|
||||
{
|
||||
"max_pages": 350 # for 300 estimated
|
||||
}
|
||||
```
|
||||
|
||||
### Issue: "No content extracted"
|
||||
|
||||
```bash
|
||||
# Wrong selectors
|
||||
# Test selectors manually:
|
||||
curl -s https://docs.example.com/ | grep -i 'article\|main\|content'
|
||||
|
||||
# Common selectors:
|
||||
"main_content": "article"
|
||||
"main_content": "main"
|
||||
"main_content": ".content"
|
||||
"main_content": "#main-content"
|
||||
"main_content": "div[role=\"main\"]"
|
||||
|
||||
# Update config with correct selector
|
||||
```
|
||||
|
||||
### Issue: "Tests failing"
|
||||
|
||||
```bash
|
||||
# Run specific failing test
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_name -v
|
||||
|
||||
# Check error message
|
||||
# Verify expectations match implementation
|
||||
```
|
||||
|
||||
### Issue: "Enhancement fails"
|
||||
|
||||
```bash
|
||||
# Local enhancement:
|
||||
# Make sure Claude Code is running
|
||||
# Check terminal output
|
||||
|
||||
# API enhancement:
|
||||
# Verify API key is set:
|
||||
echo $ANTHROPIC_API_KEY
|
||||
|
||||
# Or use inline:
|
||||
python3 cli/enhance_skill.py output/react/ --api-key sk-ant-...
|
||||
```
|
||||
|
||||
### Issue: "Package fails"
|
||||
|
||||
```bash
|
||||
# Verify SKILL.md exists
|
||||
ls output/my-skill/SKILL.md
|
||||
|
||||
# If missing, build first:
|
||||
python3 cli/doc_scraper.py --config configs/my-skill.json --skip-scrape
|
||||
```
|
||||
|
||||
### Issue: "Can't find output"
|
||||
|
||||
```bash
|
||||
# Check output directory
|
||||
ls output/
|
||||
|
||||
# Skill data (cached):
|
||||
ls output/{name}_data/
|
||||
|
||||
# Built skill:
|
||||
ls output/{name}/
|
||||
|
||||
# Packaged skill:
|
||||
ls output/{name}.zip
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Selectors
|
||||
|
||||
```json
|
||||
{
|
||||
"selectors": {
|
||||
"main_content": "div.documentation",
|
||||
"title": "h1.page-title",
|
||||
"code_blocks": "pre.highlight code",
|
||||
"navigation": "nav.sidebar"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### URL Pattern Filtering
|
||||
|
||||
```json
|
||||
{
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/docs/",
|
||||
"/guide/",
|
||||
"/api/",
|
||||
"/tutorial/"
|
||||
],
|
||||
"exclude": [
|
||||
"/blog/",
|
||||
"/news/",
|
||||
"/community/",
|
||||
"/showcase/"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Custom Categories
|
||||
|
||||
```json
|
||||
{
|
||||
"categories": {
|
||||
"getting_started": ["intro", "tutorial", "quickstart", "installation"],
|
||||
"core_concepts": ["concept", "fundamental", "architecture"],
|
||||
"api": ["reference", "api", "method", "function"],
|
||||
"guides": ["guide", "how-to", "example"],
|
||||
"advanced": ["advanced", "expert", "performance"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multiple Start URLs
|
||||
|
||||
```json
|
||||
{
|
||||
"start_urls": [
|
||||
"https://docs.example.com/getting-started/",
|
||||
"https://docs.example.com/api/",
|
||||
"https://docs.example.com/guides/",
|
||||
"https://docs.example.com/examples/"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Estimate first**: Save 20-40 minutes by validating config
|
||||
2. **Use dry-run**: Test selectors before full scrape
|
||||
3. **Cache data**: Use `--skip-scrape` for fast rebuilds
|
||||
4. **Adjust rate_limit**: Balance speed vs politeness
|
||||
5. **Set appropriate max_pages**: Don't scrape more than needed
|
||||
6. **Use start_urls**: Target specific documentation sections
|
||||
7. **Filter URLs**: Use include/exclude patterns
|
||||
8. **Run tests**: Catch issues early
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Anthropic API key (for API enhancement)
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Optional: Set custom output directory
|
||||
export SKILL_SEEKER_OUTPUT_DIR=/path/to/output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- `0`: Success
|
||||
- `1`: Error (general)
|
||||
- `2`: Warning (estimation hit limit)
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
```
|
||||
Skill_Seekers/
|
||||
├── doc_scraper.py # Main tool
|
||||
├── estimate_pages.py # Estimator
|
||||
├── enhance_skill.py # API enhancement
|
||||
├── enhance_skill_local.py # Local enhancement
|
||||
├── package_skill.py # Packager
|
||||
├── run_tests.py # Test runner
|
||||
├── configs/ # Preset configs
|
||||
├── tests/ # Test suite
|
||||
├── docs/ # Documentation
|
||||
└── output/ # Generated output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
```bash
|
||||
# Tool-specific help
|
||||
python3 cli/doc_scraper.py --help
|
||||
python3 cli/estimate_pages.py --help
|
||||
python3 cli/run_tests.py --help
|
||||
|
||||
# Documentation
|
||||
cat CLAUDE.md # Quick reference for Claude Code
|
||||
cat docs/CLAUDE.md # Detailed technical docs
|
||||
cat docs/TESTING.md # Testing guide
|
||||
cat docs/USAGE.md # This file
|
||||
cat docs/ENHANCEMENT.md # Enhancement guide
|
||||
cat docs/UPLOAD_GUIDE.md # Upload instructions
|
||||
cat README.md # Project overview
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Essential Commands:**
|
||||
```bash
|
||||
python3 cli/estimate_pages.py configs/react.json # Estimate
|
||||
python3 cli/doc_scraper.py --config configs/react.json # Scrape
|
||||
python3 cli/enhance_skill_local.py output/react/ # Enhance
|
||||
python3 cli/package_skill.py output/react/ # Package
|
||||
python3 cli/run_tests.py # Test
|
||||
```
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
pip3 install requests beautifulsoup4
|
||||
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
|
||||
python3 cli/package_skill.py output/react/
|
||||
# Upload output/react.zip to Claude!
|
||||
```
|
||||
|
||||
Happy skill creating! 🚀
|
||||
Reference in New Issue
Block a user