ba9a8ff8b5
Documentation restructure: - New docs/getting-started/ guide (4 files: install, quick-start, first-skill, next-steps) - New docs/user-guide/ section (6 files: core concepts through troubleshooting) - New docs/reference/ section (CLI_REFERENCE, CONFIG_FORMAT, ENVIRONMENT_VARIABLES, MCP_REFERENCE) - New docs/advanced/ section (custom-workflows, mcp-server, multi-source) - New docs/ARCHITECTURE.md - system architecture overview - Archived legacy files (QUICKSTART.md, QUICK_REFERENCE.md, docs/guides/USAGE.md) to docs/archive/legacy/ Chinese (zh-CN) translations: - Full zh-CN mirror of all user-facing docs (getting-started, user-guide, reference, advanced) - GitHub Actions workflow for translation sync (.github/workflows/translate-docs.yml) - Translation sync checker script (scripts/check_translation_sync.sh) - Translation helper script (scripts/translate_doc.py) Content updates: - CHANGELOG.md: [Unreleased] → [3.1.0] - 2026-02-22 - README.md: updated with new doc structure links - AGENTS.md: updated agent documentation - docs/features/UNIFIED_SCRAPING.md: updated for unified scraper workflow JSON config Analysis/planning artifacts (kept for reference): - DOCUMENTATION_OVERHAUL_PLAN.md, DOCUMENTATION_OVERHAUL_SUMMARY.md - FEATURE_GAP_ANALYSIS.md, IMPLEMENTATION_GAPS_ANALYSIS.md, CREATE_COMMAND_COVERAGE_ANALYSIS.md - CHINESE_TRANSLATION_IMPLEMENTATION_SUMMARY.md, ISSUE_260_UPDATE.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
61 lines
1.4 KiB
Markdown
61 lines
1.4 KiB
Markdown
# llms.txt Support
|
|
|
|
## Overview
|
|
|
|
Skill_Seekers now automatically detects and uses llms.txt files when available, providing 10x faster documentation ingestion.
|
|
|
|
## What is llms.txt?
|
|
|
|
The llms.txt convention is a growing standard where documentation sites provide pre-formatted, LLM-ready markdown files:
|
|
|
|
- `llms-full.txt` - Complete documentation
|
|
- `llms.txt` - Standard balanced version
|
|
- `llms-small.txt` - Quick reference
|
|
|
|
## How It Works
|
|
|
|
1. Before HTML scraping, Skill_Seekers checks for llms.txt files
|
|
2. If found, downloads and parses the markdown
|
|
3. If not found, falls back to HTML scraping
|
|
4. Zero config changes needed
|
|
|
|
## Configuration
|
|
|
|
### Automatic Detection (Recommended)
|
|
|
|
No config changes needed. Just run normally:
|
|
|
|
```bash
|
|
python3 cli/doc_scraper.py --config configs/hono.json
|
|
```
|
|
|
|
### Explicit URL
|
|
|
|
Optionally specify llms.txt URL:
|
|
|
|
```json
|
|
{
|
|
"name": "hono",
|
|
"llms_txt_url": "https://hono.dev/llms-full.txt",
|
|
"base_url": "https://hono.dev/docs"
|
|
}
|
|
```
|
|
|
|
## Performance Comparison
|
|
|
|
| Method | Time | Requests |
|
|
|--------|------|----------|
|
|
| HTML Scraping (20 pages) | 20-60s | 20+ |
|
|
| llms.txt | < 5s | 1 |
|
|
|
|
## Supported Sites
|
|
|
|
Sites known to provide llms.txt:
|
|
|
|
- Hono: https://hono.dev/llms-full.txt
|
|
- (More to be discovered)
|
|
|
|
## Fallback Behavior
|
|
|
|
If llms.txt download or parsing fails, automatically falls back to HTML scraping with no user intervention required.
|