docs: add active skills design for demand-driven documentation
This commit is contained in:
Edgar I.
867
docs/plans/2025-10-24-active-skills-design.md
Normal file
867
docs/plans/2025-10-24-active-skills-design.md
Normal file
@@ -0,0 +1,867 @@
|
||||
# Active Skills Design - Demand-Driven Documentation Loading
|
||||
|
||||
**Date:** 2025-10-24
|
||||
**Type:** Architecture Design
|
||||
**Status:** Proposed
|
||||
**Author:** Edgar + Claude (Brainstorming Session)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Transform Skill_Seekers from creating **passive documentation dumps** into **active, intelligent skills** that load documentation on-demand. This eliminates context bloat (300k → 5-10k per query) while maintaining full access to complete documentation.
|
||||
|
||||
**Key Innovation:** Skills become lightweight routers with heavy tools in `scripts/`, not documentation repositories.
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current Architecture: Passive Skills
|
||||
|
||||
**What happens today:**
|
||||
```
|
||||
Agent: "How do I use Hono middleware?"
|
||||
↓
|
||||
Skill: *Claude loads 203k llms-txt.md into context*
|
||||
↓
|
||||
Agent: *answers using loaded docs*
|
||||
↓
|
||||
Result: Context bloat, slower performance, hits limits
|
||||
```
|
||||
|
||||
**Issues:**
|
||||
1. **Context Bloat**: 319k llms-full.txt loaded entirely into context
|
||||
2. **Wasted Resources**: Agent needs 5k but gets 319k
|
||||
3. **Truncation Loss**: 36% of content lost (319k → 203k) due to size limits
|
||||
4. **File Extension Bug**: llms.txt files stored as .txt instead of .md
|
||||
5. **Single Variant**: Only downloads one file (usually llms-full.txt)
|
||||
|
||||
### Current File Structure
|
||||
|
||||
```
|
||||
output/hono/
|
||||
├── SKILL.md ──────────► Documentation dump + instructions
|
||||
├── references/
|
||||
│ └── llms-txt.md ───► 203k (36% truncated from 319k original)
|
||||
├── scripts/ ──────────► EMPTY (placeholder only!)
|
||||
└── assets/ ───────────► EMPTY (placeholder only!)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture: Active Skills
|
||||
|
||||
### Core Concept
|
||||
|
||||
**Skills = Routers + Tools**, not documentation dumps.
|
||||
|
||||
**New workflow:**
|
||||
```
|
||||
Agent: "How do I use Hono middleware?"
|
||||
↓
|
||||
Skill: *runs scripts/search.py "middleware"*
|
||||
↓
|
||||
Script: *loads llms-full.md, extracts middleware section, returns 8k*
|
||||
↓
|
||||
Agent: *answers using ONLY 8k* (CLEAN CONTEXT!)
|
||||
↓
|
||||
Result: 40x less context, no truncation, full access to docs
|
||||
```
|
||||
|
||||
### Benefits
|
||||
|
||||
| Metric | Before | After | Improvement |
|
||||
|--------|--------|-------|-------------|
|
||||
| Context per query | 203k | 5-10k | **20-40x reduction** |
|
||||
| Content loss | 36% truncated | 0% (no truncation) | **Full fidelity** |
|
||||
| Variants available | 1 | 3 | **User choice** |
|
||||
| File format | .txt (wrong) | .md (correct) | **Fixed** |
|
||||
| Agent workflow | Passive read | Active tools | **Autonomous** |
|
||||
|
||||
---
|
||||
|
||||
## Design Components
|
||||
|
||||
### Component 1: Multi-Variant Download
|
||||
|
||||
**Change:** Download ALL 3 variants, not just one.
|
||||
|
||||
**File naming (FIXED):**
|
||||
- `https://hono.dev/llms-full.txt` → `llms-full.md` ✅
|
||||
- `https://hono.dev/llms.txt` → `llms.md` ✅
|
||||
- `https://hono.dev/llms-small.txt` → `llms-small.md` ✅
|
||||
|
||||
**Sizes (Hono example):**
|
||||
- `llms-full.md` - 319k (complete documentation)
|
||||
- `llms-small.md` - 176k (curated essentials)
|
||||
- `llms.md` - 5.4k (quick reference)
|
||||
|
||||
**Storage:**
|
||||
```
|
||||
output/hono/references/
|
||||
├── llms-full.md # 319k - everything (RENAMED from .txt)
|
||||
├── llms-small.md # 176k - curated (RENAMED from .txt)
|
||||
├── llms.md # 5.4k - quick ref (RENAMED from .txt)
|
||||
└── catalog.json # Generated index (NEW)
|
||||
```
|
||||
|
||||
**Implementation in `_try_llms_txt()`:**
|
||||
```python
|
||||
def _try_llms_txt(self) -> bool:
|
||||
"""Download ALL llms.txt variants for active skills"""
|
||||
|
||||
# 1. Detect all available variants
|
||||
detector = LlmsTxtDetector(self.base_url)
|
||||
variants = detector.detect_all() # NEW method
|
||||
|
||||
downloaded = {}
|
||||
for variant_info in variants:
|
||||
url = variant_info['url'] # https://hono.dev/llms-full.txt
|
||||
variant = variant_info['variant'] # 'full', 'standard', 'small'
|
||||
|
||||
downloader = LlmsTxtDownloader(url)
|
||||
content = downloader.download()
|
||||
|
||||
if content:
|
||||
# ✨ FIX: Rename .txt → .md immediately
|
||||
clean_name = f"llms-{variant}.md"
|
||||
downloaded[variant] = {
|
||||
'content': content,
|
||||
'filename': clean_name
|
||||
}
|
||||
|
||||
# 2. Save ALL variants (not just one)
|
||||
for variant, data in downloaded.items():
|
||||
path = os.path.join(self.skill_dir, "references", data['filename'])
|
||||
with open(path, 'w', encoding='utf-8') as f:
|
||||
f.write(data['content'])
|
||||
|
||||
# 3. Generate catalog from smallest variant
|
||||
if 'small' in downloaded:
|
||||
self._generate_catalog(downloaded['small']['content'])
|
||||
|
||||
return True
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Component 2: The Catalog System
|
||||
|
||||
**Purpose:** Lightweight index of what exists, not the content itself.
|
||||
|
||||
**File:** `assets/catalog.json`
|
||||
|
||||
**Structure:**
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"framework": "hono",
|
||||
"version": "auto-detected",
|
||||
"generated": "2025-10-24T14:30:00Z",
|
||||
"total_sections": 93,
|
||||
"variants": {
|
||||
"quick": "llms-small.md",
|
||||
"standard": "llms.md",
|
||||
"complete": "llms-full.md"
|
||||
}
|
||||
},
|
||||
"sections": [
|
||||
{
|
||||
"id": "routing",
|
||||
"title": "Routing",
|
||||
"h1_marker": "# Routing",
|
||||
"topics": ["routes", "path", "params", "wildcard"],
|
||||
"size_bytes": 4800,
|
||||
"variants": ["quick", "complete"],
|
||||
"complexity": "beginner"
|
||||
},
|
||||
{
|
||||
"id": "middleware",
|
||||
"title": "Middleware",
|
||||
"h1_marker": "# Middleware",
|
||||
"topics": ["cors", "auth", "logging", "compression"],
|
||||
"size_bytes": 8200,
|
||||
"variants": ["quick", "complete"],
|
||||
"complexity": "intermediate"
|
||||
}
|
||||
],
|
||||
"search_index": {
|
||||
"cors": ["middleware"],
|
||||
"routing": ["routing", "path-parameters"],
|
||||
"authentication": ["middleware", "jwt"],
|
||||
"context": ["context-handling"],
|
||||
"streaming": ["streaming-responses"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Generation (from llms-small.md):**
|
||||
```python
|
||||
def _generate_catalog(self, llms_small_content):
|
||||
"""Generate catalog.json from llms-small.md TOC"""
|
||||
catalog = {
|
||||
"metadata": {...},
|
||||
"sections": [],
|
||||
"search_index": {}
|
||||
}
|
||||
|
||||
# Split by h1 headers
|
||||
sections = re.split(r'\n# ', llms_small_content)
|
||||
|
||||
for section_text in sections[1:]:
|
||||
lines = section_text.split('\n')
|
||||
title = lines[0].strip()
|
||||
|
||||
# Extract h2 topics
|
||||
topics = re.findall(r'^## (.+)$', section_text, re.MULTILINE)
|
||||
topics = [t.strip().lower() for t in topics]
|
||||
|
||||
section_info = {
|
||||
"id": title.lower().replace(' ', '-'),
|
||||
"title": title,
|
||||
"h1_marker": f"# {title}",
|
||||
"topics": topics + [title.lower()],
|
||||
"size_bytes": len(section_text),
|
||||
"variants": ["quick", "complete"]
|
||||
}
|
||||
|
||||
catalog["sections"].append(section_info)
|
||||
|
||||
# Build search index
|
||||
for topic in section_info["topics"]:
|
||||
if topic not in catalog["search_index"]:
|
||||
catalog["search_index"][topic] = []
|
||||
catalog["search_index"][topic].append(section_info["id"])
|
||||
|
||||
# Save to assets/catalog.json
|
||||
catalog_path = os.path.join(self.skill_dir, "assets", "catalog.json")
|
||||
with open(catalog_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(catalog, f, indent=2)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Component 3: Active Scripts
|
||||
|
||||
**Location:** `scripts/` directory (currently empty)
|
||||
|
||||
#### Script 1: `scripts/search.py`
|
||||
|
||||
**Purpose:** Search and return only relevant documentation sections.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ABOUTME: Searches framework documentation and returns relevant sections
|
||||
ABOUTME: Loads only what's needed - keeps agent context clean
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def search(query, detail="auto"):
|
||||
"""
|
||||
Search documentation and return relevant sections.
|
||||
|
||||
Args:
|
||||
query: Search term (e.g., "middleware", "cors", "routing")
|
||||
detail: "quick" | "standard" | "complete" | "auto"
|
||||
|
||||
Returns:
|
||||
Markdown text of relevant sections only
|
||||
"""
|
||||
# Load catalog
|
||||
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
|
||||
catalog = json.load(open(catalog_path))
|
||||
|
||||
# 1. Find matching sections using search index
|
||||
query_lower = query.lower()
|
||||
matching_section_ids = set()
|
||||
|
||||
for keyword, section_ids in catalog["search_index"].items():
|
||||
if query_lower in keyword or keyword in query_lower:
|
||||
matching_section_ids.update(section_ids)
|
||||
|
||||
# Get section details
|
||||
matches = [s for s in catalog["sections"] if s["id"] in matching_section_ids]
|
||||
|
||||
if not matches:
|
||||
return f"❌ No sections found for '{query}'. Try: python scripts/list_topics.py"
|
||||
|
||||
# 2. Determine detail level
|
||||
if detail == "auto":
|
||||
# Use quick for overview, complete for deep dive
|
||||
total_size = sum(s["size_bytes"] for s in matches)
|
||||
if total_size > 50000: # > 50k
|
||||
variant = "quick"
|
||||
else:
|
||||
variant = "complete"
|
||||
else:
|
||||
variant = detail
|
||||
|
||||
variant_file = catalog["metadata"]["variants"].get(variant, "complete")
|
||||
|
||||
# 3. Load documentation file
|
||||
doc_path = Path(__file__).parent.parent / "references" / variant_file
|
||||
doc_content = open(doc_path, 'r', encoding='utf-8').read()
|
||||
|
||||
# 4. Extract matched sections
|
||||
results = []
|
||||
for match in matches:
|
||||
h1_marker = match["h1_marker"]
|
||||
|
||||
# Find section boundaries
|
||||
start = doc_content.find(h1_marker)
|
||||
if start == -1:
|
||||
continue
|
||||
|
||||
# Find next h1 (or end of file)
|
||||
next_h1 = doc_content.find("\n# ", start + len(h1_marker))
|
||||
if next_h1 == -1:
|
||||
section_text = doc_content[start:]
|
||||
else:
|
||||
section_text = doc_content[start:next_h1]
|
||||
|
||||
results.append({
|
||||
'title': match['title'],
|
||||
'size': len(section_text),
|
||||
'content': section_text
|
||||
})
|
||||
|
||||
# 5. Format output
|
||||
output = [f"# Search Results for '{query}' ({len(results)} sections found)\n"]
|
||||
output.append(f"**Variant used:** {variant} ({variant_file})")
|
||||
output.append(f"**Total size:** {sum(r['size'] for r in results):,} bytes\n")
|
||||
output.append("---\n")
|
||||
|
||||
for result in results:
|
||||
output.append(result['content'])
|
||||
output.append("\n---\n")
|
||||
|
||||
return '\n'.join(output)
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python search.py <query> [detail]")
|
||||
print("Example: python search.py middleware")
|
||||
print("Example: python search.py routing --detail quick")
|
||||
sys.exit(1)
|
||||
|
||||
query = sys.argv[1]
|
||||
detail = sys.argv[2] if len(sys.argv) > 2 else "auto"
|
||||
|
||||
print(search(query, detail))
|
||||
```
|
||||
|
||||
#### Script 2: `scripts/list_topics.py`
|
||||
|
||||
**Purpose:** Show all available documentation sections.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ABOUTME: Lists all available documentation sections with sizes
|
||||
ABOUTME: Helps agent discover what documentation exists
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
def list_topics():
|
||||
"""List all available documentation sections."""
|
||||
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
|
||||
catalog = json.load(open(catalog_path))
|
||||
|
||||
print(f"# Available Documentation Topics ({catalog['metadata']['framework']})\n")
|
||||
print(f"**Total sections:** {catalog['metadata']['total_sections']}")
|
||||
print(f"**Variants:** {', '.join(catalog['metadata']['variants'].keys())}\n")
|
||||
print("---\n")
|
||||
|
||||
# Group by complexity if available
|
||||
by_complexity = {}
|
||||
for section in catalog["sections"]:
|
||||
complexity = section.get("complexity", "general")
|
||||
if complexity not in by_complexity:
|
||||
by_complexity[complexity] = []
|
||||
by_complexity[complexity].append(section)
|
||||
|
||||
for complexity in ["beginner", "intermediate", "advanced", "general"]:
|
||||
if complexity not in by_complexity:
|
||||
continue
|
||||
|
||||
sections = by_complexity[complexity]
|
||||
print(f"## {complexity.title()} ({len(sections)} sections)\n")
|
||||
|
||||
for section in sections:
|
||||
size_kb = section["size_bytes"] / 1024
|
||||
topics_str = ", ".join(section["topics"][:3])
|
||||
print(f"- **{section['title']}** ({size_kb:.1f}k)")
|
||||
print(f" Topics: {topics_str}")
|
||||
print(f" Search: `python scripts/search.py {section['id']}`\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
list_topics()
|
||||
```
|
||||
|
||||
#### Script 3: `scripts/get_section.py`
|
||||
|
||||
**Purpose:** Extract a complete section by exact title.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ABOUTME: Extracts a complete documentation section by title
|
||||
ABOUTME: Returns full section from llms-full.md (no truncation)
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
def get_section(title, variant="complete"):
|
||||
"""
|
||||
Get a complete section by exact title.
|
||||
|
||||
Args:
|
||||
title: Section title (e.g., "Middleware", "Routing")
|
||||
variant: Which file to use (quick/standard/complete)
|
||||
|
||||
Returns:
|
||||
Complete section content
|
||||
"""
|
||||
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
|
||||
catalog = json.load(open(catalog_path))
|
||||
|
||||
# Find section
|
||||
section = None
|
||||
for s in catalog["sections"]:
|
||||
if s["title"].lower() == title.lower():
|
||||
section = s
|
||||
break
|
||||
|
||||
if not section:
|
||||
return f"❌ Section '{title}' not found. Try: python scripts/list_topics.py"
|
||||
|
||||
# Load doc
|
||||
variant_file = catalog["metadata"]["variants"].get(variant, "complete")
|
||||
doc_path = Path(__file__).parent.parent / "references" / variant_file
|
||||
doc_content = open(doc_path, 'r', encoding='utf-8').read()
|
||||
|
||||
# Extract section
|
||||
h1_marker = section["h1_marker"]
|
||||
start = doc_content.find(h1_marker)
|
||||
|
||||
if start == -1:
|
||||
return f"❌ Section '{title}' not found in {variant_file}"
|
||||
|
||||
next_h1 = doc_content.find("\n# ", start + len(h1_marker))
|
||||
if next_h1 == -1:
|
||||
section_text = doc_content[start:]
|
||||
else:
|
||||
section_text = doc_content[start:next_h1]
|
||||
|
||||
return section_text
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python get_section.py <title> [variant]")
|
||||
print("Example: python get_section.py Middleware")
|
||||
print("Example: python get_section.py Routing quick")
|
||||
sys.exit(1)
|
||||
|
||||
title = sys.argv[1]
|
||||
variant = sys.argv[2] if len(sys.argv) > 2 else "complete"
|
||||
|
||||
print(get_section(title, variant))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Component 4: Active SKILL.md Template
|
||||
|
||||
**New template for llms.txt-based skills:**
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: {name}
|
||||
description: {description}
|
||||
type: active
|
||||
---
|
||||
|
||||
# {Name} Skill
|
||||
|
||||
**⚡ This is an ACTIVE skill** - Uses scripts to load documentation on-demand instead of dumping everything into context.
|
||||
|
||||
## 🎯 Strategy: Demand-Driven Documentation
|
||||
|
||||
**Traditional approach:**
|
||||
- Load 300k+ documentation into context
|
||||
- Agent reads everything to answer one question
|
||||
- Context bloat, slower performance
|
||||
|
||||
**Active approach:**
|
||||
- Load 5-10k of relevant sections on-demand
|
||||
- Agent calls scripts to fetch what's needed
|
||||
- Clean context, faster performance
|
||||
|
||||
## 📚 Available Documentation
|
||||
|
||||
This skill provides access to {num_sections} documentation sections across 3 detail levels:
|
||||
|
||||
- **Quick Reference** (`llms-small.md`): {small_size}k - Curated essentials
|
||||
- **Standard** (`llms.md`): {standard_size}k - Core concepts
|
||||
- **Complete** (`llms-full.md`): {full_size}k - Everything
|
||||
|
||||
## 🔧 Tools Available
|
||||
|
||||
### 1. Search Documentation
|
||||
Find and load only relevant sections:
|
||||
|
||||
```bash
|
||||
python scripts/search.py "middleware"
|
||||
python scripts/search.py "routing" --detail quick
|
||||
```
|
||||
|
||||
**Returns:** 5-10k of relevant content (not 300k!)
|
||||
|
||||
### 2. List All Topics
|
||||
See what documentation exists:
|
||||
|
||||
```bash
|
||||
python scripts/list_topics.py
|
||||
```
|
||||
|
||||
**Returns:** Table of contents with section sizes and search hints
|
||||
|
||||
### 3. Get Complete Section
|
||||
Extract a full section by title:
|
||||
|
||||
```bash
|
||||
python scripts/get_section.py "Middleware"
|
||||
python scripts/get_section.py "Routing" quick
|
||||
```
|
||||
|
||||
**Returns:** Complete section from chosen variant
|
||||
|
||||
## 💡 Recommended Workflow
|
||||
|
||||
1. **Discover:** `python scripts/list_topics.py` to see what's available
|
||||
2. **Search:** `python scripts/search.py "your topic"` to find relevant sections
|
||||
3. **Deep Dive:** Use returned content to answer questions in detail
|
||||
4. **Iterate:** Search more specific topics as needed
|
||||
|
||||
## ⚠️ Important
|
||||
|
||||
**DON'T:** Read `references/*.md` files directly into context
|
||||
**DO:** Use scripts to fetch only what you need
|
||||
|
||||
This keeps your context clean and focused!
|
||||
|
||||
## 📊 Index
|
||||
|
||||
Complete section catalog available in `assets/catalog.json` with search mappings and size information.
|
||||
|
||||
## 🔄 Updating
|
||||
|
||||
To refresh with latest documentation:
|
||||
```bash
|
||||
python3 cli/doc_scraper.py --config configs/{name}.json
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Foundation (Quick Fixes)
|
||||
|
||||
**Tasks:**
|
||||
1. Fix `.txt` → `.md` renaming in downloader
|
||||
2. Download all 3 variants (not just one)
|
||||
3. Store all variants in `references/` with correct names
|
||||
4. Remove content truncation (2500 chars → unlimited)
|
||||
|
||||
**Time:** 1-2 hours
|
||||
**Files:** `cli/doc_scraper.py`, `cli/llms_txt_downloader.py`
|
||||
|
||||
### Phase 2: Catalog System
|
||||
|
||||
**Tasks:**
|
||||
1. Implement `_generate_catalog()` method
|
||||
2. Parse llms-small.md to extract sections
|
||||
3. Build search index from topics
|
||||
4. Generate `assets/catalog.json`
|
||||
|
||||
**Time:** 2-3 hours
|
||||
**Files:** `cli/doc_scraper.py`
|
||||
|
||||
### Phase 3: Active Scripts
|
||||
|
||||
**Tasks:**
|
||||
1. Create `scripts/search.py`
|
||||
2. Create `scripts/list_topics.py`
|
||||
3. Create `scripts/get_section.py`
|
||||
4. Make scripts executable (`chmod +x`)
|
||||
|
||||
**Time:** 2-3 hours
|
||||
**Files:** New scripts in `scripts/` template directory
|
||||
|
||||
### Phase 4: Template Updates
|
||||
|
||||
**Tasks:**
|
||||
1. Create new active SKILL.md template
|
||||
2. Update `create_enhanced_skill_md()` to use active template for llms.txt skills
|
||||
3. Update documentation to explain active skills
|
||||
|
||||
**Time:** 1 hour
|
||||
**Files:** `cli/doc_scraper.py`, `README.md`, `CLAUDE.md`
|
||||
|
||||
### Phase 5: Testing & Refinement
|
||||
|
||||
**Tasks:**
|
||||
1. Test with Hono skill (has all 3 variants)
|
||||
2. Test search accuracy
|
||||
3. Measure context reduction
|
||||
4. Document examples
|
||||
|
||||
**Time:** 2-3 hours
|
||||
|
||||
**Total Estimated Time:** 8-12 hours
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
**Existing skills:** No changes (passive skills still work)
|
||||
**New llms.txt skills:** Automatically use active architecture
|
||||
**User choice:** Can disable via config flag
|
||||
|
||||
### Config Option
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "hono",
|
||||
"llms_txt_url": "https://hono.dev/llms-full.txt",
|
||||
"active_skill": true, // NEW: Enable active architecture (default: true)
|
||||
"base_url": "https://hono.dev/docs"
|
||||
}
|
||||
```
|
||||
|
||||
### Detection Logic
|
||||
|
||||
```python
|
||||
# In _try_llms_txt()
|
||||
active_mode = self.config.get('active_skill', True) # Default true
|
||||
|
||||
if active_mode:
|
||||
# Download all variants, generate catalog, create scripts
|
||||
self._build_active_skill(downloaded)
|
||||
else:
|
||||
# Traditional: single file, no scripts
|
||||
self._build_passive_skill(downloaded)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Benefits Analysis
|
||||
|
||||
### Context Efficiency
|
||||
|
||||
| Scenario | Passive Skill | Active Skill | Improvement |
|
||||
|----------|---------------|--------------|-------------|
|
||||
| Simple query | 203k loaded | 5k loaded | **40x reduction** |
|
||||
| Multi-topic query | 203k loaded | 15k loaded | **13x reduction** |
|
||||
| Deep dive | 203k loaded | 30k loaded | **6x reduction** |
|
||||
|
||||
### Data Fidelity
|
||||
|
||||
| Aspect | Passive | Active |
|
||||
|--------|---------|--------|
|
||||
| Content truncation | 36% lost | 0% lost |
|
||||
| Code truncation | 600 chars max | Unlimited |
|
||||
| Variants available | 1 | 3 |
|
||||
|
||||
### Agent Capabilities
|
||||
|
||||
**Passive Skills:**
|
||||
- ❌ Cannot choose detail level
|
||||
- ❌ Cannot search efficiently
|
||||
- ❌ Must read entire context
|
||||
- ❌ Limited by context window
|
||||
|
||||
**Active Skills:**
|
||||
- ✅ Chooses appropriate detail level
|
||||
- ✅ Searches catalog efficiently
|
||||
- ✅ Loads only what's needed
|
||||
- ✅ Unlimited documentation access
|
||||
|
||||
---
|
||||
|
||||
## Trade-offs
|
||||
|
||||
### Advantages
|
||||
|
||||
1. **Massive context reduction** (20-40x less per query)
|
||||
2. **No content loss** (all 3 variants preserved)
|
||||
3. **Correct file format** (.md not .txt)
|
||||
4. **Agent autonomy** (tools to fetch docs)
|
||||
5. **Scalable** (works with 1MB+ docs)
|
||||
|
||||
### Disadvantages
|
||||
|
||||
1. **Complexity** (scripts + catalog vs simple files)
|
||||
2. **Initial overhead** (catalog generation)
|
||||
3. **Agent learning curve** (must learn to use scripts)
|
||||
4. **Dependency** (Python required to run scripts)
|
||||
|
||||
### Risk Mitigation
|
||||
|
||||
**Risk:** Scripts don't work in Claude's sandbox
|
||||
**Mitigation:** Test thoroughly, provide fallback to passive mode
|
||||
|
||||
**Risk:** Catalog generation fails
|
||||
**Mitigation:** Graceful degradation to single-file mode
|
||||
|
||||
**Risk:** Agent doesn't use scripts
|
||||
**Mitigation:** Clear SKILL.md instructions, examples in quick reference
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Technical Metrics
|
||||
|
||||
- ✅ Context per query < 20k (down from 203k)
|
||||
- ✅ All 3 variants downloaded and named correctly
|
||||
- ✅ 0% content truncation
|
||||
- ✅ Catalog generation < 5 seconds
|
||||
- ✅ Search script < 1 second response time
|
||||
|
||||
### User Experience Metrics
|
||||
|
||||
- ✅ Agent successfully uses scripts without prompting
|
||||
- ✅ Answers are equally or more accurate than passive mode
|
||||
- ✅ Agent can handle queries about all documentation sections
|
||||
- ✅ No "context limit exceeded" errors
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Phase 6: Smart Caching
|
||||
|
||||
Cache frequently accessed sections in SKILL.md quick reference:
|
||||
```python
|
||||
# Track access frequency in catalog.json
|
||||
"sections": [
|
||||
{
|
||||
"id": "middleware",
|
||||
"access_count": 47, # NEW: Track usage
|
||||
"last_accessed": "2025-10-24T14:30:00Z"
|
||||
}
|
||||
]
|
||||
|
||||
# Include top 10 most-accessed sections directly in SKILL.md
|
||||
```
|
||||
|
||||
### Phase 7: Semantic Search
|
||||
|
||||
Use embeddings for better search:
|
||||
```python
|
||||
# Generate embeddings for each section
|
||||
"sections": [
|
||||
{
|
||||
"id": "middleware",
|
||||
"embedding": [...], # NEW: Vector embedding
|
||||
"topics": ["cors", "auth"]
|
||||
}
|
||||
]
|
||||
|
||||
# In search.py: Use cosine similarity for better matches
|
||||
```
|
||||
|
||||
### Phase 8: Progressive Loading
|
||||
|
||||
Load increasingly detailed docs:
|
||||
```python
|
||||
# First: Load llms.md (5.4k - overview)
|
||||
# If insufficient: Load llms-small.md section (15k)
|
||||
# If still insufficient: Load llms-full.md section (30k)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Active skills represent a fundamental shift from **documentation repositories** to **documentation routers**. By treating skills as intelligent intermediaries rather than static dumps, we can:
|
||||
|
||||
1. **Eliminate context bloat** (40x reduction)
|
||||
2. **Preserve full fidelity** (0% truncation)
|
||||
3. **Enable agent autonomy** (tools to fetch docs)
|
||||
4. **Scale indefinitely** (no size limits)
|
||||
|
||||
This design maintains backward compatibility while unlocking new capabilities for modern, LLM-optimized documentation sources like llms.txt.
|
||||
|
||||
**Recommendation:** Implement in phases, starting with foundation fixes, then catalog system, then active scripts. Test thoroughly with Hono before making it the default for all llms.txt-based skills.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Original brainstorming session: 2025-10-24
|
||||
- llms.txt convention: https://llmstxt.org/
|
||||
- Hono example: https://hono.dev/llms-full.txt
|
||||
- Skill_Seekers repository: Current project
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Example Workflows
|
||||
|
||||
### Example 1: Agent Searches for "Middleware"
|
||||
|
||||
```bash
|
||||
# Agent runs:
|
||||
python scripts/search.py "middleware"
|
||||
|
||||
# Script returns ~8k of middleware documentation from llms-full.md
|
||||
# Agent uses that 8k to answer the question
|
||||
# Total context used: 8k (not 319k!)
|
||||
```
|
||||
|
||||
### Example 2: Agent Explores Documentation
|
||||
|
||||
```bash
|
||||
# 1. Agent lists topics
|
||||
python scripts/list_topics.py
|
||||
# Returns: Table of contents (2k)
|
||||
|
||||
# 2. Agent picks a topic
|
||||
python scripts/get_section.py "Routing"
|
||||
# Returns: Complete Routing section (5k)
|
||||
|
||||
# 3. Agent searches related topics
|
||||
python scripts/search.py "path parameters"
|
||||
# Returns: Routing + Path section (7k)
|
||||
|
||||
# Total context used across 3 queries: 14k (not 3 × 319k = 957k!)
|
||||
```
|
||||
|
||||
### Example 3: Agent Needs Quick Answer
|
||||
|
||||
```bash
|
||||
# Agent uses quick variant for overview
|
||||
python scripts/search.py "cors" --detail quick
|
||||
|
||||
# Returns: Short CORS explanation from llms-small.md (2k)
|
||||
# If insufficient, agent can follow up with:
|
||||
python scripts/get_section.py "Middleware" # Full section from llms-full.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Document Status:** Ready for review and implementation planning.
|
||||
Reference in New Issue
Block a user