firefrost-gaming/skill-seekers-reference
3
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

feat: AI enhancement multi-repo support + critical bug fix

Browse Source 
CRITICAL BUG FIX:
- Fixed documentation scraper overwriting list with dict
- Changed self.scraped_data['documentation'] = {...} to .append({...})
- Bug was breaking unified skill builder reference generation

AI ENHANCEMENT UPDATES:
- Added repo_id extraction in utils.py for multi-repo support
- Enhanced grouping by (source, repo_id) tuple in both enhancement files
- Added MULTI-REPOSITORY HANDLING section to AI prompts
- AI now correctly identifies and synthesizes multiple repos

CHANGES:
1. src/skill_seekers/cli/utils.py:
   - _determine_source_metadata() now returns (source, confidence, repo_id)
   - Extracts repo_id from codebase_analysis/{repo_id}/ paths
   - Added repo_id field to reference metadata dict

2. src/skill_seekers/cli/enhance_skill_local.py:
   - Group references by (source_type, repo_id) instead of just source_type
   - Display repo identity in prompt sections
   - Detect multiple repos and add explicit guidance to AI

3. src/skill_seekers/cli/enhance_skill.py:
   - Same grouping and display logic as local enhancement
   - Multi-repository handling section added

4. src/skill_seekers/cli/unified_scraper.py:
   - FIX: Documentation scraper now appends to list instead of overwriting
   - Added source_id, base_url, refs_dir to documentation metadata
   - Update refs_dir after moving to cache

TESTING:
- All 57 tests passing (unified, C3, utilities)
- Single-source verified: httpx comprehensive (219→749 lines after enhancement)
- Multi-source verified: encode/httpx + encode/httpcore (523 lines)
- AI enhancement working: Professional output with source attribution

QUALITY:
- Enhanced httpx SKILL.md: 749 lines, 19KB, A+ quality
- Source attribution working correctly
- Multi-repo synthesis transparent and accurate
- Reference structure clean and organized

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
yusyus
2026-01-12 22:05:34 +03:00
parent 52cf99136a
commit 72dde1ba08
4 changed files with 151 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
src/skill_seekers/cli/enhance_skill.py
View File

@@ -138,18 +138,24 @@ This skill combines knowledge from {len(sources_found)} source type(s):
"""
# Group references by source type
# Group references by (source_type, repo_id) for multi-source support
by_source = {}
for filename, metadata in references.items():
source = metadata['source']
if source not in by_source:
by_source[source] = []
by_source[source].append((filename, metadata))
repo_id = metadata.get('repo_id') # None for single-source
key = (source, repo_id) if repo_id else (source, None)
# Add source breakdown
for source in sorted(by_source.keys()):
files = by_source[source]
prompt += f"\n**{source.upper()} ({len(files)} file(s))**\n"
if key not in by_source:
by_source[key] = []
by_source[key].append((filename, metadata))
# Add source breakdown with repo identity
for (source, repo_id) in sorted(by_source.keys()):
files = by_source[(source, repo_id)]
if repo_id:
prompt += f"\n**{source.upper()} - {repo_id} ({len(files)} file(s))**\n"
else:
prompt += f"\n**{source.upper()} ({len(files)} file(s))**\n"
for filename, metadata in files[:5]: # Top 5 per source
prompt += f"- {filename} (confidence: {metadata['confidence']}, {metadata['size']:,} chars)\n"
if len(files) > 5:
@@ -157,17 +163,24 @@ This skill combines knowledge from {len(sources_found)} source type(s):
prompt += "\n\nREFERENCE DOCUMENTATION:\n"
# Add references grouped by source with metadata
for source in sorted(by_source.keys()):
prompt += f"\n### {source.upper()} SOURCES\n\n"
for filename, metadata in by_source[source]:
# Add references grouped by (source, repo_id) with metadata
for (source, repo_id) in sorted(by_source.keys()):
if repo_id:
prompt += f"\n### {source.upper()} SOURCES - {repo_id}\n\n"
else:
prompt += f"\n### {source.upper()} SOURCES\n\n"
for filename, metadata in by_source[(source, repo_id)]:
content = metadata['content']
# Limit per-file to 30K
if len(content) > 30000:
content = content[:30000] + "\n\n[Content truncated for size...]"
prompt += f"\n#### {filename}\n"
prompt += f"*Source: {metadata['source']}, Confidence: {metadata['confidence']}*\n\n"
if repo_id:
prompt += f"*Source: {metadata['source']} ({repo_id}), Confidence: {metadata['confidence']}*\n\n"
else:
prompt += f"*Source: {metadata['source']}, Confidence: {metadata['confidence']}*\n\n"
prompt += f"```markdown\n{content}\n```\n"
prompt += """
@@ -178,6 +191,34 @@ REFERENCE PRIORITY (when sources differ):
3. **GitHub issues**: Real-world usage and known problems
4. **PDF documentation**: Additional context and tutorials
MULTI-REPOSITORY HANDLING:
"""
# Detect multiple repos from same source type
repo_ids = set()
for metadata in references.values():
if metadata.get('repo_id'):
repo_ids.add(metadata['repo_id'])
if len(repo_ids) > 1:
prompt += f"""
⚠️ MULTIPLE REPOSITORIES DETECTED: {', '.join(sorted(repo_ids))}
This skill combines codebase analysis from {len(repo_ids)} different repositories.
Each repo has its own ARCHITECTURE.md, patterns, examples, and configuration.
When synthesizing:
- Clearly identify which content comes from which repo
- Compare and contrast patterns across repos (e.g., "httpx uses Strategy pattern 50 times, httpcore uses it 32 times")
- Highlight relationships (e.g., "httpx is a client library built on top of httpcore")
- Present examples from BOTH repos to show different use cases
- If repos serve different purposes, explain when to use each
"""
else:
prompt += "\nSingle repository - standard synthesis applies.\n"
prompt += """
YOUR TASK:
Create an enhanced SKILL.md that synthesizes knowledge from multiple sources:

67
src/skill_seekers/cli/enhance_skill_local.py
View File

@@ -259,18 +259,24 @@ This skill combines knowledge from {len(sources_found)} source type(s):
"""
# Group references by source type
# Group references by (source_type, repo_id) for multi-source support
by_source = {}
for filename, metadata in references.items():
source = metadata['source']
if source not in by_source:
by_source[source] = []
by_source[source].append((filename, metadata))
repo_id = metadata.get('repo_id') # None for single-source
key = (source, repo_id) if repo_id else (source, None)
# Add source breakdown
for source in sorted(by_source.keys()):
files = by_source[source]
prompt += f"\n**{source.upper()} ({len(files)} file(s))**\n"
if key not in by_source:
by_source[key] = []
by_source[key].append((filename, metadata))
# Add source breakdown with repo identity
for (source, repo_id) in sorted(by_source.keys()):
files = by_source[(source, repo_id)]
if repo_id:
prompt += f"\n**{source.upper()} - {repo_id} ({len(files)} file(s))**\n"
else:
prompt += f"\n**{source.upper()} ({len(files)} file(s))**\n"
for filename, metadata in files[:5]: # Top 5 per source
prompt += f"- {filename} (confidence: {metadata['confidence']}, {metadata['size']:,} chars)\n"
if len(files) > 5:
@@ -283,10 +289,14 @@ REFERENCE DOCUMENTATION:
{'-'*60}
"""
# Add references grouped by source with metadata
for source in sorted(by_source.keys()):
prompt += f"\n### {source.upper()} SOURCES\n\n"
for filename, metadata in by_source[source]:
# Add references grouped by (source, repo_id) with metadata
for (source, repo_id) in sorted(by_source.keys()):
if repo_id:
prompt += f"\n### {source.upper()} SOURCES - {repo_id}\n\n"
else:
prompt += f"\n### {source.upper()} SOURCES\n\n"
for filename, metadata in by_source[(source, repo_id)]:
# Further limit per-file to 12K to be safe
content = metadata['content']
max_per_file = 12000
@@ -294,7 +304,10 @@ REFERENCE DOCUMENTATION:
content = content[:max_per_file] + "\n\n[Content truncated for size...]"
prompt += f"\n#### {filename}\n"
prompt += f"*Source: {metadata['source']}, Confidence: {metadata['confidence']}*\n\n"
if repo_id:
prompt += f"*Source: {metadata['source']} ({repo_id}), Confidence: {metadata['confidence']}*\n\n"
else:
prompt += f"*Source: {metadata['source']}, Confidence: {metadata['confidence']}*\n\n"
prompt += f"{content}\n"
prompt += f"""
@@ -306,6 +319,34 @@ REFERENCE PRIORITY (when sources differ):
3. **GitHub issues**: Real-world usage and known problems
4. **PDF documentation**: Additional context and tutorials
MULTI-REPOSITORY HANDLING:
"""
# Detect multiple repos from same source type
repo_ids = set()
for metadata in references.values():
if metadata.get('repo_id'):
repo_ids.add(metadata['repo_id'])
if len(repo_ids) > 1:
prompt += f"""
⚠️ MULTIPLE REPOSITORIES DETECTED: {', '.join(sorted(repo_ids))}
This skill combines codebase analysis from {len(repo_ids)} different repositories.
Each repo has its own ARCHITECTURE.md, patterns, examples, and configuration.
When synthesizing:
- Clearly identify which content comes from which repo
- Compare and contrast patterns across repos (e.g., "httpx uses Strategy pattern 50 times, httpcore uses it 32 times")
- Highlight relationships (e.g., "httpx is a client library built on top of httpcore")
- Present examples from BOTH repos to show different use cases
- If repos serve different purposes, explain when to use each
"""
else:
prompt += "\nSingle repository - standard synthesis applies.\n"
prompt += """
YOUR TASK:
Create an EXCELLENT SKILL.md file that synthesizes knowledge from multiple sources.

16
src/skill_seekers/cli/unified_scraper.py
View File

@@ -216,10 +216,15 @@ class UnifiedScraper:
with open(docs_data_file, 'r', encoding='utf-8') as f:
summary = json.load(f)
self.scraped_data['documentation'] = {
# Append to documentation list (multi-source support)
self.scraped_data['documentation'].append({
'source_id': doc_config['name'],
'base_url': source['base_url'],
'pages': summary.get('pages', []),
'data_file': docs_data_file
}
'total_pages': summary.get('total_pages', 0),
'data_file': docs_data_file,
'refs_dir': '' # Will be set after moving to cache
})
logger.info(f"✅ Documentation: {summary.get('total_pages', 0)} pages scraped")
else:
@@ -240,6 +245,11 @@ class UnifiedScraper:
shutil.move(docs_output_dir, cache_docs_dir)
logger.info(f"📦 Moved docs output to cache: {cache_docs_dir}")
# Update refs_dir in scraped_data with cache location
refs_dir_path = os.path.join(cache_docs_dir, 'references')
if self.scraped_data['documentation']:
self.scraped_data['documentation'][-1]['refs_dir'] = refs_dir_path
if os.path.exists(docs_data_dir):
cache_data_dir = os.path.join(self.data_dir, f"{doc_config['name']}_data")
if os.path.exists(cache_data_dir):

48
src/skill_seekers/cli/utils.py
View File

@@ -197,6 +197,7 @@ def read_reference_files(skill_dir: Union[str, Path], max_chars: int = 100000, p
- 'source': Source type (documentation/github/pdf/api/codebase_analysis)
- 'confidence': Confidence level (high/medium/low)
- 'path': Relative path from references directory
- 'repo_id': Repository identifier for multi-source (e.g., 'encode_httpx'), None for single-source
Example:
>>> refs = read_reference_files('output/react/', max_chars=50000)
@@ -215,58 +216,68 @@ def read_reference_files(skill_dir: Union[str, Path], max_chars: int = 100000, p
print(f"⚠ No references directory found at {references_dir}")
return references
def _determine_source_metadata(relative_path: Path) -> Tuple[str, str]:
"""Determine source type and confidence level from path.
def _determine_source_metadata(relative_path: Path) -> Tuple[str, str, Optional[str]]:
"""Determine source type, confidence level, and repo_id from path.
For multi-source support, extracts repo_id from paths like:
- codebase_analysis/encode_httpx/ARCHITECTURE.md -> repo_id='encode_httpx'
- github/README.md -> repo_id=None (single source)
Returns:
tuple: (source_type, confidence_level)
tuple: (source_type, confidence_level, repo_id)
"""
path_str = str(relative_path)
repo_id = None # Default: no repo identity
# Documentation sources (official docs)
if path_str.startswith('documentation/'):
return 'documentation', 'high'
return 'documentation', 'high', None
# GitHub sources
elif path_str.startswith('github/'):
# README and releases are medium confidence
if 'README' in path_str or 'releases' in path_str:
return 'github', 'medium'
return 'github', 'medium', None
# Issues are low confidence (user reports)
elif 'issues' in path_str:
return 'github', 'low'
return 'github', 'low', None
else:
return 'github', 'medium'
return 'github', 'medium', None
# PDF sources (books, manuals)
elif path_str.startswith('pdf/'):
return 'pdf', 'high'
return 'pdf', 'high', None
# Merged API (synthesized from multiple sources)
elif path_str.startswith('api/'):
return 'api', 'high'
return 'api', 'high', None
# Codebase analysis (C3.x automated analysis)
elif path_str.startswith('codebase_analysis/'):
# Extract repo_id from path: codebase_analysis/{repo_id}/...
parts = Path(path_str).parts
if len(parts) >= 2:
repo_id = parts[1] # e.g., 'encode_httpx', 'encode_httpcore'
# ARCHITECTURE.md is high confidence (comprehensive)
if 'ARCHITECTURE' in path_str:
return 'codebase_analysis', 'high'
return 'codebase_analysis', 'high', repo_id
# Patterns and examples are medium (heuristic-based)
elif 'patterns' in path_str or 'examples' in path_str:
return 'codebase_analysis', 'medium'
return 'codebase_analysis', 'medium', repo_id
# Configuration is high (direct extraction)
elif 'configuration' in path_str:
return 'codebase_analysis', 'high'
return 'codebase_analysis', 'high', repo_id
else:
return 'codebase_analysis', 'medium'
return 'codebase_analysis', 'medium', repo_id
# Conflicts report (discrepancy detection)
elif 'conflicts' in path_str:
return 'conflicts', 'medium'
return 'conflicts', 'medium', None
# Fallback
else:
return 'unknown', 'medium'
return 'unknown', 'medium', None
total_chars = 0
# Search recursively for all .md files (including subdirectories like github/README.md)
@@ -284,16 +295,17 @@ def read_reference_files(skill_dir: Union[str, Path], max_chars: int = 100000, p
# Use relative path from references_dir as key for nested files
relative_path = ref_file.relative_to(references_dir)
source_type, confidence = _determine_source_metadata(relative_path)
source_type, confidence, repo_id = _determine_source_metadata(relative_path)
# Build enriched metadata
# Build enriched metadata (with repo_id for multi-source support)
references[str(relative_path)] = {
'content': content,
'source': source_type,
'confidence': confidence,
'path': str(relative_path),
'truncated': truncated,
'size': len(content)
'size': len(content),
'repo_id': repo_id # None for single-source, repo identifier for multi-source
}
total_chars += len(content)