Implements hybrid smart extraction + improved fallback templates for
skill descriptions across all scrapers.
Changes:
- github_scraper.py:
* Added extract_description_from_readme() helper
* Extracts from README first paragraph (60 lines)
* Updates description after README extraction
* Fallback: "Use when working with {name}"
* Updated 3 locations (GitHubScraper, GitHubToSkillConverter, main)
- doc_scraper.py:
* Added infer_description_from_docs() helper
* Extracts from meta tags or first paragraph (65 lines)
* Tries: meta description, og:description, first content paragraph
* Fallback: "Use when working with {name}"
* Updated 2 locations (create_enhanced_skill_md, get_configuration)
- pdf_scraper.py:
* Added infer_description_from_pdf() helper
* Extracts from PDF metadata (subject, title)
* Fallback: "Use when referencing {name} documentation"
* Updated 3 locations (PDFToSkillConverter, main x2)
- generate_router.py:
* Updated 2 locations with improved router descriptions
* "Use when working with {name} development and programming"
All changes:
- Only apply to NEW skill generations (don't modify existing)
- No API calls (free/offline)
- Smart extraction when metadata/README available
- Improved "Use when..." fallbacks instead of generic templates
- 612 tests passing (100%)
Fixes#191
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixes#209 - UnicodeDecodeError on Windows with non-ASCII characters
**Problem:**
Windows users with non-English locales (Chinese, Japanese, Korean, etc.)
experienced GBK/SHIFT-JIS codec errors when the system default encoding
is not UTF-8.
Error: 'gbk' codec can't decode byte 0xac in position 206: illegal
multibyte sequence
**Root Cause:**
File operations using open() without explicit encoding parameter use
the system default encoding, which on Windows Chinese edition is GBK.
JSON files contain UTF-8 encoded characters that fail to decode with GBK.
**Solution:**
Added encoding='utf-8' to ALL file operations across:
- doc_scraper.py (4 instances):
* load_config() - line 1310
* check_existing_data() - line 1416
* save_checkpoint() - line 173
* load_checkpoint() - line 186
- github_scraper.py (1 instance):
* main() config loading - line 922
- unified_scraper.py (10 instances):
* All JSON read/write operations - lines 134, 153, 205, 239, 275,
278, 325, 328, 342, 364
**Test Results:**
- ✅ All 612 tests passing (100% pass rate)
- ✅ Backward compatible (UTF-8 is standard on Linux/macOS)
- ✅ Fixes Windows locale issues
**Impact:**
- ✅ Works on ALL Windows locales (Chinese, Japanese, Korean, etc.)
- ✅ Maintains compatibility with Linux/macOS
- ✅ Prevents future encoding issues
**Thanks to:** @my5icol for the detailed bug report and fix suggestion!
Fixes#214 - Local enhancement now handles large skills automatically
**Problem:**
- Claude CLI has undocumented ~30-40K character limit
- Large skills (>30K chars) fail silently during local enhancement
- Users experience "Claude finished but SKILL.md was not updated" error
**Solution:**
- Auto-detect large skills (>30K chars)
- Apply intelligent summarization to reduce content size
- Preserve critical content:
* First 20% (introduction/overview)
* Up to 5 best code blocks
* Up to 10 section headings with context
- Target ~30% of original size
- Show clear warnings when summarization is applied
**Implementation:**
- Added `summarize_reference()` method to LocalSkillEnhancer
- Modified `create_enhancement_prompt()` to accept summarization parameters
- Updated `run()` method to auto-enable summarization for large skills
- Added comprehensive test suite (6 tests)
**Test Results:**
- ✅ All 612 tests passing (100% pass rate)
- ✅ 6 new smart summarization tests
- ✅ E2E test: 60K skill → 17K prompt (within limits)
- ✅ Code block preservation verified
**User Experience:**
When enhancement is triggered on a large skill:
```
⚠️ LARGE SKILL DETECTED
📊 Reference content: 60,072 characters
💡 Claude CLI limit: ~30,000-40,000 characters
🔧 Applying smart summarization to ensure success...
• Keeping introductions and overviews
• Extracting best code examples
• Preserving key concepts and headings
• Target: ~30% of original size
✓ Reduced from 60,072 to 15,685 chars (26%)
✓ Prompt created and optimized (17,804 characters)
✓ Ready for Claude CLI (within safe limits)
```
**Backward Compatibility:**
- No breaking changes
- Works with existing skills
- Falls back gracefully for normal-sized skills
Add automatic skill installation to 10+ AI coding agents with a single command.
New Features:
- New install-agent command for installing skills to any AI agent
- Support for 10+ agents: Claude Code, Cursor, VS Code, Amp, Goose, OpenCode, Letta, Aide, Windsurf
- Smart path resolution (global ~/.agent vs project-relative .agent/)
- Fuzzy agent name matching with suggestions
- --agent all flag to install to all agents at once
- --force flag to overwrite existing installations
- --dry-run flag to preview installations
- Comprehensive error handling and user feedback
Implementation:
- Created install_agent.py (379 lines) with core installation logic
- Updated main.py with install-agent subcommand
- Updated pyproject.toml with entry point
- Added 32 comprehensive tests (all passing, 603 total)
- No regressions in existing functionality
Documentation:
- Updated README.md with multi-agent installation guide
- Updated CLAUDE.md with install-agent examples
- Updated CHANGELOG.md with v2.3.0 release notes
- Added agent compatibility table
Technical Details:
- 100% own implementation (no external dependencies)
- Pure Python using stdlib (shutil, pathlib, argparse)
- Compatible with Agent Skills open standard (agentskills.io)
- Works offline
Closes#210🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Created LanguageDetector class supporting 20+ programming languages
- Confidence-based detection with customizable thresholds (min_confidence parameter)
- Replaces duplicate language detection code in doc_scraper and pdf_extractor
- Comprehensive test suite with 100+ test cases
Changes:
- NEW: src/skill_seekers/cli/language_detector.py (17 KB)
- Unified detector with pattern matching for 20+ languages
- Confidence scoring (0.0-1.0 scale)
- Supports: Python, JavaScript, TypeScript, Java, C++, C#, Go, Rust, PHP, Ruby, Swift, Kotlin, Shell, SQL, HTML, CSS, JSON, YAML, XML, and more
- NEW: tests/test_language_detector.py (20 KB)
- 100+ test cases covering all supported languages
- Edge case testing (mixed code, low confidence, etc.)
- MODIFIED: src/skill_seekers/cli/doc_scraper.py
- Removed 80+ lines of duplicate detection code
- Now uses shared LanguageDetector instance
- MODIFIED: src/skill_seekers/cli/pdf_extractor_poc.py
- Removed 130+ lines of duplicate detection code
- Now uses shared LanguageDetector instance
- MODIFIED: tests/test_pdf_extractor.py
- Fixed imports to use proper package paths
- Added manual detector initialization in test setup
Benefits:
- DRY: Single source of truth for language detection
- Maintainability: Add new languages in one place
- Consistency: Same detection logic across all scrapers
- Testability: Comprehensive test coverage
- Extensibility: Easy to add new languages or improve patterns
Addresses technical debt from having duplicate detection logic in multiple files.
Add retry_with_backoff() and retry_with_backoff_async() for network operations.
Features:
- Configurable max attempts (default: 3)
- Exponential backoff with configurable base delay
- Operation name for meaningful log messages
- Both sync and async versions
Addresses E2.6: Add retry logic for network failures
Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>
This commit fixes three critical limitations discovered during local repository skill extraction testing:
**Fix 1: Code Analyzer Import Issue**
- Changed unified_scraper.py to use absolute imports instead of relative imports
- Fixed: `from github_scraper import` → `from skill_seekers.cli.github_scraper import`
- Fixed: `from pdf_scraper import` → `from skill_seekers.cli.pdf_scraper import`
- Result: CodeAnalyzer now available during extraction, deep analysis works
**Fix 2: Unity Library Exclusions**
- Updated should_exclude_dir() to accept and check full directory paths
- Updated _extract_file_tree_local() to pass both dir name and full path
- Added exclusion config passing from unified_scraper to github_scraper
- Result: exclude_dirs_additional now works (297 files excluded in test)
**Fix 3: AI Enhancement for Single Sources**
- Changed read_reference_files() to use rglob() for recursive search
- Now finds reference files in subdirectories (e.g., references/github/README.md)
- Result: AI enhancement works with unified skills that have nested references
**Test Results:**
- Code Analyzer: ✅ Working (deep analysis running)
- Unity Exclusions: ✅ Working (297 files excluded from 679)
- AI Enhancement: ✅ Working (finds and reads nested references)
**Files Changed:**
- src/skill_seekers/cli/unified_scraper.py (Fix 1 & 2)
- src/skill_seekers/cli/github_scraper.py (Fix 2)
- src/skill_seekers/cli/utils.py (Fix 3)
**Test Artifacts:**
- configs/deck_deck_go_local.json (test configuration)
- docs/LOCAL_REPO_TEST_RESULTS.md (comprehensive test report)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Issue: #11 (A1.3 test failures)
## Problem
3/8 tests were failing because ConfigValidator only validates structure
and required fields, NOT format validation (names, URLs, etc.).
## Root Cause
ConfigValidator checks:
- Required fields (name, description, sources/base_url)
- Source types validity
- Field types (arrays, integers)
ConfigValidator does NOT check:
- Name format (alphanumeric, hyphens, underscores)
- URL format (http:// or https://)
## Solution
Added additional format validation in submit_config_tool after ConfigValidator:
1. Name format validation using regex: `^[a-zA-Z0-9_-]+$`
2. URL format validation (must start with http:// or https://)
3. Validates both legacy (base_url) and unified (sources.base_url) formats
## Test Results
Before: 5/8 tests passing, 3 failing
After: 8/8 tests passing ✅
Full suite: 427 tests passing, 40 skipped ✅
## Changes Made
- src/skill_seekers/mcp/server.py:
* Added `import re` at top of file
* Added name format validation (line 1280-1281)
* Added URL format validation for legacy configs (line 1285-1289)
* Added URL format validation for unified configs (line 1291-1296)
- tests/test_mcp_server.py:
* Updated test_submit_config_validates_required_fields to accept
ConfigValidator's correct error message ("cannot detect" instead of "description")
## Validation Examples
Invalid name: "React@2024!" → ❌ "Invalid name format"
Invalid URL: "not-a-url" → ❌ "Invalid base_url format"
Valid name: "react-docs" → ✅
Valid URL: "https://react.dev/" → ✅🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements A1.2 - Add MCP tool to download configs from API
Features:
- Download config files from api.skillseekersweb.com
- List all available configs (24 configs)
- Filter configs by category
- Download specific config by name
- Save to local configs directory
- Display config metadata (category, tags, type, source, last_updated)
- Error handling for 404 and network errors
Usage:
- List configs: fetch_config with list_available=true
- Filter by category: fetch_config with list_available=true, category='web-frameworks'
- Download config: fetch_config with config_name='react'
- Custom destination: fetch_config with config_name='react', destination='my_configs/'
Technical:
- Uses httpx AsyncClient for HTTP requests
- Connects to https://api.skillseekersweb.com
- Returns formatted TextContent responses
- Supports GET /api/configs and GET /api/download endpoints
- Proper error handling for HTTP and JSON errors
Tests:
- ✅ List all configs (24 total)
- ✅ List by category filter (12 web-frameworks)
- ✅ Download specific config (react.json)
- ✅ Handle nonexistent config (404 error)
Issue: N/A (from roadmap task A1.2)
Major improvements:
- Configurable directory exclusions (Issue #203)
- Unlimited local repository analysis
- Skip llms.txt option (PR #198)
- 10+ bug fixes for GitHub scraper
- Test suite expanded to 427 tests
See CHANGELOG.md for full details.
Merges feat/add-skip-llm-to-config by @sogoiii.
This PR adds a valuable configuration option to explicitly skip llms.txt
detection, useful when a site's llms.txt is incomplete, incorrect, or when
specific HTML scraping is needed.
Key features:
- New 'skip_llms_txt' config option (default: false, backward compatible)
- Boolean type validation with warning for invalid values
- Support in both sync and async scraping modes
- 17 comprehensive tests (15 feature tests + 2 config validation tests)
All tests passing after fixing import paths to use proper package names.
Test results: ✅ 17/17 tests passing
Full test suite: ✅ 391 tests passing
Co-authored-by: sogoiii <sogoiii@users.noreply.github.com>
Fixes Issue #190 - "name 'logger' is not defined" error
**Problem:**
- Logger was used at line 40 (in code_analyzer import exception)
- Logger was defined at line 47
- Caused runtime error when code_analyzer import failed
**Solution:**
- Moved logging.basicConfig() and logger initialization to lines 34-39
- Now logger is defined BEFORE the code_analyzer import block
- Warning message now works correctly when code_analyzer is missing
**Testing:**
- ✅ All 22 GitHub scraper tests pass
- ✅ Logger warning appears correctly when code_analyzer missing
- ✅ No similar issues found in other CLI files
Closes#190
Fixes#193 - PDF scraping broken for PyPI users
Changed 3 files from absolute to relative imports to fix
ModuleNotFoundError when package is installed via pip:
1. pdf_scraper.py:22
- from pdf_extractor_poc import → from .pdf_extractor_poc import
- Fixes: skill-seekers pdf command failed with import error
2. github_scraper.py:36
- from code_analyzer import → from .code_analyzer import
- Proactive fix: prevents future import errors
3. test_unified_simple.py:17
- from config_validator import → from .config_validator import
- Proactive fix: test helper file
These absolute imports worked locally due to sys.path differences
but failed when installed via PyPI (pip install skill-seekers).
Tested with:
- skill-seekers pdf command now works ✅
- Extracted 32-page Godot Farming PDF successfully
All CLI commands should now work correctly when installed from PyPI.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add skip_llms_txt config option (default: False)
- Validate value is boolean, warn and default to False if not
- Support in both sync and async scraping modes
- Add 17 tests for config, behavior, and edge cases
The MCP server's package_skill_tool was failing in CI because the quality
checker was prompting for user input, which doesn't exist in CI/MCP contexts.
Fix:
- Add --skip-quality-check flag to package_skill command in MCP server
- This prevents interactive prompts that cause EOFError in CI
- MCP tools should skip interactive checks since they run in background
Impact:
- All 25 MCP server tests now pass
- All 391 tests passing
- CI builds will succeed
Context:
- Quality checks are interactive by default for CLI users
- MCP server runs commands programmatically without user input
- This is the correct behavior: interactive for CLI, non-interactive for MCP
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 & 3: Quality assurance before packaging
New module: quality_checker.py
- Enhancement verification (checks for template text, code examples, sections)
- Structure validation (SKILL.md, references/ directory)
- Content quality checks (YAML frontmatter, language tags, "When to Use" section)
- Link validation (internal markdown links)
- Quality scoring system (0-100 score + A-F grade)
- Detailed reporting with errors, warnings, and info messages
- CLI with --verbose and --strict modes
Integration in package_skill.py:
- Automatic quality checks before packaging
- Display quality report with score and grade
- Ask user to confirm if warnings/errors found
- Add --skip-quality-check flag to bypass checks
- Updated help examples
Benefits:
- Catch quality issues before packaging
- Ensure SKILL.md is properly enhanced
- Validate all links work
- Give users confidence in skill quality
- Comprehensive quality reports
Addresses user request: "check some sort of quality check at the end
like is links working, skill is good etc and give report the user"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 1: Fix race condition where main console exits before enhancement completes
Changes to enhance_skill_local.py:
- Add headless mode (default) using subprocess.run() which WAITS for completion
- Add timeout protection (default 10 minutes, configurable)
- Verify SKILL.md was actually updated (check mtime and size)
- Add --interactive-enhancement flag to use old terminal mode
- Detailed progress messages and error handling
- Clean up temp files after completion
Changes to doc_scraper.py:
- Use skill-seekers-enhance entry point instead of direct python path
- Pass --interactive-enhancement flag through if requested
- Update help text to reflect new headless default behavior
- Show proper status messages (HEADLESS vs INTERACTIVE)
Benefits:
- Main console now waits for enhancement to complete
- No more "Package your skill" message while enhancement is running
- Timeout prevents infinite hangs
- Terminal mode still available for users who want it
- Better error messages and progress tracking
Fixes user request: "make sure 1. console wait for it to finish"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes#182
Changes:
- Add 'anthropic-beta: skills-2025-10-02' header (required by Anthropic Skills API)
- Change multipart field name from 'skill' to 'files[]' (correct API format)
Without these fixes, all upload attempts returned 404 errors.
Verified:
- All 379 tests passing (100%)
- No regressions in test suite
- Upload functionality corrected per API requirements
Co-authored-by: Straughter "BatmanOsama" Guthrie <straughterguthrie@gmail.com>
Original PR: #183
