firefrost-gaming/skill-seekers-reference
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
344 Commits 1 Branch 0 Tags
6439c85cdec87bcf93cf3c9d2cfd1d2ff18f2a5a
Commit Graph

30 Commits

Author SHA1 Message Date
yusyus
c89f059712 feat(v2.7.0): Smart Rate Limit Management & Multi-Token Configuration 
Major Features:
- Multi-profile GitHub token system with secure storage
- Smart rate limit handler with 4 strategies (prompt/wait/switch/fail)
- Interactive configuration wizard with browser integration
- Configurable timeout (default 30 min) per profile
- Automatic profile switching on rate limits
- Live countdown timers with real-time progress
- Non-interactive mode for CI/CD (--non-interactive flag)
- Progress tracking and resume capability (skeleton)
- Comprehensive test suite (16 tests, all passing)

Solves:
- Indefinite waiting on GitHub rate limits
- Confusing GitHub token setup

Files Added:
- src/skill_seekers/cli/config_manager.py (~490 lines)
- src/skill_seekers/cli/config_command.py (~400 lines)
- src/skill_seekers/cli/rate_limit_handler.py (~450 lines)
- src/skill_seekers/cli/resume_command.py (~150 lines)
- tests/test_rate_limit_handler.py (16 tests)

Files Modified:
- src/skill_seekers/cli/github_fetcher.py (rate limit integration)
- src/skill_seekers/cli/github_scraper.py (--non-interactive, --profile flags)
- src/skill_seekers/cli/main.py (config, resume subcommands)
- pyproject.toml (version 2.7.0)
- CHANGELOG.md, README.md, CLAUDE.md (documentation)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-17 18:38:31 +03:00
yusyus
2019a02b51 docs: Update CLAUDE.md to v2.6.0 with complete C3.x suite 
Updates:
- Version: v2.5.2 → v2.6.0
- Added complete C3.x feature documentation (C3.1-C3.8)
- Updated Recent Achievements section with v2.6.0 release info
- Expanded C3.x descriptions with all 8 features
- Documented C3.8 Standalone Codebase Scraper

C3.x Suite Now Complete:
- C3.1: Design pattern detection (10 GoF patterns, 9 languages, 87% precision)
- C3.2: Test example extraction (5 categories, AST-based)
- C3.3: How-to guide generation with AI enhancement
- C3.4: Configuration pattern extraction
- C3.5: Architectural overview & router skill generation
- C3.6: AI enhancement for patterns and tests (Claude API integration)
- C3.7: Architectural pattern detection (8 patterns, framework-aware)
- C3.8: Standalone codebase scraper (300+ line SKILL.md from code alone)

Release History Updated:
- v2.6.0 (Latest - January 14, 2026) - C3.x suite complete
- v2.5.2 - UX improvements (opt-out flags)
- v2.5.0 - Multi-platform support
- v2.1.0 - Unified multi-source scraping
- v1.0.0 - Production release

Benefits:
- Accurate version information for Claude Code
- Complete C3.x feature documentation
- Clear release history
- Better developer onboarding

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-14 22:52:35 +03:00
yusyus
48b8544dea docs: Consolidate roadmaps and refactor documentation structure 
MAJOR REFACTORING: Merge 3 roadmap files into single comprehensive ROADMAP.md

Changes:
- Merged ROADMAP.md + FLEXIBLE_ROADMAP.md + FUTURE_RELEASES.md → ROADMAP.md
- Consolidated 1,008 lines across 3 files into 429 lines (single source of truth)
- Removed duplicate/overlapping content
- Cleaned up docs archive structure

New ROADMAP.md Structure:
- Current Status (v2.6.0)
- Development Philosophy (task-based approach)
- Task-Based Roadmap (136 tasks, 10 categories)
- Release History (v1.0.0, v2.1.0, v2.6.0)
- Release Planning (v2.7-v2.9)
- Long-term Vision (v3.0+)
- Metrics & Goals
- Contribution guidelines

Deleted Files:
- FLEXIBLE_ROADMAP.md (merged into ROADMAP.md)
- FUTURE_RELEASES.md (merged into ROADMAP.md)
- docs/archive/temp/TERMINAL_SELECTION.md (temporary file)
- docs/archive/temp/TESTING.md (temporary file)

Moved Files:
- docs/plans/*.md → docs/archive/plans/ (dated planning docs)

Updated References:
- CLAUDE.md: FLEXIBLE_ROADMAP.md → ROADMAP.md
- docs/README.md: Removed duplicate roadmap references
- CHANGELOG.md: Updated documentation references

Benefits:
- Single source of truth for roadmap
- No duplicate maintenance
- Cleaner repository structure
- Better discoverability
- Historical context preserved in archive/

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-14 22:36:03 +03:00
yusyus
a99e22c639 feat: Multi-Source Synthesis Architecture - Rich Standalone Skills + Smart Combination 
BREAKING CHANGE: Major architectural improvements to multi-source skill generation

This commit implements the complete "Multi-Source Synthesis Architecture" where
each source (documentation, GitHub, PDF) generates a rich standalone SKILL.md
file before being intelligently synthesized with source-specific formulas.

## 🎯 Core Architecture Changes

### 1. Rich Standalone SKILL.md Generation (Source Parity)

Each source now generates comprehensive, production-quality SKILL.md files that
can stand alone OR be synthesized with other sources.

**GitHub Scraper Enhancements** (+263 lines):
- Now generates 300+ line SKILL.md (was ~50 lines)
- Integrates C3.x codebase analysis data:
  - C2.5: API Reference extraction
  - C3.1: Design pattern detection (27 high-confidence patterns)
  - C3.2: Test example extraction (215 examples)
  - C3.7: Architectural pattern analysis
- Enhanced sections:
  - ⚡ Quick Reference with pattern summaries
  - 📝 Code Examples from real repository tests
  - 🔧 API Reference from codebase analysis
  - 🏗️ Architecture Overview with design patterns
  - ⚠️ Known Issues from GitHub issues
- Location: src/skill_seekers/cli/github_scraper.py

**PDF Scraper Enhancements** (+205 lines):
- Now generates 200+ line SKILL.md (was ~50 lines)
- Enhanced content extraction:
  - 📖 Chapter Overview (PDF structure breakdown)
  - 🔑 Key Concepts (extracted from headings)
  - ⚡ Quick Reference (pattern extraction)
  - 📝 Code Examples: Top 15 (was top 5), grouped by language
  - Quality scoring and intelligent truncation
- Better formatting and organization
- Location: src/skill_seekers/cli/pdf_scraper.py

**Result**: All 3 sources (docs, GitHub, PDF) now have equal capability to
generate rich, comprehensive standalone skills.

### 2. File Organization & Caching System

**Problem**: output/ directory cluttered with intermediate files, data, and logs.

**Solution**: New `.skillseeker-cache/` hidden directory for all intermediate files.

**New Structure**:
```
.skillseeker-cache/{skill_name}/
├── sources/          # Standalone SKILL.md from each source
│   ├── httpx_docs/
│   ├── httpx_github/
│   └── httpx_pdf/
├── data/             # Raw scraped data (JSON)
├── repos/            # Cloned GitHub repositories (cached for reuse)
└── logs/             # Session logs with timestamps

output/{skill_name}/  # CLEAN: Only final synthesized skill
├── SKILL.md
└── references/
```

**Benefits**:
- ✅ Clean output/ directory (only final product)
- ✅ Intermediate files preserved for debugging
- ✅ Repository clones cached and reused (faster re-runs)
- ✅ Timestamped logs for each scraping session
- ✅ All cache dirs added to .gitignore

**Changes**:
- .gitignore: Added `.skillseeker-cache/` entry
- unified_scraper.py: Complete reorganization (+238 lines)
  - Added cache directory structure
  - File logging with timestamps
  - Repository cloning with caching/reuse
  - Cleaner intermediate file management
  - Better subprocess logging and error handling

### 3. Config Repository Migration

**Moved to separate config repository**: https://github.com/yusufkaraaslan/skill-seekers-configs

**Deleted from this repo** (35 config files):
- ansible-core.json, astro.json, claude-code.json
- django.json, django_unified.json, fastapi.json, fastapi_unified.json
- godot.json, godot_unified.json, godot_github.json, godot-large-example.json
- react.json, react_unified.json, react_github.json, react_github_example.json
- vue.json, kubernetes.json, laravel.json, tailwind.json, hono.json
- svelte_cli_unified.json, steam-economy-complete.json
- deck_deck_go_local.json, python-tutorial-test.json, example_pdf.json
- test-manual.json, fastapi_unified_test.json, fastmcp_github_example.json
- example-team/ directory (4 files)

**Kept as reference example**:
- configs/httpx_comprehensive.json (complete multi-source example)

**Rationale**:
- Cleaner repository (979+ lines added, 1680 deleted)
- Configs managed separately with versioning
- Official presets available via `fetch-config` command
- Users can maintain private config repos

### 4. AI Enhancement Improvements

**enhance_skill.py** (+125 lines):
- Better integration with multi-source synthesis
- Enhanced prompt generation for synthesized skills
- Improved error handling and logging
- Support for source metadata in enhancement

### 5. Documentation Updates

**CLAUDE.md** (+252 lines):
- Comprehensive project documentation
- Architecture explanations
- Development workflow guidelines
- Testing requirements
- Multi-source synthesis patterns

**SKILL_QUALITY_ANALYSIS.md** (new):
- Quality assessment framework
- Before/after analysis of httpx skill
- Grading rubric for skill quality
- Metrics and benchmarks

### 6. Testing & Validation Scripts

**test_httpx_skill.sh** (new):
- Complete httpx skill generation test
- Multi-source synthesis validation
- Quality metrics verification

**test_httpx_quick.sh** (new):
- Quick validation script
- Subset of features for rapid testing

## 📊 Quality Improvements

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| GitHub SKILL.md lines | ~50 | 300+ | +500% |
| PDF SKILL.md lines | ~50 | 200+ | +300% |
| GitHub C3.x integration | ❌ No | ✅ Yes | New feature |
| PDF pattern extraction | ❌ No | ✅ Yes | New feature |
| File organization | Messy | Clean cache | Major improvement |
| Repository cloning | Always fresh | Cached reuse | Faster re-runs |
| Logging | Console only | Timestamped files | Better debugging |
| Config management | In-repo | Separate repo | Cleaner separation |

## 🧪 Testing

All existing tests pass:
- test_c3_integration.py: Updated for new architecture
- 700+ tests passing
- Multi-source synthesis validated with httpx example

## 🔧 Technical Details

**Modified Core Files**:
1. src/skill_seekers/cli/github_scraper.py (+263 lines)
   - _generate_skill_md(): Rich content with C3.x integration
   - _format_pattern_summary(): Design pattern summaries
   - _format_code_examples(): Test example formatting
   - _format_api_reference(): API reference from codebase
   - _format_architecture(): Architectural pattern analysis

2. src/skill_seekers/cli/pdf_scraper.py (+205 lines)
   - _generate_skill_md(): Enhanced with rich content
   - _format_key_concepts(): Extract concepts from headings
   - _format_patterns_from_content(): Pattern extraction
   - Code examples: Top 15, grouped by language, better quality scoring

3. src/skill_seekers/cli/unified_scraper.py (+238 lines)
   - __init__(): Cache directory structure
   - _setup_logging(): File logging with timestamps
   - _clone_github_repo(): Repository caching system
   - _scrape_documentation(): Move to cache, better logging
   - Better subprocess handling and error reporting

4. src/skill_seekers/cli/enhance_skill.py (+125 lines)
   - Multi-source synthesis awareness
   - Enhanced prompt generation
   - Better error handling

**Minor Updates**:
- src/skill_seekers/cli/codebase_scraper.py (+3 lines): Minor improvements
- src/skill_seekers/cli/test_example_extractor.py: Quality scoring adjustments
- tests/test_c3_integration.py: Test updates for new architecture

## 🚀 Migration Guide

**For users with existing configs**:
No action required - all existing configs continue to work.

**For users wanting official presets**:
```bash
# Fetch from official config repo
skill-seekers fetch-config --name react --target unified

# Or use existing local configs
skill-seekers unified --config configs/httpx_comprehensive.json
```

**Cache directory**:
New `.skillseeker-cache/` directory will be created automatically.
Safe to delete - will be regenerated on next run.

## 📈 Next Steps

This architecture enables:
- ✅ Source parity: All sources generate rich standalone skills
- ✅ Smart synthesis: Each combination has optimal formula
- ✅ Better debugging: Cached files and logs preserved
- ✅ Faster iteration: Repository caching, clean output
- 🔄 Future: Multi-platform enhancement (Gemini, GPT-4) - planned
- 🔄 Future: Conflict detection between sources - planned
- 🔄 Future: Source prioritization rules - planned

## 🎓 Example: httpx Skill Quality

**Before**: 186 lines, basic synthesis, missing data
**After**: 640 lines with AI enhancement, A- (9/10) quality

**What changed**:
- All C3.x analysis data integrated (patterns, tests, API, architecture)
- GitHub metadata included (stars, topics, languages)
- PDF chapter structure visible
- Professional formatting with emojis and clear sections
- Real-world code examples from test suite
- Design patterns explained with confidence scores
- Known issues with impact assessment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-11 23:01:07 +03:00
yusyus
48370a1963 docs: Update CLAUDE.md with streamlined developer guidance 
- Reduced from 1116 to 526 lines (53% reduction)
- Focused on architecture and testing requirements
- Removed redundant user-facing documentation
- Added critical development notes and workflows

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-01 18:57:29 +03:00
yusyus
5e166c40b9 chore: Bump version to v2.5.1 - Critical PyPI Bug Fix 
Version Updates:
- pyproject.toml: 2.5.0 → 2.5.1
- src/skill_seekers/__init__.py: 2.0.0 → 2.5.1
- src/skill_seekers/cli/__init__.py: 2.0.0 → 2.5.1
- src/skill_seekers/cli/main.py: 2.4.0 → 2.5.1
- src/skill_seekers/mcp/__init__.py: 2.4.0 → 2.5.1
- src/skill_seekers/mcp/tools/__init__.py: 2.4.0 → 2.5.1

CHANGELOG:
- Added v2.5.1 release notes documenting PR #221 fix
- Critical: Fixed missing skill_seekers.cli.adaptors package
- Impact: Restores all multi-platform features for PyPI users

Documentation:
- Updated CLAUDE.md to v2.5.0 with multi-platform details
- Added platform adaptor architecture documentation
- Updated test architecture and environment variables

Related: PR #221 (merged), Issue #222 (py.typed follow-up)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2025-12-30 23:22:30 +03:00
yusyus
72611af87d feat(v2.3.0): Add multi-agent installation support 
Add automatic skill installation to 10+ AI coding agents with a single command.

New Features:
- New install-agent command for installing skills to any AI agent
- Support for 10+ agents: Claude Code, Cursor, VS Code, Amp, Goose, OpenCode, Letta, Aide, Windsurf
- Smart path resolution (global ~/.agent vs project-relative .agent/)
- Fuzzy agent name matching with suggestions
- --agent all flag to install to all agents at once
- --force flag to overwrite existing installations
- --dry-run flag to preview installations
- Comprehensive error handling and user feedback

Implementation:
- Created install_agent.py (379 lines) with core installation logic
- Updated main.py with install-agent subcommand
- Updated pyproject.toml with entry point
- Added 32 comprehensive tests (all passing, 603 total)
- No regressions in existing functionality

Documentation:
- Updated README.md with multi-agent installation guide
- Updated CLAUDE.md with install-agent examples
- Updated CHANGELOG.md with v2.3.0 release notes
- Added agent compatibility table

Technical Details:
- 100% own implementation (no external dependencies)
- Pure Python using stdlib (shutil, pathlib, argparse)
- Compatible with Agent Skills open standard (agentskills.io)
- Works offline

Closes #210

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2025-12-22 02:04:32 +03:00
yusyus
b7cd317efb feat(A1.7): Add install_skill MCP tool for one-command workflow automation 
Implements complete end-to-end skill installation in a single command:
fetch_config → scrape_docs → enhance_skill_local → package_skill → upload_skill

Changes:
- MCP Tool: Added install_skill_tool() to server.py (~300 lines)
  - Input validation (config_name XOR config_path)
  - 5-phase orchestration with error handling
  - Dry-run mode for workflow preview
  - Mandatory AI enhancement (30-60 sec, 3/10→9/10 quality boost)
  - Auto-upload to Claude (if ANTHROPIC_API_KEY set)

- CLI Integration: New install command
  - Created install_skill.py CLI wrapper (~150 lines)
  - Updated main.py with install subcommand
  - Added entry point to pyproject.toml

- Testing: Comprehensive test suite
  - Created test_install_skill.py with 13 tests
  - Tests cover validation, dry-run, orchestration, error handling
  - All tests passing (13/13)

- Documentation: Updated all user-facing docs
  - CLAUDE.md: Added MCP tool (10 tools total) and CLI examples
  - README.md: Added prominent one-command workflow section
  - FLEXIBLE_ROADMAP.md: Marked A1.7 as complete

Features:
- Zero friction: One command instead of 5 separate steps
- Quality guaranteed: Mandatory enhancement ensures 9/10 quality
- Complete automation: From config to uploaded skill
- Intelligent: Auto-detects config type (name vs path)
- Flexible: Dry-run, unlimited, no-upload modes
- Well-tested: 13 unit tests with mocking

Usage:
  skill-seekers install --config react
  skill-seekers install --config configs/custom.json --no-upload
  skill-seekers install --config django --unlimited
  skill-seekers install --config react --dry-run

Closes #204

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2025-12-21 20:17:59 +03:00
yusyus
3c8603e6b7 docs: Update test architecture and CLI details in CLAUDE.md 2025-12-21 14:17:12 +03:00
yusyus
cbacdb0e66 release: v2.1.1 - GitHub Repository Analysis Enhancements 
Major improvements:
- Configurable directory exclusions (Issue #203)
- Unlimited local repository analysis
- Skip llms.txt option (PR #198)
- 10+ bug fixes for GitHub scraper
- Test suite expanded to 427 tests

See CHANGELOG.md for full details.
 2025-11-30 12:22:28 +03:00
yusyus
bd2b201aa5 docs: Update all documentation for v2.1.0 release 
Updates across all major documentation files to reflect v2.1.0 release
status and recent completions.

Changes:
- CLAUDE.md:
  * Updated version from v2.0.0 to v2.1.0
  * Updated date to November 29, 2025
  * Updated test count from 391 to 427
  * Moved completed PRs (#195, #198) and Issue #203 to "Completed" section
  * Updated "Next Up" priorities

- README.md:
  * Updated version badge from 2.0.0 to 2.1.0
  * Updated test badge from 379 to 427 passing

- CHANGELOG.md:
  * Added Issue #203 (Configurable EXCLUDED_DIRS) to Unreleased section
  * Documented 19 comprehensive tests for exclude_dirs feature
  * Listed both extend and replace modes

- FUTURE_RELEASES.md:
  * Marked v2.1.0 as "Released" (November 29, 2025)
  * Moved "Fix 12 unified tests" to completed
  * Updated release schedule table

- FLEXIBLE_ROADMAP.md:
  * Updated current status from v1.0.0 to v2.1.0
  * Added latest release date
  * Expanded "What Works" section with new features
  * Updated test count to 427

All documentation now accurately reflects:
- v2.1.0 release status ✅
- 427 tests passing (up from 391) ✅  - Issue #203 completion ✅
- PR #195 and #198 merged status ✅

Related: #203
 2025-11-30 01:06:21 +03:00
yusyus
ea289cebe1 feat: Make EXCLUDED_DIRS configurable for local repository analysis 
Closes #203

Adds configuration options to customize directory exclusions during local
repository analysis, while maintaining backward compatibility with smart
defaults.

**New Config Options:**

1. `exclude_dirs_additional` - Extend defaults (most common)
   - Adds custom directories to default exclusions
   - Example: ["proprietary", "legacy", "third_party"]
   - Total exclusions = defaults + additional

2. `exclude_dirs` - Replace defaults (advanced users)
   - Completely overrides default exclusions
   - Example: ["node_modules", ".git", "custom_vendor"]
   - Gives full control over exclusions

**Implementation:**

- Modified GitHubScraper.__init__() to parse exclude_dirs config
- Changed should_exclude_dir() to use instance variable instead of global
- Added logging for custom exclusions (INFO for extend, WARNING for replace)
- Maintains backward compatibility (no config = use defaults)

**Testing:**

- Added 12 comprehensive tests in test_excluded_dirs_config.py
  - 3 tests for defaults (backward compatibility)
  - 3 tests for extend mode
  - 3 tests for replace mode
  - 1 test for precedence
  - 2 tests for edge cases
- All 12 new tests passing ✅
- All 22 existing github_scraper tests passing ✅

**Documentation:**

- Updated CLAUDE.md config parameters section
- Added detailed "Configurable Directory Exclusions" feature section
- Included examples for both modes
- Listed common use cases (monorepos, enterprise, legacy codebases)

**Use Cases:**

- Monorepos with custom directory structures
- Enterprise projects with non-standard naming conventions
- Including unusual directories for analysis
- Minimal exclusions for small/simple projects

**Backward Compatibility:**

✅ Fully backward compatible - existing configs work unchanged
✅ Smart defaults maintained when no config provided
✅ All existing tests pass

Co-authored-by: jimmy058910 <jimmy058910@users.noreply.github.com>
 2025-11-29 23:53:27 +03:00
yusyus
bd20b32470 Merge PR #198: Skip llms.txt Config Option 
Merges feat/add-skip-llm-to-config by @sogoiii.

This PR adds a valuable configuration option to explicitly skip llms.txt
detection, useful when a site's llms.txt is incomplete, incorrect, or when
specific HTML scraping is needed.

Key features:
- New 'skip_llms_txt' config option (default: false, backward compatible)
- Boolean type validation with warning for invalid values
- Support in both sync and async scraping modes
- 17 comprehensive tests (15 feature tests + 2 config validation tests)

All tests passing after fixing import paths to use proper package names.

Test results: ✅ 17/17 tests passing
Full test suite: ✅ 391 tests passing

Co-authored-by: sogoiii <sogoiii@users.noreply.github.com>
 2025-11-29 22:56:46 +03:00
yusyus
cf77f9e392 docs: Update test status - all 391 tests passing including unified tests 
All unified scraping tests are now passing! Updated documentation to reflect current status.

**Changes:**

1. **CLAUDE.md** - Updated test status throughout
   - Changed "⚠️ 12 unified tests need fixes" to "✅ All 22 unified tests passing"
   - Updated test count from 379 to 391 tests
   - Marked unified configs as ✅ (all 5 working and tested)
   - Updated "Next Up" section with completed items
   - Updated last verification date to Nov 29, 2025

2. **README.md** - Updated test count
   - Changed "379 tests" to "391 tests"

3. **docs/CLAUDE.md** - Updated test documentation
   - Updated test counts throughout
   - Removed outdated warnings about failing tests

**Test Status:**
- ✅ tests/test_unified.py: 18/18 passing
- ✅ tests/test_unified_mcp_integration.py: 4/4 passing
- ✅ Total: 391 tests passing, 32 skipped

**Unified Scraping:**
- All 5 unified configs verified and working
- Conflict detection fully tested
- Rule-based and AI merge modes tested
- Feature is production-ready

Task 2.2 Complete - No code changes needed, tests were already passing!
 2025-11-29 22:20:43 +03:00
sogoiii
91692db87c 📝 docs: add skip_llms_txt to config parameters documentation 2025-11-20 14:00:55 -08:00
yusyus
5ee07a2181 docs: Update CLAUDE.md for v2.0.0 PyPI release 
Major updates for v2.0.0:
- Added PyPI publication status and installation instructions
- Updated to reflect modern Python packaging (src/ layout, pyproject.toml)
- Updated all commands to use 'skill-seekers' CLI instead of python3 cli/*
- Updated file structure section for src/ layout
- Updated key code locations with new paths
- Added FUTURE_RELEASES.md to documentation list
- Updated test count (379 passing, all CI checks green)
- Updated date to November 11, 2025
- Added development workflow section
- Reorganized Additional Documentation into categories

All sections now reflect the post-PyPI publication state of the project.
 2025-11-11 23:27:48 +03:00
yusyus
693294be8e docs: Update CLAUDE.md with new unified CLI commands 
Updated all command examples to use new entry points:
- skill-seekers scrape (was: python3 cli/doc_scraper.py)
- skill-seekers unified (was: python3 cli/unified_scraper.py)
- skill-seekers estimate (was: python3 cli/estimate_pages.py)
- skill-seekers package (was: python3 cli/package_skill.py)
- skill-seekers enhance (was: python3 cli/enhance_skill_local.py)
- skill-seekers upload (was: python3 cli/upload_skill.py)

All 44+ command examples now use modern entry point syntax.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-07 01:25:40 +03:00
yusyus
13b19c2b06 Update CLAUDE.md with current project status 
- Update date from October 26 to November 6, 2025
- Update test count: 390 tests total, 378 passing, 12 unified tests failing
- Update configs inventory: 24 total configs (14 single-source, 5 unified, 5 test)
- Add priority task: Fix 12 failing unified tests
- Update status: Core functionality stable, unified tests need attention
- Add detailed config breakdown by category
- Update available configs section with complete categorization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-06 23:23:12 +03:00
yusyus
1e277f80d2 Update documentation for unified multi-source scraping (v2.0.0) 
Major documentation update explaining the new unified scraping system that combines documentation + GitHub + PDF sources in a single skill with automatic conflict detection.

## Changes:

**README.md:**
- Update version badge to v2.0.0
- Add "Unified Multi-Source Scraping" to Key Features section
- Add comprehensive Option 5 section showing:
  - Problem statement (documentation drift)
  - Solution with code example
  - Conflict detection types and severity levels
  - Transparent reporting with side-by-side comparison
  - List of advantages (identifies gaps, catches changes, single source of truth)
  - Available unified configs
  - Link to full guide (docs/UNIFIED_SCRAPING.md)

**CLAUDE.md:**
- Update Current Status to v2.0.0
- Add "Major Release: Unified Multi-Source Scraping" in Recent Updates
- Update configs count from 11/11 to 15/15 (added 4 unified configs)
- Add new "Unified Multi-Source Scraping" section under Core Commands
- Include command examples and feature highlights
- Explain what makes unified scraping special

**QUICKSTART.md:**
- Add Option D: Unified Multi-Source to Step 2
- Add unified configs to Available Presets section
- Show react_unified, django_unified, fastapi_unified, godot_unified examples

## Value:
This documentation update explains how unified scraping helps developers:
- Mix documentation + code in one skill
- Automatically detect conflicts (missing_in_docs, missing_in_code, signature_mismatch)
- Get transparent side-by-side comparisons with ⚠️ warnings
- Identify documentation gaps and outdated docs
- Create a single source of truth combining both sources

Related to: Phase 7-11 unified scraper implementation (commit 5d8c7e3)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-26 16:41:58 +03:00
yusyus
319331f5a6 feat: Complete refactoring with async support, type safety, and package structure 
This comprehensive refactoring improves code quality, performance, and maintainability
while maintaining 100% backwards compatibility.

## Major Features Added

### 🚀 Async/Await Support (2-3x Performance Boost)
- Added `--async` flag for parallel scraping using asyncio
- Implemented `scrape_page_async()` with httpx.AsyncClient
- Implemented `scrape_all_async()` with asyncio.gather()
- Connection pooling for better resource management
- Performance: 18 pg/s → 55 pg/s (3x faster)
- Memory: 120 MB → 40 MB (66% reduction)
- Full documentation in ASYNC_SUPPORT.md

### 📦 Python Package Structure (Phase 0 Complete)
- Created cli/__init__.py for clean imports
- Created skill_seeker_mcp/__init__.py (renamed from mcp/)
- Created skill_seeker_mcp/tools/__init__.py
- Proper package imports: `from cli import constants`
- Better IDE support and autocomplete

### ⚙️ Centralized Configuration
- Created cli/constants.py with 18 configuration constants
- DEFAULT_ASYNC_MODE, DEFAULT_RATE_LIMIT, DEFAULT_MAX_PAGES
- Enhancement limits, categorization scores, file limits
- All magic numbers now centralized and configurable

### 🔧 Code Quality Improvements
- Converted 71 print() statements to proper logging
- Added type hints to all DocToSkillConverter methods
- Fixed all mypy type checking issues
- Installed types-requests for better type safety
- Code quality: 5.5/10 → 6.5/10

## Testing
- Test count: 207 → 299 tests (92 new tests)
- 11 comprehensive async tests (all passing)
- 16 constants tests (all passing)
- Fixed test isolation issues
- 100% pass rate maintained (299/299 passing)

## Documentation
- Updated README.md with async examples and test count
- Updated CLAUDE.md with async usage guide
- Created ASYNC_SUPPORT.md (292 lines)
- Updated CHANGELOG.md with all changes
- Cleaned up temporary refactoring documents

## Cleanup
- Removed temporary planning/status documents
- Moved test_pr144_concerns.py to tests/ folder
- Updated .gitignore for test artifacts
- Better repository organization

## Breaking Changes
None - all changes are backwards compatible.
Async mode is opt-in via --async flag.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-26 13:05:39 +03:00
Preston Brown
de5344caf9 Add virtual environment setup and minimal dependencies (#149) 
## Changes
- Add virtual environment setup instructions to all docs
- Create requirements.txt with minimal dependencies (13 packages)
- Make anthropic optional (only needed for API enhancement)
- Clarify path notation (~ = $HOME, /Users/yourname examples)
- Add venv activation reminders throughout documentation

## Files Changed
- README.md: Added venv setup section to CLI method
- BULLETPROOF_QUICKSTART.md: Replaced Step 4 with venv setup
- CLAUDE.md: Updated Prerequisites with venv instructions
- requirements.txt: Created with minimal deps (requests, beautifulsoup4, pytest)

## Why
- Prevents package conflicts and permission issues
- Standard Python development practice
- Enables proper pytest usage without pipx complications
- Makes setup clearer for beginners
 2025-10-22 21:54:05 +03:00
yusyus
ff148cf98f Update documentation for new Ansible config 
Added ansible-core.json config to available presets list in:
- README.md: Added to preset table and usage examples
- CLAUDE.md: Added to production configs list with details

Changes:
- Total configs: 11 → 12
- New category: DevOps & Automation
- Reorganized config list for better categorization

Related: PR #147
 2025-10-22 21:51:45 +03:00
yusyus
831ea67d58 Update task tracking and CLAUDE.md with latest progress 
Documentation Updates:
======================

TODO.md:
--------
✅ Added "Completed This Week" section:
   - H1.1: Issue #8 fixed (bulletproof docs + MCP setup)
   - H1.2: Issue #7 fixed (11/11 configs working)
   - H1.4: Issue #4 linked to roadmap
   - PR #5: Reviewed and approved

✅ Updated "Immediate Tasks" list:
   - Removed completed tasks
   - Added H1.3 (example project) as next priority

✅ Updated Progress Tracking:
   - 10 items completed this week
   - Clear visibility of accomplishments
   - Next steps clearly defined

NEXT_TASKS.md:
--------------
✅ Marked completed tasks in Starter Pack:
   - H1.1 (Issue #8) - DONE
   - H1.2 (Issue #7) - DONE
   - H1.4 (Issue #4) - DONE
   - PR #5 Review - DONE

✅ Updated Current Sprint (Oct 20-27):
   - Monday/Tuesday: 4/4 tasks completed ✅
   - Wednesday/Thursday: 3 tasks remaining
   - Progress: 4/10 tasks (40%)

✅ Added specific accomplishments:
   - Community engaged (3 issues)
   - All configs fixed (11/11)
   - PR security verified
   - Bulletproof documentation

CLAUDE.md:
----------
✅ Added "Current Status" section at top:
   - Version: v1.0.0
   - Recent updates this week
   - Community response wins
   - Next priorities

✅ Added configs status:
   - 11/11 verified working (100%)
   - New Laravel config
   - All selectors tested

✅ Added roadmap reference:
   - 134 tasks in 22 groups
   - Project board link
   - Clear next steps

✅ Added Laravel to Quick Start examples

✅ Added "Available Production Configs" section:
   - All 11 configs listed with selectors
   - Content extraction stats
   - Organized by category
   - Verification date

✅ Updated Additional Documentation:
   - Added BULLETPROOF_QUICKSTART.md
   - Added TROUBLESHOOTING.md
   - Added FLEXIBLE_ROADMAP.md
   - Added NEXT_TASKS.md
   - Added TODO.md

Impact:
-------
- Clear visibility of progress (4 major items this week)
- Updated guidance for Claude Code
- Accurate config information (11 working configs)
- Better onboarding with new docs
- Transparent roadmap tracking

Files modified: TODO.md, NEXT_TASKS.md, CLAUDE.md
 2025-10-21 00:42:36 +03:00
yusyus
b83f276621 Update Python requirement to 3.10+ for MCP compatibility 
The MCP package requires Python 3.10 or higher. Updated:
- GitHub Actions workflow to test Python 3.10, 3.11, 3.12
- README.md badge to Python 3.10+
- CLAUDE.md prerequisites
- CONTRIBUTING.md prerequisites
- docs/MCP_SETUP.md prerequisites

This fixes the MCP installation error in CI:
'ERROR: No matching distribution found for mcp>=1.0.0'

MCP package versions 0.9.1+ all require Python 3.10+.
 2025-10-19 22:53:28 +03:00
yusyus
9ce78e9a16 Fix GitHub Actions workflow: Update Python version requirements 
- Update CI workflow to Python 3.9-3.12 (from 3.7-3.11)
- Python 3.7 and 3.8 no longer available on ubuntu-latest (Ubuntu 24.04)
- Add fail-fast: false to continue testing on failures
- Update all documentation to reflect Python 3.9+ requirement

Files updated:
- .github/workflows/tests.yml - New Python versions
- README.md - Badge updated to Python 3.9+
- CLAUDE.md - Prerequisites updated
- CONTRIBUTING.md - Prerequisites updated
- docs/MCP_SETUP.md - Prerequisites updated

This fixes the failing GitHub Actions tests.
 2025-10-19 22:49:14 +03:00
yusyus
d8cc92cd46 Add smart auto-upload feature with API key detection 
Features:
- New upload_skill.py for automatic API-based upload
- Smart detection: upload if API key available, helpful message if not
- Enhanced package_skill.py with --upload flag
- New MCP tool: upload_skill (9 total MCP tools now)
- Enhanced MCP tool: package_skill with smart auto-upload
- Cross-platform folder opening in utils.py
- Graceful error handling throughout

Fixes:
- Fix missing import os in mcp/server.py
- Fix package_skill.py exit code (now 0 when API key missing)
- Improve UX with helpful messages instead of errors

Tests: 14/14 passed (100%)
- CLI tests: 8/8 passed
- MCP tests: 6/6 passed

Files: +4 new, 5 modified, ~600 lines added
 2025-10-19 22:17:23 +03:00
yusyus
1c5801d121 Update documentation for MCP integration 
Comprehensive documentation updates reflecting MCP integration:

README.md:
- Add MCP Integration and Tests Passing badges
- Enhance MCP section with "Tested and Working" status
- Add links to both setup and testing guides

docs/MCP_SETUP.md:
- Update status to reflect production testing
- Add integration testing verification notes
- Confirm all 6 tools working with natural language

CLAUDE.md:
- Add prominent MCP Integration section at top
- List all 6 available MCP tools with descriptions
- Add setup instructions and production status

docs/TEST_MCP_IN_CLAUDE_CODE.md (moved from root):
- Relocate testing guide to docs/ for better organization
- Provides step-by-step MCP integration testing workflow
- Documents complete test suite for all 6 tools

All documentation now accurately reflects the fully tested and
working MCP integration verified in production Claude Code environment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-19 19:44:47 +03:00
yusyus
b69f57b60a Add comprehensive MCP setup guide and integration test template 
**Documentation Added:**
- docs/MCP_SETUP.md: Complete 400+ line setup guide
  - Prerequisites and installation steps
  - Configuration examples for Claude Code
  - Verification and troubleshooting
  - 3 usage examples and advanced configuration
  - End-to-end workflow and quick reference

- tests/mcp_integration_test.md: Comprehensive test template
  - 10 test cases covering all MCP tools
  - Performance metrics table
  - Issue tracking and environment setup
  - Setup and cleanup scripts

- .claude/mcp_config.example.json: Example MCP configuration

**Documentation Updated:**
- STRUCTURE.md: Complete monorepo structure documentation
- CLAUDE.md: All Python script paths updated to cli/ prefix
- docs/USAGE.md: All command examples updated for monorepo
- TODO.md: Current sprint status and completed tasks

**Summary:**
- Issues #2 and #3 handled (MCP setup guide + integration tests)
- All documentation now reflects monorepo structure (cli/ + mcp/)
- Tests: 71/71 passing (100%)
- Ready for MCP server testing with Claude Code

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-19 17:01:37 +03:00
yusyus
9c1a133c51 Add page count estimator for fast config validation 
- Add estimate_pages.py script (~270 lines)
- Fast estimation without downloading content (HEAD requests only)
- Shows estimated total pages and recommended max_pages
- Validates URL patterns work correctly
- Estimates scraping time based on rate_limit
- Update CLAUDE.md with estimator workflow and commands
- Update README.md features section with estimation benefits
- Usage: python3 estimate_pages.py configs/react.json
- Time: 1-2 minutes vs 20-40 minutes for full scrape

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-19 02:44:50 +03:00
yusyus
f8c75a3b2d Add comprehensive CLAUDE.md for Claude Code integration 
- Add root-level CLAUDE.md with complete guidance for Claude Code
- Include Python 3.7+ requirement
- Add first-time user workflow with all commands
- Include CSS selector testing with BeautifulSoup examples
- Add output quality verification commands
- Document force re-scrape instructions
- Fix package_skill.py path (remove hardcoded /mnt/skills reference)
- Add complete config file structure with real examples
- Include testing section for selector validation
- Add performance metrics table
- Document all key code locations with line numbers
- Organize by: quick start → architecture → workflows → troubleshooting
- Preserve existing docs/CLAUDE.md as detailed technical reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-19 01:43:02 +03:00