docs: Consolidate roadmaps and refactor documentation structure
MAJOR REFACTORING: Merge 3 roadmap files into single comprehensive ROADMAP.md Changes: - Merged ROADMAP.md + FLEXIBLE_ROADMAP.md + FUTURE_RELEASES.md → ROADMAP.md - Consolidated 1,008 lines across 3 files into 429 lines (single source of truth) - Removed duplicate/overlapping content - Cleaned up docs archive structure New ROADMAP.md Structure: - Current Status (v2.6.0) - Development Philosophy (task-based approach) - Task-Based Roadmap (136 tasks, 10 categories) - Release History (v1.0.0, v2.1.0, v2.6.0) - Release Planning (v2.7-v2.9) - Long-term Vision (v3.0+) - Metrics & Goals - Contribution guidelines Deleted Files: - FLEXIBLE_ROADMAP.md (merged into ROADMAP.md) - FUTURE_RELEASES.md (merged into ROADMAP.md) - docs/archive/temp/TERMINAL_SELECTION.md (temporary file) - docs/archive/temp/TESTING.md (temporary file) Moved Files: - docs/plans/*.md → docs/archive/plans/ (dated planning docs) Updated References: - CLAUDE.md: FLEXIBLE_ROADMAP.md → ROADMAP.md - docs/README.md: Removed duplicate roadmap references - CHANGELOG.md: Updated documentation references Benefits: - Single source of truth for roadmap - No duplicate maintenance - Cleaner repository structure - Better discoverability - Historical context preserved in archive/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
yusyus
@@ -1246,7 +1246,7 @@ This is a major milestone release featuring complete restructuring for modern Py
|
||||
|
||||
#### Documentation
|
||||
- **Updated README.md** - PyPI badges, reordered installation options
|
||||
- **FUTURE_RELEASES.md** - Roadmap for upcoming features
|
||||
- **ROADMAP.md** - Comprehensive roadmap with task-based approach
|
||||
- **Installation guides** - Simplified with PyPI as primary method
|
||||
- **Testing documentation** - How to run full test suite
|
||||
|
||||
|
||||
@@ -618,7 +618,7 @@ pytest tests/test_file.py --cov=src/skill_seekers --cov-report=term-missing
|
||||
|
||||
**For Developers:**
|
||||
- [CHANGELOG.md](CHANGELOG.md) - Release history
|
||||
- [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md) - 134 tasks across 22 feature groups
|
||||
- [ROADMAP.md](ROADMAP.md) - 136 tasks across 10 categories
|
||||
- [docs/UNIFIED_SCRAPING.md](docs/UNIFIED_SCRAPING.md) - Multi-source scraping
|
||||
- [docs/MCP_SETUP.md](docs/MCP_SETUP.md) - MCP server setup
|
||||
- [docs/ENHANCEMENT_MODES.md](docs/ENHANCEMENT_MODES.md) - AI enhancement modes
|
||||
|
||||
@@ -1,450 +0,0 @@
|
||||
# Flexible Development Roadmap
|
||||
**Philosophy:** Small incremental tasks → Pick one → Complete → Move to next
|
||||
**No big milestones, just continuous progress!**
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Current Status: v2.1.0 Released ✅
|
||||
|
||||
**Latest Release:** v2.1.0 (November 29, 2025)
|
||||
|
||||
**What Works:**
|
||||
- ✅ Documentation scraping (HTML websites)
|
||||
- ✅ GitHub repository scraping with unlimited local analysis
|
||||
- ✅ PDF extraction and conversion
|
||||
- ✅ Unified multi-source scraping (docs + GitHub + PDF)
|
||||
- ✅ 9 MCP tools fully functional
|
||||
- ✅ Auto-upload to Claude
|
||||
- ✅ 24 preset configs (including 5 unified configs)
|
||||
- ✅ Large docs support (40K+ pages)
|
||||
- ✅ Configurable directory exclusions
|
||||
- ✅ 427 tests passing
|
||||
|
||||
---
|
||||
|
||||
## 📋 Task Categories (Pick Any, Any Order)
|
||||
|
||||
### 🌐 **Category A: Community & Sharing**
|
||||
Small tasks that build community features incrementally
|
||||
|
||||
#### A1: Config Sharing (Website Feature)
|
||||
- [x] **Task A1.1:** Create simple JSON API endpoint to list configs ✅ **COMPLETE** (Issue #9)
|
||||
- **Status:** Live at https://api.skillseekersweb.com
|
||||
- **Features:** 6 REST endpoints, auto-categorization, auto-tags, filtering, SSL enabled
|
||||
- **Branch:** `feature/a1-config-sharing`
|
||||
- **Deployment:** Render with custom domain
|
||||
- [x] **Task A1.2:** Add MCP tool `fetch_config` to download from website ✅ **COMPLETE**
|
||||
- **Status:** Implemented in MCP server
|
||||
- **Features:** List 24 configs, filter by category, download by name, save to local directory
|
||||
- **Commands:** `list_available=true`, `category='web-frameworks'`, `config_name='react'`
|
||||
- **Branch:** `feature/a1-config-sharing`
|
||||
- [ ] **Task A1.3:** Add MCP tool `submit_config` to submit custom configs (Issue #11)
|
||||
- **Purpose:** Allow users to submit custom configs via MCP (creates GitHub issue)
|
||||
- **Features:** Validate config JSON, create GitHub issue, auto-label, return issue URL
|
||||
- **Approach:** GitHub Issues backend (safe, uses GitHub auth/spam detection)
|
||||
- **Time:** 2-3 hours
|
||||
- [ ] **Task A1.4:** Create static config catalog website (GitHub Pages) (Issue #12)
|
||||
- **Purpose:** Read-only catalog to browse/search configs (like npm registry)
|
||||
- **Features:** Static HTML/JS, pulls from API, search/filter, copy JSON button
|
||||
- **Architecture:** Website = browse, MCP = download/submit/manage
|
||||
- **Time:** 2-3 hours
|
||||
- [ ] **Task A1.5:** Add config rating/voting system (Issue #13)
|
||||
- **Purpose:** Community feedback on config quality
|
||||
- **Features:** Star ratings, vote counts, sort by rating, "most popular" section
|
||||
- **Options:** GitHub reactions, backend database, or localStorage
|
||||
- **Time:** 3-4 hours
|
||||
- [ ] **Task A1.6:** Admin review queue for submitted configs (Issue #14)
|
||||
- **Purpose:** Review community-submitted configs before publishing
|
||||
- **Approach:** Use GitHub Issues with labels (no custom code needed)
|
||||
- **Workflow:** Review → Validate → Test → Approve/Reject
|
||||
- **Time:** 1-2 hours (GitHub Issues) or 4-6 hours (custom dashboard)
|
||||
- [x] **Task A1.7:** Add MCP tool `install_skill` for one-command workflow (Issue #204) ✅ **COMPLETE!**
|
||||
- **Purpose:** Complete one-command workflow: fetch → scrape → **enhance** → package → upload
|
||||
- **Features:** Single command install, smart config detection, automatic AI enhancement (LOCAL)
|
||||
- **Workflow:** fetch_config → scrape_docs → enhance_skill_local → package_skill → upload_skill
|
||||
- **Critical:** Always includes AI enhancement step (30-60 sec, 3/10→9/10 quality boost)
|
||||
- **Time:** 3-4 hours
|
||||
- **Completed:** December 21, 2025 - 10 tools total, 13 tests passing, full automation working
|
||||
- [ ] **Task A1.8:** Add smart skill detection and auto-install (Issue #205)
|
||||
- **Purpose:** Auto-detect missing skills from user queries and offer to install them
|
||||
- **Features:** Topic extraction, skill gap analysis, API search, smart suggestions
|
||||
- **Modes:** Ask first (default), Auto-install, Suggest only, Manual
|
||||
- **Example:** User asks about React → Claude detects → Suggests installing React skill
|
||||
- **Time:** 4-6 hours
|
||||
|
||||
**Start Small:** ~~Pick A1.1 first (simple JSON endpoint)~~ ✅ A1.1 Complete! ~~Pick A1.2 next (MCP tool)~~ ✅ A1.2 Complete! Pick A1.3 next (MCP submit tool)
|
||||
|
||||
#### A2: Knowledge Sharing (Website Feature)
|
||||
- [ ] **Task A2.1:** Design knowledge database schema
|
||||
- [ ] **Task A2.2:** Create API endpoint to upload knowledge (.zip files)
|
||||
- [ ] **Task A2.3:** Add MCP tool `fetch_knowledge` to download from site
|
||||
- [ ] **Task A2.4:** Add knowledge preview/description
|
||||
- [ ] **Task A2.5:** Add knowledge categorization (by framework/topic)
|
||||
- [ ] **Task A2.6:** Add knowledge search functionality
|
||||
|
||||
**Start Small:** Pick A2.1 first (schema design, no coding)
|
||||
|
||||
#### A3: Simple Website Foundation
|
||||
- [ ] **Task A3.1:** Create single-page static site (GitHub Pages)
|
||||
- [ ] **Task A3.2:** Add config gallery view (display existing 12 configs)
|
||||
- [ ] **Task A3.3:** Add "Submit Config" link (opens GitHub issue for now)
|
||||
- [ ] **Task A3.4:** Add basic stats (total configs, downloads, etc.)
|
||||
- [ ] **Task A3.5:** Add simple blog using GitHub Issues
|
||||
- [ ] **Task A3.6:** Add RSS feed for updates
|
||||
|
||||
**Start Small:** Pick A3.1 first (single HTML page on GitHub Pages)
|
||||
|
||||
---
|
||||
|
||||
### 🛠️ **Category B: New Input Formats**
|
||||
Add support for non-HTML documentation sources
|
||||
|
||||
#### B1: PDF Documentation Support
|
||||
- [ ] **Task B1.1:** Research PDF parsing libraries (PyPDF2, pdfplumber, etc.)
|
||||
- [ ] **Task B1.2:** Create simple PDF text extractor (proof of concept)
|
||||
- [ ] **Task B1.3:** Add PDF page detection and chunking
|
||||
- [ ] **Task B1.4:** Extract code blocks from PDFs (syntax detection)
|
||||
- [ ] **Task B1.5:** Add PDF image extraction (diagrams, screenshots)
|
||||
- [ ] **Task B1.6:** Create `pdf_scraper.py` CLI tool
|
||||
- [ ] **Task B1.7:** Add MCP tool `scrape_pdf`
|
||||
- [ ] **Task B1.8:** Create PDF config format (similar to web configs)
|
||||
|
||||
**Start Small:** Pick B1.1 first (just research, document findings)
|
||||
|
||||
#### B2: Microsoft Word (.docx) Support
|
||||
- [ ] **Task B2.1:** Research .docx parsing (python-docx library)
|
||||
- [ ] **Task B2.2:** Create simple .docx text extractor
|
||||
- [ ] **Task B2.3:** Extract headings and create categories
|
||||
- [ ] **Task B2.4:** Extract code blocks from Word docs
|
||||
- [ ] **Task B2.5:** Extract tables and convert to markdown
|
||||
- [ ] **Task B2.6:** Create `docx_scraper.py` CLI tool
|
||||
- [ ] **Task B2.7:** Add MCP tool `scrape_docx`
|
||||
|
||||
**Start Small:** Pick B2.1 first (research only)
|
||||
|
||||
#### B3: Excel/Spreadsheet (.xlsx) Support
|
||||
- [ ] **Task B3.1:** Research Excel parsing (openpyxl, pandas)
|
||||
- [ ] **Task B3.2:** Create simple sheet → markdown converter
|
||||
- [ ] **Task B3.3:** Add table detection and formatting
|
||||
- [ ] **Task B3.4:** Extract API reference from spreadsheets (common pattern)
|
||||
- [ ] **Task B3.5:** Create `xlsx_scraper.py` CLI tool
|
||||
- [ ] **Task B3.6:** Add MCP tool `scrape_xlsx`
|
||||
|
||||
**Start Small:** Pick B3.1 first (research only)
|
||||
|
||||
#### B4: Markdown Files Support
|
||||
- [ ] **Task B4.1:** Create markdown file crawler (for local docs)
|
||||
- [ ] **Task B4.2:** Extract front matter (title, category, etc.)
|
||||
- [ ] **Task B4.3:** Build category tree from folder structure
|
||||
- [ ] **Task B4.4:** Add link resolution (internal references)
|
||||
- [ ] **Task B4.5:** Create `markdown_scraper.py` CLI tool
|
||||
- [ ] **Task B4.6:** Add MCP tool `scrape_markdown_dir`
|
||||
|
||||
**Start Small:** Pick B4.1 first (simple file walker)
|
||||
|
||||
---
|
||||
|
||||
### 💻 **Category C: Codebase Knowledge**
|
||||
Generate skills from actual code repositories
|
||||
|
||||
#### C1: GitHub Repository Scraping
|
||||
- [ ] **Task C1.1:** Create GitHub API client (fetch repo structure)
|
||||
- [ ] **Task C1.2:** Extract README.md files
|
||||
- [ ] **Task C1.3:** Extract code comments and docstrings
|
||||
- [ ] **Task C1.4:** Detect programming language per file
|
||||
- [ ] **Task C1.5:** Extract function/class signatures
|
||||
- [ ] **Task C1.6:** Build usage examples from tests
|
||||
- [ ] **Task C1.7:** Extract GitHub Issues (open/closed, labels, milestones)
|
||||
- [ ] **Task C1.8:** Extract CHANGELOG.md and release notes
|
||||
- [ ] **Task C1.9:** Extract GitHub Releases with version history
|
||||
- [ ] **Task C1.10:** Create `github_scraper.py` CLI tool
|
||||
- [ ] **Task C1.11:** Add MCP tool `scrape_github`
|
||||
- [ ] **Task C1.12:** Add config format for GitHub repos
|
||||
|
||||
**Start Small:** Pick C1.1 first (basic GitHub API connection)
|
||||
|
||||
#### C2: Local Codebase Scraping
|
||||
- [ ] **Task C2.1:** Create file tree walker (with .gitignore support)
|
||||
- [ ] **Task C2.2:** Extract docstrings (Python, JS, etc.)
|
||||
- [ ] **Task C2.3:** Extract function signatures and types
|
||||
- [ ] **Task C2.4:** Build API reference from code
|
||||
- [ ] **Task C2.5:** Extract inline comments as notes
|
||||
- [ ] **Task C2.6:** Create dependency graph
|
||||
- [ ] **Task C2.7:** Create `codebase_scraper.py` CLI tool
|
||||
- [ ] **Task C2.8:** Add MCP tool `scrape_codebase`
|
||||
|
||||
**Start Small:** Pick C2.1 first (simple file walker)
|
||||
|
||||
#### C3: Code Pattern Recognition
|
||||
- [x] **Task C3.1:** Detect common patterns (singleton, factory, etc.) ✅ **v2.6.0** - Completed Jan 2026
|
||||
- 10 GoF patterns: Singleton, Factory, Observer, Strategy, Decorator, Builder, Adapter, Command, Template Method, Chain of Responsibility
|
||||
- 9 languages: Python (AST), JavaScript, TypeScript, C++, C, C#, Go, Rust, Java
|
||||
- 3 detection levels: Surface (naming), Deep (structure), Full (behavior)
|
||||
- CLI tool, MCP integration, 24 tests, 87% precision
|
||||
- See: `docs/PATTERN_DETECTION.md`, Issue #71
|
||||
- [x] **Task C3.2:** Extract usage examples from test files ✅ **v2.6.0** - Completed Jan 2026
|
||||
- 5 categories: instantiation, method_call, config, setup, workflow
|
||||
- 9 languages: Python (AST-based), JavaScript, TypeScript, Go, Rust, Java, C#, PHP, Ruby
|
||||
- Quality filtering with confidence scoring (removes trivial patterns)
|
||||
- CLI tool, MCP integration, 19 tests, 80%+ high-confidence examples
|
||||
- See: `docs/TEST_EXAMPLE_EXTRACTION.md`, Issue #72
|
||||
- [ ] **Task C3.3:** Build "how to" guides from code
|
||||
- [ ] **Task C3.4:** Extract configuration patterns
|
||||
- [ ] **Task C3.5:** Create architectural overview
|
||||
- [x] **Task C3.6:** AI Enhancement for Pattern Detection and Test Examples ✅ **v2.6.0** - Completed Jan 2026
|
||||
- Enhances C3.1 and C3.2 with AI-powered insights using Claude API
|
||||
- Pattern enhancement: Explains detection, suggests improvements, identifies issues
|
||||
- Test example enhancement: Adds context, groups tutorials, identifies best practices
|
||||
- Auto-activation when ANTHROPIC_API_KEY is set, graceful offline degradation
|
||||
- Batch processing (5 items/call) to minimize API costs
|
||||
- See: `src/skill_seekers/cli/ai_enhancer.py`, Issue #234
|
||||
- [x] **Task C3.7:** Architectural Pattern Detection ✅ **v2.6.0** - Completed Jan 2026
|
||||
- Detects 8 architectural patterns: MVC, MVVM, MVP, Repository, Service Layer, Layered, Clean Architecture
|
||||
- Framework detection: Django, Flask, Spring, ASP.NET, Rails, Laravel, Angular, React, Vue.js
|
||||
- Multi-file analysis with directory structure pattern matching
|
||||
- Evidence-based detection with confidence scoring
|
||||
- AI-enhanced architectural insights (integrates with C3.6)
|
||||
- See: `src/skill_seekers/cli/architectural_pattern_detector.py`, Issue #235
|
||||
|
||||
**Start Small:** Pick C3.3 next (build "how to" guides from workflow examples)
|
||||
|
||||
---
|
||||
|
||||
### 🔌 **Category D: Context7 Integration**
|
||||
Explore integration with Context7 for enhanced context management
|
||||
|
||||
#### D1: Context7 Research & Planning
|
||||
- [ ] **Task D1.1:** Research Context7 API and capabilities
|
||||
- [ ] **Task D1.2:** Document potential use cases for Skill Seeker
|
||||
- [ ] **Task D1.3:** Create integration design proposal
|
||||
- [ ] **Task D1.4:** Identify which features benefit most
|
||||
|
||||
**Start Small:** Pick D1.1 first (pure research, no code)
|
||||
|
||||
#### D2: Context7 Basic Integration
|
||||
- [ ] **Task D2.1:** Create Context7 API client
|
||||
- [ ] **Task D2.2:** Test basic context storage/retrieval
|
||||
- [ ] **Task D2.3:** Store scraped documentation in Context7
|
||||
- [ ] **Task D2.4:** Query Context7 during skill building
|
||||
- [ ] **Task D2.5:** Add MCP tool `sync_to_context7`
|
||||
|
||||
**Start Small:** Pick D2.1 first (basic API connection)
|
||||
|
||||
---
|
||||
|
||||
### 🚀 **Category E: MCP Enhancements**
|
||||
Small improvements to existing MCP tools
|
||||
|
||||
#### E1: New MCP Tools
|
||||
- [ ] **Task E1.1:** Add `fetch_config` MCP tool (download from website)
|
||||
- [ ] **Task E1.2:** Add `fetch_knowledge` MCP tool (download skills)
|
||||
- [x] **Task E1.3:** Add `scrape_pdf` MCP tool (✅ COMPLETED v1.0.0)
|
||||
- [ ] **Task E1.4:** Add `scrape_docx` MCP tool
|
||||
- [ ] **Task E1.5:** Add `scrape_xlsx` MCP tool
|
||||
- [ ] **Task E1.6:** Add `scrape_github` MCP tool (see C1.11)
|
||||
- [ ] **Task E1.7:** Add `scrape_codebase` MCP tool (see C2.8)
|
||||
- [ ] **Task E1.8:** Add `scrape_markdown_dir` MCP tool (see B4.6)
|
||||
- [ ] **Task E1.9:** Add `sync_to_context7` MCP tool (see D2.5)
|
||||
|
||||
**Start Small:** Pick E1.1 first (once A1.2 is done)
|
||||
|
||||
#### E2: MCP Quality Improvements
|
||||
- [ ] **Task E2.1:** Add error handling to all tools
|
||||
- [ ] **Task E2.2:** Add structured logging
|
||||
- [ ] **Task E2.3:** Add progress indicators for long operations
|
||||
- [ ] **Task E2.4:** Add validation for all inputs
|
||||
- [ ] **Task E2.5:** Add helpful error messages
|
||||
- [x] **Task E2.6:** Add retry logic for network failures *(Utilities ready via PR #208, integration pending)*
|
||||
|
||||
**Start Small:** Pick E2.1 first (one tool at a time)
|
||||
|
||||
---
|
||||
|
||||
### ⚡ **Category F: Performance & Reliability**
|
||||
Technical improvements to existing features
|
||||
|
||||
#### F1: Core Scraper Improvements
|
||||
- [ ] **Task F1.1:** Add URL normalization (remove query params)
|
||||
- [ ] **Task F1.2:** Add duplicate page detection
|
||||
- [ ] **Task F1.3:** Add memory-efficient streaming for large docs
|
||||
- [ ] **Task F1.4:** Add HTML parser fallback (lxml → html5lib)
|
||||
- [x] **Task F1.5:** Add network retry with exponential backoff *(Utilities ready via PR #208, scraper integration pending)*
|
||||
- [ ] **Task F1.6:** Fix package path output bug
|
||||
|
||||
**Start Small:** Pick F1.1 first (URL normalization only)
|
||||
|
||||
#### F2: Incremental Updates
|
||||
- [ ] **Task F2.1:** Track page modification times (Last-Modified header)
|
||||
- [ ] **Task F2.2:** Store page checksums/hashes
|
||||
- [ ] **Task F2.3:** Compare on re-run, skip unchanged pages
|
||||
- [ ] **Task F2.4:** Update only changed content
|
||||
- [ ] **Task F2.5:** Preserve local annotations/edits
|
||||
|
||||
**Start Small:** Pick F2.1 first (just tracking, no logic)
|
||||
|
||||
---
|
||||
|
||||
### 🎨 **Category G: Tools & Utilities**
|
||||
Small standalone tools that add value
|
||||
|
||||
#### G1: Config Tools
|
||||
- [ ] **Task G1.1:** Create `validate_config.py` (enhanced validation)
|
||||
- [ ] **Task G1.2:** Create `test_selectors.py` (interactive selector tester)
|
||||
- [ ] **Task G1.3:** Create `auto_detect_selectors.py` (AI-powered)
|
||||
- [ ] **Task G1.4:** Create `compare_configs.py` (diff two configs)
|
||||
- [ ] **Task G1.5:** Create `optimize_config.py` (suggest improvements)
|
||||
|
||||
**Start Small:** Pick G1.1 first (simple validation script)
|
||||
|
||||
#### G2: Skill Quality Tools
|
||||
- [ ] **Task G2.1:** Create `analyze_skill.py` (quality metrics)
|
||||
- [ ] **Task G2.2:** Add code example counter
|
||||
- [ ] **Task G2.3:** Add readability scoring
|
||||
- [ ] **Task G2.4:** Add completeness checker
|
||||
- [ ] **Task G2.5:** Create quality report generator
|
||||
|
||||
**Start Small:** Pick G2.1 first (basic metrics)
|
||||
|
||||
---
|
||||
|
||||
### 📚 **Category H: Community Response**
|
||||
Respond to existing GitHub issues
|
||||
|
||||
#### H1: Address Open Issues
|
||||
- [ ] **Task H1.1:** Respond to Issue #8: Prereqs to Getting Started
|
||||
- [ ] **Task H1.2:** Investigate Issue #7: Laravel scraping issue
|
||||
- [ ] **Task H1.3:** Create example project (Issue #4)
|
||||
- [ ] **Task H1.4:** Answer Issue #3: Pro plan compatibility
|
||||
- [ ] **Task H1.5:** Create self-documenting skill (Issue #1)
|
||||
|
||||
**Start Small:** Pick H1.1 first (just respond, don't solve)
|
||||
|
||||
---
|
||||
|
||||
### 🎓 **Category I: Content & Documentation**
|
||||
Educational content and guides
|
||||
|
||||
#### I1: Video Tutorials
|
||||
- [ ] **Task I1.1:** Write script for "Quick Start" video
|
||||
- [ ] **Task I1.2:** Record "Quick Start" (5 min)
|
||||
- [ ] **Task I1.3:** Write script for "MCP Setup" video
|
||||
- [ ] **Task I1.4:** Record "MCP Setup" (8 min)
|
||||
- [ ] **Task I1.5:** Write script for "Custom Config" video
|
||||
- [ ] **Task I1.6:** Record "Custom Config" (10 min)
|
||||
|
||||
**Start Small:** Pick I1.1 first (just write script, no recording)
|
||||
|
||||
#### I2: Written Guides
|
||||
- [ ] **Task I2.1:** Write troubleshooting guide
|
||||
- [ ] **Task I2.2:** Write best practices guide
|
||||
- [ ] **Task I2.3:** Write performance optimization guide
|
||||
- [ ] **Task I2.4:** Write community config contribution guide
|
||||
- [ ] **Task I2.5:** Write codebase scraping guide
|
||||
|
||||
**Start Small:** Pick I2.1 first (common issues + solutions)
|
||||
|
||||
---
|
||||
|
||||
### 🧪 **Category J: Testing & Quality**
|
||||
Improve test coverage and quality
|
||||
|
||||
#### J1: Test Expansion
|
||||
- [ ] **Task J1.1:** Install MCP package: `pip install mcp`
|
||||
- [ ] **Task J1.2:** Verify all 14 tests pass
|
||||
- [ ] **Task J1.3:** Add tests for new MCP tools (as they're created)
|
||||
- [ ] **Task J1.4:** Add integration tests for PDF scraper
|
||||
- [ ] **Task J1.5:** Add integration tests for GitHub scraper
|
||||
- [ ] **Task J1.6:** Add end-to-end workflow tests
|
||||
|
||||
**Start Small:** Pick J1.1 first (just install package)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Recommended Starting Tasks (Pick 3-5)
|
||||
|
||||
### Quick Wins (1-2 hours each):
|
||||
1. **H1.1** - Respond to Issue #8 (community engagement)
|
||||
2. **J1.1** - Install MCP package (fix tests)
|
||||
3. **A3.1** - Create simple GitHub Pages site (single HTML)
|
||||
4. **B1.1** - Research PDF parsing (no coding, just notes)
|
||||
5. **F1.1** - Add URL normalization (small code fix)
|
||||
|
||||
### Medium Tasks (3-5 hours each):
|
||||
6. ~~**A1.1** - Create JSON API for configs (simple endpoint)~~ ✅ **COMPLETE**
|
||||
7. **G1.1** - Create config validator script
|
||||
8. **C1.1** - GitHub API client (basic connection)
|
||||
9. **I1.1** - Write Quick Start video script
|
||||
10. **E2.1** - Add error handling to one MCP tool
|
||||
|
||||
### Bigger Tasks (5-10 hours each):
|
||||
11. **B1.2-B1.6** - Complete PDF scraper
|
||||
12. **C1.7-C1.9** - Complete GitHub scraper
|
||||
13. **A2.1-A2.3** - Knowledge sharing foundation
|
||||
14. **I1.2** - Record and publish Quick Start video
|
||||
|
||||
---
|
||||
|
||||
## 📊 Progress Tracking
|
||||
|
||||
**Completed Tasks:** 3 (A1.1 ✅, A1.2 ✅, A1.7 ✅)
|
||||
**In Progress:** 0
|
||||
**Total Available Tasks:** 136
|
||||
|
||||
### Current Sprint: Choose Your Own Adventure!
|
||||
**Pick 1-3 tasks** from any category that interest you most.
|
||||
|
||||
**No pressure, no deadlines, just progress!** ✨
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Flexibility Rules
|
||||
|
||||
1. **Pick any task, any order** - No dependencies (mostly)
|
||||
2. **Start small** - Research tasks before implementation
|
||||
3. **One task at a time** - Focus, complete, move on
|
||||
4. **Switch anytime** - Not enjoying it? Pick another!
|
||||
5. **Document as you go** - Each task should update docs
|
||||
6. **Test incrementally** - Each task should have a quick test
|
||||
7. **Ship early** - Don't wait for "complete" features
|
||||
|
||||
---
|
||||
|
||||
## 🚀 How to Use This Roadmap
|
||||
|
||||
### Step 1: Pick a Task
|
||||
- Read through categories
|
||||
- Pick something that sounds interesting
|
||||
- Check estimated time
|
||||
- Choose 1-3 tasks for this week
|
||||
|
||||
### Step 2: Create Issue (Optional)
|
||||
- Create GitHub issue for tracking
|
||||
- Add labels (category, priority)
|
||||
- Add to project board
|
||||
|
||||
### Step 3: Work on It
|
||||
- Complete the task
|
||||
- Test it
|
||||
- Document it
|
||||
- Mark as done ✅
|
||||
|
||||
### Step 4: Ship It
|
||||
- Commit changes
|
||||
- Update changelog
|
||||
- Tag version (if significant)
|
||||
- Announce on GitHub
|
||||
|
||||
### Step 5: Repeat
|
||||
- Pick next task
|
||||
- Keep moving forward!
|
||||
|
||||
---
|
||||
|
||||
**Philosophy:**
|
||||
**Small steps → Consistent progress → Compound results**
|
||||
|
||||
**No rigid milestones. No big releases. Just continuous improvement!** 🎯
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** October 20, 2025
|
||||
@@ -1,292 +0,0 @@
|
||||
# Future Releases Roadmap
|
||||
|
||||
This document outlines planned features, improvements, and the vision for upcoming releases of Skill Seekers.
|
||||
|
||||
## Release Philosophy
|
||||
|
||||
We follow semantic versioning (MAJOR.MINOR.PATCH) and maintain backward compatibility wherever possible. Each release focuses on delivering value to users while maintaining code quality and test coverage.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Release: v2.1.0 (Released: November 29, 2025)
|
||||
|
||||
**Focus:** Test Coverage & Quality Improvements
|
||||
|
||||
### Completed Features
|
||||
|
||||
#### Testing & Quality
|
||||
- [x] **Fix 12 unified scraping tests** ✅ - Complete test coverage for unified multi-source scraping
|
||||
- ConfigValidator expecting dict instead of file path
|
||||
- ConflictDetector expecting dict pages, not list
|
||||
- Full integration test suite for unified workflow
|
||||
|
||||
### Planned Features (Future v2.2.0)
|
||||
|
||||
#### Testing & Quality
|
||||
|
||||
- [ ] **Improve test coverage to 60%+** (currently 39%)
|
||||
- Write tests for 0% coverage files:
|
||||
- `generate_router.py` (110 lines) - Router skill generator
|
||||
- `split_config.py` (165 lines) - Config splitter
|
||||
- `unified_scraper.py` (208 lines) - Unified scraping CLI
|
||||
- `package_multi.py` (37 lines) - Multi-package tool
|
||||
- Improve coverage for low-coverage files:
|
||||
- `mcp/server.py` (9% → 60%)
|
||||
- `enhance_skill.py` (11% → 60%)
|
||||
- `code_analyzer.py` (19% → 60%)
|
||||
|
||||
- [ ] **Fix MCP test skipping issue** - 29 MCP tests pass individually but skip in full suite
|
||||
- Resolve pytest isolation issue
|
||||
- Ensure all tests run in CI/CD
|
||||
|
||||
#### Features
|
||||
- [ ] **Task H1.3: Create example project folder**
|
||||
- Real-world example projects using Skill Seekers
|
||||
- Step-by-step tutorials
|
||||
- Before/after comparisons
|
||||
|
||||
- [ ] **Task J1.1: Install MCP package for testing**
|
||||
- Better MCP integration testing
|
||||
- Automated MCP server tests in CI
|
||||
|
||||
- [ ] **Enhanced error handling**
|
||||
- Better error messages for common issues
|
||||
- Graceful degradation for missing dependencies
|
||||
- Recovery from partial failures
|
||||
|
||||
### Documentation
|
||||
- [ ] Video tutorials for common workflows
|
||||
- [ ] Troubleshooting guide expansion
|
||||
- [ ] Performance optimization guide
|
||||
|
||||
---
|
||||
|
||||
## Release: v2.2.0 (Estimated: Q1 2026)
|
||||
|
||||
**Focus:** Web Presence & Community Growth
|
||||
|
||||
### Planned Features
|
||||
|
||||
#### Community & Documentation
|
||||
- [ ] **Task A3.1: GitHub Pages website** (skillseekersweb.com)
|
||||
- Interactive documentation
|
||||
- Live demos and examples
|
||||
- Getting started wizard
|
||||
- Community showcase
|
||||
|
||||
- [ ] **Plugin system foundation**
|
||||
- Allow custom scrapers via plugins
|
||||
- Plugin discovery and installation
|
||||
- Plugin documentation generator
|
||||
|
||||
#### Enhancements
|
||||
- [ ] **Support for additional documentation formats**
|
||||
- Sphinx documentation
|
||||
- Docusaurus sites
|
||||
- GitBook
|
||||
- Read the Docs
|
||||
- MkDocs Material
|
||||
|
||||
- [ ] **Improved caching strategies**
|
||||
- Intelligent cache invalidation
|
||||
- Differential scraping (only changed pages)
|
||||
- Cache compression
|
||||
- Cross-session cache sharing
|
||||
|
||||
#### Performance
|
||||
- [ ] **Scraping performance improvements**
|
||||
- Connection pooling optimizations
|
||||
- Smart rate limiting based on server response
|
||||
- Adaptive concurrency
|
||||
- Memory usage optimization for large docs
|
||||
|
||||
---
|
||||
|
||||
## Release: v2.3.0 (Estimated: Q2 2026)
|
||||
|
||||
**Focus:** Developer Experience & Integrations
|
||||
|
||||
### Planned Features
|
||||
|
||||
#### Developer Tools
|
||||
- [ ] **Web UI for config generation**
|
||||
- Visual config builder
|
||||
- Real-time preview
|
||||
- Template library
|
||||
- Export/import configs
|
||||
|
||||
- [ ] **CI/CD integration examples**
|
||||
- GitHub Actions workflows
|
||||
- GitLab CI
|
||||
- Jenkins pipelines
|
||||
- Automated skill updates on doc changes
|
||||
|
||||
- [ ] **Docker containerization**
|
||||
- Official Docker images
|
||||
- docker-compose examples
|
||||
- Kubernetes deployment guides
|
||||
|
||||
#### API & Integrations
|
||||
- [ ] **GraphQL API support**
|
||||
- Scrape GraphQL documentation
|
||||
- Extract schema and queries
|
||||
- Generate interactive examples
|
||||
|
||||
- [ ] **REST API documentation formats**
|
||||
- OpenAPI/Swagger
|
||||
- Postman collections
|
||||
- API Blueprint
|
||||
|
||||
---
|
||||
|
||||
## Long-term Vision (v3.0+)
|
||||
|
||||
### Major Features Under Consideration
|
||||
|
||||
#### Advanced Scraping
|
||||
- [ ] **Real-time documentation monitoring**
|
||||
- Watch for documentation changes
|
||||
- Automatic skill updates
|
||||
- Change notifications
|
||||
- Version diff reports
|
||||
|
||||
- [ ] **Multi-language documentation**
|
||||
- Automatic language detection
|
||||
- Combined multi-language skills
|
||||
- Translation quality checking
|
||||
|
||||
#### Collaboration
|
||||
- [ ] **Collaborative skill curation**
|
||||
- Shared skill repositories
|
||||
- Community ratings and reviews
|
||||
- Collaborative editing
|
||||
- Fork and merge workflows
|
||||
|
||||
- [ ] **Skill marketplace**
|
||||
- Discover community-created skills
|
||||
- Share your skills
|
||||
- Quality ratings
|
||||
- Usage statistics
|
||||
|
||||
#### AI & Intelligence
|
||||
- [ ] **Enhanced AI analysis**
|
||||
- Better conflict detection algorithms
|
||||
- Automatic documentation quality scoring
|
||||
- Suggested improvements
|
||||
- Code example validation
|
||||
|
||||
- [ ] **Semantic understanding**
|
||||
- Natural language queries for skill content
|
||||
- Intelligent categorization
|
||||
- Auto-generated summaries
|
||||
- Concept relationship mapping
|
||||
|
||||
---
|
||||
|
||||
## Backlog Ideas
|
||||
|
||||
### Features Requested by Community
|
||||
- [ ] Support for video tutorial transcription
|
||||
- [ ] Integration with Notion, Confluence, and other wikis
|
||||
- [ ] Jupyter notebook scraping and conversion
|
||||
- [ ] Live documentation preview during scraping
|
||||
- [ ] Skill versioning and update management
|
||||
- [ ] A/B testing for skill quality
|
||||
- [ ] Analytics dashboard (scraping stats, error rates, etc.)
|
||||
|
||||
### Technical Improvements
|
||||
- [ ] Migration to modern async framework (httpx everywhere)
|
||||
- [ ] Improved type safety (full mypy strict mode)
|
||||
- [ ] Better logging and debugging tools
|
||||
- [ ] Performance profiling dashboard
|
||||
- [ ] Memory optimization for very large docs (100K+ pages)
|
||||
|
||||
### Ecosystem
|
||||
- [ ] VS Code extension
|
||||
- [ ] IntelliJ/PyCharm plugin
|
||||
- [ ] Command-line interactive mode (TUI)
|
||||
- [ ] Skill diff tool (compare versions)
|
||||
- [ ] Skill merge tool (combine multiple skills)
|
||||
|
||||
---
|
||||
|
||||
## How to Influence the Roadmap
|
||||
|
||||
### Priority System
|
||||
|
||||
Features are prioritized based on:
|
||||
1. **User impact** - How many users will benefit?
|
||||
2. **Technical feasibility** - How complex is the implementation?
|
||||
3. **Community interest** - How many upvotes/requests?
|
||||
4. **Strategic alignment** - Does it fit our vision?
|
||||
|
||||
### Ways to Contribute
|
||||
|
||||
#### 1. Vote on Features
|
||||
- ⭐ Star feature request issues
|
||||
- 💬 Comment with your use case
|
||||
- 🔼 Upvote discussions
|
||||
|
||||
#### 2. Contribute Code
|
||||
See our [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md) for:
|
||||
- **134 tasks** across 22 feature groups
|
||||
- Tasks categorized by difficulty and area
|
||||
- Clear acceptance criteria
|
||||
- Estimated effort levels
|
||||
|
||||
Pick any task and submit a PR! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
||||
|
||||
#### 3. Share Feedback
|
||||
- Open issues for bugs or feature requests
|
||||
- Share your success stories
|
||||
- Suggest improvements to existing features
|
||||
- Report performance issues
|
||||
|
||||
#### 4. Help with Documentation
|
||||
- Write tutorials
|
||||
- Improve existing docs
|
||||
- Translate documentation
|
||||
- Create video guides
|
||||
|
||||
---
|
||||
|
||||
## Release Schedule
|
||||
|
||||
We aim for predictable releases:
|
||||
|
||||
- **Patch releases (2.0.x)**: As needed for critical bugs
|
||||
- **Minor releases (2.x.0)**: Every 2-3 months
|
||||
- **Major releases (x.0.0)**: Annually, with breaking changes announced 3 months in advance
|
||||
|
||||
### Current Schedule
|
||||
|
||||
| Version | Focus | ETA | Status |
|
||||
|---------|-------|-----|--------|
|
||||
| v2.0.0 | PyPI Publication | 2025-11-11 | ✅ Released |
|
||||
| v2.1.0 | Test Coverage & Quality | 2025-11-29 | ✅ Released |
|
||||
| v2.2.0 | Web Presence | Q1 2026 | 📋 Planned |
|
||||
| v2.3.0 | Developer Experience | Q2 2026 | 📋 Planned |
|
||||
| v3.0.0 | Major Evolution | 2026 | 💡 Conceptual |
|
||||
|
||||
---
|
||||
|
||||
## Stay Updated
|
||||
|
||||
- 📋 **Project Board**: https://github.com/users/yusufkaraaslan/projects/2
|
||||
- 📚 **Full Roadmap**: [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md)
|
||||
- 📝 **Changelog**: [CHANGELOG.md](CHANGELOG.md)
|
||||
- 💬 **Discussions**: https://github.com/yusufkaraaslan/Skill_Seekers/discussions
|
||||
- 🐛 **Issues**: https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
Have questions about the roadmap or want to suggest a feature?
|
||||
|
||||
1. Check if it's already in our [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md)
|
||||
2. Search [existing discussions](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)
|
||||
3. Open a new discussion or issue
|
||||
4. Reach out in our community channels
|
||||
|
||||
**Together, we're building the future of documentation-to-AI skill conversion!** 🚀
|
||||
576
ROADMAP.md
576
ROADMAP.md
@@ -1,256 +1,405 @@
|
||||
# Skill Seeker Development Roadmap
|
||||
# Skill Seekers Roadmap
|
||||
|
||||
## Vision
|
||||
Transform Skill Seeker into the easiest way to create Claude AI skills from **any knowledge source** - documentation websites, PDFs, codebases, GitHub repos, Office docs, and more - with both CLI and MCP interfaces.
|
||||
Transform Skill Seekers into the easiest way to create Claude AI skills from **any knowledge source** - documentation websites, PDFs, codebases, GitHub repos, Office docs, and more - with both CLI and MCP interfaces.
|
||||
|
||||
## 🎯 New Approach: Flexible, Incremental Development
|
||||
---
|
||||
|
||||
**Philosophy:** Small tasks → Pick one → Complete → Move on
|
||||
## 🎯 Current Status: v2.6.0 ✅
|
||||
|
||||
Instead of rigid milestones, we now use a **flexible task-based approach**:
|
||||
- 100+ small, independent tasks across 10 categories
|
||||
**Latest Release:** v2.6.0 (January 14, 2026)
|
||||
|
||||
**What Works:**
|
||||
- ✅ Documentation scraping (HTML websites with llms.txt support)
|
||||
- ✅ GitHub repository scraping with C3.x codebase analysis
|
||||
- ✅ PDF extraction with OCR and image support
|
||||
- ✅ Unified multi-source scraping (docs + GitHub + PDF)
|
||||
- ✅ 18 MCP tools fully functional
|
||||
- ✅ Multi-platform support (Claude, Gemini, OpenAI, Markdown)
|
||||
- ✅ Auto-upload to all platforms
|
||||
- ✅ 24 preset configs (including 7 unified configs)
|
||||
- ✅ Large docs support (40K+ pages with router skills)
|
||||
- ✅ C3.x codebase analysis suite (C3.1-C3.8)
|
||||
- ✅ 700+ tests passing
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Development Philosophy
|
||||
|
||||
**Small tasks → Pick one → Complete → Move on**
|
||||
|
||||
Instead of rigid milestones, we use a **flexible task-based approach**:
|
||||
- 136 small, independent tasks across 10 categories
|
||||
- Pick any task, any order
|
||||
- Start small, ship often
|
||||
- No deadlines, just continuous progress
|
||||
|
||||
**See:** [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md) for the complete task list!
|
||||
**Philosophy:** Small steps → Consistent progress → Compound results
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Milestones
|
||||
## 📋 Task-Based Roadmap (136 Tasks, 10 Categories)
|
||||
|
||||
### ✅ v1.0 - Production Release (COMPLETED - Oct 19, 2025)
|
||||
**Released:** October 19, 2025 | **Tag:** v1.0.0
|
||||
### 🌐 **Category A: Community & Sharing**
|
||||
Small tasks that build community features incrementally
|
||||
|
||||
#### Core Features ✅
|
||||
- [x] Documentation scraping with BFS
|
||||
- [x] Smart categorization
|
||||
- [x] Language detection
|
||||
- [x] Pattern extraction
|
||||
- [x] 12 preset configurations (Godot, React, Vue, Django, FastAPI, Tailwind, Kubernetes, Astro, etc.)
|
||||
- [x] Comprehensive test suite (14 tests, 100% pass rate)
|
||||
#### A1: Config Sharing (Website Feature)
|
||||
- [x] **Task A1.1:** Create simple JSON API endpoint to list configs ✅ **COMPLETE**
|
||||
- **Status:** Live at https://api.skillseekersweb.com
|
||||
- **Features:** 6 REST endpoints, auto-categorization, auto-tags, filtering, SSL enabled
|
||||
- [x] **Task A1.2:** Add MCP tool `fetch_config` to download from website ✅ **COMPLETE**
|
||||
- **Features:** List 24 configs, filter by category, download by name
|
||||
- [ ] **Task A1.3:** Add MCP tool `submit_config` to submit custom configs
|
||||
- **Purpose:** Allow users to submit custom configs via MCP (creates GitHub issue)
|
||||
- **Time:** 2-3 hours
|
||||
- [ ] **Task A1.4:** Create static config catalog website (GitHub Pages)
|
||||
- **Purpose:** Read-only catalog to browse/search configs
|
||||
- **Time:** 2-3 hours
|
||||
- [ ] **Task A1.5:** Add config rating/voting system
|
||||
- **Purpose:** Community feedback on config quality
|
||||
- **Time:** 3-4 hours
|
||||
- [ ] **Task A1.6:** Admin review queue for submitted configs
|
||||
- **Approach:** Use GitHub Issues with labels
|
||||
- **Time:** 1-2 hours
|
||||
- [x] **Task A1.7:** Add MCP tool `install_skill` for one-command workflow ✅ **COMPLETE**
|
||||
- **Features:** fetch → scrape → enhance → package → upload
|
||||
- **Completed:** December 21, 2025
|
||||
- [ ] **Task A1.8:** Add smart skill detection and auto-install
|
||||
- **Purpose:** Auto-detect missing skills from user queries
|
||||
- **Time:** 4-6 hours
|
||||
|
||||
#### MCP Integration ✅
|
||||
- [x] Monorepo refactor (cli/ and mcp/)
|
||||
- [x] MCP server with 9 tools (fully functional)
|
||||
- [x] All MCP tools tested and working
|
||||
- [x] Complete MCP documentation
|
||||
- [x] Setup automation (setup_mcp.sh)
|
||||
**Start Next:** Pick A1.3 (MCP submit tool)
|
||||
|
||||
#### Large Documentation Support ✅
|
||||
- [x] Config splitting for 40K+ page docs
|
||||
- [x] Router/hub skill generation
|
||||
- [x] Checkpoint/resume functionality
|
||||
- [x] Parallel scraping support
|
||||
#### A2: Knowledge Sharing (Website Feature)
|
||||
- [ ] **Task A2.1:** Design knowledge database schema
|
||||
- [ ] **Task A2.2:** Create API endpoint to upload knowledge (.zip files)
|
||||
- [ ] **Task A2.3:** Add MCP tool `fetch_knowledge` to download from site
|
||||
- [ ] **Task A2.4:** Add knowledge preview/description
|
||||
- [ ] **Task A2.5:** Add knowledge categorization (by framework/topic)
|
||||
- [ ] **Task A2.6:** Add knowledge search functionality
|
||||
|
||||
#### Auto-Upload Feature ✅
|
||||
- [x] Smart API key detection
|
||||
- [x] Automatic upload to Claude
|
||||
- [x] Cross-platform folder opening
|
||||
- [x] Graceful fallback to manual upload
|
||||
**Start Small:** Pick A2.1 first (schema design, no coding)
|
||||
|
||||
**Statistics:**
|
||||
- 9 MCP tools (fully working)
|
||||
#### A3: Simple Website Foundation
|
||||
- [ ] **Task A3.1:** Create single-page static site (GitHub Pages)
|
||||
- [ ] **Task A3.2:** Add config gallery view
|
||||
- [ ] **Task A3.3:** Add "Submit Config" link
|
||||
- [ ] **Task A3.4:** Add basic stats
|
||||
- [ ] **Task A3.5:** Add simple blog using GitHub Issues
|
||||
- [ ] **Task A3.6:** Add RSS feed for updates
|
||||
|
||||
**Start Small:** Pick A3.1 first (single HTML page)
|
||||
|
||||
---
|
||||
|
||||
### 🛠️ **Category B: New Input Formats**
|
||||
Add support for non-HTML documentation sources
|
||||
|
||||
#### B1: PDF Documentation Support
|
||||
- [ ] **Task B1.1:** Research PDF parsing libraries
|
||||
- [ ] **Task B1.2:** Create simple PDF text extractor (POC)
|
||||
- [ ] **Task B1.3:** Add PDF page detection and chunking
|
||||
- [ ] **Task B1.4:** Extract code blocks from PDFs
|
||||
- [ ] **Task B1.5:** Add PDF image extraction
|
||||
- [ ] **Task B1.6:** Create `pdf_scraper.py` CLI tool
|
||||
- [ ] **Task B1.7:** Add MCP tool `scrape_pdf`
|
||||
- [ ] **Task B1.8:** Create PDF config format
|
||||
|
||||
**Start Small:** Pick B1.1 first (research only)
|
||||
|
||||
#### B2: Microsoft Word (.docx) Support
|
||||
- [ ] **Task B2.1-B2.7:** Word document parsing and scraping
|
||||
|
||||
#### B3: Excel/Spreadsheet (.xlsx) Support
|
||||
- [ ] **Task B3.1-B3.6:** Spreadsheet parsing and API extraction
|
||||
|
||||
#### B4: Markdown Files Support
|
||||
- [ ] **Task B4.1-B4.6:** Local markdown directory scraping
|
||||
|
||||
---
|
||||
|
||||
### 💻 **Category C: Codebase Knowledge**
|
||||
Generate skills from actual code repositories
|
||||
|
||||
#### C1: GitHub Repository Scraping
|
||||
- [ ] **Task C1.1-C1.12:** GitHub API integration and code analysis
|
||||
|
||||
#### C2: Local Codebase Scraping
|
||||
- [ ] **Task C2.1-C2.8:** Local directory analysis and API extraction
|
||||
|
||||
#### C3: Code Pattern Recognition
|
||||
- [x] **Task C3.1:** Detect common patterns (singleton, factory, etc.) ✅ **v2.6.0**
|
||||
- 10 GoF patterns, 9 languages, 87% precision
|
||||
- [x] **Task C3.2:** Extract usage examples from test files ✅ **v2.6.0**
|
||||
- 5 categories, 9 languages, 80%+ high-confidence examples
|
||||
- [ ] **Task C3.3:** Build "how to" guides from code
|
||||
- [ ] **Task C3.4:** Extract configuration patterns
|
||||
- [ ] **Task C3.5:** Create architectural overview
|
||||
- [x] **Task C3.6:** AI Enhancement for Pattern Detection ✅ **v2.6.0**
|
||||
- Claude API integration for enhanced insights
|
||||
- [x] **Task C3.7:** Architectural Pattern Detection ✅ **v2.6.0**
|
||||
- Detects 8 architectural patterns, framework-aware
|
||||
|
||||
**Start Next:** Pick C3.3 (build guides from workflow examples)
|
||||
|
||||
---
|
||||
|
||||
### 🔌 **Category D: Context7 Integration**
|
||||
- [ ] **Task D1.1-D1.4:** Research and planning
|
||||
- [ ] **Task D2.1-D2.5:** Basic integration
|
||||
|
||||
---
|
||||
|
||||
### 🚀 **Category E: MCP Enhancements**
|
||||
Small improvements to existing MCP tools
|
||||
|
||||
#### E1: New MCP Tools
|
||||
- [x] **Task E1.3:** Add `scrape_pdf` MCP tool ✅
|
||||
- [ ] **Task E1.1:** Add `fetch_config` MCP tool
|
||||
- [ ] **Task E1.2:** Add `fetch_knowledge` MCP tool
|
||||
- [ ] **Task E1.4-E1.9:** Additional format scrapers
|
||||
|
||||
#### E2: MCP Quality Improvements
|
||||
- [ ] **Task E2.1:** Add error handling to all tools
|
||||
- [ ] **Task E2.2:** Add structured logging
|
||||
- [ ] **Task E2.3:** Add progress indicators
|
||||
- [ ] **Task E2.4:** Add validation for all inputs
|
||||
- [ ] **Task E2.5:** Add helpful error messages
|
||||
- [x] **Task E2.6:** Add retry logic for network failures ✅ **Utilities ready**
|
||||
|
||||
---
|
||||
|
||||
### ⚡ **Category F: Performance & Reliability**
|
||||
Technical improvements to existing features
|
||||
|
||||
#### F1: Core Scraper Improvements
|
||||
- [ ] **Task F1.1:** Add URL normalization
|
||||
- [ ] **Task F1.2:** Add duplicate page detection
|
||||
- [ ] **Task F1.3:** Add memory-efficient streaming
|
||||
- [ ] **Task F1.4:** Add HTML parser fallback
|
||||
- [x] **Task F1.5:** Add network retry with exponential backoff ✅
|
||||
- [ ] **Task F1.6:** Fix package path output bug
|
||||
|
||||
#### F2: Incremental Updates
|
||||
- [ ] **Task F2.1-F2.5:** Track modifications, update only changed content
|
||||
|
||||
---
|
||||
|
||||
### 🎨 **Category G: Tools & Utilities**
|
||||
Small standalone tools that add value
|
||||
|
||||
#### G1: Config Tools
|
||||
- [ ] **Task G1.1:** Create `validate_config.py`
|
||||
- [ ] **Task G1.2:** Create `test_selectors.py`
|
||||
- [ ] **Task G1.3:** Create `auto_detect_selectors.py` (AI-powered)
|
||||
- [ ] **Task G1.4:** Create `compare_configs.py`
|
||||
- [ ] **Task G1.5:** Create `optimize_config.py`
|
||||
|
||||
#### G2: Skill Quality Tools
|
||||
- [ ] **Task G2.1-G2.5:** Quality analysis and reporting
|
||||
|
||||
---
|
||||
|
||||
### 📚 **Category H: Community Response**
|
||||
- [ ] **Task H1.1-H1.5:** Address open GitHub issues
|
||||
|
||||
---
|
||||
|
||||
### 🎓 **Category I: Content & Documentation**
|
||||
- [ ] **Task I1.1-I1.6:** Video tutorials
|
||||
- [ ] **Task I2.1-I2.5:** Written guides
|
||||
|
||||
---
|
||||
|
||||
### 🧪 **Category J: Testing & Quality**
|
||||
- [ ] **Task J1.1-J1.6:** Test expansion and coverage
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Recommended Starting Tasks
|
||||
|
||||
### Quick Wins (1-2 hours each):
|
||||
1. **H1.1** - Respond to Issue #8
|
||||
2. **J1.1** - Install MCP package
|
||||
3. **A3.1** - Create GitHub Pages site
|
||||
4. **B1.1** - Research PDF parsing
|
||||
5. **F1.1** - Add URL normalization
|
||||
|
||||
### Medium Tasks (3-5 hours each):
|
||||
6. ✅ **A1.1** - JSON API for configs (COMPLETE)
|
||||
7. **G1.1** - Config validator script
|
||||
8. **C1.1** - GitHub API client
|
||||
9. **I1.1** - Video script writing
|
||||
10. **E2.1** - Error handling for MCP tools
|
||||
|
||||
---
|
||||
|
||||
## 📊 Release History
|
||||
|
||||
### ✅ v2.6.0 - C3.x Codebase Analysis Suite (January 14, 2026)
|
||||
**Focus:** Complete codebase analysis with multi-platform support
|
||||
|
||||
**Completed Features:**
|
||||
- C3.x suite (C3.1-C3.8): Pattern detection, test extraction, architecture analysis
|
||||
- Multi-platform support: Claude, Gemini, OpenAI, Markdown
|
||||
- Platform adaptor architecture
|
||||
- 18 MCP tools (up from 9)
|
||||
- 700+ tests passing
|
||||
- Unified multi-source scraping maturity
|
||||
|
||||
### ✅ v2.1.0 - Test Coverage & Quality (November 29, 2025)
|
||||
**Focus:** Test coverage and unified scraping
|
||||
|
||||
**Completed Features:**
|
||||
- Fixed 12 unified scraping tests
|
||||
- GitHub repository scraping with unlimited local analysis
|
||||
- PDF extraction and conversion
|
||||
- 427 tests passing
|
||||
|
||||
### ✅ v1.0.0 - Production Release (October 19, 2025)
|
||||
**First stable release**
|
||||
|
||||
**Core Features:**
|
||||
- Documentation scraping with BFS
|
||||
- Smart categorization
|
||||
- Language detection
|
||||
- Pattern extraction
|
||||
- 12 preset configurations
|
||||
- 14/14 tests passing (100%)
|
||||
- ~3,800 lines of code
|
||||
- Complete documentation suite
|
||||
- MCP server with 9 tools
|
||||
- Large documentation support (40K+ pages)
|
||||
- Auto-upload functionality
|
||||
|
||||
---
|
||||
|
||||
## 📋 Task Categories (Flexible Development)
|
||||
## 📅 Release Planning
|
||||
|
||||
See [FLEXIBLE_ROADMAP.md](FLEXIBLE_ROADMAP.md) for detailed task breakdown.
|
||||
### Release: v2.7.0 (Estimated: February 2026)
|
||||
**Focus:** Router Quality Improvements & Multi-Source Maturity
|
||||
|
||||
### Category Summary:
|
||||
- **🌐 Community & Sharing** - Config/knowledge sharing website features
|
||||
- **🛠️ New Input Formats** - PDF, Word, Excel, Markdown support
|
||||
- **💻 Codebase Knowledge** - GitHub repos, local code scraping
|
||||
- **🔌 Context7 Integration** - Enhanced context management
|
||||
- **🚀 MCP Enhancements** - New tools and quality improvements
|
||||
- **⚡ Performance & Reliability** - Core improvements
|
||||
- **🎨 Tools & Utilities** - Standalone helper tools
|
||||
- **📚 Community Response** - Address GitHub issues
|
||||
- **🎓 Content & Documentation** - Videos and guides
|
||||
- **🧪 Testing & Quality** - Test coverage expansion
|
||||
**Planned Features:**
|
||||
- Router skill quality improvements
|
||||
- Enhanced multi-source synthesis
|
||||
- Source-parity for all scrapers
|
||||
- AI enhancement improvements
|
||||
- Documentation refinements
|
||||
|
||||
---
|
||||
### Release: v2.8.0 (Estimated: Q1 2026)
|
||||
**Focus:** Web Presence & Community Growth
|
||||
|
||||
### ~~📋 v1.1 - Website Launch (PLANNED)~~ → Now flexible tasks!
|
||||
**Goal:** Create professional website and community presence
|
||||
**Timeline:** November 2025 (Due: Nov 3, 2025)
|
||||
**Planned Features:**
|
||||
- GitHub Pages website (skillseekersweb.com)
|
||||
- Interactive documentation
|
||||
- Config submission workflow
|
||||
- Community showcase
|
||||
- Video tutorials
|
||||
|
||||
**Features:**
|
||||
- Professional landing page (skillseekersweb.com)
|
||||
- Documentation migration to website
|
||||
- Preset showcase gallery (interactive)
|
||||
- Blog with release notes and tutorials
|
||||
- SEO optimization
|
||||
- Analytics integration
|
||||
### Release: v2.9.0 (Estimated: Q2 2026)
|
||||
**Focus:** Developer Experience & Integrations
|
||||
|
||||
**Community:**
|
||||
- Video tutorial series
|
||||
- Contributing guidelines
|
||||
- Issue templates and workflows
|
||||
- GitHub Project board
|
||||
- Community engagement
|
||||
|
||||
---
|
||||
|
||||
### 📋 v1.2 - Core Improvements (PLANNED)
|
||||
**Goal:** Address technical debt and performance
|
||||
**Timeline:** Late November 2025
|
||||
|
||||
**Technical Enhancements:**
|
||||
- URL normalization/deduplication
|
||||
- Memory optimization for large docs
|
||||
- HTML parser fallback (lxml)
|
||||
- Selector validation tool
|
||||
- Incremental update system
|
||||
|
||||
**MCP Enhancements:**
|
||||
- Interactive config wizard via MCP
|
||||
- Real-time progress updates
|
||||
- Auto-detect documentation patterns
|
||||
- Enhanced error handling and logging
|
||||
- Batch operations
|
||||
|
||||
---
|
||||
|
||||
### 📋 v2.0 - Intelligence Layer (PLANNED)
|
||||
**Goal:** Smart defaults and auto-configuration
|
||||
**Timeline:** December 2025
|
||||
|
||||
**Features:**
|
||||
- **Auto-detection:**
|
||||
- Automatically find best selectors
|
||||
- Detect documentation framework (Docusaurus, GitBook, etc.)
|
||||
- Suggest optimal rate_limit and max_pages
|
||||
|
||||
- **Quality Metrics:**
|
||||
- Analyze generated SKILL.md quality
|
||||
- Suggest improvements
|
||||
- Validate code examples
|
||||
|
||||
- **Templates:**
|
||||
- Pre-built configs for popular frameworks
|
||||
- Community config sharing
|
||||
- One-click generation for common docs
|
||||
|
||||
**Example:**
|
||||
```
|
||||
User: "Create skill from https://tailwindcss.com/docs"
|
||||
Tool: Auto-detects Tailwind, uses template, generates in 30 seconds
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 💭 v3.0 - Platform Features (IDEAS)
|
||||
**Goal:** Build ecosystem around skill generation
|
||||
|
||||
**Possible Features:**
|
||||
**Planned Features:**
|
||||
- Web UI for config generation
|
||||
- GitHub Actions integration
|
||||
- CI/CD integration examples
|
||||
- Docker containerization
|
||||
- Enhanced scraping formats (Sphinx, Docusaurus detection)
|
||||
- Performance optimizations
|
||||
|
||||
---
|
||||
|
||||
## 🔮 Long-term Vision (v3.0+)
|
||||
|
||||
### Major Features Under Consideration
|
||||
|
||||
#### Advanced Scraping
|
||||
- Real-time documentation monitoring
|
||||
- Automatic skill updates
|
||||
- Change notifications
|
||||
- Multi-language documentation support
|
||||
|
||||
#### Collaboration
|
||||
- Collaborative skill curation
|
||||
- Shared skill repositories
|
||||
- Community ratings and reviews
|
||||
- Skill marketplace
|
||||
- Analytics dashboard
|
||||
- API for programmatic access
|
||||
|
||||
#### AI & Intelligence
|
||||
- Enhanced AI analysis
|
||||
- Better conflict detection algorithms
|
||||
- Automatic documentation quality scoring
|
||||
- Semantic understanding and natural language queries
|
||||
|
||||
#### Ecosystem
|
||||
- VS Code extension
|
||||
- IntelliJ/PyCharm plugin
|
||||
- Interactive TUI mode
|
||||
- Skill diff and merge tools
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Feature Ideas
|
||||
## 📈 Metrics & Goals
|
||||
|
||||
### High Priority
|
||||
1. **Selector Auto-Detection** - Analyze page, suggest selectors
|
||||
2. **Progress Streaming** - Real-time updates during scraping
|
||||
3. **Config Validation UI** - Visual feedback on config quality
|
||||
4. **Batch Processing** - Handle multiple sites at once
|
||||
### Current State (v2.6.0) ✅
|
||||
- ✅ 24 preset configs (14 official + 10 test/examples)
|
||||
- ✅ 700+ tests (excellent coverage)
|
||||
- ✅ 18 MCP tools
|
||||
- ✅ 4 platform adaptors (Claude, Gemini, OpenAI, Markdown)
|
||||
- ✅ C3.x codebase analysis suite complete
|
||||
- ✅ Multi-source synthesis with conflict detection
|
||||
|
||||
### Medium Priority
|
||||
5. **Skill Quality Score** - Rate generated skills
|
||||
6. **Enhanced SKILL.md** - Better templates, more examples
|
||||
7. **Documentation Framework Detection** - Auto-detect Docusaurus, VuePress, etc.
|
||||
8. **Custom Categories AI** - Use AI to suggest categories
|
||||
|
||||
### Low Priority
|
||||
9. **Web Dashboard** - Browser-based interface
|
||||
10. **Skill Analytics** - Track usage, quality metrics
|
||||
11. **Community Configs** - Share and discover configs
|
||||
12. **Plugin System** - Extend with custom scrapers
|
||||
|
||||
---
|
||||
|
||||
## 🔬 Research Areas
|
||||
|
||||
### MCP Enhancements
|
||||
- [ ] Investigate MCP progress/streaming APIs
|
||||
- [ ] Test MCP with large documentation sites
|
||||
- [ ] Explore MCP caching strategies
|
||||
|
||||
### AI Integration
|
||||
- [ ] Use Claude to auto-generate categories
|
||||
- [ ] AI-powered selector detection
|
||||
- [ ] Quality analysis with LLMs
|
||||
|
||||
### Performance
|
||||
- [ ] Parallel scraping
|
||||
- [ ] Incremental updates
|
||||
- [ ] Smart caching
|
||||
|
||||
---
|
||||
|
||||
## 📊 Metrics & Goals
|
||||
|
||||
### Current State (Oct 20, 2025) ✅
|
||||
- ✅ 12 preset configs (Godot, React, Vue, Django, FastAPI, Tailwind, Kubernetes, Astro, etc.)
|
||||
- ✅ 14/14 tests (100% pass rate)
|
||||
- ✅ 9 MCP tools (fully functional)
|
||||
- ✅ ~3,800 lines of code
|
||||
- ✅ Complete documentation suite
|
||||
- ✅ Production-ready v1.0.0 release
|
||||
- ✅ Auto-upload functionality
|
||||
- ✅ Large documentation support (40K+ pages)
|
||||
|
||||
### Goals for v1.1 (Website Launch)
|
||||
### Goals for v2.7-v2.9
|
||||
- 🎯 Professional website live
|
||||
- 🎯 Video tutorial series (5 videos)
|
||||
- 🎯 20+ GitHub stars
|
||||
- 🎯 Community engagement started
|
||||
- 🎯 Documentation site migration
|
||||
|
||||
### Goals for v1.2 (Core Improvements)
|
||||
- 🎯 Enhanced MCP features
|
||||
- 🎯 Performance optimization
|
||||
- 🎯 Better error handling
|
||||
- 🎯 Incremental update system
|
||||
|
||||
### Goals for v2.0 (Intelligence)
|
||||
- 🎯 50+ preset configs
|
||||
- 🎯 Video tutorial series (5+ videos)
|
||||
- 🎯 100+ GitHub stars
|
||||
- 🎯 Community contributions flowing
|
||||
|
||||
### Goals for v3.0+
|
||||
- 🎯 Auto-detection for 80%+ of sites
|
||||
- 🎯 <1 minute skill generation
|
||||
- 🎯 Community contributions
|
||||
- 🎯 Active community marketplace
|
||||
- 🎯 Quality scoring system
|
||||
- 🎯 Real-time monitoring
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
## 🤝 How to Influence the Roadmap
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for:
|
||||
- How to add new MCP tools
|
||||
- Testing guidelines
|
||||
- Code style
|
||||
- PR process
|
||||
### Priority System
|
||||
|
||||
Features are prioritized based on:
|
||||
1. **User impact** - How many users will benefit?
|
||||
2. **Technical feasibility** - How complex is the implementation?
|
||||
3. **Community interest** - How many upvotes/requests?
|
||||
4. **Strategic alignment** - Does it fit our vision?
|
||||
|
||||
### Ways to Contribute
|
||||
|
||||
1. **Vote on Features** - ⭐ Star feature request issues
|
||||
2. **Contribute Code** - Pick any task from the 136 available
|
||||
3. **Share Feedback** - Open issues, share success stories
|
||||
4. **Help with Documentation** - Write tutorials, improve docs
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
|
||||
|
||||
---
|
||||
|
||||
## 📅 Release Schedule
|
||||
## 🎨 Flexibility Rules
|
||||
|
||||
| Version | Target Date | Status | Focus |
|
||||
|---------|-------------|--------|-------|
|
||||
| v1.0.0 | Oct 19, 2025 | ✅ **RELEASED** | Core CLI + MCP Integration |
|
||||
| v1.1.0 | Nov 3, 2025 | 📋 Planned | Website Launch |
|
||||
| v1.2.0 | Late Nov 2025 | 📋 Planned | Core Improvements |
|
||||
| v2.0.0 | Dec 2025 | 📋 Planned | Intelligence Layer |
|
||||
| v3.0.0 | Q1 2026 | 💭 Ideas | Platform Features |
|
||||
1. **Pick any task, any order** - No rigid dependencies
|
||||
2. **Start small** - Research tasks before implementation
|
||||
3. **One task at a time** - Focus, complete, move on
|
||||
4. **Switch anytime** - Not enjoying it? Pick another!
|
||||
5. **Document as you go** - Each task should update docs
|
||||
6. **Test incrementally** - Each task should have a quick test
|
||||
7. **Ship early** - Don't wait for "complete" features
|
||||
|
||||
---
|
||||
|
||||
## 📊 Progress Tracking
|
||||
|
||||
**Completed Tasks:** 10+ (C3.1, C3.2, C3.6, C3.7, A1.1, A1.2, A1.7, E1.3, E2.6, F1.5)
|
||||
**In Progress:** Router quality improvements (v2.7.0)
|
||||
**Total Available Tasks:** 136
|
||||
|
||||
**No pressure, no deadlines, just progress!** ✨
|
||||
|
||||
---
|
||||
|
||||
@@ -263,4 +412,17 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for:
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** October 20, 2025
|
||||
## 📚 Learn More
|
||||
|
||||
- **Project Board**: https://github.com/users/yusufkaraaslan/projects/2
|
||||
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
|
||||
- **Contributing**: [CONTRIBUTING.md](CONTRIBUTING.md)
|
||||
- **Discussions**: https://github.com/yusufkaraaslan/Skill_Seekers/discussions
|
||||
- **Issues**: https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** January 14, 2026
|
||||
**Philosophy:** Small steps → Consistent progress → Compound results
|
||||
|
||||
**Together, we're building the future of documentation-to-AI skill conversion!** 🚀
|
||||
|
||||
@@ -80,9 +80,7 @@ Historical documentation and completed features:
|
||||
|
||||
Want to contribute? See:
|
||||
- [Contributing Guide](../CONTRIBUTING.md) - Contribution guidelines
|
||||
- [Roadmap](../ROADMAP.md) - Project roadmap
|
||||
- [Flexible Roadmap](../FLEXIBLE_ROADMAP.md) - Detailed task list (134 tasks)
|
||||
- [Future Releases](../FUTURE_RELEASES.md) - Planned features
|
||||
- [Roadmap](../ROADMAP.md) - Comprehensive roadmap with 136 tasks
|
||||
|
||||
## 📝 Changelog
|
||||
|
||||
|
||||
@@ -1,94 +0,0 @@
|
||||
# Terminal Selection Guide
|
||||
|
||||
When using `--enhance-local`, Skill Seeker opens a new terminal window to run Claude Code. This guide explains how to control which terminal app is used.
|
||||
|
||||
## Priority Order
|
||||
|
||||
The script automatically detects which terminal to use in this order:
|
||||
|
||||
1. **`SKILL_SEEKER_TERMINAL` environment variable** (highest priority)
|
||||
2. **`TERM_PROGRAM` environment variable** (inherit current terminal)
|
||||
3. **Terminal.app** (fallback default)
|
||||
|
||||
## Setting Your Preferred Terminal
|
||||
|
||||
### Option 1: Set Environment Variable (Recommended)
|
||||
|
||||
Add this to your shell config (`~/.zshrc` or `~/.bashrc`):
|
||||
|
||||
```bash
|
||||
# For Ghostty users
|
||||
export SKILL_SEEKER_TERMINAL="Ghostty"
|
||||
|
||||
# For iTerm users
|
||||
export SKILL_SEEKER_TERMINAL="iTerm"
|
||||
|
||||
# For WezTerm users
|
||||
export SKILL_SEEKER_TERMINAL="WezTerm"
|
||||
```
|
||||
|
||||
Then reload your shell:
|
||||
```bash
|
||||
source ~/.zshrc # or source ~/.bashrc
|
||||
```
|
||||
|
||||
### Option 2: Set Per-Session
|
||||
|
||||
Set the variable before running the command:
|
||||
|
||||
```bash
|
||||
SKILL_SEEKER_TERMINAL="Ghostty" python3 cli/doc_scraper.py --config configs/react.json --enhance-local
|
||||
```
|
||||
|
||||
### Option 3: Inherit Current Terminal (Automatic)
|
||||
|
||||
If you run the script from Ghostty, iTerm2, or WezTerm, it will automatically open the enhancement in the same terminal app.
|
||||
|
||||
**Note:** IDE terminals (VS Code, Zed, JetBrains) use unique `TERM_PROGRAM` values, so they fall back to Terminal.app unless you set `SKILL_SEEKER_TERMINAL`.
|
||||
|
||||
## Supported Terminals
|
||||
|
||||
- **Ghostty** (`ghostty`)
|
||||
- **iTerm2** (`iTerm.app`)
|
||||
- **Terminal.app** (`Apple_Terminal`)
|
||||
- **WezTerm** (`WezTerm`)
|
||||
|
||||
## Example Output
|
||||
|
||||
When terminal detection works:
|
||||
```
|
||||
🚀 Launching Claude Code in new terminal...
|
||||
Using terminal: Ghostty (from SKILL_SEEKER_TERMINAL)
|
||||
```
|
||||
|
||||
When running from an IDE terminal:
|
||||
```
|
||||
🚀 Launching Claude Code in new terminal...
|
||||
⚠️ unknown TERM_PROGRAM (zed)
|
||||
→ Using Terminal.app as fallback
|
||||
```
|
||||
|
||||
**Tip:** Set `SKILL_SEEKER_TERMINAL` to avoid the fallback behavior.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Q: The wrong terminal opens even though I set `SKILL_SEEKER_TERMINAL`**
|
||||
|
||||
A: Make sure you reloaded your shell after editing `~/.zshrc`:
|
||||
```bash
|
||||
source ~/.zshrc
|
||||
```
|
||||
|
||||
**Q: I want to use a different terminal temporarily**
|
||||
|
||||
A: Set the variable inline:
|
||||
```bash
|
||||
SKILL_SEEKER_TERMINAL="iTerm" python3 cli/doc_scraper.py --enhance-local ...
|
||||
```
|
||||
|
||||
**Q: Can I use a custom terminal app?**
|
||||
|
||||
A: Yes! Just use the app name as it appears in `/Applications/`:
|
||||
```bash
|
||||
export SKILL_SEEKER_TERMINAL="Alacritty"
|
||||
```
|
||||
@@ -1,716 +0,0 @@
|
||||
# Testing Guide for Skill Seeker
|
||||
|
||||
Comprehensive testing documentation for the Skill Seeker project.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python3 run_tests.py
|
||||
|
||||
# Run all tests with verbose output
|
||||
python3 run_tests.py -v
|
||||
|
||||
# Run specific test suite
|
||||
python3 run_tests.py --suite config
|
||||
python3 run_tests.py --suite features
|
||||
python3 run_tests.py --suite integration
|
||||
|
||||
# Stop on first failure
|
||||
python3 run_tests.py --failfast
|
||||
|
||||
# List all available tests
|
||||
python3 run_tests.py --list
|
||||
```
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── __init__.py # Test package marker
|
||||
├── test_config_validation.py # Config validation tests (30+ tests)
|
||||
├── test_scraper_features.py # Core feature tests (25+ tests)
|
||||
├── test_integration.py # Integration tests (15+ tests)
|
||||
├── test_pdf_extractor.py # PDF extraction tests (23 tests)
|
||||
├── test_pdf_scraper.py # PDF workflow tests (18 tests)
|
||||
└── test_pdf_advanced_features.py # PDF advanced features (26 tests) NEW
|
||||
```
|
||||
|
||||
## Test Suites
|
||||
|
||||
### 1. Config Validation Tests (`test_config_validation.py`)
|
||||
|
||||
Tests the `validate_config()` function with comprehensive coverage.
|
||||
|
||||
**Test Categories:**
|
||||
- ✅ Valid configurations (minimal and complete)
|
||||
- ✅ Missing required fields (`name`, `base_url`)
|
||||
- ✅ Invalid name formats (special characters)
|
||||
- ✅ Valid name formats (alphanumeric, hyphens, underscores)
|
||||
- ✅ Invalid URLs (missing protocol)
|
||||
- ✅ Valid URL protocols (http, https)
|
||||
- ✅ Selector validation (structure and recommended fields)
|
||||
- ✅ URL patterns validation (include/exclude lists)
|
||||
- ✅ Categories validation (structure and keywords)
|
||||
- ✅ Rate limit validation (range 0-10, type checking)
|
||||
- ✅ Max pages validation (range 1-10000, type checking)
|
||||
- ✅ Start URLs validation (format and protocol)
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_valid_complete_config(self):
|
||||
"""Test valid complete configuration"""
|
||||
config = {
|
||||
'name': 'godot',
|
||||
'base_url': 'https://docs.godotengine.org/en/stable/',
|
||||
'selectors': {
|
||||
'main_content': 'div[role="main"]',
|
||||
'title': 'title',
|
||||
'code_blocks': 'pre code'
|
||||
},
|
||||
'rate_limit': 0.5,
|
||||
'max_pages': 500
|
||||
}
|
||||
errors = validate_config(config)
|
||||
self.assertEqual(len(errors), 0)
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 run_tests.py --suite config -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Scraper Features Tests (`test_scraper_features.py`)
|
||||
|
||||
Tests core scraper functionality including URL validation, language detection, pattern extraction, and categorization.
|
||||
|
||||
**Test Categories:**
|
||||
|
||||
**URL Validation:**
|
||||
- ✅ URL matching include patterns
|
||||
- ✅ URL matching exclude patterns
|
||||
- ✅ Different domain rejection
|
||||
- ✅ No pattern configuration
|
||||
|
||||
**Language Detection:**
|
||||
- ✅ Detection from CSS classes (`language-*`, `lang-*`)
|
||||
- ✅ Detection from parent elements
|
||||
- ✅ Python detection (import, from, def)
|
||||
- ✅ JavaScript detection (const, let, arrow functions)
|
||||
- ✅ GDScript detection (func, var)
|
||||
- ✅ C++ detection (#include, int main)
|
||||
- ✅ Unknown language fallback
|
||||
|
||||
**Pattern Extraction:**
|
||||
- ✅ Extraction with "Example:" marker
|
||||
- ✅ Extraction with "Usage:" marker
|
||||
- ✅ Pattern limit (max 5)
|
||||
|
||||
**Categorization:**
|
||||
- ✅ Categorization by URL keywords
|
||||
- ✅ Categorization by title keywords
|
||||
- ✅ Categorization by content keywords
|
||||
- ✅ Fallback to "other" category
|
||||
- ✅ Empty category removal
|
||||
|
||||
**Text Cleaning:**
|
||||
- ✅ Multiple spaces normalization
|
||||
- ✅ Newline normalization
|
||||
- ✅ Tab normalization
|
||||
- ✅ Whitespace stripping
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_detect_python_from_heuristics(self):
|
||||
"""Test Python detection from code content"""
|
||||
html = '<code>import os\nfrom pathlib import Path</code>'
|
||||
elem = BeautifulSoup(html, 'html.parser').find('code')
|
||||
lang = self.converter.detect_language(elem, elem.get_text())
|
||||
self.assertEqual(lang, 'python')
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 run_tests.py --suite features -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Integration Tests (`test_integration.py`)
|
||||
|
||||
Tests complete workflows and interactions between components.
|
||||
|
||||
**Test Categories:**
|
||||
|
||||
**Dry-Run Mode:**
|
||||
- ✅ No directories created in dry-run mode
|
||||
- ✅ Dry-run flag properly set
|
||||
- ✅ Normal mode creates directories
|
||||
|
||||
**Config Loading:**
|
||||
- ✅ Load valid configuration files
|
||||
- ✅ Invalid JSON error handling
|
||||
- ✅ Nonexistent file error handling
|
||||
- ✅ Validation errors during load
|
||||
|
||||
**Real Config Validation:**
|
||||
- ✅ Godot config validation
|
||||
- ✅ React config validation
|
||||
- ✅ Vue config validation
|
||||
- ✅ Django config validation
|
||||
- ✅ FastAPI config validation
|
||||
- ✅ Steam Economy config validation
|
||||
|
||||
**URL Processing:**
|
||||
- ✅ URL normalization
|
||||
- ✅ Start URLs fallback to base_url
|
||||
- ✅ Multiple start URLs handling
|
||||
|
||||
**Content Extraction:**
|
||||
- ✅ Empty content handling
|
||||
- ✅ Basic content extraction
|
||||
- ✅ Code sample extraction with language detection
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_dry_run_no_directories_created(self):
|
||||
"""Test that dry-run mode doesn't create directories"""
|
||||
converter = DocToSkillConverter(self.config, dry_run=True)
|
||||
|
||||
data_dir = Path(f"output/{self.config['name']}_data")
|
||||
skill_dir = Path(f"output/{self.config['name']}")
|
||||
|
||||
self.assertFalse(data_dir.exists())
|
||||
self.assertFalse(skill_dir.exists())
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 run_tests.py --suite integration -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. PDF Extraction Tests (`test_pdf_extractor.py`) **NEW**
|
||||
|
||||
Tests PDF content extraction functionality (B1.2-B1.5).
|
||||
|
||||
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). They will be skipped if not installed.
|
||||
|
||||
**Test Categories:**
|
||||
|
||||
**Language Detection (5 tests):**
|
||||
- ✅ Python detection with confidence scoring
|
||||
- ✅ JavaScript detection with confidence
|
||||
- ✅ C++ detection with confidence
|
||||
- ✅ Unknown language returns low confidence
|
||||
- ✅ Confidence always between 0 and 1
|
||||
|
||||
**Syntax Validation (5 tests):**
|
||||
- ✅ Valid Python syntax validation
|
||||
- ✅ Invalid Python indentation detection
|
||||
- ✅ Unbalanced brackets detection
|
||||
- ✅ Valid JavaScript syntax validation
|
||||
- ✅ Natural language fails validation
|
||||
|
||||
**Quality Scoring (4 tests):**
|
||||
- ✅ Quality score between 0 and 10
|
||||
- ✅ High-quality code gets good score (>7)
|
||||
- ✅ Low-quality code gets low score (<4)
|
||||
- ✅ Quality considers multiple factors
|
||||
|
||||
**Chapter Detection (4 tests):**
|
||||
- ✅ Detect chapters with numbers
|
||||
- ✅ Detect uppercase chapter headers
|
||||
- ✅ Detect section headings (e.g., "2.1")
|
||||
- ✅ Normal text not detected as chapter
|
||||
|
||||
**Code Block Merging (2 tests):**
|
||||
- ✅ Merge code blocks split across pages
|
||||
- ✅ Don't merge different languages
|
||||
|
||||
**Code Detection Methods (2 tests):**
|
||||
- ✅ Pattern-based detection (keywords)
|
||||
- ✅ Indent-based detection
|
||||
|
||||
**Quality Filtering (1 test):**
|
||||
- ✅ Filter by minimum quality threshold
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_detect_python_with_confidence(self):
|
||||
"""Test Python detection returns language and confidence"""
|
||||
extractor = self.PDFExtractor.__new__(self.PDFExtractor)
|
||||
code = "def hello():\n print('world')\n return True"
|
||||
|
||||
language, confidence = extractor.detect_language_from_code(code)
|
||||
|
||||
self.assertEqual(language, "python")
|
||||
self.assertGreater(confidence, 0.7)
|
||||
self.assertLessEqual(confidence, 1.0)
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 -m pytest tests/test_pdf_extractor.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. PDF Workflow Tests (`test_pdf_scraper.py`) **NEW**
|
||||
|
||||
Tests PDF to skill conversion workflow (B1.6).
|
||||
|
||||
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). They will be skipped if not installed.
|
||||
|
||||
**Test Categories:**
|
||||
|
||||
**PDFToSkillConverter (3 tests):**
|
||||
- ✅ Initialization with name and PDF path
|
||||
- ✅ Initialization with config file
|
||||
- ✅ Requires name or config_path
|
||||
|
||||
**Categorization (3 tests):**
|
||||
- ✅ Categorize by keywords
|
||||
- ✅ Categorize by chapters
|
||||
- ✅ Handle missing chapters
|
||||
|
||||
**Skill Building (3 tests):**
|
||||
- ✅ Create required directory structure
|
||||
- ✅ Create SKILL.md with metadata
|
||||
- ✅ Create reference files for categories
|
||||
|
||||
**Code Block Handling (2 tests):**
|
||||
- ✅ Include code blocks in references
|
||||
- ✅ Prefer high-quality code
|
||||
|
||||
**Image Handling (2 tests):**
|
||||
- ✅ Save images to assets directory
|
||||
- ✅ Reference images in markdown
|
||||
|
||||
**Error Handling (3 tests):**
|
||||
- ✅ Handle missing PDF files
|
||||
- ✅ Handle invalid config JSON
|
||||
- ✅ Handle missing required config fields
|
||||
|
||||
**JSON Workflow (2 tests):**
|
||||
- ✅ Load from extracted JSON
|
||||
- ✅ Build from JSON without extraction
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_build_skill_creates_structure(self):
|
||||
"""Test that build_skill creates required directory structure"""
|
||||
converter = self.PDFToSkillConverter(
|
||||
name="test_skill",
|
||||
pdf_path="test.pdf",
|
||||
output_dir=self.temp_dir
|
||||
)
|
||||
|
||||
converter.extracted_data = {
|
||||
"pages": [{"page_number": 1, "text": "Test", "code_blocks": [], "images": []}],
|
||||
"total_pages": 1
|
||||
}
|
||||
converter.categories = {"test": [converter.extracted_data["pages"][0]]}
|
||||
|
||||
converter.build_skill()
|
||||
|
||||
skill_dir = Path(self.temp_dir) / "test_skill"
|
||||
self.assertTrue(skill_dir.exists())
|
||||
self.assertTrue((skill_dir / "references").exists())
|
||||
self.assertTrue((skill_dir / "scripts").exists())
|
||||
self.assertTrue((skill_dir / "assets").exists())
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 -m pytest tests/test_pdf_scraper.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. PDF Advanced Features Tests (`test_pdf_advanced_features.py`) **NEW**
|
||||
|
||||
Tests advanced PDF features (Priority 2 & 3).
|
||||
|
||||
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). OCR tests also require pytesseract and Pillow. They will be skipped if not installed.
|
||||
|
||||
**Test Categories:**
|
||||
|
||||
**OCR Support (5 tests):**
|
||||
- ✅ OCR flag initialization
|
||||
- ✅ OCR disabled behavior
|
||||
- ✅ OCR only triggers for minimal text
|
||||
- ✅ Warning when pytesseract unavailable
|
||||
- ✅ OCR extraction triggered correctly
|
||||
|
||||
**Password Protection (4 tests):**
|
||||
- ✅ Password parameter initialization
|
||||
- ✅ Encrypted PDF detection
|
||||
- ✅ Wrong password handling
|
||||
- ✅ Missing password error
|
||||
|
||||
**Table Extraction (5 tests):**
|
||||
- ✅ Table extraction flag initialization
|
||||
- ✅ No extraction when disabled
|
||||
- ✅ Basic table extraction
|
||||
- ✅ Multiple tables per page
|
||||
- ✅ Error handling during extraction
|
||||
|
||||
**Caching (5 tests):**
|
||||
- ✅ Cache initialization
|
||||
- ✅ Set and get cached values
|
||||
- ✅ Cache miss returns None
|
||||
- ✅ Caching can be disabled
|
||||
- ✅ Cache overwrite
|
||||
|
||||
**Parallel Processing (4 tests):**
|
||||
- ✅ Parallel flag initialization
|
||||
- ✅ Disabled by default
|
||||
- ✅ Worker count auto-detection
|
||||
- ✅ Custom worker count
|
||||
|
||||
**Integration (3 tests):**
|
||||
- ✅ Full initialization with all features
|
||||
- ✅ Various feature combinations
|
||||
- ✅ Page data includes tables
|
||||
|
||||
**Example Test:**
|
||||
```python
|
||||
def test_table_extraction_basic(self):
|
||||
"""Test basic table extraction"""
|
||||
extractor = self.PDFExtractor.__new__(self.PDFExtractor)
|
||||
extractor.extract_tables = True
|
||||
extractor.verbose = False
|
||||
|
||||
# Create mock table
|
||||
mock_table = Mock()
|
||||
mock_table.extract.return_value = [
|
||||
["Header 1", "Header 2", "Header 3"],
|
||||
["Data 1", "Data 2", "Data 3"]
|
||||
]
|
||||
mock_table.bbox = (0, 0, 100, 100)
|
||||
|
||||
mock_tables = Mock()
|
||||
mock_tables.tables = [mock_table]
|
||||
|
||||
mock_page = Mock()
|
||||
mock_page.find_tables.return_value = mock_tables
|
||||
|
||||
tables = extractor.extract_tables_from_page(mock_page)
|
||||
|
||||
self.assertEqual(len(tables), 1)
|
||||
self.assertEqual(tables[0]['row_count'], 2)
|
||||
self.assertEqual(tables[0]['col_count'], 3)
|
||||
```
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
python3 -m pytest tests/test_pdf_advanced_features.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Runner Features
|
||||
|
||||
The custom test runner (`run_tests.py`) provides:
|
||||
|
||||
### Colored Output
|
||||
- 🟢 Green for passing tests
|
||||
- 🔴 Red for failures and errors
|
||||
- 🟡 Yellow for skipped tests
|
||||
|
||||
### Detailed Summary
|
||||
```
|
||||
======================================================================
|
||||
TEST SUMMARY
|
||||
======================================================================
|
||||
|
||||
Total Tests: 70
|
||||
✓ Passed: 68
|
||||
✗ Failed: 2
|
||||
⊘ Skipped: 0
|
||||
|
||||
Success Rate: 97.1%
|
||||
|
||||
Test Breakdown by Category:
|
||||
TestConfigValidation: 28/30 passed
|
||||
TestURLValidation: 6/6 passed
|
||||
TestLanguageDetection: 10/10 passed
|
||||
TestPatternExtraction: 3/3 passed
|
||||
TestCategorization: 5/5 passed
|
||||
TestDryRunMode: 3/3 passed
|
||||
TestConfigLoading: 4/4 passed
|
||||
TestRealConfigFiles: 6/6 passed
|
||||
TestContentExtraction: 3/3 passed
|
||||
|
||||
======================================================================
|
||||
```
|
||||
|
||||
### Command-Line Options
|
||||
|
||||
```bash
|
||||
# Verbose output (show each test name)
|
||||
python3 run_tests.py -v
|
||||
|
||||
# Quiet output (minimal)
|
||||
python3 run_tests.py -q
|
||||
|
||||
# Stop on first failure
|
||||
python3 run_tests.py --failfast
|
||||
|
||||
# Run specific suite
|
||||
python3 run_tests.py --suite config
|
||||
|
||||
# List all tests
|
||||
python3 run_tests.py --list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running Individual Tests
|
||||
|
||||
### Run Single Test File
|
||||
```bash
|
||||
python3 -m unittest tests.test_config_validation
|
||||
python3 -m unittest tests.test_scraper_features
|
||||
python3 -m unittest tests.test_integration
|
||||
```
|
||||
|
||||
### Run Single Test Class
|
||||
```bash
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation
|
||||
python3 -m unittest tests.test_scraper_features.TestLanguageDetection
|
||||
```
|
||||
|
||||
### Run Single Test Method
|
||||
```bash
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_valid_complete_config
|
||||
python3 -m unittest tests.test_scraper_features.TestLanguageDetection.test_detect_python_from_heuristics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### Current Coverage
|
||||
|
||||
| Component | Tests | Coverage |
|
||||
|-----------|-------|----------|
|
||||
| Config Validation | 30+ | 100% |
|
||||
| URL Validation | 6 | 95% |
|
||||
| Language Detection | 10 | 90% |
|
||||
| Pattern Extraction | 3 | 85% |
|
||||
| Categorization | 5 | 90% |
|
||||
| Text Cleaning | 4 | 100% |
|
||||
| Dry-Run Mode | 3 | 100% |
|
||||
| Config Loading | 4 | 95% |
|
||||
| Real Configs | 6 | 100% |
|
||||
| Content Extraction | 3 | 80% |
|
||||
| **PDF Extraction** | **23** | **90%** |
|
||||
| **PDF Workflow** | **18** | **85%** |
|
||||
| **PDF Advanced Features** | **26** | **95%** |
|
||||
|
||||
**Total: 142 tests (75 passing + 67 PDF tests)**
|
||||
|
||||
**Note:** PDF tests (67 total) require PyMuPDF and will be skipped if not installed. When PyMuPDF is available, all 142 tests run.
|
||||
|
||||
### Not Yet Covered
|
||||
- Network operations (actual scraping)
|
||||
- Enhancement scripts (`enhance_skill.py`, `enhance_skill_local.py`)
|
||||
- Package creation (`package_skill.py`)
|
||||
- Interactive mode
|
||||
- SKILL.md generation
|
||||
- Reference file creation
|
||||
- PDF extraction with real PDF files (tests use mocked data)
|
||||
|
||||
---
|
||||
|
||||
## Writing New Tests
|
||||
|
||||
### Test Template
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test suite for [feature name]
|
||||
Tests [description of what's being tested]
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import unittest
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from doc_scraper import DocToSkillConverter
|
||||
|
||||
|
||||
class TestYourFeature(unittest.TestCase):
|
||||
"""Test [feature] functionality"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test fixtures"""
|
||||
self.config = {
|
||||
'name': 'test',
|
||||
'base_url': 'https://example.com/',
|
||||
'selectors': {
|
||||
'main_content': 'article',
|
||||
'title': 'h1',
|
||||
'code_blocks': 'pre code'
|
||||
},
|
||||
'rate_limit': 0.1,
|
||||
'max_pages': 10
|
||||
}
|
||||
self.converter = DocToSkillConverter(self.config, dry_run=True)
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up after tests"""
|
||||
pass
|
||||
|
||||
def test_your_feature(self):
|
||||
"""Test description"""
|
||||
# Arrange
|
||||
test_input = "something"
|
||||
|
||||
# Act
|
||||
result = self.converter.some_method(test_input)
|
||||
|
||||
# Assert
|
||||
self.assertEqual(result, expected_value)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use descriptive test names**: `test_valid_name_formats` not `test1`
|
||||
2. **Follow AAA pattern**: Arrange, Act, Assert
|
||||
3. **One assertion per test** when possible
|
||||
4. **Test edge cases**: empty inputs, invalid inputs, boundary values
|
||||
5. **Use setUp/tearDown**: for common initialization and cleanup
|
||||
6. **Mock external dependencies**: don't make real network calls
|
||||
7. **Keep tests independent**: tests should not depend on each other
|
||||
8. **Use dry_run=True**: for converter tests to avoid file creation
|
||||
|
||||
---
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
### GitHub Actions (Future)
|
||||
|
||||
```yaml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.7'
|
||||
- run: pip install requests beautifulsoup4
|
||||
- run: python3 run_tests.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Fail with Import Errors
|
||||
```bash
|
||||
# Make sure you're in the repository root
|
||||
cd /path/to/Skill_Seekers
|
||||
|
||||
# Run tests from root directory
|
||||
python3 run_tests.py
|
||||
```
|
||||
|
||||
### Tests Create Output Directories
|
||||
```bash
|
||||
# Clean up test artifacts
|
||||
rm -rf output/test-*
|
||||
|
||||
# Make sure tests use dry_run=True
|
||||
# Check test setUp methods
|
||||
```
|
||||
|
||||
### Specific Test Keeps Failing
|
||||
```bash
|
||||
# Run only that test with verbose output
|
||||
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_name -v
|
||||
|
||||
# Check the error message carefully
|
||||
# Verify test expectations match implementation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
Test execution times:
|
||||
- **Config Validation**: ~0.1 seconds (30 tests)
|
||||
- **Scraper Features**: ~0.3 seconds (25 tests)
|
||||
- **Integration Tests**: ~0.5 seconds (15 tests)
|
||||
- **Total**: ~1 second (70 tests)
|
||||
|
||||
---
|
||||
|
||||
## Contributing Tests
|
||||
|
||||
When adding new features:
|
||||
|
||||
1. Write tests **before** implementing the feature (TDD)
|
||||
2. Ensure tests cover:
|
||||
- ✅ Happy path (valid inputs)
|
||||
- ✅ Edge cases (empty, null, boundary values)
|
||||
- ✅ Error cases (invalid inputs)
|
||||
3. Run tests before committing:
|
||||
```bash
|
||||
python3 run_tests.py
|
||||
```
|
||||
4. Aim for >80% coverage for new code
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- **unittest documentation**: https://docs.python.org/3/library/unittest.html
|
||||
- **pytest** (alternative): https://pytest.org/ (more powerful, but requires installation)
|
||||
- **Test-Driven Development**: https://en.wikipedia.org/wiki/Test-driven_development
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **142 comprehensive tests** covering all major features (75 + 67 PDF)
|
||||
✅ **PDF support testing** with 67 tests for B1 tasks + Priority 2 & 3
|
||||
✅ **Colored test runner** with detailed summaries
|
||||
✅ **Fast execution** (~1 second for full suite)
|
||||
✅ **Easy to extend** with clear patterns and templates
|
||||
✅ **Good coverage** of critical paths
|
||||
|
||||
**PDF Tests Status:**
|
||||
- 23 tests for PDF extraction (language detection, syntax validation, quality scoring, chapter detection)
|
||||
- 18 tests for PDF workflow (initialization, categorization, skill building, code/image handling)
|
||||
- **26 tests for advanced features (OCR, passwords, tables, parallel, caching)** NEW!
|
||||
- Tests are skipped gracefully when PyMuPDF is not installed
|
||||
- Full test coverage when PyMuPDF + optional dependencies are available
|
||||
|
||||
**Advanced PDF Features Tested:**
|
||||
- ✅ OCR support for scanned PDFs (5 tests)
|
||||
- ✅ Password-protected PDFs (4 tests)
|
||||
- ✅ Table extraction (5 tests)
|
||||
- ✅ Parallel processing (4 tests)
|
||||
- ✅ Caching (5 tests)
|
||||
- ✅ Integration (3 tests)
|
||||
|
||||
Run tests frequently to catch bugs early! 🚀
|
||||
Reference in New Issue
Block a user