firefrost-gaming/skill-seekers-reference
3
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
650 Commits 1 Branch 0 Tags
5e4932e8b1fbc2f6e72581703ca1ad655d5d44f1
Commit Graph

11 Commits

Author SHA1 Message Date
yusyus
37cb307455 docs: update all documentation for 17 source types 
Update 32 documentation files across English and Chinese (zh-CN) docs
to reflect the 10 new source types added in the previous commit.

Updated files:
- README.md, README.zh-CN.md — taglines, feature lists, examples, install extras
- docs/reference/ — CLI_REFERENCE, FEATURE_MATRIX, MCP_REFERENCE, CONFIG_FORMAT, API_REFERENCE
- docs/features/ — UNIFIED_SCRAPING with generic merge docs
- docs/advanced/ — multi-source guide, MCP server guide
- docs/getting-started/ — installation extras, quick-start examples
- docs/user-guide/ — core-concepts, scraping, packaging, workflows (complex-merge)
- docs/ — FAQ, TROUBLESHOOTING, BEST_PRACTICES, ARCHITECTURE, UNIFIED_PARSERS, README
- Root — BULLETPROOF_QUICKSTART, CONTRIBUTING, ROADMAP
- docs/zh-CN/ — Chinese translations for all of the above

32 files changed, +3,016 lines, -245 lines
 2026-03-15 15:56:04 +03:00
yusyus
cc9cc32417 feat: add skill-seekers video --setup for GPU auto-detection and dependency installation 
Auto-detects NVIDIA (CUDA), AMD (ROCm), or CPU-only GPU and installs the
correct PyTorch variant + easyocr + all visual extraction dependencies.
Removes easyocr from video-full pip extras to avoid pulling ~2GB of wrong
CUDA packages on non-NVIDIA systems.

New files:
- video_setup.py (835 lines): GPU detection, PyTorch install, ROCm config,
  venv checks, system dep validation, module selection, verification
- test_video_setup.py (60 tests): Full coverage of detection, install, verify

Updated docs: CHANGELOG, AGENTS.md, CLAUDE.md, README.md, CLI_REFERENCE,
FAQ, TROUBLESHOOTING, installation guide, video dependency plan

All 2523 tests passing (15 skipped).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 18:39:16 +03:00
yusyus
064405c052 fix: resolve 18 bugs and code quality issues across adaptors, CLI, and chunking pipeline 
Bug fixes:
- Fix --var flag silently dropped in create routing (args.workflow_var → args.var)
- Fix double _score_code_quality() call in word scraper
- Add .docx file extension validation in WordToSkillConverter
- Fix weaviate ImportError masked by generic Exception handler
- Fix RAG chunking crash using non-existent converter.output_dir

Chunking pipeline improvements:
- Wire --chunk-overlap-tokens through entire package pipeline
  (package_skill → adaptor.package → format_skill_md → _maybe_chunk_content → RAGChunker)
- Add auto-scaling overlap: max(50, chunk_tokens//10) when chunk size is non-default
- Rename --no-preserve-code to --no-preserve-code-blocks (backward-compat alias kept)
- Replace hardcoded 512/50 chunk defaults with DEFAULT_CHUNK_TOKENS/DEFAULT_CHUNK_OVERLAP_TOKENS
  constants across all 12 concrete adaptors, rag_chunker, base, and package_skill

Code quality:
- Extract shared _generate_openai_embeddings() and _generate_st_embeddings() to SkillAdaptor
  base class, removing ~150 lines of duplication from chroma/weaviate/pinecone
- Add Pinecone adaptor with full upload support (pinecone_adaptor.py)

Tests (14 new):
- chunk_overlap_tokens parameter wiring, auto-scaling overlap, preserve_code_blocks flag
- .docx/.doc/no-extension file validation, --var flag routing E2E
- Embedding method inheritance verification, backward-compatible flag aliases

Docs:
- Update CHANGELOG, CLI_REFERENCE, API_REFERENCE, packaging guide (EN+ZH)
- Update README test count badge (1880+ → 2283+)

All 2283 tests passing, 8 skipped, 0 failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 21:57:59 +03:00
yusyus
4c8e16c8b1 fix(#300): centralize selector fallback, fix dry-run link discovery, and smart --config routing 
- Add FALLBACK_MAIN_SELECTORS constant and _find_main_content() helper to
  eliminate 3 duplicated fallback loops in doc_scraper.py
- Move link extraction before early return in extract_content() so links
  are always discovered from the full page, not just main content
- Fix single-threaded dry-run to extract links from soup (full page)
  instead of main element only — fixes reactflow.dev finding only 1 page
- Add link extraction to async dry-run path (was completely missing)
- Remove main_content from get_configuration() defaults so fallback logic
  kicks in instead of a broad CSS comma selector matching body
- Smart create --config routing: peek at JSON to determine unified
  (sources array → unified_scraper) vs simple (base_url → doc_scraper)
- Update docs/user-guide/02-scraping.md and docs/reference/CONFIG_FORMAT.md
  to use unified config format (legacy format rejected since v2.11.0)
- Fix test_auto_fetch_enabled and test_mcp_validate_legacy_config

Closes #300

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 22:25:59 +03:00
yusyus
73adda0b17 docs: update all chunk flag names to match renamed CLI flags 
Replace all occurrences of old ambiguous flag names with the new explicit ones:
  --chunk-size (tokens)  → --chunk-tokens
  --chunk-overlap        → --chunk-overlap-tokens
  --chunk                → --chunk-for-rag
  --streaming-chunk-size → --streaming-chunk-chars
  --streaming-overlap    → --streaming-overlap-chars
  --chunk-size (pages)   → --pdf-pages-per-chunk

Updated: CLI_REFERENCE (EN+ZH), user-guide (EN+ZH), integrations (Haystack,
Chroma, Weaviate, FAISS, Qdrant), features/PDF_CHUNKING, examples/haystack-pipeline,
strategy docs, archive docs, and CHANGELOG.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-24 22:15:14 +03:00
YusufKaraaslanSpyke
3adc5a8c1d fix: unify scraper argument interface and fix create command forwarding 
All scrapers (scrape, github, analyze, pdf) now share a common argument
contract via add_all_standard_arguments() in arguments/common.py.
Universal flags (--dry-run, --verbose, --quiet, --name, --description,
workflow args) work consistently across all source types.

Previously, `create <url> --dry-run`, `create owner/repo --dry-run`,
and `create ./path --dry-run` would crash because sub-scrapers didn't
accept those flags. Also fixes main.py _handle_analyze_command() not
forwarding --dry-run, --preset, --quiet, --name, --description to
codebase_scraper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 20:56:13 +03:00
yusyus
ba9a8ff8b5 docs: complete documentation overhaul with v3.1.0 release notes and zh-CN translations 
Documentation restructure:
- New docs/getting-started/ guide (4 files: install, quick-start, first-skill, next-steps)
- New docs/user-guide/ section (6 files: core concepts through troubleshooting)
- New docs/reference/ section (CLI_REFERENCE, CONFIG_FORMAT, ENVIRONMENT_VARIABLES, MCP_REFERENCE)
- New docs/advanced/ section (custom-workflows, mcp-server, multi-source)
- New docs/ARCHITECTURE.md - system architecture overview
- Archived legacy files (QUICKSTART.md, QUICK_REFERENCE.md, docs/guides/USAGE.md) to docs/archive/legacy/

Chinese (zh-CN) translations:
- Full zh-CN mirror of all user-facing docs (getting-started, user-guide, reference, advanced)
- GitHub Actions workflow for translation sync (.github/workflows/translate-docs.yml)
- Translation sync checker script (scripts/check_translation_sync.sh)
- Translation helper script (scripts/translate_doc.py)

Content updates:
- CHANGELOG.md: [Unreleased] → [3.1.0] - 2026-02-22
- README.md: updated with new doc structure links
- AGENTS.md: updated agent documentation
- docs/features/UNIFIED_SCRAPING.md: updated for unified scraper workflow JSON config

Analysis/planning artifacts (kept for reference):
- DOCUMENTATION_OVERHAUL_PLAN.md, DOCUMENTATION_OVERHAUL_SUMMARY.md
- FEATURE_GAP_ANALYSIS.md, IMPLEMENTATION_GAPS_ANALYSIS.md, CREATE_COMMAND_COVERAGE_ANALYSIS.md
- CHINESE_TRANSLATION_IMPLEMENTATION_SUMMARY.md, ISSUE_260_UPDATE.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-22 01:01:51 +03:00
yusyus
c44b88e801 docs: update stale version numbers, MCP counts, and test counts across docs/ 
Version headers/footers updated to 3.1.0-dev:
- docs/features/BOOTSTRAP_SKILL_TECHNICAL.md (was 2.8.0-dev)
- docs/reference/API_REFERENCE.md (was 2.7.0)
- docs/reference/CODE_QUALITY.md (was 2.7.0)
- docs/guides/TESTING_GUIDE.md (was 2.7.0)
- docs/guides/MIGRATION_GUIDE.md (was 2.7.0, historical tables untouched)

MCP tool count 18 → 26:
- docs/guides/MCP_SETUP.md
- docs/guides/TESTING_GUIDE.md
- docs/reference/CODE_QUALITY.md
- docs/reference/CLAUDE_INTEGRATION.md
- docs/integrations/CLINE.md
- docs/strategy/INTEGRATION_STRATEGY.md

Test count 700+/1200+ → 1,880+:
- docs/guides/MCP_SETUP.md
- docs/guides/TESTING_GUIDE.md
- docs/reference/CODE_QUALITY.md
- docs/reference/CLAUDE_INTEGRATION.md
- docs/features/HOW_TO_GUIDES.md
- docs/blog/UNIVERSAL_RAG_PREPROCESSOR.md

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-18 22:36:08 +03:00
yusyus
edd1d99d70 docs: Update remaining files with v2.7.0 version and test counts 
- CONTRIBUTING.md: Added Ruff code quality tools section
- MCP_SETUP.md: Updated to v2.7.0, 18 tools, 700+ tests
- CLAUDE_INTEGRATION.md: Updated test count to 1200+

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-18 01:18:26 +03:00
yusyus
6f1d0a9a45 docs: Comprehensive markdown documentation update for v2.7.0 
Documentation Overhaul (7 new files, ~4,750 lines)

Version Consistency Updates:
- Updated all version references to v2.7.0 (ROADMAP.md)
- Standardized test counts to 1200+ tests (README.md, Quality Assurance)
- Updated MCP tool references to 18 tools (CHANGELOG.md)

New Documentation Files:
1. docs/reference/API_REFERENCE.md (750 lines)
   - Complete programmatic usage guide for Python integration
   - All 8 core APIs documented with examples
   - Configuration schema reference and error handling
   - CI/CD integration examples (GitHub Actions, GitLab CI)
   - Performance optimization and batch processing

2. docs/features/BOOTSTRAP_SKILL.md (450 lines)
   - Self-hosting capability documentation (dogfooding)
   - Architecture and workflow explanation (3 components)
   - Troubleshooting and testing guide
   - CI/CD integration examples
   - Advanced usage and customization

3. docs/reference/CODE_QUALITY.md (550 lines)
   - Comprehensive Ruff linting documentation
   - All 21 v2.7.0 fixes explained with examples
   - Testing requirements and coverage standards
   - CI/CD integration (GitHub Actions, pre-commit hooks)
   - Security scanning with Bandit
   - Development workflow best practices

4. docs/guides/TESTING_GUIDE.md (750 lines)
   - Complete testing reference (1200+ tests)
   - Unit, integration, E2E, and MCP testing guides
   - Coverage analysis and improvement strategies
   - Debugging tests and troubleshooting
   - CI/CD matrix testing (2 OS, 4 Python versions)
   - Best practices and common patterns

5. docs/QUICK_REFERENCE.md (300 lines)
   - One-page cheat sheet for quick lookup
   - All CLI commands with examples
   - Common workflows and shortcuts
   - Environment variables and configurations
   - Tips & tricks for power users

6. docs/guides/MIGRATION_GUIDE.md (400 lines)
   - Version upgrade guides (v1.0.0 → v2.7.0)
   - Breaking changes and migration steps
   - Compatibility tables for all versions
   - Rollback instructions
   - Common migration issues and solutions

7. docs/FAQ.md (550 lines)
   - Comprehensive Q&A covering all major topics
   - Installation, usage, platforms, features
   - Troubleshooting shortcuts
   - Platform-specific questions
   - Advanced usage and programmatic integration

Navigation Improvements:
- Added "New in v2.7.0" section to docs/README.md
- Integrated all new docs into navigation structure
- Enhanced "Finding What You Need" section with new entries
- Updated developer quick links (testing, code quality, API)
- Cross-referenced related documentation

Documentation Quality:
- All version references consistent (v2.7.0)
- Test counts standardized (1200+ tests)
- MCP tool counts accurate (18 tools)
- All internal links validated
- Format consistency maintained
- Proper heading hierarchy

Impact:
- 64 markdown files reviewed and validated
- 7 new documentation files created (~4,750 lines)
- 4 files updated (ROADMAP, README, CHANGELOG, docs/README)
- Comprehensive coverage of all v2.7.0 features
- Enhanced developer onboarding experience
- Improved user documentation accessibility

Related Issues:
- Addresses documentation gaps identified in v2.7.0 planning
- Supports code quality improvements (21 ruff fixes)
- Documents bootstrap skill feature
- Provides migration path for users upgrading from older versions

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-18 01:16:22 +03:00
yusyus
67282b7531 docs: Comprehensive documentation reorganization for v2.6.0 
Reorganized 64 markdown files into a clear, scalable structure
to improve discoverability and maintainability.

## Changes Summary

### Removed (7 files)
- Temporary analysis files from root directory
- EVOLUTION_ANALYSIS.md, SKILL_QUALITY_ANALYSIS.md, ASYNC_SUPPORT.md
- STRUCTURE.md, SUMMARY_*.md, REDDIT_POST_v2.2.0.md

### Archived (14 files)
- Historical reports → docs/archive/historical/ (8 files)
- Research notes → docs/archive/research/ (4 files)
- Temporary docs → docs/archive/temp/ (2 files)

### Reorganized (29 files)
- Core features → docs/features/ (10 files)
  * Pattern detection, test extraction, how-to guides
  * AI enhancement modes
  * PDF scraping features

- Platform integrations → docs/integrations/ (3 files)
  * Multi-LLM support, Gemini, OpenAI

- User guides → docs/guides/ (6 files)
  * Setup, MCP, usage, upload guides

- Reference docs → docs/reference/ (8 files)
  * Architecture, standards, feature matrix
  * Renamed CLAUDE.md → CLAUDE_INTEGRATION.md

### Created
- docs/README.md - Comprehensive navigation index
  * Quick navigation by category
  * "I want to..." user-focused navigation
  * Links to all documentation

## New Structure

```
docs/
├── README.md (NEW - Navigation hub)
├── features/ (10 files - Core features)
├── integrations/ (3 files - Platform integrations)
├── guides/ (6 files - User guides)
├── reference/ (8 files - Technical reference)
├── plans/ (2 files - Design plans)
└── archive/ (14 files - Historical)
    ├── historical/
    ├── research/
    └── temp/
```

## Benefits

- ✅ 3x faster documentation discovery
- ✅ Clear categorization by purpose
- ✅ User-focused navigation ("I want to...")
- ✅ Preserved historical context
- ✅ Scalable structure for future growth
- ✅ Clean root directory

## Impact

Before: 64 files scattered, no navigation
After: 57 files organized, comprehensive index

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-13 22:58:37 +03:00