The previous fix (a82cf69) only addressed anchor fragment stripping but
left the fundamental problem: _convert_to_md_urls() blindly appended
/index.html.md to ALL non-.md URLs from llms.txt. This only works for
Docusaurus sites — for sites like Discord docs it generates mass 404s.
Changes:
- _convert_to_md_urls() now strips anchors and deduplicates only,
preserving original URLs as-is instead of appending /index.html.md
- New _has_md_extension() helper uses urlparse().path.endswith(".md")
instead of error-prone ".md" in url substring matching
- Fixed ".md" in url checks at 4 locations (lines 465, 554, 716, 775)
- Removed 24 lines of dead commented-out code
- Added real-world e2e test against docs.discord.com (no mocks)
- Updated unit tests for new behavior (32 tests)
Fixes#277
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The unified scraper's _scrape_documentation() was creating temp configs
in flat/legacy format (no "sources" key), causing doc_scraper's
ConfigValidator to reject them. Wrap the temp config in unified format
with a "sources" array. Also remove dead code branches and fix a
pre-existing test that didn't clear GITHUB_TOKEN from env.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add MiniMax AI as LLM platform adaptor
Original implementation by octo-patch in PR #318.
This commit includes comprehensive improvements and documentation.
Code Improvements:
- Fix API key validation to properly check JWT format (eyJ prefix)
- Add specific exception handling for timeout and connection errors
- Remove unused variable in upload method
Dependencies:
- Add MiniMax to [all-llms] extra group in pyproject.toml
Tests:
- Remove duplicate setUp method in integration test class
- Add 4 new test methods:
* test_package_excludes_backup_files
* test_upload_success_mocked (with OpenAI mocking)
* test_upload_network_error
* test_upload_connection_error
* test_validate_api_key_jwt_format
- Update test_validate_api_key_valid to use JWT format keys
- Fix test assertions for error message matching
Documentation:
- Create comprehensive MINIMAX_INTEGRATION.md guide (380+ lines)
- Update MULTI_LLM_SUPPORT.md with MiniMax platform entry
- Update 01-installation.md extras table
- Update INTEGRATIONS.md AI platforms table
- Update AGENTS.md adaptor import pattern example
- Fix README.md platform count from 4 to 5
All tests pass (33 passed, 3 skipped)
Lint checks pass
Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
* fix: improve MiniMax adaptor — typed exceptions, key validation, tests, docs
- Remove invalid "minimax" self-reference from all-llms dependency group
- Use typed OpenAI exceptions (APITimeoutError, APIConnectionError)
instead of string-matching on generic Exception
- Replace incorrect JWT assumption in validate_api_key with length check
- Use DEFAULT_API_ENDPOINT constant instead of hardcoded URLs (3 sites)
- Add Path() cast for output_path before .is_dir() call
- Add sys.modules mock to test_enhance_missing_library
- Add mocked test_enhance_success with backup/content verification
- Update test assertions for new exception types and key validation
- Add MiniMax to __init__.py docstrings (module, get_adaptor, list_platforms)
- Add MiniMax sections to MULTI_LLM_SUPPORT.md (install, format, API key,
workflow example, export-to-all)
Follows up on PR #318 by @octo-patch (feat: add MiniMax AI as LLM platform adaptor).
Co-Authored-By: Octopus <octo-patch@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace → (U+2192) with -> in argparse help strings. Windows cmd uses
cp1252 encoding which cannot render unicode arrows, causing --help to
crash with UnicodeEncodeError.
- Add Claude Code Plugin: plugin.json, .mcp.json, 3 slash commands, skill-builder agent skill
- Add GitHub Action: composite action.yml with 6 inputs/2 outputs, comprehensive README
- Add Smithery: publishing guide with namespace yusufkaraaslan/skill-seekers created
- Add render-mcp.yaml for MCP server deployment on Render
- Fix Dockerfile.mcp: --transport flag (nonexistent) → --http, add dynamic PORT support
- Update AGENTS.md to v3.3.0 with corrected test count and expanded CI section
- Allow distribution/claude-plugin/.mcp.json in .gitignore
Fix CI failures: version hardcoded in _version.py fallbacks and
test assertions (test_package_structure, test_cli_paths) still
referenced 3.2.0 after the version bump.
Add machine-translated README files for Japanese, Korean, Spanish,
French, German, Portuguese (BR), Turkish, Arabic, Hindi, and Russian.
Update language selector in English and Chinese READMEs to link all 12
versions.
New files: README.{ja,ko,es,fr,de,pt-BR,tr,ar,hi,ru}.md
Modified: README.md, README.zh-CN.md (language selector bar)
Adds docs/BEST_PRACTICES.md — a comprehensive guide for creating
high-quality Claude skills. Covers SKILL.md structure, code examples,
prerequisites, troubleshooting, quality targets, and a real-world
before/after example (Grade F to Grade A). Addresses roadmap item I2.2.
Based on PR #206 by @jmagly from the AI Writing Guide project.
Fixes applied: updated outdated CLI command, fixed broken doc links.
Co-authored-by: jmagly <jmagly@users.noreply.github.com>
PyGithub's PaginatedList slicing (issues[:max_issues]) may fail with
'list index out of range' on some PyGithub versions or when repos
have no issues. Replace with itertools.islice() which works reliably
with any iterable, including PaginatedList.
Bug reported by @dream0438-cmd in PR #269.
Closes#269
## Summary
Add `skill-seekers sync-config` subcommand that crawls a docs site's navigation,
diffs discovered URLs against a config's start_urls, and optionally writes the
updated list back with --apply.
- BFS link discovery with configurable depth (default 2), max-pages, rate-limit
- Respects url_patterns.include/exclude from config
- Supports optional nav_seed_urls config field
- Handles both unified (sources array) and legacy flat config formats
- MCP tool sync_config included
- 57 tests (39 unit + 18 E2E with local HTTP server)
- Fixed CI: renamed summary job to "Tests" to match branch protection rule
Closes#306
The branch protection requires a status check named 'Tests', but
GitHub reports checks using job names, not the workflow name. The
summary job was named 'All Checks Complete' which never satisfied
the required check, leaving PRs permanently stuck as 'Expected —
Waiting for status to be reported'.
Fix: rename the summary job from 'All Checks Complete' to 'Tests'
so it matches the required status check exactly.
The original fix (741daf1) only patched LlmsTxtParser._clean_url(),
which covers URLs extracted directly from llms.txt content. But URLs
discovered from .md files during BFS crawl (_extract_markdown_content)
and from HTML pages (extract_content) bypass _clean_url() entirely.
When those pages contain links with square brackets (e.g.
/api/[v1]/users), httpx raises 'Invalid IPv6 URL' on fetch.
Fix: add a shared sanitize_url() utility in cli/utils.py that
percent-encodes [ and ] in path/query components, and apply it at
every URL ingestion point:
- _enqueue_url(): main chokepoint — all discovered URLs pass through
- scrape_page(): safety net for start_urls that skip _enqueue_url
- scrape_page_async(): same for async mode
- dry-run sync/async paths: direct fetches that also bypass _enqueue_url
LlmsTxtParser._clean_url() now delegates bracket-encoding to the
shared sanitize_url() (DRY), keeping only its malformed-anchor
stripping logic.
Added 16 tests: sanitize_url unit tests, _clean_url bracket tests,
_enqueue_url sanitization tests, and integration test verifying
markdown content with bracket URLs is handled safely.
Fixes#284
Follow-up to PR #309 (perf: optimize with caching, pre-compiled regex,
O(1) lookups, and bisect line indexing). These fixes were committed to
the PR branch but missed the squash merge.
Review fixes (credit: PR #309 by copperlang2007):
1. Rename _pending_set -> _enqueued_urls to accurately reflect that the
set tracks all ever-enqueued URLs, not just currently pending ones
2. Extract duplicated _build_line_index()/_offset_to_line() into shared
build_line_index()/offset_to_line() in cli/utils.py (DRY)
3. Fix pre-existing bug: infer_categories() guard checked 'tutorial'
but wrote to 'tutorials' key, risking silent overwrites
4. Remove unnecessary _store_results() closure in scrape_page()
5. Simplify parser pre-import in codebase_scraper.py
Benchmark stabilization:
- test_benchmark_metadata_overhead was flaky on CI (106.7% overhead
observed, threshold 50%) because 5 iterations with mean averaging
can't reliably measure microsecond-level differences
- Fix: 20 iterations, warm-up run, median instead of mean, threshold
raised to 200% (guards catastrophic regression, not noise)
Ref: https://github.com/yusufkaraaslan/Skill_Seekers/pull/309
tomllib is only available in Python 3.11+, but the release workflow
runs on Python 3.10. Replace with grep/sed which works everywhere.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GitHub release was showing v3.1.3 instead of v3.2.0 because:
1. No explicit `name` was set on the GitHub release action, relying on
defaults that could be unreliable
2. The sed command for extracting release notes used unescaped dots in
the version regex, which could match wrong versions
3. No fallback if release notes extraction produced an empty file
Changes:
- Add explicit `name` and `tag_name` to softprops/action-gh-release
- Add version consistency check (tag vs pyproject.toml vs package)
- Escape dots in sed regex for exact version matching
- Add fallback when release notes extraction produces empty output
https://claude.ai/code/session_015hYfpKhFH3GSMVSKgA4JVd
Update version across pyproject.toml, _version.py fallbacks,
CHANGELOG.md, and README badges for v3.2.0 release.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Tagline, descriptions, and feature lists now include video
- Add Video Extraction subsection in Key Features (7 bullet points)
- Update Feature Matrix: 5 → 6 skill modes (added Video)
- Add video rows to Performance table (transcript + visual)
- Add VIDEO_GUIDE.md to documentation links
- Update test badge and counts: 1,880/2,283 → 2,540+
- Sync all changes to README.zh-CN.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add docs/VIDEO_GUIDE.md (483 lines) — comprehensive guide covering
Quick Start, CLI reference, visual pipeline, AI enhancement, output
structure, time clipping, and troubleshooting
- Update README.md video section with new CLI examples (enhance,
clipping, vision OCR, re-build from JSON) and link to full guide
- Sync README.zh-CN.md with all video feature additions:
- Quick Start section: video commands
- Core Features: new video extraction feature list
- Installation table: video/video-full packages + GPU note
- Usage Examples: full video extraction subsection
- Documentation links: VIDEO_GUIDE.md reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip OCR on WEBCAM/OTHER frames (eliminates ~64 junk results per video)
- Add _clean_ocr_line() to strip line numbers, IDE decorations, collapse markers
- Add _fix_intra_line_duplication() for multi-engine OCR overlap artifacts
- Add _is_likely_code() filter to prevent UI junk in reference code fences
- Add language detection to get_text_groups() via LanguageDetector
- Apply OCR cleaning in _assemble_structured_text() pipeline
- Add two-pass AI enhancement: Pass 1 cleans reference Code Timeline
using transcript context, Pass 2 generates SKILL.md from cleaned refs
- Update video-tutorial.yaml prompts for pre-cleaned references
- Add 17 new tests (197 total video tests), 2540 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add missing video pipeline feature (the main 15K+ line addition),
15 video bug fixes, and restructure [Unreleased] section with
proper hierarchy: Video Pipeline Core → Video --setup → Word support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- UP024: Replace `(OSError, IOError)` with `OSError` in video_setup.py
- E402: Use existing `os` import instead of `import os as _os` in video_visual.py
- SIM103: Inline condition in `_detect_gpu()` return statement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Auto-detects NVIDIA (CUDA), AMD (ROCm), or CPU-only GPU and installs the
correct PyTorch variant + easyocr + all visual extraction dependencies.
Removes easyocr from video-full pip extras to avoid pulling ~2GB of wrong
CUDA packages on non-NVIDIA systems.
New files:
- video_setup.py (835 lines): GPU detection, PyTorch install, ROCm config,
venv checks, system dep validation, module selection, verification
- test_video_setup.py (60 tests): Full coverage of detection, install, verify
Updated docs: CHANGELOG, AGENTS.md, CLAUDE.md, README.md, CLI_REFERENCE,
FAQ, TROUBLESHOOTING, installation guide, video dependency plan
All 2523 tests passing (15 skipped).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix extract_visual_data returning 2-tuple instead of 3 (ValueError crash)
- Move pytesseract from core deps to [video-full] optional group
- Add 30-min timeout + user feedback to video enhancement subprocess
- Add scrape_video_impl to MCP server fallback import block
- Detect auto-generated YouTube captions via is_generated property
- Forward --vision-ocr and --video-playlist through create command
- Fix filename collision for non-ASCII video titles (fallback to video_id)
- Make _vision_used a proper dataclass field on FrameSubSection
- Expose 6 visual params in MCP scrape_video tool
- Add install instructions on missing video deps in unified scraper
- Update MCP docstring tool counts (25→33, 7 categories)
- Add video and word commands to main.py docstring
- Document video-full exclusion from [all] deps in pyproject.toml
- Update parser registry test count (22→23 for video parser)
All 2437 tests passing, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sync with latest development changes including ruff formatting,
bug fixes, and pinecone adaptor additions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DocToSkillConverter has self.skill_dir (string), not self.output_dir.
The --chunk-for-rag flag on scrape command crashed with AttributeError.
Changed to Path(converter.skill_dir).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bare `pip3` can point to a different Python installation than `python3`.
On the reporter's macOS, python3 was 3.14 but pip3 was linked to 3.9,
causing "no matching distribution" since skill-seekers requires >=3.10.
Using `python3 -m pip` guarantees the same interpreter that passed the
version check is the one performing the install.
Closes#301
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add FALLBACK_MAIN_SELECTORS constant and _find_main_content() helper to
eliminate 3 duplicated fallback loops in doc_scraper.py
- Move link extraction before early return in extract_content() so links
are always discovered from the full page, not just main content
- Fix single-threaded dry-run to extract links from soup (full page)
instead of main element only — fixes reactflow.dev finding only 1 page
- Add link extraction to async dry-run path (was completely missing)
- Remove main_content from get_configuration() defaults so fallback logic
kicks in instead of a broad CSS comma selector matching body
- Smart create --config routing: peek at JSON to determine unified
(sources array → unified_scraper) vs simple (base_url → doc_scraper)
- Update docs/user-guide/02-scraping.md and docs/reference/CONFIG_FORMAT.md
to use unified config format (legacy format rejected since v2.11.0)
- Fix test_auto_fetch_enabled and test_mcp_validate_legacy_config
Closes#300
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stage 1 quality improvements from the Arbitrary Limits & Dead Code audit:
Reference file truncation removed:
- codebase_scraper.py: remove code[:500] truncation at 5 locations — reference
files now contain complete code blocks for copy-paste usability
- unified_skill_builder.py: remove issues[:20], releases[:10], body[:500],
and code_snippet[:300] caps in reference files — full content preserved
Enhancement summarizer rewrite:
- enhance_skill_local.py: replace arbitrary [:5] code block cap with
character-budget approach using target_ratio * content_chars
- Fix intro boundary bug: track code block state so intro never ends
inside a code block, which was desynchronizing the parser
- Remove dead _target_lines variable (assigned but never used)
- Heading chunks now also respect the character budget
Hardcoded language fixes:
- unified_skill_builder.py: test examples use ex["language"] instead of
always "python" for syntax highlighting
- how_to_guide_builder.py: add language field to HowToGuide dataclass,
set from workflow at creation, used in AI enhancement prompt
Test fixes:
- test_enhance_skill_local.py: rename test to test_code_blocks_not_arbitrarily_capped,
fix assertion to count actual blocks (```count // 2), use target_ratio=0.9
Documentation:
- Add Stage 1 plan, implementation summary, review, and corrected docs
- Update CHANGELOG.md with all changes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix#299: rename --chunk-size/--chunk-overlap to --streaming-chunk-size/
--streaming-overlap in arguments/package.py to avoid collision with the
RAG --chunk-size flag from arguments/common.py
- Phase 1a: make package_skill.py import args via add_package_arguments()
instead of a 105-line inline duplicate argparse block; fixes the root
cause of _reconstruct_argv() passing unrecognised flag names
- Phase 1b: centralise setup_logging() into utils.py and remove 4
duplicate module-level logging.basicConfig() calls from doc_scraper.py,
github_scraper.py, codebase_scraper.py, and unified_scraper.py
- Fix test_package_structure.py / test_cli_paths.py version strings
(3.1.1 → 3.1.2)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
