yusyus 83b9a695ba feat: add sync-config command to detect and update config start_urls (#306) ...

## Summary

Add `skill-seekers sync-config` subcommand that crawls a docs site's navigation,
diffs discovered URLs against a config's start_urls, and optionally writes the
updated list back with --apply.

- BFS link discovery with configurable depth (default 2), max-pages, rate-limit
- Respects url_patterns.include/exclude from config
- Supports optional nav_seed_urls config field
- Handles both unified (sources array) and legacy flat config formats
- MCP tool sync_config included
- 57 tests (39 unit + 18 E2E with local HTTP server)
- Fixed CI: renamed summary job to "Tests" to match branch protection rule

Closes #306

2026-03-15 02:16:32 +03:00

..

adaptors

style: ruff format remaining 14 files

2026-03-01 10:54:45 +03:00

arguments

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

parsers

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

presets

chore: bump version to 3.1.0 and update CHANGELOG

2026-02-22 21:52:04 +03:00

storage

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

__init__.py

refactor: Centralize version management in single source of truth

2026-01-31 17:03:40 +03:00

ai_enhancer.py

style: Format code with ruff formatter

2026-02-03 21:37:54 +03:00

api_reference_builder.py

change max lenght

2026-01-17 17:48:15 +00:00

architectural_pattern_detector.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

benchmark_cli.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

cloud_storage_cli.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

code_analyzer.py

fix: apply review fixes from PR #309 and stabilize flaky benchmark test

2026-03-14 23:39:23 +03:00

codebase_scraper.py

fix: apply review fixes from PR #309 and stabilize flaky benchmark test

2026-03-14 23:39:23 +03:00

config_command.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

config_enhancer.py

style: Format code with ruff formatter

2026-02-03 21:37:54 +03:00

config_extractor.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

config_fetcher.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

config_manager.py

fix: smart enhancement dispatcher — Gemini/API mode + root/Docker detection

2026-02-22 01:26:19 +03:00

config_validator.py

feat: add video tutorial scraping pipeline with per-panel OCR and AI enhancement

2026-02-27 23:10:19 +03:00

conflict_detector.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

constants.py

run ruff

2026-01-17 17:29:21 +00:00

create_command.py

feat: add skill-seekers video --setup for GPU auto-detection and dependency installation

2026-03-01 18:39:16 +03:00

dependency_analyzer.py

fix: apply review fixes from PR #309 and stabilize flaky benchmark test

2026-03-14 23:39:23 +03:00

doc_scraper.py

fix: centralize bracket-encoding to prevent 'Invalid IPv6 URL' on all code paths (#284)

2026-03-14 23:53:47 +03:00

embedding_pipeline.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

enhance_command.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

enhance_skill_local.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

enhance_skill.py

feat: add video tutorial scraping pipeline with per-panel OCR and AI enhancement

2026-02-27 23:10:19 +03:00

enhance_status.py

change max lenght

2026-01-17 17:48:15 +00:00

enhancement_workflow.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

estimate_pages.py

fix: resolve all CI ruff linting errors (F401, F821, ARG001, SIM117, SIM105, C408)

2026-02-22 22:30:52 +03:00

generate_router.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

github_fetcher.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

github_scraper.py

perf: optimize with caching, pre-compiled regex, O(1) lookups, and bisect line indexing (#309)

2026-03-14 23:35:39 +03:00

guide_enhancer.py

feat: Add GLM-4.7 support and fix PDF scraper issues (#266)

2026-01-27 21:10:40 +03:00

how_to_guide_builder.py

fix: remove arbitrary limits, fix hardcoded languages, and fix summarizer bugs

2026-02-26 00:30:40 +03:00

incremental_updater.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

install_agent.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

install_skill.py

fix: resolve all CI ruff linting errors (F401, F821, ARG001, SIM117, SIM105, C408)

2026-02-22 22:30:52 +03:00

language_detector.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

llms_txt_detector.py

run ruff

2026-01-17 17:29:21 +00:00

llms_txt_downloader.py

style: Run black formatter on 16 files

2026-01-17 23:56:24 +03:00

llms_txt_parser.py

fix: centralize bracket-encoding to prevent 'Invalid IPv6 URL' on all code paths (#284)

2026-03-14 23:53:47 +03:00

main.py

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

markdown_cleaner.py

run ruff

2026-01-17 17:29:21 +00:00

merge_sources.py

change max lenght

2026-01-17 17:48:15 +00:00

multilang_support.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

package_multi.py

run ruff

2026-01-17 17:29:21 +00:00

package_skill.py

fix: resolve 18 bugs and code quality issues across adaptors, CLI, and chunking pipeline

2026-02-28 21:57:59 +03:00

pattern_recognizer.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

pdf_extractor_poc.py

style: auto-format 6 files with ruff format (CI formatting check)

2026-02-24 22:28:11 +03:00

pdf_scraper.py

feat(B2): add Microsoft Word (.docx) support

2026-02-25 21:47:30 +03:00

quality_checker.py

fix: Fix list comprehension variable names (NameError in CI)

2026-01-17 23:33:34 +03:00

quality_metrics.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

rag_chunker.py

style: ruff format remaining 14 files

2026-03-01 10:54:45 +03:00

rate_limit_handler.py

change max lenght

2026-01-17 17:48:15 +00:00

resume_command.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

run_tests.py

change max lenght

2026-01-17 17:48:15 +00:00

setup_wizard.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

signal_flow_analyzer.py

style: Format code with ruff formatter

2026-02-03 21:37:54 +03:00

source_detector.py

style: ruff format 4 video pipeline files

2026-03-01 19:48:02 +03:00

split_config.py

change max lenght

2026-01-17 17:48:15 +00:00

streaming_ingest.py

style: auto-format 6 files with ruff format (CI formatting check)

2026-02-24 22:28:11 +03:00

swift_patterns.py

change max lenght

2026-01-17 17:48:15 +00:00

sync_cli.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

sync_config.py

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

test_example_extractor.py

style: Format code with ruff formatter

2026-02-03 21:37:54 +03:00

test_unified_simple.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

unified_codebase_analyzer.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

unified_enhancer.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

unified_scraper.py

fix: resolve 15 bugs and gaps in video scraper pipeline

2026-03-01 12:39:21 +03:00

unified_skill_builder.py

style: ruff format remaining 14 files

2026-03-01 10:54:45 +03:00

upload_skill.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

utils.py

fix: centralize bracket-encoding to prevent 'Invalid IPv6 URL' on all code paths (#284)

2026-03-14 23:53:47 +03:00

video_metadata.py

feat: add video tutorial scraping pipeline with per-panel OCR and AI enhancement

2026-02-27 23:10:19 +03:00

video_models.py

feat: add skill-seekers video --setup for GPU auto-detection and dependency installation

2026-03-01 18:39:16 +03:00

video_scraper.py

feat: video pipeline OCR quality fixes + two-pass AI enhancement

2026-03-01 21:48:21 +03:00

video_segmenter.py

feat: add skill-seekers video --setup for GPU auto-detection and dependency installation

2026-03-01 18:39:16 +03:00

video_setup.py

style: ruff format 4 video pipeline files

2026-03-01 19:48:02 +03:00

video_transcript.py

fix: resolve 15 bugs and gaps in video scraper pipeline

2026-03-01 12:39:21 +03:00

video_visual.py

feat: video pipeline OCR quality fixes + two-pass AI enhancement

2026-03-01 21:48:21 +03:00

word_scraper.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

workflow_runner.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

workflows_command.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00