yusyus a82cf6967a fix: Strip anchor fragments in URL conversion to prevent 404 errors (fixes #277) ...

Critical bug fix for llms.txt URL parsing:

Problem:
- URLs with anchor fragments (e.g., #synchronous-initialization) were
  malformed when converting to .md format
- Example: https://example.com/api#method → https://example.com/api#method/index.html.md ❌
- Caused 404 errors and duplicate requests for same page with different anchors

Solution:
1. Parse URLs with urllib.parse.urlparse() to extract fragments
2. Strip anchor fragments before appending /index.html.md
3. Deduplicate base URLs (multiple anchors → single request)
4. Fix .md detection: '.md' in url → url.endswith('.md')
   - Prevents false matches on URLs like /cmd-line or /AMD-processors

Changes:
- src/skill_seekers/cli/doc_scraper.py (_convert_to_md_urls)
  - Added URL parsing to remove fragments
  - Added deduplication with seen_base_urls set
  - Fixed .md extension detection
  - Updated log message to show deduplicated count
- tests/test_url_conversion.py (NEW)
  - 12 comprehensive tests covering all edge cases
  - Real-world MikroORM case validation
  - 54/54 tests passing (42 existing + 12 new)
- CHANGELOG.md
  - Documented bug fix and solution

Reported-by: @devjones <https://github.com/yusufkaraaslan/Skill_Seekers/issues/277>

2026-02-04 21:16:13 +03:00

..

fixtures

Add unified scraping tests and example conflict data

2025-10-29 23:19:32 +03:00

test_adaptors

change max lenght

2026-01-17 17:48:15 +00:00

__init__.py

Add comprehensive test system with 71 tests (100% pass rate)

2025-10-19 02:08:58 +03:00

conftest.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

mcp_integration_test.md

Add comprehensive MCP setup guide and integration test template

2025-10-19 17:01:37 +03:00

test_analyze_command.py

style: Format code with ruff formatter

2026-02-03 21:37:54 +03:00

test_analyze_e2e.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

test_api_reference_builder.py

change max lenght

2026-01-17 17:48:15 +00:00

test_architecture_scenarios.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_async_scraping.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_bootstrap_skill_e2e.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_bootstrap_skill.py

docs: Update all documentation to use new 'analyze' command

2026-01-29 22:56:33 +03:00

test_c3_integration.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

test_cli_paths.py

fix: Update test version checks to 2.9.0 and remove whitespace

2026-02-03 21:00:34 +03:00

test_code_analyzer.py

run ruff

2026-01-17 17:29:21 +00:00

test_codebase_scraper.py

feat: C3.9 documentation extraction, AI enhancement optimization, and C# support

2026-01-31 14:56:00 +03:00

test_config_extractor.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_config_fetcher.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

test_config_validation.py

change max lenght

2026-01-17 17:48:15 +00:00

test_constants.py

change max lenght

2026-01-17 17:48:15 +00:00

test_dependency_analyzer.py

fix: Resolve 61 critical linting errors

2026-01-17 22:54:40 +03:00

test_e2e_three_stream_pipeline.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_enhance_skill_local.py

Add custom agent validation and tests

2026-02-04 10:14:20 +01:00

test_estimate_pages.py

change max lenght

2026-01-17 17:48:15 +00:00

test_excluded_dirs_config.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_generate_router_github.py

change max lenght

2026-01-17 17:48:15 +00:00

test_git_repo.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_git_sources_e2e.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_github_fetcher.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_github_scraper.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_guide_enhancer.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_how_to_guide_builder.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

test_install_agent.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_install_multiplatform.py

change max lenght

2026-01-17 17:48:15 +00:00

test_install_skill_e2e.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_install_skill.py

change max lenght

2026-01-17 17:48:15 +00:00

test_integration.py

chore: Post-merge cleanup - remove client docs and fix linter errors

2026-01-31 14:58:09 +03:00

test_issue_219_e2e.py

style: Format test_issue_219_e2e.py with ruff

2026-01-18 12:11:01 +03:00

test_language_detector.py

run ruff

2026-01-17 17:29:21 +00:00

test_llms_txt_detector.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_llms_txt_downloader.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_llms_txt_parser.py

run ruff

2026-01-17 17:29:21 +00:00

test_markdown_parsing.py

fix: Fix list comprehension variable names (NameError in CI)

2026-01-17 23:33:34 +03:00

test_mcp_fastmcp.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_mcp_git_sources.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_mcp_server.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_merge_sources_github.py

change max lenght

2026-01-17 17:48:15 +00:00

test_multi_source.py

change max lenght

2026-01-17 17:48:15 +00:00

test_package_skill.py

change max lenght

2026-01-17 17:48:15 +00:00

test_package_structure.py

fix: Update test version checks to 2.9.0 and remove whitespace

2026-02-03 21:00:34 +03:00

test_parallel_scraping.py

change max lenght

2026-01-17 17:48:15 +00:00

test_pattern_recognizer.py

change max lenght

2026-01-17 17:48:15 +00:00

test_pdf_advanced_features.py

fix: Resolve 61 critical linting errors

2026-01-17 22:54:40 +03:00

test_pdf_extractor.py

Merge PR #275: Add Dart, Scala, SCSS, SASS, Elixir, Lua, Perl language detection

2026-02-04 21:00:49 +03:00

test_pdf_scraper.py

fix: Remove duplicate import os statements causing UnboundLocalError

2026-01-27 21:11:04 +03:00

test_pr144_concerns.py

run ruff

2026-01-17 17:29:21 +00:00

test_quality_checker.py

change max lenght

2026-01-17 17:48:15 +00:00

test_rate_limit_handler.py

change max lenght

2026-01-17 17:48:15 +00:00

test_real_world_fastmcp.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_scraper_features.py

change max lenght

2026-01-17 17:48:15 +00:00

test_server_fastmcp_http.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

test_setup_scripts.py

style: Format test_setup_scripts.py with ruff

2026-01-18 13:48:37 +03:00

test_skip_llms_txt.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_smart_summarization.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_source_manager.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_swift_detection.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_terminal_detection.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_test_example_extractor.py

test: Add GDScript test extraction test case

2026-02-02 23:08:25 +03:00

test_unified_analyzer.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_unified_mcp_integration.py

fix: Resolve 61 critical linting errors

2026-01-17 22:54:40 +03:00

test_unified.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

test_upload_skill.py

change max lenght

2026-01-17 17:48:15 +00:00

test_url_conversion.py

fix: Strip anchor fragments in URL conversion to prevent 404 errors (fixes #277)

2026-02-04 21:16:13 +03:00

test_utilities.py

run ruff

2026-01-17 17:29:21 +00:00