yusyus 2ef6e59d06 fix: stop blindly appending /index.html.md to non-.md URLs (#277) ...

The previous fix (a82cf69) only addressed anchor fragment stripping but
left the fundamental problem: _convert_to_md_urls() blindly appended
/index.html.md to ALL non-.md URLs from llms.txt. This only works for
Docusaurus sites — for sites like Discord docs it generates mass 404s.

Changes:
- _convert_to_md_urls() now strips anchors and deduplicates only,
  preserving original URLs as-is instead of appending /index.html.md
- New _has_md_extension() helper uses urlparse().path.endswith(".md")
  instead of error-prone ".md" in url substring matching
- Fixed ".md" in url checks at 4 locations (lines 465, 554, 716, 775)
- Removed 24 lines of dead commented-out code
- Added real-world e2e test against docs.discord.com (no mocks)
- Updated unit tests for new behavior (32 tests)

Fixes #277

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-20 23:44:35 +03:00

..

fixtures

Add unified scraping tests and example conflict data

2025-10-29 23:19:32 +03:00

test_adaptors

fix: improve MiniMax adaptor from PR #318 review (#319)

2026-03-20 22:12:23 +03:00

__init__.py

Add comprehensive test system with 71 tests (100% pass rate)

2025-10-19 02:08:58 +03:00

conftest.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

docker-compose.test.yml

feat: Add integration testing with real vector databases (Phase 5)

2026-02-07 22:55:02 +03:00

mcp_integration_test.md

Add comprehensive MCP setup guide and integration test template

2025-10-19 17:01:37 +03:00

test_adaptor_benchmarks.py

fix: apply review fixes from PR #309 and stabilize flaky benchmark test

2026-03-14 23:39:23 +03:00

test_analyze_command.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_analyze_e2e.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

test_api_reference_builder.py

change max lenght

2026-01-17 17:48:15 +00:00

test_architecture_scenarios.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_async_scraping.py

fix: Resolve deprecation warnings in Pydantic and asyncio

2026-02-08 13:34:48 +03:00

test_benchmark.py

test: Skip tests requiring optional dependencies

2026-02-08 14:49:45 +03:00

test_bootstrap_skill_e2e.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_bootstrap_skill.py

docs: Update all documentation to use new 'analyze' command

2026-01-29 22:56:33 +03:00

test_c3_integration.py

fix: Fix remaining 61 ruff linting errors (SIM102, SIM117)

2026-01-17 23:25:12 +03:00

test_chunking_integration.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

test_cli_parsers.py

feat: add 10 new skill source types (17 total) with full pipeline integration

2026-03-15 15:30:15 +03:00

test_cli_paths.py

fix: update version strings and test expectations from 3.2.0 to 3.3.0

2026-03-16 00:53:35 +03:00

test_cli_refactor_e2e.py

style: ruff format remaining 14 files

2026-03-01 10:54:45 +03:00

test_cloud_storage.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_code_analyzer.py

run ruff

2026-01-17 17:29:21 +00:00

test_codebase_scraper.py

fix: Remove duplicate documentation directories to save disk space (fixes #279)

2026-02-05 21:27:41 +03:00

test_config_extractor.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_config_fetcher.py

fix(#300): centralize selector fallback, fix dry-run link discovery, and smart --config routing

2026-02-26 22:25:59 +03:00

test_config_validation.py

change max lenght

2026-01-17 17:48:15 +00:00

test_constants.py

change max lenght

2026-01-17 17:48:15 +00:00

test_create_arguments.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

test_create_integration_basic.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_dependency_analyzer.py

fix: Resolve 61 critical linting errors

2026-01-17 22:54:40 +03:00

test_e2e_three_stream_pipeline.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_embedding_pipeline.py

test: Skip tests requiring optional dependencies

2026-02-08 14:49:45 +03:00

test_embedding.py

test: Skip tests requiring optional dependencies

2026-02-08 14:49:45 +03:00

test_enhance_command.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_enhance_skill_local.py

fix: remove arbitrary limits, fix hardcoded languages, and fix summarizer bugs

2026-02-26 00:30:40 +03:00

test_epub_scraper.py

feat: add EPUB input support (#310)

2026-03-15 02:34:41 +03:00

test_estimate_pages.py

change max lenght

2026-01-17 17:48:15 +00:00

test_excluded_dirs_config.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_framework_detection.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_generate_router_github.py

change max lenght

2026-01-17 17:48:15 +00:00

test_git_repo.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_git_sources_e2e.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_github_fetcher.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_github_scraper.py

fix: unified scraper temp config uses unified format for doc_scraper (#317)

2026-03-20 22:35:12 +03:00

test_guide_enhancer.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_how_to_guide_builder.py

fix: Resolve PDF processing (#267), How-To Guide (#242), Chinese README (#260) + code quality (#273)

2026-01-31 21:30:00 +03:00

test_incremental_updates.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_install_agent.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_install_multiplatform.py

change max lenght

2026-01-17 17:48:15 +00:00

test_install_skill_e2e.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_install_skill.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_integration_adaptors.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_integration.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_issue_219_e2e.py

style: Auto-format 48 files with ruff format

2026-02-15 20:24:32 +03:00

test_issue_277_discord_e2e.py

fix: stop blindly appending /index.html.md to non-.md URLs (#277)

2026-03-20 23:44:35 +03:00

test_issue_277_real_world.py

fix: stop blindly appending /index.html.md to non-.md URLs (#277)

2026-03-20 23:44:35 +03:00

test_language_detector.py

run ruff

2026-01-17 17:29:21 +00:00

test_llms_txt_detector.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_llms_txt_downloader.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_llms_txt_parser.py

run ruff

2026-01-17 17:29:21 +00:00

test_markdown_parsing.py

fix: centralize bracket-encoding to prevent 'Invalid IPv6 URL' on all code paths (#284)

2026-03-14 23:53:47 +03:00

test_mcp_fastmcp.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_mcp_git_sources.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_mcp_server.py

fix: Update MCP tests for unified config format

2026-02-08 14:44:46 +03:00

test_mcp_vector_dbs.py

fix: Enforce min_chunk_size in RAG chunker

2026-02-07 20:59:03 +03:00

test_mcp_workflow_tools.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_merge_sources_github.py

change max lenght

2026-01-17 17:48:15 +00:00

test_multi_source.py

change max lenght

2026-01-17 17:48:15 +00:00

test_multilang_support.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_new_source_types.py

feat: add 10 new skill source types (17 total) with full pipeline integration

2026-03-15 15:30:15 +03:00

test_package_skill.py

change max lenght

2026-01-17 17:48:15 +00:00

test_package_structure.py

fix: update version strings and test expectations from 3.2.0 to 3.3.0

2026-03-16 00:53:35 +03:00

test_parallel_scraping.py

change max lenght

2026-01-17 17:48:15 +00:00

test_parser_sync.py

style: Auto-format 48 files with ruff format

2026-02-15 20:24:32 +03:00

test_pattern_recognizer.py

fix: Add skipif for HTTP server tests & finalize test suite fixes

2026-02-08 13:33:15 +03:00

test_pdf_advanced_features.py

fix: Resolve 61 critical linting errors

2026-01-17 22:54:40 +03:00

test_pdf_extractor.py

Merge PR #275: Add Dart, Scala, SCSS, SASS, Elixir, Lua, Perl language detection

2026-02-04 21:00:49 +03:00

test_pdf_scraper.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_pinecone_adaptor.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

test_pr144_concerns.py

run ruff

2026-01-17 17:29:21 +00:00

test_preset_system.py

chore: bump version to 3.1.0 and update CHANGELOG

2026-02-22 21:52:04 +03:00

test_quality_checker.py

change max lenght

2026-01-17 17:48:15 +00:00

test_quality_metrics.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_rag_chunker.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_rate_limit_handler.py

change max lenght

2026-01-17 17:48:15 +00:00

test_real_world_fastmcp.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_scraper_features.py

fix: centralize bracket-encoding to prevent 'Invalid IPv6 URL' on all code paths (#284)

2026-03-14 23:53:47 +03:00

test_server_fastmcp_http.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_setup_scripts.py

style: Format test_setup_scripts.py with ruff

2026-01-18 13:48:37 +03:00

test_skip_llms_txt.py

style: Run ruff format on 15 files (CI fix)

2026-01-18 00:01:30 +03:00

test_smart_summarization.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_source_detector.py

style: Auto-format 48 files with ruff format

2026-02-15 20:24:32 +03:00

test_source_manager.py

fix: Resolve remaining 188 linting errors (249 total fixed)

2026-01-17 23:02:11 +03:00

test_streaming_ingestion.py

style: Format all Python files with ruff

2026-02-08 14:42:27 +03:00

test_swift_detection.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_sync_config_e2e.py

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

test_sync_config.py

feat: add sync-config command to detect and update config start_urls (#306)

2026-03-15 02:16:32 +03:00

test_terminal_detection.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_test_example_extractor.py

test: Add GDScript test extraction test case

2026-02-02 23:08:25 +03:00

test_unified_analyzer.py

fix: resolve all test failures — 2115 passing, 0 failures

2026-02-22 20:43:17 +03:00

test_unified_mcp_integration.py

fix(#300): centralize selector fallback, fix dry-run link discovery, and smart --config routing

2026-02-26 22:25:59 +03:00

test_unified_parsers.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

test_unified_scraper_orchestration.py

fix: unified scraper temp config uses unified format for doc_scraper (#317)

2026-03-20 22:35:12 +03:00

test_unified.py

style: auto-format 12 files with ruff format (CI formatting check)

2026-02-22 22:32:31 +03:00

test_upload_integration.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

test_upload_skill.py

change max lenght

2026-01-17 17:48:15 +00:00

test_url_conversion.py

fix: stop blindly appending /index.html.md to non-.md URLs (#277)

2026-03-20 23:44:35 +03:00

test_utilities.py

run ruff

2026-01-17 17:29:21 +00:00

test_video_scraper.py

feat: video pipeline OCR quality fixes + two-pass AI enhancement

2026-03-01 21:48:21 +03:00

test_video_setup.py

style: ruff format 4 video pipeline files

2026-03-01 19:48:02 +03:00

test_word_scraper.py

style: fix ruff lint and formatting errors

2026-03-01 10:54:32 +03:00

test_workflow_runner.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

test_workflow_tools_mcp.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00

test_workflows_command.py

style: apply ruff format to all source and test files

2026-02-18 22:50:05 +03:00