2a14309342
- Add CHANGELOG.md entry for v3.5.0 with all PR #336 changes - Update README.md: version 3.5.0, agent-agnostic examples, marketplace pipeline, SPA discovery - Update CLAUDE.md: AgentClient architecture, 40 MCP tools, new modules - Update docs/: UML architecture, MCP reference (40 tools, new tool categories), enhancement modes (multi-provider/multi-agent), FAQ - Update src/skill_seekers/mcp/README.md: accurate tool count and paths Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
152 lines
10 KiB
Markdown
152 lines
10 KiB
Markdown
# Skill Seekers Architecture
|
|
|
|
> Generated 2026-03-22 | StarUML project: `docs/UML/skill_seekers.mdj`
|
|
|
|
## Overview
|
|
|
|
Skill Seekers converts documentation from 17 source types into production-ready formats for 24+ AI platforms. The architecture follows a layered module design with 8 core modules and 5 utility modules.
|
|
|
|
## Package Diagram
|
|
|
|
|
|
|
|
**Core Modules** (upper area):
|
|
- **CLICore** -- Git-style command dispatcher, entry point for all `skill-seekers` commands
|
|
- **Scrapers** -- 17 source-type extractors (web, GitHub, PDF, Word, EPUB, video, etc.)
|
|
- **Adaptors** -- Strategy+Factory pattern for 20+ output platforms (Claude, Gemini, OpenAI, RAG frameworks)
|
|
- **Analysis** -- C3.x codebase analysis pipeline (AST parsing, 10 GoF pattern detectors, guide builders)
|
|
- **Enhancement** -- AI-powered skill improvement via `AgentClient` (API mode: Anthropic/Kimi/Gemini/OpenAI + LOCAL mode: Claude Code/Kimi/Codex/Copilot/OpenCode/custom, --enhance-level 0-3)
|
|
- **Packaging** -- Package, upload, and install skills to AI agent directories
|
|
- **MCP** -- FastMCP server exposing 40 tools via stdio/HTTP transport (includes marketplace and config publishing)
|
|
- **Sync** -- Documentation change detection and re-scraping triggers
|
|
|
|
**Utility Modules** (lower area):
|
|
- **Parsers** -- CLI argument parsers (30+ SubcommandParser subclasses)
|
|
- **Storage** -- Cloud storage abstraction (S3, GCS, Azure)
|
|
- **Embedding** -- Multi-provider vector embedding generation
|
|
- **Benchmark** -- Performance measurement framework
|
|
- **Utilities** -- Shared helpers (LanguageDetector, RAGChunker, MarkdownCleaner, etc.)
|
|
|
|
## Core Module Diagrams
|
|
|
|
### CLICore
|
|
|
|
|
|
Entry point: `skill-seekers` CLI. `CLIDispatcher` maps subcommands to modules via `COMMAND_MODULES` dict. `CreateCommand` auto-detects source type via `SourceDetector`.
|
|
|
|
### Scrapers
|
|
|
|
|
|
18 scraper classes implementing `IScraper`. Each has a `main()` entry point. Notable: `GitHubScraper` (3-stream fetcher) + `GitHubToSkillConverter` (builder), `UnifiedScraper` (multi-source orchestrator).
|
|
|
|
### Adaptors
|
|
|
|
|
|
`SkillAdaptor` ABC with 3 abstract methods: `format_skill_md()`, `package()`, `upload()`. Two-level hierarchy: direct subclasses (Claude, Gemini, OpenAI, Markdown, OpenCode, RAG adaptors) and `OpenAICompatibleAdaptor` intermediate (MiniMax, Kimi, DeepSeek, Qwen, OpenRouter, Together, Fireworks).
|
|
|
|
### Analysis (C3.x Pipeline)
|
|
|
|
|
|
`UnifiedCodebaseAnalyzer` controller orchestrates: `CodeAnalyzer` (AST, 9 languages), `PatternRecognizer` (10 GoF detectors via `BasePatternDetector`), `TestExampleExtractor`, `HowToGuideBuilder`, `ConfigExtractor`, `SignalFlowAnalyzer`, `DependencyAnalyzer`, `ArchitecturalPatternDetector`.
|
|
|
|
### Enhancement
|
|
|
|
|
|
Two enhancement hierarchies: `AIEnhancer` (API mode, multi-provider via `AgentClient`) and `UnifiedEnhancer` (C3.x pipeline enhancers). Each has specialized subclasses for patterns, test examples, guides, and configs. `WorkflowEngine` orchestrates multi-stage `EnhancementWorkflow`. The `AgentClient` (`cli/agent_client.py`) centralizes all AI invocations, supporting API mode (Anthropic, Moonshot/Kimi, Gemini, OpenAI) and LOCAL mode (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents).
|
|
|
|
### Packaging
|
|
|
|
|
|
`PackageSkill` delegates to adaptors for format-specific packaging. `UploadSkill` handles platform API uploads. `InstallSkill`/`InstallAgent` install to AI agent directories. `OpenCodeSkillSplitter` handles large file splitting.
|
|
|
|
### MCP Server
|
|
|
|
|
|
`SkillSeekerMCPServer` (FastMCP) with 40 tools in 10 categories. Supporting classes: `SourceManager` (config CRUD), `AgentDetector` (environment detection), `GitConfigRepo` (community configs), `MarketplacePublisher` (publish skills to marketplace repos), `MarketplaceManager` (marketplace registry CRUD), `ConfigPublisher` (push configs to registered source repos).
|
|
|
|
### Sync
|
|
|
|
|
|
`SyncMonitor` controller schedules periodic checks via `ChangeDetector` (SHA-256 hashing, HTTP headers, content diffing). `Notifier` sends alerts when changes are found. Pydantic models: `PageChange`, `ChangeReport`, `SyncConfig`, `SyncState`.
|
|
|
|
## Utility Module Diagrams
|
|
|
|
### Parsers
|
|
|
|
|
|
`SubcommandParser` ABC with 27 subclasses -- one per CLI subcommand (Create, Scrape, GitHub, PDF, Word, EPUB, Video, Unified, Analyze, Enhance, Package, Upload, Jupyter, HTML, OpenAPI, AsciiDoc, Pptx, RSS, ManPage, Confluence, Notion, Chat, Config, Estimate, Install, Stream, Quality, SyncConfig).
|
|
|
|
### Storage
|
|
|
|
|
|
`BaseStorageAdaptor` ABC with `S3StorageAdaptor`, `GCSStorageAdaptor`, `AzureStorageAdaptor`. `StorageObject` dataclass for file metadata.
|
|
|
|
### Embedding
|
|
|
|
|
|
`EmbeddingGenerator` (multi-provider: OpenAI, Sentence Transformers, Voyage AI). `EmbeddingPipeline` coordinates provider, caching, and cost tracking. `EmbeddingProvider` ABC with OpenAI and Local implementations.
|
|
|
|
### Benchmark
|
|
|
|
|
|
`BenchmarkRunner` orchestrates `Benchmark` instances. `BenchmarkResult` collects timings/memory/metrics and produces `BenchmarkReport`. Supporting data types: `Metric`, `TimingResult`, `MemoryUsage`, `ComparisonReport`.
|
|
|
|
### Utilities
|
|
|
|
|
|
16 shared helper classes: `LanguageDetector`, `MarkdownCleaner`, `RAGChunker`, `RateLimitHandler`, `ConfigManager`, `ConfigValidator`, `SkillQualityChecker`, `QualityAnalyzer`, `LlmsTxtDetector`/`Downloader`/`Parser`, `ConfigSplitter`, `ConflictDetector`, `IncrementalUpdater`, `MultiLanguageManager`, `StreamingIngester`.
|
|
|
|
## Key Design Patterns
|
|
|
|
| Pattern | Where | Classes |
|
|
|---------|-------|---------|
|
|
| Strategy + Factory | Adaptors | `SkillAdaptor` ABC + `get_adaptor()` factory + 20+ implementations |
|
|
| Strategy + Factory | Storage | `BaseStorageAdaptor` ABC + S3/GCS/Azure |
|
|
| Strategy + Factory | Embedding | `EmbeddingProvider` ABC + OpenAI/Local |
|
|
| Command | CLI | `CLIDispatcher` + `COMMAND_MODULES` lazy dispatch |
|
|
| Template Method | Pattern Detection | `BasePatternDetector` + 10 GoF detectors |
|
|
| Template Method | Parsers | `SubcommandParser` + 27 subclasses |
|
|
|
|
## Behavioral Diagrams
|
|
|
|
### Create Pipeline Sequence
|
|
|
|
|
|
`CreateCommand` is a dispatcher, not a pipeline orchestrator. Flow: User → `execute()` → `SourceDetector.detect(source)` → `validate_source()` → `_validate_arguments()` → `_route_to_scraper()` → `scraper.main(argv)`. The 5 phases (scrape, build_skill, enhance, package, upload) all happen *inside* each scraper's `main()` — CreateCommand only sees the exit code.
|
|
|
|
### GitHub Unified Flow + C3.x
|
|
|
|
|
|
`UnifiedScraper` orchestrates GitHub scraping (3-stream fetch) then delegates to `analyze_codebase(enhance_level)` for C3.x analysis. Shows all 5 C3.x stages: `PatternRecognizer` (C3.1), `TestExampleExtractor` (C3.2), `HowToGuideBuilder` with examples from C3.2 (C3.3), `ConfigExtractor` (C3.4), and `ArchitecturalPatternDetector` (C3.5). Note: `enhance_level` is the sole AI control parameter — `enhance_with_ai`/`ai_mode` are internal to C3.x classes only.
|
|
|
|
### Source Auto-Detection
|
|
|
|
|
|
Activity diagram showing `source_detector.py` decision tree in correct code order: file extension first (.json config, .pdf/.docx/.epub/.ipynb/.html/.pptx/etc) → video URL → `os.path.isdir()` (Codebase) → GitHub pattern (owner/repo or github.com URL) → http/https URL (Web) → bare domain inference → error.
|
|
|
|
### MCP Tool Invocation
|
|
|
|
|
|
MCP Client (Claude Code/Cursor) → FastMCPServer (stdio/HTTP) with two invocation paths: **Path A** (scraping tools) uses `subprocess.run(["skill-seekers", ...])`, **Path B** (packaging/config tools) uses direct Python imports (`get_adaptor()`, `sync_config()`). Both return TextContent → JSON-RPC.
|
|
|
|
### Enhancement Pipeline
|
|
|
|
|
|
`--enhance-level` decision flow with precise internal variable mapping: Level 0 sets `ai_mode=none`, skips all AI. Level >= 1 selects `ai_mode=api` (if any supported API key set: Anthropic, Moonshot/Kimi, Gemini, OpenAI) or `ai_mode=local` (via `AgentClient` with configurable agent: Claude Code, Kimi, Codex, Copilot, OpenCode, or custom), then SKILL.md enhancement happens post-build via `enhance_command`. Level >= 2 enables `enhance_config=True`, `enhance_architecture=True` inside `analyze_codebase()`. Level 3 adds `enhance_patterns=True`, `enhance_tests=True`.
|
|
|
|
### Runtime Components
|
|
|
|
|
|
Component diagram with corrected runtime dependencies. Key flows: `CLI Core` dispatches to `Scrapers` (via `scraper.main(argv)`) and to `Adaptors` (via package/upload commands). `Scrapers` call `Codebase Analysis` via `analyze_codebase(enhance_level)`. `Codebase Analysis` uses `C3.x Classes` internally and `Enhancement` when level ≥ 2. `MCP Server` reaches `Scrapers` via subprocess and `Adaptors` via direct import. `Scrapers` optionally use `Browser Renderer (Playwright)` via `render_page()` when `--browser` flag is set for JavaScript SPA sites.
|
|
|
|
### Browser Rendering Flow
|
|
|
|
|
|
When `--browser` flag is set, `DocScraper.scrape_page()` delegates to `BrowserRenderer.render_page(url)` instead of `requests.get()`. The renderer auto-installs Chromium on first use, navigates with `wait_until='networkidle'` to let JavaScript execute, then returns the fully-rendered HTML. The rest of the pipeline (BeautifulSoup → `extract_content()` → `save_page()`) remains unchanged. Optional dependency: `pip install "skill-seekers[browser]"`.
|
|
|
|
## File Locations
|
|
|
|
- **StarUML project**: `docs/UML/skill_seekers.mdj`
|
|
- **Diagram exports**: `docs/UML/exports/*.png`
|
|
- **Source code**: `src/skill_seekers/`
|