fix: Update setup_mcp.sh for v2.0.0 src/ layout + test fixes (#201)
Merges setup_mcp.sh fix for v2.0.0 src/ layout + test updates. Original fix by @501981732 in PR #197. Test updates to make CI pass. Closes #192
This commit is contained in:
yusyus
GitHub
2
.github/workflows/release.yml
vendored
2
.github/workflows/release.yml
vendored
@@ -25,6 +25,8 @@ jobs:
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
if [ -f skill_seeker_mcp/requirements.txt ]; then pip install -r skill_seeker_mcp/requirements.txt; fi
|
||||
# Install package in editable mode for tests (required for src/ layout)
|
||||
pip install -e .
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
|
||||
148
CHANGELOG.md
148
CHANGELOG.md
@@ -12,6 +12,154 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
---
|
||||
|
||||
## [2.1.0] - 2025-11-12
|
||||
|
||||
### 🎉 Major Enhancement: Quality Assurance + Race Condition Fixes
|
||||
|
||||
This release focuses on quality and reliability improvements, adding comprehensive quality checks and fixing critical race conditions in the enhancement workflow.
|
||||
|
||||
### 🚀 Major Features
|
||||
|
||||
#### Comprehensive Quality Checker
|
||||
- **Automatic quality checks before packaging** - Validates skill quality before upload
|
||||
- **Quality scoring system** - 0-100 score with A-F grades
|
||||
- **Enhancement verification** - Checks for template text, code examples, sections
|
||||
- **Structure validation** - Validates SKILL.md, references/ directory
|
||||
- **Content quality checks** - YAML frontmatter, language tags, "When to Use" section
|
||||
- **Link validation** - Validates internal markdown links
|
||||
- **Detailed reporting** - Errors, warnings, and info messages with file locations
|
||||
- **CLI tool** - `skill-seekers-quality-checker` with verbose and strict modes
|
||||
|
||||
#### Headless Enhancement Mode (Default)
|
||||
- **No terminal windows** - Runs enhancement in background by default
|
||||
- **Proper waiting** - Main console waits for enhancement to complete
|
||||
- **Timeout protection** - 10-minute default timeout (configurable)
|
||||
- **Verification** - Checks that SKILL.md was actually updated
|
||||
- **Progress messages** - Clear status updates during enhancement
|
||||
- **Interactive mode available** - `--interactive-enhancement` flag for terminal mode
|
||||
|
||||
### Added
|
||||
|
||||
#### New CLI Tools
|
||||
- **quality_checker.py** - Comprehensive skill quality validation
|
||||
- Structure checks (SKILL.md, references/)
|
||||
- Enhancement verification (code examples, sections)
|
||||
- Content validation (frontmatter, language tags)
|
||||
- Link validation (internal markdown links)
|
||||
- Quality scoring (0-100 + A-F grade)
|
||||
|
||||
#### New Features
|
||||
- **Headless enhancement** - `skill-seekers-enhance` runs in background by default
|
||||
- **Quality checks in packaging** - Automatic validation before creating .zip
|
||||
- **MCP quality skip** - MCP server skips interactive checks
|
||||
- **Enhanced error handling** - Better error messages and timeout handling
|
||||
|
||||
#### Tests
|
||||
- **+12 quality checker tests** - Comprehensive validation testing
|
||||
- **391 total tests passing** - Up from 379 in v2.0.0
|
||||
- **0 test failures** - All tests green
|
||||
- **CI improvements** - Fixed macOS terminal detection tests
|
||||
|
||||
### Changed
|
||||
|
||||
#### Enhancement Workflow
|
||||
- **Default mode changed** - Headless mode is now default (was terminal mode)
|
||||
- **Waiting behavior** - Main console waits for enhancement completion
|
||||
- **No race conditions** - Fixed "Package your skill" message appearing too early
|
||||
- **Better progress** - Clear status messages during enhancement
|
||||
|
||||
#### Package Workflow
|
||||
- **Quality checks added** - Automatic validation before packaging
|
||||
- **User confirmation** - Ask to continue if warnings/errors found
|
||||
- **Skip option** - `--skip-quality-check` flag to bypass checks
|
||||
- **MCP context** - Automatically skips checks in non-interactive contexts
|
||||
|
||||
#### CLI Arguments
|
||||
- **doc_scraper.py:**
|
||||
- Updated `--enhance-local` help text (mentions headless mode)
|
||||
- Added `--interactive-enhancement` flag
|
||||
- **enhance_skill_local.py:**
|
||||
- Changed default to `headless=True`
|
||||
- Added `--interactive-enhancement` flag
|
||||
- Added `--timeout` flag (default: 600 seconds)
|
||||
- **package_skill.py:**
|
||||
- Added `--skip-quality-check` flag
|
||||
|
||||
### Fixed
|
||||
|
||||
#### Critical Bugs
|
||||
- **Enhancement race condition** - Main console no longer exits before enhancement completes
|
||||
- **MCP stdin errors** - MCP server now skips interactive prompts
|
||||
- **Terminal detection tests** - Fixed for headless mode default
|
||||
|
||||
#### Enhancement Issues
|
||||
- **Process detachment** - subprocess.run() now waits properly instead of Popen()
|
||||
- **Timeout handling** - Added timeout protection to prevent infinite hangs
|
||||
- **Verification** - Checks file modification time and size to verify success
|
||||
- **Error messages** - Better error handling and user-friendly messages
|
||||
|
||||
#### Test Fixes
|
||||
- **package_skill tests** - Added skip_quality_check=True to prevent stdin errors
|
||||
- **Terminal detection tests** - Updated to use headless=False for interactive tests
|
||||
- **MCP server tests** - Fixed to skip quality checks in non-interactive context
|
||||
|
||||
### Technical Details
|
||||
|
||||
#### New Modules
|
||||
- `src/skill_seekers/cli/quality_checker.py` - Quality validation engine
|
||||
- `tests/test_quality_checker.py` - 12 comprehensive tests
|
||||
|
||||
#### Modified Modules
|
||||
- `src/skill_seekers/cli/enhance_skill_local.py` - Added headless mode
|
||||
- `src/skill_seekers/cli/doc_scraper.py` - Updated enhancement integration
|
||||
- `src/skill_seekers/cli/package_skill.py` - Added quality checks
|
||||
- `src/skill_seekers/mcp/server.py` - Skip quality checks in MCP context
|
||||
- `tests/test_package_skill.py` - Updated for quality checker
|
||||
- `tests/test_terminal_detection.py` - Updated for headless default
|
||||
|
||||
#### Commits in This Release
|
||||
- `e279ed6` - Phase 1: Enhancement race condition fix (headless mode)
|
||||
- `3272f9c` - Phases 2 & 3: Quality checker implementation
|
||||
- `2dd1027` - Phase 4: Tests (+12 quality checker tests)
|
||||
- `befcb89` - CI Fix: Skip quality checks in MCP context
|
||||
- `67ab627` - CI Fix: Update terminal tests for headless default
|
||||
|
||||
### Upgrade Notes
|
||||
|
||||
#### Breaking Changes
|
||||
- **Headless mode default** - Enhancement now runs in background by default
|
||||
- Use `--interactive-enhancement` if you want the old terminal mode
|
||||
- Affects: `skill-seekers-enhance` and `skill-seekers scrape --enhance-local`
|
||||
|
||||
#### New Behavior
|
||||
- **Quality checks** - Packaging now runs quality checks by default
|
||||
- May prompt for confirmation if warnings/errors found
|
||||
- Use `--skip-quality-check` to bypass (not recommended)
|
||||
|
||||
#### Recommendations
|
||||
- **Try headless mode** - Faster and more reliable than terminal mode
|
||||
- **Review quality reports** - Fix warnings before packaging
|
||||
- **Update scripts** - Add `--skip-quality-check` to automated packaging scripts if needed
|
||||
|
||||
### Migration Guide
|
||||
|
||||
**If you want the old terminal mode behavior:**
|
||||
```bash
|
||||
# Old (v2.0.0): Default was terminal mode
|
||||
skill-seekers-enhance output/react/
|
||||
|
||||
# New (v2.1.0): Use --interactive-enhancement
|
||||
skill-seekers-enhance output/react/ --interactive-enhancement
|
||||
```
|
||||
|
||||
**If you want to skip quality checks:**
|
||||
```bash
|
||||
# Add --skip-quality-check to package command
|
||||
skill-seekers-package output/react/ --skip-quality-check
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [2.0.0] - 2025-11-11
|
||||
|
||||
### 🎉 Major Release: PyPI Publication + Modern Python Packaging
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "skill-seekers"
|
||||
version = "2.0.0"
|
||||
version = "2.1.0"
|
||||
description = "Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
26
setup_mcp.sh
26
setup_mcp.sh
@@ -77,15 +77,9 @@ read -p "Continue? (y/n) " -n 1 -r
|
||||
echo ""
|
||||
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Installing MCP server dependencies..."
|
||||
$PIP_INSTALL_CMD -r skill_seeker_mcp/requirements.txt || {
|
||||
echo -e "${RED}❌ Failed to install MCP dependencies${NC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "Installing CLI tool dependencies..."
|
||||
$PIP_INSTALL_CMD requests beautifulsoup4 || {
|
||||
echo -e "${RED}❌ Failed to install CLI dependencies${NC}"
|
||||
echo "Installing package in editable mode..."
|
||||
$PIP_INSTALL_CMD -e . || {
|
||||
echo -e "${RED}❌ Failed to install package${NC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
@@ -97,7 +91,7 @@ echo ""
|
||||
|
||||
# Step 4: Test MCP server
|
||||
echo "Step 4: Testing MCP server..."
|
||||
timeout 3 python3 skill_seeker_mcp/server.py 2>/dev/null || {
|
||||
timeout 3 python3 src/skill_seekers/mcp/server.py 2>/dev/null || {
|
||||
if [ $? -eq 124 ]; then
|
||||
echo -e "${GREEN}✓${NC} MCP server starts correctly (timeout expected)"
|
||||
else
|
||||
@@ -147,7 +141,7 @@ echo " \"mcpServers\": {"
|
||||
echo " \"skill-seeker\": {"
|
||||
echo " \"command\": \"python3\","
|
||||
echo " \"args\": ["
|
||||
echo " \"$REPO_PATH/skill_seeker_mcp/server.py\""
|
||||
echo " \"$REPO_PATH/src/skill_seekers/mcp/server.py\""
|
||||
echo " ],"
|
||||
echo " \"cwd\": \"$REPO_PATH\""
|
||||
echo " }"
|
||||
@@ -188,7 +182,7 @@ if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
"skill-seeker": {
|
||||
"command": "python3",
|
||||
"args": [
|
||||
"$REPO_PATH/skill_seeker_mcp/server.py"
|
||||
"$REPO_PATH/src/skill_seekers/mcp/server.py"
|
||||
],
|
||||
"cwd": "$REPO_PATH"
|
||||
}
|
||||
@@ -203,10 +197,10 @@ EOF
|
||||
echo ""
|
||||
|
||||
# Verify the path exists
|
||||
if [ -f "$REPO_PATH/skill_seeker_mcp/server.py" ]; then
|
||||
echo -e "${GREEN}✓${NC} Verified: MCP server file exists at $REPO_PATH/skill_seeker_mcp/server.py"
|
||||
if [ -f "$REPO_PATH/src/skill_seekers/mcp/server.py" ]; then
|
||||
echo -e "${GREEN}✓${NC} Verified: MCP server file exists at $REPO_PATH/src/skill_seekers/mcp/server.py"
|
||||
else
|
||||
echo -e "${RED}❌ Warning: MCP server not found at $REPO_PATH/skill_seeker_mcp/server.py${NC}"
|
||||
echo -e "${RED}❌ Warning: MCP server not found at $REPO_PATH/src/skill_seekers/mcp/server.py${NC}"
|
||||
echo "Please check the path!"
|
||||
fi
|
||||
else
|
||||
@@ -266,7 +260,7 @@ echo " • Full docs: ${YELLOW}README.md${NC}"
|
||||
echo ""
|
||||
echo "Troubleshooting:"
|
||||
echo " • Check logs: ~/Library/Logs/Claude Code/ (macOS)"
|
||||
echo " • Test server: python3 skill_seeker_mcp/server.py"
|
||||
echo " • Test server: python3 src/skill_seekers/mcp/server.py"
|
||||
echo " • Run tests: python3 -m pytest tests/test_mcp_server.py -v"
|
||||
echo ""
|
||||
echo "Happy skill creating! 🚀"
|
||||
|
||||
@@ -1506,7 +1506,9 @@ def setup_argument_parser() -> argparse.ArgumentParser:
|
||||
parser.add_argument('--enhance', action='store_true',
|
||||
help='Enhance SKILL.md using Claude API after building (requires API key)')
|
||||
parser.add_argument('--enhance-local', action='store_true',
|
||||
help='Enhance SKILL.md using Claude Code in new terminal (no API key needed)')
|
||||
help='Enhance SKILL.md using Claude Code (no API key needed, runs in background)')
|
||||
parser.add_argument('--interactive-enhancement', action='store_true',
|
||||
help='Open terminal window for enhancement (use with --enhance-local)')
|
||||
parser.add_argument('--api-key', type=str,
|
||||
help='Anthropic API key for --enhance (or set ANTHROPIC_API_KEY)')
|
||||
parser.add_argument('--resume', action='store_true',
|
||||
@@ -1740,16 +1742,25 @@ def execute_enhancement(config: Dict[str, Any], args: argparse.Namespace) -> Non
|
||||
# Optional enhancement with Claude Code (local, no API key)
|
||||
if args.enhance_local:
|
||||
logger.info("\n" + "=" * 60)
|
||||
logger.info("ENHANCING SKILL.MD WITH CLAUDE CODE (LOCAL)")
|
||||
if args.interactive_enhancement:
|
||||
logger.info("ENHANCING SKILL.MD WITH CLAUDE CODE (INTERACTIVE)")
|
||||
else:
|
||||
logger.info("ENHANCING SKILL.MD WITH CLAUDE CODE (HEADLESS)")
|
||||
logger.info("=" * 60 + "\n")
|
||||
|
||||
try:
|
||||
enhance_cmd = ['python3', 'cli/enhance_skill_local.py', f'output/{config["name"]}/']
|
||||
subprocess.run(enhance_cmd, check=True)
|
||||
enhance_cmd = ['skill-seekers-enhance', f'output/{config["name"]}/']
|
||||
if args.interactive_enhancement:
|
||||
enhance_cmd.append('--interactive-enhancement')
|
||||
|
||||
result = subprocess.run(enhance_cmd, check=True)
|
||||
|
||||
if result.returncode == 0:
|
||||
logger.info("\n✅ Enhancement complete!")
|
||||
except subprocess.CalledProcessError:
|
||||
logger.warning("\n⚠ Enhancement failed, but skill was still built")
|
||||
except FileNotFoundError:
|
||||
logger.warning("\n⚠ enhance_skill_local.py not found. Run manually:")
|
||||
logger.warning("\n⚠ skill-seekers-enhance command not found. Run manually:")
|
||||
logger.info(" skill-seekers-enhance output/%s/", config['name'])
|
||||
|
||||
# Print packaging instructions
|
||||
@@ -1759,10 +1770,11 @@ def execute_enhancement(config: Dict[str, Any], args: argparse.Namespace) -> Non
|
||||
# Suggest enhancement if not done
|
||||
if not args.enhance and not args.enhance_local:
|
||||
logger.info("\n💡 Optional: Enhance SKILL.md with Claude:")
|
||||
logger.info(" API-based: skill-seekers-enhance output/%s/", config['name'])
|
||||
logger.info(" or re-run with: --enhance")
|
||||
logger.info(" Local (no API key): skill-seekers-enhance output/%s/", config['name'])
|
||||
logger.info(" or re-run with: --enhance-local")
|
||||
logger.info(" Local (recommended): skill-seekers-enhance output/%s/", config['name'])
|
||||
logger.info(" or re-run with: --enhance-local")
|
||||
logger.info(" API-based: skill-seekers-enhance-api output/%s/", config['name'])
|
||||
logger.info(" or re-run with: --enhance")
|
||||
logger.info("\n💡 Tip: Use --interactive-enhancement with --enhance-local to open terminal window")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
||||
@@ -166,8 +166,13 @@ First, backup the original to: {self.skill_md_path.with_suffix('.md.backup').abs
|
||||
|
||||
return prompt
|
||||
|
||||
def run(self):
|
||||
"""Main enhancement workflow"""
|
||||
def run(self, headless=True, timeout=600):
|
||||
"""Main enhancement workflow
|
||||
|
||||
Args:
|
||||
headless: If True, run claude directly without opening terminal (default: True)
|
||||
timeout: Maximum time to wait for enhancement in seconds (default: 600 = 10 minutes)
|
||||
"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"LOCAL ENHANCEMENT: {self.skill_dir.name}")
|
||||
print(f"{'='*60}\n")
|
||||
@@ -207,7 +212,11 @@ First, backup the original to: {self.skill_md_path.with_suffix('.md.backup').abs
|
||||
|
||||
print(f" ✓ Prompt saved ({len(prompt):,} characters)\n")
|
||||
|
||||
# Launch Claude Code in new terminal
|
||||
# Headless mode: Run claude directly without opening terminal
|
||||
if headless:
|
||||
return self._run_headless(prompt_file, timeout)
|
||||
|
||||
# Terminal mode: Launch Claude Code in new terminal
|
||||
print("🚀 Launching Claude Code in new terminal...")
|
||||
print(" This will:")
|
||||
print(" 1. Open a new terminal window")
|
||||
@@ -281,20 +290,159 @@ rm {prompt_file}
|
||||
|
||||
return True
|
||||
|
||||
def _run_headless(self, prompt_file, timeout):
|
||||
"""Run Claude enhancement in headless mode (no terminal window)
|
||||
|
||||
Args:
|
||||
prompt_file: Path to prompt file
|
||||
timeout: Maximum seconds to wait
|
||||
|
||||
Returns:
|
||||
bool: True if enhancement succeeded
|
||||
"""
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
print("✨ Running Claude Code enhancement (headless mode)...")
|
||||
print(f" Timeout: {timeout} seconds ({timeout//60} minutes)")
|
||||
print()
|
||||
|
||||
# Record initial state
|
||||
initial_mtime = self.skill_md_path.stat().st_mtime if self.skill_md_path.exists() else 0
|
||||
initial_size = self.skill_md_path.stat().st_size if self.skill_md_path.exists() else 0
|
||||
|
||||
# Start timer
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Run claude command directly (this WAITS for completion)
|
||||
print(" Running: claude {prompt_file}")
|
||||
print(" ⏳ Please wait...")
|
||||
print()
|
||||
|
||||
result = subprocess.run(
|
||||
['claude', prompt_file],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
# Check if successful
|
||||
if result.returncode == 0:
|
||||
# Verify SKILL.md was actually updated
|
||||
if self.skill_md_path.exists():
|
||||
new_mtime = self.skill_md_path.stat().st_mtime
|
||||
new_size = self.skill_md_path.stat().st_size
|
||||
|
||||
if new_mtime > initial_mtime and new_size > initial_size:
|
||||
print(f"✅ Enhancement complete! ({elapsed:.1f} seconds)")
|
||||
print(f" SKILL.md updated: {new_size:,} bytes")
|
||||
print()
|
||||
|
||||
# Clean up prompt file
|
||||
try:
|
||||
os.unlink(prompt_file)
|
||||
except:
|
||||
pass
|
||||
|
||||
return True
|
||||
else:
|
||||
print(f"⚠️ Claude finished but SKILL.md was not updated")
|
||||
print(f" This might indicate an error during enhancement")
|
||||
print()
|
||||
return False
|
||||
else:
|
||||
print(f"❌ SKILL.md not found after enhancement")
|
||||
return False
|
||||
else:
|
||||
print(f"❌ Claude Code returned error (exit code: {result.returncode})")
|
||||
if result.stderr:
|
||||
print(f" Error: {result.stderr[:200]}")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
elapsed = time.time() - start_time
|
||||
print(f"\n⚠️ Enhancement timed out after {elapsed:.0f} seconds")
|
||||
print(f" Timeout limit: {timeout} seconds")
|
||||
print()
|
||||
print(" Possible reasons:")
|
||||
print(" - Skill is very large (many references)")
|
||||
print(" - Claude is taking longer than usual")
|
||||
print(" - Network issues")
|
||||
print()
|
||||
print(" Try:")
|
||||
print(" 1. Use terminal mode: --interactive-enhancement")
|
||||
print(" 2. Reduce reference content")
|
||||
print(" 3. Try again later")
|
||||
|
||||
# Clean up
|
||||
try:
|
||||
os.unlink(prompt_file)
|
||||
except:
|
||||
pass
|
||||
|
||||
return False
|
||||
|
||||
except FileNotFoundError:
|
||||
print("❌ 'claude' command not found")
|
||||
print()
|
||||
print(" Make sure Claude Code CLI is installed:")
|
||||
print(" See: https://docs.claude.com/claude-code")
|
||||
print()
|
||||
print(" Try terminal mode instead: --interactive-enhancement")
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Unexpected error: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: skill-seekers enhance <skill_directory>")
|
||||
print()
|
||||
print("Examples:")
|
||||
print(" skill-seekers enhance output/steam-inventory/")
|
||||
print(" skill-seekers enhance output/react/")
|
||||
sys.exit(1)
|
||||
import argparse
|
||||
|
||||
skill_dir = sys.argv[1]
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Enhance a skill with Claude Code (local)",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Headless mode (default - runs in background)
|
||||
skill-seekers enhance output/react/
|
||||
|
||||
enhancer = LocalSkillEnhancer(skill_dir)
|
||||
success = enhancer.run()
|
||||
# Interactive mode (opens terminal window)
|
||||
skill-seekers enhance output/react/ --interactive-enhancement
|
||||
|
||||
# Custom timeout
|
||||
skill-seekers enhance output/react/ --timeout 1200
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'skill_directory',
|
||||
help='Path to skill directory (e.g., output/react/)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--interactive-enhancement',
|
||||
action='store_true',
|
||||
help='Open terminal window for enhancement (default: headless mode)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--timeout',
|
||||
type=int,
|
||||
default=600,
|
||||
help='Timeout in seconds for headless mode (default: 600 = 10 minutes)'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Run enhancement
|
||||
enhancer = LocalSkillEnhancer(args.skill_directory)
|
||||
headless = not args.interactive_enhancement # Invert: default is headless
|
||||
success = enhancer.run(headless=headless, timeout=args.timeout)
|
||||
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@ try:
|
||||
format_file_size,
|
||||
validate_skill_directory
|
||||
)
|
||||
from quality_checker import SkillQualityChecker, print_report
|
||||
except ImportError:
|
||||
# If running from different directory, add cli to path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
@@ -32,15 +33,17 @@ except ImportError:
|
||||
format_file_size,
|
||||
validate_skill_directory
|
||||
)
|
||||
from quality_checker import SkillQualityChecker, print_report
|
||||
|
||||
|
||||
def package_skill(skill_dir, open_folder_after=True):
|
||||
def package_skill(skill_dir, open_folder_after=True, skip_quality_check=False):
|
||||
"""
|
||||
Package a skill directory into a .zip file
|
||||
|
||||
Args:
|
||||
skill_dir: Path to skill directory
|
||||
open_folder_after: Whether to open the output folder after packaging
|
||||
skip_quality_check: Skip quality checks before packaging
|
||||
|
||||
Returns:
|
||||
tuple: (success, zip_path) where success is bool and zip_path is Path or None
|
||||
@@ -53,6 +56,30 @@ def package_skill(skill_dir, open_folder_after=True):
|
||||
print(f"❌ Error: {error_msg}")
|
||||
return False, None
|
||||
|
||||
# Run quality checks (unless skipped)
|
||||
if not skip_quality_check:
|
||||
print("\n" + "=" * 60)
|
||||
print("QUALITY CHECK")
|
||||
print("=" * 60)
|
||||
|
||||
checker = SkillQualityChecker(skill_path)
|
||||
report = checker.check_all()
|
||||
|
||||
# Print report
|
||||
print_report(report, verbose=False)
|
||||
|
||||
# If there are errors or warnings, ask user to confirm
|
||||
if report.has_errors or report.has_warnings:
|
||||
print("=" * 60)
|
||||
response = input("\nContinue with packaging? (y/n): ").strip().lower()
|
||||
if response != 'y':
|
||||
print("\n❌ Packaging cancelled by user")
|
||||
return False, None
|
||||
print()
|
||||
else:
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# Create zip filename
|
||||
skill_name = skill_path.name
|
||||
zip_path = skill_path.parent / f"{skill_name}.zip"
|
||||
@@ -95,12 +122,18 @@ def main():
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Package skill and open folder
|
||||
# Package skill with quality checks (recommended)
|
||||
skill-seekers package output/react/
|
||||
|
||||
# Package skill without opening folder
|
||||
skill-seekers package output/react/ --no-open
|
||||
|
||||
# Skip quality checks (faster, but not recommended)
|
||||
skill-seekers package output/react/ --skip-quality-check
|
||||
|
||||
# Package and auto-upload to Claude
|
||||
skill-seekers package output/react/ --upload
|
||||
|
||||
# Get help
|
||||
skill-seekers package --help
|
||||
"""
|
||||
@@ -117,6 +150,12 @@ Examples:
|
||||
help='Do not open the output folder after packaging'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--skip-quality-check',
|
||||
action='store_true',
|
||||
help='Skip quality checks before packaging'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--upload',
|
||||
action='store_true',
|
||||
@@ -125,7 +164,11 @@ Examples:
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
success, zip_path = package_skill(args.skill_dir, open_folder_after=not args.no_open)
|
||||
success, zip_path = package_skill(
|
||||
args.skill_dir,
|
||||
open_folder_after=not args.no_open,
|
||||
skip_quality_check=args.skip_quality_check
|
||||
)
|
||||
|
||||
if not success:
|
||||
sys.exit(1)
|
||||
|
||||
480
src/skill_seekers/cli/quality_checker.py
Normal file
480
src/skill_seekers/cli/quality_checker.py
Normal file
@@ -0,0 +1,480 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quality Checker for Claude Skills
|
||||
Validates skill quality, checks links, and generates quality reports.
|
||||
|
||||
Usage:
|
||||
python3 quality_checker.py output/react/
|
||||
python3 quality_checker.py output/godot/ --verbose
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Tuple, Optional
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class QualityIssue:
|
||||
"""Represents a quality issue found during validation."""
|
||||
level: str # 'error', 'warning', 'info'
|
||||
category: str # 'enhancement', 'content', 'links', 'structure'
|
||||
message: str
|
||||
file: Optional[str] = None
|
||||
line: Optional[int] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class QualityReport:
|
||||
"""Complete quality report for a skill."""
|
||||
skill_name: str
|
||||
skill_path: Path
|
||||
errors: List[QualityIssue] = field(default_factory=list)
|
||||
warnings: List[QualityIssue] = field(default_factory=list)
|
||||
info: List[QualityIssue] = field(default_factory=list)
|
||||
|
||||
def add_error(self, category: str, message: str, file: str = None, line: int = None):
|
||||
"""Add an error to the report."""
|
||||
self.errors.append(QualityIssue('error', category, message, file, line))
|
||||
|
||||
def add_warning(self, category: str, message: str, file: str = None, line: int = None):
|
||||
"""Add a warning to the report."""
|
||||
self.warnings.append(QualityIssue('warning', category, message, file, line))
|
||||
|
||||
def add_info(self, category: str, message: str, file: str = None, line: int = None):
|
||||
"""Add info to the report."""
|
||||
self.info.append(QualityIssue('info', category, message, file, line))
|
||||
|
||||
@property
|
||||
def has_errors(self) -> bool:
|
||||
"""Check if there are any errors."""
|
||||
return len(self.errors) > 0
|
||||
|
||||
@property
|
||||
def has_warnings(self) -> bool:
|
||||
"""Check if there are any warnings."""
|
||||
return len(self.warnings) > 0
|
||||
|
||||
@property
|
||||
def is_excellent(self) -> bool:
|
||||
"""Check if quality is excellent (no errors, no warnings)."""
|
||||
return not self.has_errors and not self.has_warnings
|
||||
|
||||
@property
|
||||
def quality_score(self) -> float:
|
||||
"""Calculate quality score (0-100)."""
|
||||
# Start with perfect score
|
||||
score = 100.0
|
||||
|
||||
# Deduct points for issues
|
||||
score -= len(self.errors) * 15 # -15 per error
|
||||
score -= len(self.warnings) * 5 # -5 per warning
|
||||
|
||||
# Never go below 0
|
||||
return max(0.0, score)
|
||||
|
||||
@property
|
||||
def quality_grade(self) -> str:
|
||||
"""Get quality grade (A-F)."""
|
||||
score = self.quality_score
|
||||
if score >= 90:
|
||||
return 'A'
|
||||
elif score >= 80:
|
||||
return 'B'
|
||||
elif score >= 70:
|
||||
return 'C'
|
||||
elif score >= 60:
|
||||
return 'D'
|
||||
else:
|
||||
return 'F'
|
||||
|
||||
|
||||
class SkillQualityChecker:
|
||||
"""Validates skill quality and generates reports."""
|
||||
|
||||
def __init__(self, skill_dir: Path):
|
||||
"""Initialize quality checker.
|
||||
|
||||
Args:
|
||||
skill_dir: Path to skill directory
|
||||
"""
|
||||
self.skill_dir = Path(skill_dir)
|
||||
self.skill_md_path = self.skill_dir / "SKILL.md"
|
||||
self.references_dir = self.skill_dir / "references"
|
||||
self.report = QualityReport(
|
||||
skill_name=self.skill_dir.name,
|
||||
skill_path=self.skill_dir
|
||||
)
|
||||
|
||||
def check_all(self) -> QualityReport:
|
||||
"""Run all quality checks and return report.
|
||||
|
||||
Returns:
|
||||
QualityReport: Complete quality report
|
||||
"""
|
||||
# Basic structure checks
|
||||
self._check_skill_structure()
|
||||
|
||||
# Enhancement verification
|
||||
self._check_enhancement_quality()
|
||||
|
||||
# Content quality checks
|
||||
self._check_content_quality()
|
||||
|
||||
# Link validation
|
||||
self._check_links()
|
||||
|
||||
return self.report
|
||||
|
||||
def _check_skill_structure(self):
|
||||
"""Check basic skill structure."""
|
||||
# Check SKILL.md exists
|
||||
if not self.skill_md_path.exists():
|
||||
self.report.add_error(
|
||||
'structure',
|
||||
'SKILL.md file not found',
|
||||
str(self.skill_md_path)
|
||||
)
|
||||
return
|
||||
|
||||
# Check references directory exists
|
||||
if not self.references_dir.exists():
|
||||
self.report.add_warning(
|
||||
'structure',
|
||||
'references/ directory not found - skill may be incomplete',
|
||||
str(self.references_dir)
|
||||
)
|
||||
elif not list(self.references_dir.glob('*.md')):
|
||||
self.report.add_warning(
|
||||
'structure',
|
||||
'references/ directory is empty - no reference documentation found',
|
||||
str(self.references_dir)
|
||||
)
|
||||
|
||||
def _check_enhancement_quality(self):
|
||||
"""Check if SKILL.md was properly enhanced."""
|
||||
if not self.skill_md_path.exists():
|
||||
return
|
||||
|
||||
content = self.skill_md_path.read_text(encoding='utf-8')
|
||||
|
||||
# Check for template indicators (signs it wasn't enhanced)
|
||||
template_indicators = [
|
||||
"TODO:",
|
||||
"[Add description]",
|
||||
"[Framework specific tips]",
|
||||
"coming soon",
|
||||
]
|
||||
|
||||
for indicator in template_indicators:
|
||||
if indicator.lower() in content.lower():
|
||||
self.report.add_warning(
|
||||
'enhancement',
|
||||
f'Found template placeholder: "{indicator}" - SKILL.md may not be enhanced',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
# Check for good signs of enhancement
|
||||
enhancement_indicators = {
|
||||
'code_examples': re.compile(r'```[\w-]+\n', re.MULTILINE),
|
||||
'real_examples': re.compile(r'Example:', re.IGNORECASE),
|
||||
'sections': re.compile(r'^## .+', re.MULTILINE),
|
||||
}
|
||||
|
||||
code_blocks = len(enhancement_indicators['code_examples'].findall(content))
|
||||
real_examples = len(enhancement_indicators['real_examples'].findall(content))
|
||||
sections = len(enhancement_indicators['sections'].findall(content))
|
||||
|
||||
# Quality thresholds
|
||||
if code_blocks == 0:
|
||||
self.report.add_warning(
|
||||
'enhancement',
|
||||
'No code examples found in SKILL.md - consider enhancing',
|
||||
'SKILL.md'
|
||||
)
|
||||
elif code_blocks < 3:
|
||||
self.report.add_info(
|
||||
'enhancement',
|
||||
f'Only {code_blocks} code examples found - more examples would improve quality',
|
||||
'SKILL.md'
|
||||
)
|
||||
else:
|
||||
self.report.add_info(
|
||||
'enhancement',
|
||||
f'✓ Found {code_blocks} code examples',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
if sections < 4:
|
||||
self.report.add_warning(
|
||||
'enhancement',
|
||||
f'Only {sections} sections found - SKILL.md may be too basic',
|
||||
'SKILL.md'
|
||||
)
|
||||
else:
|
||||
self.report.add_info(
|
||||
'enhancement',
|
||||
f'✓ Found {sections} sections',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
def _check_content_quality(self):
|
||||
"""Check content quality."""
|
||||
if not self.skill_md_path.exists():
|
||||
return
|
||||
|
||||
content = self.skill_md_path.read_text(encoding='utf-8')
|
||||
|
||||
# Check YAML frontmatter
|
||||
if not content.startswith('---'):
|
||||
self.report.add_error(
|
||||
'content',
|
||||
'Missing YAML frontmatter - SKILL.md must start with ---',
|
||||
'SKILL.md',
|
||||
1
|
||||
)
|
||||
else:
|
||||
# Extract frontmatter
|
||||
try:
|
||||
frontmatter_match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
|
||||
if frontmatter_match:
|
||||
frontmatter = frontmatter_match.group(1)
|
||||
|
||||
# Check for required fields
|
||||
if 'name:' not in frontmatter:
|
||||
self.report.add_error(
|
||||
'content',
|
||||
'Missing "name:" field in YAML frontmatter',
|
||||
'SKILL.md',
|
||||
2
|
||||
)
|
||||
|
||||
# Check for description
|
||||
if 'description:' in frontmatter:
|
||||
self.report.add_info(
|
||||
'content',
|
||||
'✓ YAML frontmatter includes description',
|
||||
'SKILL.md'
|
||||
)
|
||||
else:
|
||||
self.report.add_error(
|
||||
'content',
|
||||
'Invalid YAML frontmatter format',
|
||||
'SKILL.md',
|
||||
1
|
||||
)
|
||||
except Exception as e:
|
||||
self.report.add_error(
|
||||
'content',
|
||||
f'Error parsing YAML frontmatter: {e}',
|
||||
'SKILL.md',
|
||||
1
|
||||
)
|
||||
|
||||
# Check code block language tags
|
||||
code_blocks_without_lang = re.findall(r'```\n[^`]', content)
|
||||
if code_blocks_without_lang:
|
||||
self.report.add_warning(
|
||||
'content',
|
||||
f'Found {len(code_blocks_without_lang)} code blocks without language tags',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
# Check for "When to Use" section
|
||||
if 'when to use' not in content.lower():
|
||||
self.report.add_warning(
|
||||
'content',
|
||||
'Missing "When to Use This Skill" section',
|
||||
'SKILL.md'
|
||||
)
|
||||
else:
|
||||
self.report.add_info(
|
||||
'content',
|
||||
'✓ Found "When to Use" section',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
# Check reference files
|
||||
if self.references_dir.exists():
|
||||
ref_files = list(self.references_dir.glob('*.md'))
|
||||
if ref_files:
|
||||
self.report.add_info(
|
||||
'content',
|
||||
f'✓ Found {len(ref_files)} reference files',
|
||||
'references/'
|
||||
)
|
||||
|
||||
# Check if references are mentioned in SKILL.md
|
||||
mentioned_refs = 0
|
||||
for ref_file in ref_files:
|
||||
if ref_file.name in content:
|
||||
mentioned_refs += 1
|
||||
|
||||
if mentioned_refs == 0:
|
||||
self.report.add_warning(
|
||||
'content',
|
||||
'Reference files exist but none are mentioned in SKILL.md',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
def _check_links(self):
|
||||
"""Check internal markdown links."""
|
||||
if not self.skill_md_path.exists():
|
||||
return
|
||||
|
||||
content = self.skill_md_path.read_text(encoding='utf-8')
|
||||
|
||||
# Find all markdown links [text](path)
|
||||
link_pattern = re.compile(r'\[([^\]]+)\]\(([^)]+)\)')
|
||||
links = link_pattern.findall(content)
|
||||
|
||||
broken_links = []
|
||||
|
||||
for text, link in links:
|
||||
# Skip external links (http/https)
|
||||
if link.startswith('http://') or link.startswith('https://'):
|
||||
continue
|
||||
|
||||
# Skip anchor links
|
||||
if link.startswith('#'):
|
||||
continue
|
||||
|
||||
# Check if file exists (relative to SKILL.md)
|
||||
link_path = self.skill_dir / link
|
||||
if not link_path.exists():
|
||||
broken_links.append((text, link))
|
||||
|
||||
if broken_links:
|
||||
for text, link in broken_links:
|
||||
self.report.add_warning(
|
||||
'links',
|
||||
f'Broken link: [{text}]({link})',
|
||||
'SKILL.md'
|
||||
)
|
||||
else:
|
||||
if links:
|
||||
internal_links = [l for t, l in links if not l.startswith('http')]
|
||||
if internal_links:
|
||||
self.report.add_info(
|
||||
'links',
|
||||
f'✓ All {len(internal_links)} internal links are valid',
|
||||
'SKILL.md'
|
||||
)
|
||||
|
||||
|
||||
def print_report(report: QualityReport, verbose: bool = False):
|
||||
"""Print quality report to console.
|
||||
|
||||
Args:
|
||||
report: Quality report to print
|
||||
verbose: Show all info messages
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print(f"QUALITY REPORT: {report.skill_name}")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# Quality score
|
||||
print(f"Quality Score: {report.quality_score:.1f}/100 (Grade: {report.quality_grade})")
|
||||
print()
|
||||
|
||||
# Errors
|
||||
if report.errors:
|
||||
print(f"❌ ERRORS ({len(report.errors)}):")
|
||||
for issue in report.errors:
|
||||
location = f" ({issue.file}:{issue.line})" if issue.file and issue.line else f" ({issue.file})" if issue.file else ""
|
||||
print(f" [{issue.category}] {issue.message}{location}")
|
||||
print()
|
||||
|
||||
# Warnings
|
||||
if report.warnings:
|
||||
print(f"⚠️ WARNINGS ({len(report.warnings)}):")
|
||||
for issue in report.warnings:
|
||||
location = f" ({issue.file}:{issue.line})" if issue.file and issue.line else f" ({issue.file})" if issue.file else ""
|
||||
print(f" [{issue.category}] {issue.message}{location}")
|
||||
print()
|
||||
|
||||
# Info (only in verbose mode)
|
||||
if verbose and report.info:
|
||||
print(f"ℹ️ INFO ({len(report.info)}):")
|
||||
for issue in report.info:
|
||||
location = f" ({issue.file})" if issue.file else ""
|
||||
print(f" [{issue.category}] {issue.message}{location}")
|
||||
print()
|
||||
|
||||
# Summary
|
||||
if report.is_excellent:
|
||||
print("✅ EXCELLENT! No issues found.")
|
||||
elif not report.has_errors:
|
||||
print("✓ GOOD! No errors, but some warnings to review.")
|
||||
else:
|
||||
print("❌ NEEDS IMPROVEMENT! Please fix errors before packaging.")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check skill quality and generate report",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Basic quality check
|
||||
python3 quality_checker.py output/react/
|
||||
|
||||
# Verbose mode (show all info)
|
||||
python3 quality_checker.py output/godot/ --verbose
|
||||
|
||||
# Exit with error code if issues found
|
||||
python3 quality_checker.py output/django/ --strict
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'skill_directory',
|
||||
help='Path to skill directory (e.g., output/react/)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose', '-v',
|
||||
action='store_true',
|
||||
help='Show all info messages'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--strict',
|
||||
action='store_true',
|
||||
help='Exit with error code if any warnings or errors found'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Check if directory exists
|
||||
skill_dir = Path(args.skill_directory)
|
||||
if not skill_dir.exists():
|
||||
print(f"❌ Directory not found: {skill_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
# Run quality checks
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Print report
|
||||
print_report(report, verbose=args.verbose)
|
||||
|
||||
# Exit code
|
||||
if args.strict and (report.has_errors or report.has_warnings):
|
||||
sys.exit(1)
|
||||
elif report.has_errors:
|
||||
sys.exit(1)
|
||||
else:
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -676,7 +676,8 @@ async def package_skill_tool(args: dict) -> list[TextContent]:
|
||||
sys.executable,
|
||||
str(CLI_DIR / "package_skill.py"),
|
||||
skill_dir,
|
||||
"--no-open" # Don't open folder in MCP context
|
||||
"--no-open", # Don't open folder in MCP context
|
||||
"--skip-quality-check" # Skip interactive quality checks in MCP context
|
||||
]
|
||||
|
||||
# Add upload flag only if we have API key
|
||||
|
||||
@@ -42,7 +42,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_dir = self.create_test_skill_directory(tmpdir)
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertTrue(success)
|
||||
self.assertIsNotNone(zip_path)
|
||||
@@ -55,7 +55,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_dir = self.create_test_skill_directory(tmpdir)
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertTrue(success)
|
||||
|
||||
@@ -78,7 +78,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
# Add a backup file
|
||||
(skill_dir / "SKILL.md.backup").write_text("# Backup")
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertTrue(success)
|
||||
|
||||
@@ -89,7 +89,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
|
||||
def test_package_nonexistent_directory(self):
|
||||
"""Test packaging a nonexistent directory"""
|
||||
success, zip_path = package_skill("/nonexistent/path", open_folder_after=False)
|
||||
success, zip_path = package_skill("/nonexistent/path", open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertFalse(success)
|
||||
self.assertIsNone(zip_path)
|
||||
@@ -100,7 +100,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
skill_dir = Path(tmpdir) / "invalid-skill"
|
||||
skill_dir.mkdir()
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertFalse(success)
|
||||
self.assertIsNone(zip_path)
|
||||
@@ -119,7 +119,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
(skill_dir / "scripts").mkdir()
|
||||
(skill_dir / "assets").mkdir()
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertTrue(success)
|
||||
# Zip should be in output directory, not inside skill directory
|
||||
@@ -136,7 +136,7 @@ class TestPackageSkill(unittest.TestCase):
|
||||
(skill_dir / "scripts").mkdir()
|
||||
(skill_dir / "assets").mkdir()
|
||||
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False)
|
||||
success, zip_path = package_skill(skill_dir, open_folder_after=False, skip_quality_check=True)
|
||||
|
||||
self.assertTrue(success)
|
||||
self.assertEqual(zip_path.name, "my-awesome-skill.zip")
|
||||
|
||||
297
tests/test_quality_checker.py
Normal file
297
tests/test_quality_checker.py
Normal file
@@ -0,0 +1,297 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for cli/quality_checker.py functionality
|
||||
"""
|
||||
|
||||
import unittest
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
import os
|
||||
|
||||
from skill_seekers.cli.quality_checker import SkillQualityChecker, QualityReport
|
||||
|
||||
|
||||
class TestQualityChecker(unittest.TestCase):
|
||||
"""Test quality checker functionality"""
|
||||
|
||||
def create_test_skill(self, tmpdir, skill_md_content, create_references=True):
|
||||
"""Helper to create a test skill directory"""
|
||||
skill_dir = Path(tmpdir) / "test-skill"
|
||||
skill_dir.mkdir()
|
||||
|
||||
# Create SKILL.md
|
||||
skill_md = skill_dir / "SKILL.md"
|
||||
skill_md.write_text(skill_md_content, encoding='utf-8')
|
||||
|
||||
# Create references directory
|
||||
if create_references:
|
||||
refs_dir = skill_dir / "references"
|
||||
refs_dir.mkdir()
|
||||
(refs_dir / "index.md").write_text("# Index\n\nTest reference.", encoding='utf-8')
|
||||
(refs_dir / "getting_started.md").write_text("# Getting Started\n\nHow to start.", encoding='utf-8')
|
||||
|
||||
return skill_dir
|
||||
|
||||
def test_checker_detects_missing_skill_md(self):
|
||||
"""Test that checker detects missing SKILL.md"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_dir = Path(tmpdir) / "test-skill"
|
||||
skill_dir.mkdir()
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have error about missing SKILL.md
|
||||
self.assertTrue(report.has_errors)
|
||||
self.assertTrue(any('SKILL.md' in issue.message for issue in report.errors))
|
||||
|
||||
def test_checker_detects_missing_references(self):
|
||||
"""Test that checker warns about missing references"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """---
|
||||
name: test
|
||||
---
|
||||
|
||||
# Test Skill
|
||||
|
||||
This is a test.
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md, create_references=False)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have warning about missing references
|
||||
self.assertTrue(report.has_warnings)
|
||||
self.assertTrue(any('references' in issue.message.lower() for issue in report.warnings))
|
||||
|
||||
def test_checker_detects_invalid_frontmatter(self):
|
||||
"""Test that checker detects invalid YAML frontmatter"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """# Test Skill
|
||||
|
||||
No frontmatter here!
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have error about missing frontmatter
|
||||
self.assertTrue(report.has_errors)
|
||||
self.assertTrue(any('frontmatter' in issue.message.lower() for issue in report.errors))
|
||||
|
||||
def test_checker_detects_missing_name_field(self):
|
||||
"""Test that checker detects missing name field in frontmatter"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """---
|
||||
description: test
|
||||
---
|
||||
|
||||
# Test Skill
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have error about missing name field
|
||||
self.assertTrue(report.has_errors)
|
||||
self.assertTrue(any('name' in issue.message.lower() for issue in report.errors))
|
||||
|
||||
def test_checker_detects_code_without_language(self):
|
||||
"""Test that checker warns about code blocks without language tags"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """---
|
||||
name: test
|
||||
---
|
||||
|
||||
# Test Skill
|
||||
|
||||
Here's some code:
|
||||
|
||||
```
|
||||
print("hello")
|
||||
```
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have warning about code without language
|
||||
self.assertTrue(report.has_warnings)
|
||||
self.assertTrue(any('language' in issue.message.lower() for issue in report.warnings))
|
||||
|
||||
def test_checker_approves_good_skill(self):
|
||||
"""Test that checker gives high score to well-formed skill"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """---
|
||||
name: test
|
||||
description: A test skill
|
||||
---
|
||||
|
||||
# Test Skill
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this when you need to test.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
Here are some examples:
|
||||
|
||||
```python
|
||||
def hello():
|
||||
print("hello")
|
||||
```
|
||||
|
||||
```javascript
|
||||
console.log("hello");
|
||||
```
|
||||
|
||||
## Example: Basic Usage
|
||||
|
||||
This shows how to use it.
|
||||
|
||||
## Reference Files
|
||||
|
||||
See the references directory for more:
|
||||
- [Getting Started](references/getting_started.md)
|
||||
- [Index](references/index.md)
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have no errors
|
||||
self.assertFalse(report.has_errors)
|
||||
|
||||
# Quality score should be high
|
||||
self.assertGreaterEqual(report.quality_score, 80.0)
|
||||
|
||||
def test_checker_detects_broken_links(self):
|
||||
"""Test that checker detects broken internal links"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
skill_md = """---
|
||||
name: test
|
||||
---
|
||||
|
||||
# Test Skill
|
||||
|
||||
See [this file](nonexistent.md) for more info.
|
||||
"""
|
||||
skill_dir = self.create_test_skill(tmpdir, skill_md)
|
||||
|
||||
checker = SkillQualityChecker(skill_dir)
|
||||
report = checker.check_all()
|
||||
|
||||
# Should have warning about broken link
|
||||
self.assertTrue(report.has_warnings)
|
||||
self.assertTrue(any('broken link' in issue.message.lower() for issue in report.warnings))
|
||||
|
||||
def test_quality_score_calculation(self):
|
||||
"""Test that quality score is calculated correctly"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
report = QualityReport("test", Path(tmpdir))
|
||||
|
||||
# Perfect score to start
|
||||
self.assertEqual(report.quality_score, 100.0)
|
||||
|
||||
# Add an error (should deduct 15 points)
|
||||
report.add_error('test', 'Test error')
|
||||
self.assertEqual(report.quality_score, 85.0)
|
||||
|
||||
# Add a warning (should deduct 5 points)
|
||||
report.add_warning('test', 'Test warning')
|
||||
self.assertEqual(report.quality_score, 80.0)
|
||||
|
||||
# Add more errors
|
||||
report.add_error('test', 'Another error')
|
||||
report.add_error('test', 'Yet another error')
|
||||
self.assertEqual(report.quality_score, 50.0)
|
||||
|
||||
def test_quality_grade_calculation(self):
|
||||
"""Test that quality grades are assigned correctly"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
report = QualityReport("test", Path(tmpdir))
|
||||
|
||||
# Grade A (90-100)
|
||||
self.assertEqual(report.quality_grade, 'A')
|
||||
|
||||
# Grade B (80-89)
|
||||
report.add_error('test', 'Error 1')
|
||||
self.assertEqual(report.quality_grade, 'B')
|
||||
|
||||
# Grade C (70-79)
|
||||
report.add_warning('test', 'Warning 1')
|
||||
report.add_warning('test', 'Warning 2')
|
||||
self.assertEqual(report.quality_grade, 'C')
|
||||
|
||||
# Grade D (60-69)
|
||||
report.add_warning('test', 'Warning 3')
|
||||
report.add_warning('test', 'Warning 4')
|
||||
self.assertEqual(report.quality_grade, 'D')
|
||||
|
||||
# Grade F (below 60)
|
||||
report.add_error('test', 'Error 2')
|
||||
report.add_error('test', 'Error 3')
|
||||
self.assertEqual(report.quality_grade, 'F')
|
||||
|
||||
def test_is_excellent_property(self):
|
||||
"""Test is_excellent property"""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
report = QualityReport("test", Path(tmpdir))
|
||||
|
||||
# Should be excellent with no issues
|
||||
self.assertTrue(report.is_excellent)
|
||||
|
||||
# Adding an error should make it not excellent
|
||||
report.add_error('test', 'Test error')
|
||||
self.assertFalse(report.is_excellent)
|
||||
|
||||
# Clean report
|
||||
report2 = QualityReport("test", Path(tmpdir))
|
||||
# Adding a warning should also make it not excellent
|
||||
report2.add_warning('test', 'Test warning')
|
||||
self.assertFalse(report2.is_excellent)
|
||||
|
||||
|
||||
class TestQualityCheckerCLI(unittest.TestCase):
|
||||
"""Test quality checker CLI"""
|
||||
|
||||
def test_cli_help_output(self):
|
||||
"""Test that CLI help works"""
|
||||
import subprocess
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['python3', '-m', 'skill_seekers.cli.quality_checker', '--help'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5
|
||||
)
|
||||
|
||||
# Should include usage info
|
||||
output = result.stdout + result.stderr
|
||||
self.assertTrue('usage:' in output.lower() or 'quality' in output.lower())
|
||||
except FileNotFoundError:
|
||||
self.skipTest("Module not installed")
|
||||
|
||||
def test_cli_with_nonexistent_directory(self):
|
||||
"""Test CLI behavior with nonexistent directory"""
|
||||
import subprocess
|
||||
|
||||
result = subprocess.run(
|
||||
['python3', '-m', 'skill_seekers.cli.quality_checker', '/nonexistent/path'],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
# Should fail
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -40,34 +40,50 @@ class TestSetupMCPScript:
|
||||
assert result.returncode == 0, f"Bash syntax error: {result.stderr}"
|
||||
|
||||
def test_references_correct_mcp_directory(self, script_content):
|
||||
"""Test that script references skill_seeker_mcp/ not old mcp/ directory"""
|
||||
# Should NOT reference old mcp/ directory
|
||||
old_refs = re.findall(r'(?:^|[^a-z_])mcp/(?!\.json)', script_content, re.MULTILINE)
|
||||
assert len(old_refs) == 0, f"Found {len(old_refs)} references to old 'mcp/' directory: {old_refs}"
|
||||
"""Test that script references src/skill_seekers/mcp/ (v2.0.0 layout)"""
|
||||
# Should NOT reference old mcp/ or skill_seeker_mcp/ directories
|
||||
old_mcp_refs = re.findall(r'(?:^|[^a-z_])(?<!/)mcp/(?!\.json)', script_content, re.MULTILINE)
|
||||
old_skill_seeker_refs = re.findall(r'skill_seeker_mcp/', script_content)
|
||||
|
||||
# SHOULD reference skill_seeker_mcp/
|
||||
new_refs = re.findall(r'skill_seeker_mcp/', script_content)
|
||||
assert len(new_refs) >= 6, f"Expected at least 6 references to 'skill_seeker_mcp/', found {len(new_refs)}"
|
||||
# Allow /mcp/ (as in src/skill_seekers/mcp/) but not standalone mcp/
|
||||
assert len(old_mcp_refs) == 0, f"Found {len(old_mcp_refs)} references to old 'mcp/' directory: {old_mcp_refs}"
|
||||
assert len(old_skill_seeker_refs) == 0, f"Found {len(old_skill_seeker_refs)} references to old 'skill_seeker_mcp/': {old_skill_seeker_refs}"
|
||||
|
||||
# SHOULD reference src/skill_seekers/mcp/
|
||||
new_refs = re.findall(r'src/skill_seekers/mcp/', script_content)
|
||||
assert len(new_refs) >= 6, f"Expected at least 6 references to 'src/skill_seekers/mcp/', found {len(new_refs)}"
|
||||
|
||||
def test_requirements_txt_path(self, script_content):
|
||||
"""Test that requirements.txt path is correct"""
|
||||
assert "skill_seeker_mcp/requirements.txt" in script_content, \
|
||||
"Should reference skill_seeker_mcp/requirements.txt"
|
||||
# Check for old mcp/ directory (but not skill_seeker_mcp/)
|
||||
"""Test that script uses pip install -e . (v2.0.0 modern packaging)"""
|
||||
# v2.0.0 uses '-e .' (editable install) instead of requirements files
|
||||
# The actual command is "$PIP_INSTALL_CMD -e ."
|
||||
assert " -e ." in script_content or " -e." in script_content, \
|
||||
"Should use '-e .' for editable install (modern packaging)"
|
||||
|
||||
# Should NOT reference old requirements.txt paths
|
||||
import re
|
||||
old_refs = re.findall(r'(?<!skill_seeker_)mcp/requirements\.txt', script_content)
|
||||
assert len(old_refs) == 0, \
|
||||
f"Should NOT reference old 'mcp/requirements.txt' (found {len(old_refs)}): {old_refs}"
|
||||
old_skill_seeker_refs = re.findall(r'skill_seeker_mcp/requirements\.txt', script_content)
|
||||
old_mcp_refs = re.findall(r'(?<!skill_seeker_)mcp/requirements\.txt', script_content)
|
||||
|
||||
assert len(old_skill_seeker_refs) == 0, \
|
||||
f"Should NOT reference 'skill_seeker_mcp/requirements.txt' (found {len(old_skill_seeker_refs)})"
|
||||
assert len(old_mcp_refs) == 0, \
|
||||
f"Should NOT reference old 'mcp/requirements.txt' (found {len(old_mcp_refs)})"
|
||||
|
||||
def test_server_py_path(self, script_content):
|
||||
"""Test that server.py path is correct"""
|
||||
"""Test that server.py path is correct (v2.0.0 layout)"""
|
||||
import re
|
||||
assert "skill_seeker_mcp/server.py" in script_content, \
|
||||
"Should reference skill_seeker_mcp/server.py"
|
||||
# Check for old mcp/ directory (but not skill_seeker_mcp/)
|
||||
old_refs = re.findall(r'(?<!skill_seeker_)mcp/server\.py', script_content)
|
||||
assert len(old_refs) == 0, \
|
||||
f"Should NOT reference old 'mcp/server.py' (found {len(old_refs)}): {old_refs}"
|
||||
assert "src/skill_seekers/mcp/server.py" in script_content, \
|
||||
"Should reference src/skill_seekers/mcp/server.py"
|
||||
|
||||
# Should NOT reference old paths
|
||||
old_skill_seeker_refs = re.findall(r'skill_seeker_mcp/server\.py', script_content)
|
||||
old_mcp_refs = re.findall(r'(?<!/)(?<!skill_seekers/)mcp/server\.py', script_content)
|
||||
|
||||
assert len(old_skill_seeker_refs) == 0, \
|
||||
f"Should NOT reference old 'skill_seeker_mcp/server.py' (found {len(old_skill_seeker_refs)})"
|
||||
assert len(old_mcp_refs) == 0, \
|
||||
f"Should NOT reference old 'mcp/server.py' (found {len(old_mcp_refs)})"
|
||||
|
||||
def test_referenced_files_exist(self):
|
||||
"""Test that all files referenced in setup_mcp.sh actually exist"""
|
||||
@@ -88,10 +104,10 @@ class TestSetupMCPScript:
|
||||
assert os.access(script_path, os.X_OK), "setup_mcp.sh should be executable"
|
||||
|
||||
def test_json_config_path_format(self, script_content):
|
||||
"""Test that JSON config examples use correct format"""
|
||||
"""Test that JSON config examples use correct format (v2.0.0 layout)"""
|
||||
# Check for the config path format in the script
|
||||
assert '"$REPO_PATH/skill_seeker_mcp/server.py"' in script_content, \
|
||||
"Config should show correct server.py path with $REPO_PATH variable"
|
||||
assert '"$REPO_PATH/src/skill_seekers/mcp/server.py"' in script_content, \
|
||||
"Config should show correct server.py path with $REPO_PATH variable (v2.0.0 layout)"
|
||||
|
||||
def test_no_hardcoded_paths(self, script_content):
|
||||
"""Test that script doesn't contain hardcoded absolute paths"""
|
||||
|
||||
@@ -164,9 +164,9 @@ class TestDetectTerminalApp(unittest.TestCase):
|
||||
# Mock Popen to prevent actual terminal launch
|
||||
mock_popen.return_value = MagicMock()
|
||||
|
||||
# Run enhancer
|
||||
# Run enhancer in interactive mode (not headless)
|
||||
enhancer = LocalSkillEnhancer(skill_dir)
|
||||
result = enhancer.run()
|
||||
result = enhancer.run(headless=False)
|
||||
|
||||
# Verify Popen was called
|
||||
self.assertTrue(mock_popen.called)
|
||||
@@ -239,7 +239,8 @@ class TestDetectTerminalApp(unittest.TestCase):
|
||||
old_stdout = sys.stdout
|
||||
sys.stdout = captured_output
|
||||
|
||||
result = enhancer.run()
|
||||
# Run in interactive mode (not headless) to test terminal launch
|
||||
result = enhancer.run(headless=False)
|
||||
|
||||
# Restore stdout
|
||||
sys.stdout = old_stdout
|
||||
@@ -279,7 +280,8 @@ class TestDetectTerminalApp(unittest.TestCase):
|
||||
# Mock Popen to prevent actual launch
|
||||
with patch('subprocess.Popen') as mock_popen:
|
||||
mock_popen.return_value = MagicMock()
|
||||
enhancer.run()
|
||||
# Run in interactive mode (not headless) to test terminal detection
|
||||
enhancer.run(headless=False)
|
||||
|
||||
# Restore stdout
|
||||
sys.stdout = old_stdout
|
||||
|
||||
Reference in New Issue
Block a user