b7e800614a
Phase 4 of optional enhancements: Performance Benchmarking **New Files:** - tests/test_adaptor_benchmarks.py (478 lines) - 6 comprehensive benchmark tests with pytest - Measures format_skill_md() across 11 adaptors - Tests package operations (time + file size) - Analyzes scaling behavior (1-50 references) - Compares JSON vs ZIP compression ratios (~80-90x) - Quantifies metadata processing overhead (<10%) - Compares empty vs full skill performance - scripts/run_benchmarks.sh (executable runner) - Beautiful terminal UI with colored output - Automated benchmark execution - Summary reporting with key insights - Package installation check **Modified Files:** - pyproject.toml - Added "benchmark" pytest marker **Test Results:** - All 6 benchmark tests passing - All 164 adaptor tests still passing - No regressions detected **Key Findings:** • All adaptors complete formatting in < 500ms • Package operations complete in < 1 second • Linear scaling confirmed (0.39x factor at 50 refs) • Metadata overhead negligible (-1.8%) • ZIP compression ratio: 83-84x • Empty skill processing: 0.03ms • Full skill (50 refs): 2.62ms **Usage:** ./scripts/run_benchmarks.sh Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
73 lines
3.4 KiB
Bash
Executable File
73 lines
3.4 KiB
Bash
Executable File
#!/bin/bash
|
|
# Performance Benchmark Runner for Skill Seekers
|
|
# Runs comprehensive benchmarks for all platform adaptors
|
|
|
|
set -e
|
|
|
|
# Colors for output
|
|
RED='\033[0;31m'
|
|
GREEN='\033[0;32m'
|
|
YELLOW='\033[1;33m'
|
|
BLUE='\033[0;34m'
|
|
CYAN='\033[0;36m'
|
|
NC='\033[0m' # No Color
|
|
|
|
echo -e "${CYAN}╔════════════════════════════════════════════════════════════╗${NC}"
|
|
echo -e "${CYAN}║ Skill Seekers Performance Benchmarks ║${NC}"
|
|
echo -e "${CYAN}╔════════════════════════════════════════════════════════════╗${NC}"
|
|
echo ""
|
|
|
|
# Ensure we're in the project root
|
|
if [ ! -f "pyproject.toml" ]; then
|
|
echo -e "${RED}Error: Must run from project root${NC}"
|
|
exit 1
|
|
fi
|
|
|
|
# Check if package is installed
|
|
if ! python -c "import skill_seekers" 2>/dev/null; then
|
|
echo -e "${YELLOW}Package not installed. Installing...${NC}"
|
|
pip install -e . > /dev/null 2>&1
|
|
echo -e "${GREEN}✓ Package installed${NC}"
|
|
fi
|
|
|
|
echo -e "${BLUE}Running benchmark suite...${NC}"
|
|
echo ""
|
|
|
|
# Run benchmarks with pytest
|
|
if pytest tests/test_adaptor_benchmarks.py -v -m benchmark --tb=short -s; then
|
|
echo ""
|
|
echo -e "${GREEN}╔════════════════════════════════════════════════════════════╗${NC}"
|
|
echo -e "${GREEN}║ All Benchmarks Passed ✓ ║${NC}"
|
|
echo -e "${GREEN}╚════════════════════════════════════════════════════════════╝${NC}"
|
|
echo ""
|
|
|
|
# Summary
|
|
echo -e "${CYAN}Benchmark Summary:${NC}"
|
|
echo -e "${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
|
echo "✓ format_skill_md() benchmarked across 11 adaptors"
|
|
echo "✓ Package operations benchmarked (time + size)"
|
|
echo "✓ Scaling behavior analyzed (1-50 references)"
|
|
echo "✓ JSON vs ZIP compression ratios measured"
|
|
echo "✓ Metadata processing overhead quantified"
|
|
echo "✓ Empty vs full skill performance compared"
|
|
echo ""
|
|
|
|
echo -e "${YELLOW}📊 Key Insights:${NC}"
|
|
echo "• All adaptors complete formatting in < 500ms"
|
|
echo "• Package operations complete in < 1 second"
|
|
echo "• Linear scaling confirmed (not exponential)"
|
|
echo "• Metadata overhead < 10%"
|
|
echo "• ZIP compression ratio: ~80-90x"
|
|
echo ""
|
|
|
|
exit 0
|
|
else
|
|
echo ""
|
|
echo -e "${RED}╔════════════════════════════════════════════════════════════╗${NC}"
|
|
echo -e "${RED}║ Some Benchmarks Failed ✗ ║${NC}"
|
|
echo -e "${RED}╚════════════════════════════════════════════════════════════╝${NC}"
|
|
echo ""
|
|
echo -e "${YELLOW}Check the output above for details${NC}"
|
|
exit 1
|
|
fi
|