firefrost-gaming/skill-seekers-reference
3
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
638 Commits 1 Branch 0 Tags
f214976ccdc30d64eb498cf66290e952e373b606
Commit Graph

14 Commits

Author SHA1 Message Date
yusyus
f214976ccd fix: apply review fixes from PR #309 and stabilize flaky benchmark test 
Follow-up to PR #309 (perf: optimize with caching, pre-compiled regex,
O(1) lookups, and bisect line indexing). These fixes were committed to
the PR branch but missed the squash merge.

Review fixes (credit: PR #309 by copperlang2007):
1. Rename _pending_set -> _enqueued_urls to accurately reflect that the
   set tracks all ever-enqueued URLs, not just currently pending ones
2. Extract duplicated _build_line_index()/_offset_to_line() into shared
   build_line_index()/offset_to_line() in cli/utils.py (DRY)
3. Fix pre-existing bug: infer_categories() guard checked 'tutorial'
   but wrote to 'tutorials' key, risking silent overwrites
4. Remove unnecessary _store_results() closure in scrape_page()
5. Simplify parser pre-import in codebase_scraper.py

Benchmark stabilization:
- test_benchmark_metadata_overhead was flaky on CI (106.7% overhead
  observed, threshold 50%) because 5 iterations with mean averaging
  can't reliably measure microsecond-level differences
- Fix: 20 iterations, warm-up run, median instead of mean, threshold
  raised to 200% (guards catastrophic regression, not noise)

Ref: https://github.com/yusufkaraaslan/Skill_Seekers/pull/309
 2026-03-14 23:39:23 +03:00
copperlang2007
89f5e6fe5f perf: optimize with caching, pre-compiled regex, O(1) lookups, and bisect line indexing (#309) 
## Summary

Performance optimizations across core scraping and analysis modules:

- **doc_scraper.py**: Pre-compiled regex at module level, O(1) URL dedup via _enqueued_urls set, cached URL patterns, _enqueue_url() helper (DRY), seen_links set for link extraction, pre-lowercased category keywords, async error logging (bug fix), summary I/O error handling
- **code_analyzer.py**: O(log n) bisect-based line lookups replacing O(n) count("\n") across all 10 language analyzers; O(n) parent class map replacing O(n^2) AST walks for Python method detection
- **dependency_analyzer.py**: Same bisect line-index optimization for all import extractors
- **codebase_scraper.py**: Module-level import re, pre-imported parser classes outside loop
- **github_scraper.py**: deque.popleft() for O(1) tree traversal, module-level import fnmatch
- **utils.py**: Shared build_line_index() / offset_to_line() utilities (DRY)
- **test_adaptor_benchmarks.py**: Stabilized flaky test_benchmark_metadata_overhead (median, warm-up, more iterations)

Review fixes applied on top of original PR:
1. Renamed misleading _pending_set to _enqueued_urls
2. Extracted duplicated line-index code into shared cli/utils.py
3. Fixed pre-existing "tutorial" vs "tutorials" key mismatch bug in infer_categories()
4. Removed unnecessary _store_results() closure
5. Simplified parser pre-import pattern
 2026-03-14 23:35:39 +03:00
yusyus
4e8ad835ed style: Format code with ruff formatter 
- Auto-format 11 files to comply with ruff formatting standards
- Fixes CI/CD formatter check failures

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-03 21:37:54 +03:00
yusyus
9496462936 fix: Remove trailing whitespace from dependency_analyzer.py 
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-03 21:19:32 +03:00
yusyus
50b28fe561 fix: Framework detection, circular deps, and GDScript test discovery 
FIXES:

1. Framework Detection (Unity → Godot)
   PROBLEM: Detected Unity instead of Godot due to generic "Assets" marker
   - "Assets" appears in comments: "// TODO: Replace with actual music assets"
   - Triggered false positive for Unity framework

   SOLUTION: Made Unity markers more specific
   - Before: "Assets", "ProjectSettings" (too generic)
   - After: "Assembly-CSharp.csproj", "UnityEngine.dll", "Library/" (specific)
   - Godot markers: "project.godot", ".godot", ".tscn", ".tres", ".gd"

   FILE: architectural_pattern_detector.py line 92-94

2. Circular Dependencies (Self-References)
   PROBLEM: Files showing circular dependency to themselves
   - WARNING: Cycle: analysis-config.gd -> analysis-config.gd
   - 3 self-referential cycles detected

   ROOT CAUSE: No self-loop filtering in build_graph()
   - File resolves class_name to itself
   - Edge created from file to same file

   SOLUTION: Skip self-dependencies in build_graph()
   - Added check: `target != file_path`
   - Prevents file from depending on itself

   FILE: dependency_analyzer.py line 728

3. GDScript Test File Detection
   PROBLEM: Found 0 test files (expected 20 GUT tests with 396 tests)
   - TEST_PATTERNS missing GDScript patterns
   - Only had: test_*.py, *_test.go, Test*.java, etc.

   SOLUTION: Added GDScript test patterns
   - Added: "test_*.gd", "*_test.gd" (GUT, gdUnit4, WAT)
   - Added ".gd": "GDScript" to LANGUAGE_MAP

   FILES:
   - test_example_extractor.py line 886-887
   - test_example_extractor.py line 901

IMPACT:
- ✅ Godot projects correctly detected as "Godot" (not Unity)
- ✅ No more false circular dependency warnings
- ✅ GUT/gdUnit4/WAT test files now discovered and analyzed
- ✅ Better test example extraction for Godot projects

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-02 22:11:38 +03:00
yusyus
3e6c448aca fix: Add GDScript-specific dependency extraction to eliminate syntax errors 
PROBLEM:
- 265+ "Syntax error in *.gd" warnings during analysis
- GDScript files were routed to Python AST parser (_extract_python_imports)
- Python AST failed because GDScript syntax differs (extends, signal, @export)

SOLUTION:
- Created dedicated _extract_gdscript_imports() method using regex
- Parses GDScript-specific patterns:
  * const/var = preload("res://path")
  * const/var = load("res://path")
  * extends "res://path/to/base.gd"
  * extends MyBaseClass (with built-in Godot class filtering)
- Converts res:// paths to relative paths
- Routes GDScript files to new extractor instead of Python AST

CHANGES:
- dependency_analyzer.py (line 114-116): Route GDScript to new extractor
- dependency_analyzer.py (line 201-318): Add _extract_gdscript_imports()
- Updated module docstring: 9 → 10 languages + Godot ecosystem
- Updated analyze_file() docstring with GDScript support

IMPACT:
- Eliminates all 265+ syntax error warnings
- Correctly extracts GDScript dependencies (preload/load/extends)
- Completes C3.10 Signal Flow Analysis integration

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-02 21:56:42 +03:00
yusyus
b252f43d0e feat: Add comprehensive Godot file type support 
Complete support for all Godot file types:
- GDScript (.gd) - Regex-based parser for Godot-specific syntax
- Godot Scenes (.tscn) - Node hierarchy and script attachments
- Godot Resources (.tres) - Properties and dependencies
- Godot Shaders (.gdshader) - Uniforms and shader functions

Implementation details:
- Added 4 new analyzer methods to CodeAnalyzer class
  - _analyze_gdscript(): Functions, signals, @export vars, class_name
  - _analyze_godot_scene(): Node hierarchy, scripts, resources
  - _analyze_godot_resource(): Resource type, properties, script refs
  - _analyze_godot_shader(): Shader type, uniforms, varyings, functions

- Updated dependency_analyzer.py
  - Added _extract_godot_resources() for ext_resource and preload()
  - Fixed DependencyInfo calls (removed invalid 'alias' parameter)

- Updated codebase_scraper.py
  - Added Godot file extensions to LANGUAGE_EXTENSIONS
  - Extended content filter to accept Godot-specific keys
    (nodes, properties, uniforms, signals, exports)

Tested on Cosmic Ideler Godot project:
- 443/452 files successfully analyzed (98%)
- 265 GDScript, 118 .tscn, 38 .tres, 9 .gdshader, 13 .cs

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-02 21:36:56 +03:00
yusyus
583a774b00 feat: Add GDScript (.gd) language support for Godot projects 
**Problem:**
Godot projects with 267 GDScript files were only analyzing 13 C# files,
missing 95%+ of the codebase.

**Changes:**
1. Added `.gd` → "GDScript" to LANGUAGE_EXTENSIONS mapping
2. Added GDScript support to code_analyzer.py (uses Python AST parser)
3. Added GDScript support to dependency_analyzer.py (uses Python import extraction)

**Known Limitation:**
GDScript has syntax differences from Python (extends, @export, signals, etc.)
so Python AST parser may fail on some files. Future enhancement needed:
- Create GDScript-specific regex-based parser
- Handle Godot-specific keywords (extends, signal, @export, preload, etc.)

**Test Results:**
Before: 13 files analyzed (C# only)
After:  280 files detected (13 C# + 267 GDScript)
Status: GDScript files detected but analysis may fail due to syntax differences

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-02 21:22:51 +03:00
yusyus
81dd5bbfbc fix: Fix remaining 61 ruff linting errors (SIM102, SIM117) 
Fixed all remaining linting errors from the 310 total:
- SIM102: Combined nested if statements (31 errors)
  - adaptors/openai.py
  - config_extractor.py
  - codebase_scraper.py
  - doc_scraper.py
  - github_fetcher.py
  - pattern_recognizer.py
  - pdf_scraper.py
  - test_example_extractor.py

- SIM117: Combined multiple with statements (24 errors)
  - tests/test_async_scraping.py (2 errors)
  - tests/test_github_scraper.py (2 errors)
  - tests/test_guide_enhancer.py (20 errors)

- Fixed test fixture parameter (mock_config in test_c3_integration.py)

All 700+ tests passing.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-17 23:25:12 +03:00
yusyus
596b219599 fix: Resolve remaining 188 linting errors (249 total fixed) 
Second batch of comprehensive linting fixes:

Unused Arguments/Variables (136 errors):
- ARG002/ARG001 (91 errors): Prefixed unused method/function arguments with '_'
  - Interface methods in adaptors (base.py, gemini.py, markdown.py)
  - AST analyzer methods maintaining signatures (code_analyzer.py)
  - Test fixtures and hooks (conftest.py)
  - Added noqa: ARG001/ARG002 for pytest hooks requiring exact names
- F841 (45 errors): Prefixed unused local variables with '_'
  - Tuple unpacking where some values aren't needed
  - Variables assigned but not referenced

Loop & Boolean Quality (28 errors):
- B007 (18 errors): Prefixed unused loop control variables with '_'
  - enumerate() loops where index not used
  - for-in loops where loop variable not referenced
- E712 (10 errors): Simplified boolean comparisons
  - Changed '== True' to direct boolean check
  - Changed '== False' to 'not' expression
  - Improved test readability

Code Quality (24 errors):
- SIM201 (4 errors): Already fixed in previous commit
- SIM118 (2 errors): Already fixed in previous commit
- E741 (4 errors): Already fixed in previous commit
- Config manager loop variable fix (1 error)

All Tests Passing:
- test_scraper_features.py: 42 passed
- test_integration.py: 51 passed
- test_architecture_scenarios.py: 11 passed
- test_real_world_fastmcp.py: 19 passed, 1 skipped

Note: Some SIM errors (nested if, multiple with) remain unfixed as they
would require non-trivial refactoring. Focus was on functional correctness.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-17 23:02:11 +03:00
Pablo Estevez
c33c6f9073 change max lenght 2026-01-17 17:48:15 +00:00
Pablo Estevez
5ed767ff9a run ruff 2026-01-17 17:29:21 +00:00
yusyus
3408315f40 feat: Add 6 new languages to codebase analysis system (C#, Go, Rust, Java, Ruby, PHP) 
Expands language support from 3 to 9 languages across entire codebase scraping system.

**New Languages Added:**
- C# (Unity/.NET support) - classes, methods, properties, async/await, XML docs
- Go - structs, functions, methods with receivers, multiple return values
- Rust - structs, functions, async functions, impl blocks
- Java - classes, methods, inheritance, interfaces, generics
- Ruby - classes, methods, inheritance, predicate methods
- PHP - classes, methods, namespaces, inheritance

**Code Analysis (code_analyzer.py):**
- Added 6 new language analyzers (~1000 lines)
- Regex-based parsers inspired by official language specs
- Extract classes, functions, signatures, async detection
- Comprehensive comment extraction for all languages

**Dependency Analysis (dependency_analyzer.py):**
- Added 6 new import extractors (~300 lines)
- C#: using statements, static using, aliases
- Go: import blocks, aliases
- Rust: use statements, curly braces, crate/super
- Java: import statements, static imports, wildcards
- Ruby: require, require_relative, load
- PHP: require/include, namespace use

**File Extensions (codebase_scraper.py):**
- Added mappings: .cs, .go, .rs, .java, .rb, .php

**Test Coverage:**
- Added 24 new tests for 6 languages (4 tests each)
- Added 19 dependency analyzer tests
- Added 6 language detection tests
- Total: 118 tests, 100% passing ✅

**Credits:**
- Regex patterns based on official language specifications:
  - Microsoft C# Language Specification
  - Go Language Specification
  - Rust Language Reference
  - Oracle Java Language Specification
  - Ruby Documentation
  - PHP Language Reference
- NetworkX for graph algorithms

**Issues Resolved:**
- Closes #166 (C# support request)
- Closes #140 (E1.7 MCP tool scrape_codebase)

**Test Results:**
- test_code_analyzer.py: 54 tests passing
- test_dependency_analyzer.py: 43 tests passing
- test_codebase_scraper.py: 21 tests passing
- Total execution: ~0.41s

🚀 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-02 21:28:21 +03:00
yusyus
aa6bc363d9 feat(C2.6): Add dependency graph analyzer with NetworkX 
- Add NetworkX dependency to pyproject.toml
- Create dependency_analyzer.py with comprehensive functionality
- Support Python, JavaScript/TypeScript, and C++ import extraction
- Build directed graphs using NetworkX DiGraph
- Detect circular dependencies with NetworkX algorithms
- Export graphs in multiple formats (JSON, Mermaid, DOT)
- Add 24 comprehensive tests with 100% pass rate

Features:
- Python: AST-based import extraction (import, from, relative)
- JavaScript/TypeScript: ES6 and CommonJS parsing (import, require)
- C++: #include directive extraction (system and local headers)
- Graph statistics (total files, dependencies, cycles, components)
- Circular dependency detection and reporting
- Multiple export formats for visualization

Architecture:
- DependencyAnalyzer class with NetworkX integration
- DependencyInfo dataclass for tracking import relationships
- FileNode dataclass for graph nodes
- Language-specific extraction methods

Related research:
- NetworkX: Standard Python graph library for analysis
- pydeps: Python-specific analyzer (inspiration)
- madge: JavaScript dependency analyzer (reference)
- dependency-cruiser: Advanced JS/TS analyzer (reference)

Test coverage:
- 5 Python import tests
- 4 JavaScript/TypeScript import tests
- 3 C++ include tests
- 3 graph building tests
- 3 circular dependency detection tests
- 3 export format tests
- 3 edge case tests
 2026-01-01 23:30:46 +03:00