skill-seekers-reference/docs/archive/research/PDF_EXTRACTOR_POC.md

PDF Extractor - Proof of Concept (Task B1.2)

Status: ✅ Completed Date: October 21, 2025 Task: B1.2 - Create simple PDF text extractor (proof of concept)

---

Overview

This is a proof-of-concept PDF text and code extractor built for Skill Seeker. It demonstrates the feasibility of extracting documentation content from PDF files using PyMuPDF (fitz).

Features

✅ Implemented

1. Text Extraction - Extract plain text from all PDF pages
2. Markdown Conversion - Convert PDF content to markdown format
3. Code Block Detection - Multiple detection methods:
   • Font-based: Detects monospace fonts (Courier, Mono, Consolas, etc.)
   • Indent-based: Detects consistently indented code blocks
   • Pattern-based: Detects function/class definitions, imports
4. Language Detection - Auto-detect programming language from code content
5. Heading Extraction - Extract document structure from markdown
6. Image Counting - Track diagrams and screenshots
7. JSON Output - Compatible format with existing doc_scraper.py

🎯 Detection Methods

Font-Based Detection

Analyzes font properties to find monospace fonts typically used for code:

• Courier, Courier New
• Monaco, Menlo
• Consolas
• DejaVu Sans Mono

Indentation-Based Detection

Identifies code blocks by consistent indentation patterns:

• 4 spaces or tabs
• Minimum 2 consecutive lines
• Minimum 20 characters

Pattern-Based Detection

Uses regex to find common code structures:

• Function definitions (Python, JS, Go, etc.)
• Class definitions
• Import/require statements

🔍 Language Detection

Supports detection of 19 programming languages:

• Python, JavaScript, Java, C, C++, C#
• Go, Rust, PHP, Ruby, Swift, Kotlin
• Shell, SQL, HTML, CSS
• JSON, YAML, XML

---

Installation

Prerequisites

pip install PyMuPDF

Verify Installation

python3 -c "import fitz; print(fitz.__doc__)"

---

Usage

Basic Usage

# Extract from PDF (print to stdout)
python3 cli/pdf_extractor_poc.py input.pdf

# Save to JSON file
python3 cli/pdf_extractor_poc.py input.pdf --output result.json

# Verbose mode (shows progress)
python3 cli/pdf_extractor_poc.py input.pdf --verbose

# Pretty-printed JSON
python3 cli/pdf_extractor_poc.py input.pdf --pretty

Examples

# Extract Python documentation
python3 cli/pdf_extractor_poc.py docs/python_guide.pdf -o python_extracted.json -v

# Extract with verbose and pretty output
python3 cli/pdf_extractor_poc.py manual.pdf -o manual.json -v --pretty

# Quick test (print to screen)
python3 cli/pdf_extractor_poc.py sample.pdf --pretty

---

Output Format

JSON Structure

{
  "source_file": "input.pdf",
  "metadata": {
    "title": "Documentation Title",
    "author": "Author Name",
    "subject": "Subject",
    "creator": "PDF Creator",
    "producer": "PDF Producer"
  },
  "total_pages": 50,
  "total_chars": 125000,
  "total_code_blocks": 87,
  "total_headings": 45,
  "total_images": 12,
  "languages_detected": {
    "python": 52,
    "javascript": 20,
    "sql": 10,
    "shell": 5
  },
  "pages": [
    {
      "page_number": 1,
      "text": "Plain text content...",
      "markdown": "# Heading\nContent...",
      "headings": [
        {
          "level": "h1",
          "text": "Getting Started"
        }
      ],
      "code_samples": [
        {
          "code": "def hello():\n    print('Hello')",
          "language": "python",
          "detection_method": "font",
          "font": "Courier-New"
        }
      ],
      "images_count": 2,
      "char_count": 2500,
      "code_blocks_count": 3
    }
  ]
}

Page Object

Each page contains:

• page_number - 1-indexed page number
• text - Plain text content
• markdown - Markdown-formatted content
• headings - Array of heading objects
• code_samples - Array of detected code blocks
• images_count - Number of images on page
• char_count - Character count
• code_blocks_count - Number of code blocks found

Code Sample Object

Each code sample includes:

• code - The actual code text
• language - Detected language (or 'unknown')
• detection_method - How it was found ('font', 'indent', or 'pattern')
• font - Font name (if detected by font method)
• pattern_type - Type of pattern (if detected by pattern method)

---

Technical Details

Detection Accuracy

Font-based detection: ⭐⭐⭐⭐⭐ (Best)

• Highly accurate for well-formatted PDFs
• Relies on proper font usage in source document
• Works with: Technical docs, programming books, API references

Indent-based detection: ⭐⭐⭐⭐ (Good)

• Good for structured code blocks
• May capture non-code indented content
• Works with: Tutorials, guides, examples

Pattern-based detection: ⭐⭐⭐ (Fair)

• Captures specific code constructs
• May miss complex or unusual code
• Works with: Code snippets, function examples

Language Detection Accuracy

• High confidence: Python, JavaScript, Java, Go, SQL
• Medium confidence: C++, Rust, PHP, Ruby, Swift
• Basic detection: Shell, JSON, YAML, XML

Detection based on keyword patterns, not AST parsing.

Performance

Tested on various PDF sizes:

• Small (1-10 pages): < 1 second
• Medium (10-100 pages): 1-5 seconds
• Large (100-500 pages): 5-30 seconds
• Very Large (500+ pages): 30+ seconds

Memory usage: ~50-200 MB depending on PDF size and image content.

---

Limitations

Current Limitations

1. No OCR - Cannot extract text from scanned/image PDFs
2. No Table Extraction - Tables are treated as plain text
3. No Image Extraction - Only counts images, doesn't extract them
4. Simple Deduplication - May miss some duplicate code blocks
5. No Multi-column Support - May jumble multi-column layouts

Known Issues

1. Code Split Across Pages - Code blocks spanning pages may be split
2. Complex Layouts - May struggle with complex PDF layouts
3. Non-standard Fonts - May miss code in non-standard monospace fonts
4. Unicode Issues - Some special characters may not preserve correctly

---

Comparison with Web Scraper

Feature	Web Scraper	PDF Extractor POC
Content source	HTML websites	PDF files
Code detection	CSS selectors	Font/indent/pattern
Language detection	CSS classes + heuristics	Pattern matching
Structure	Excellent	Good
Links	Full support	Not supported
Images	Referenced	Counted only
Categories	Auto-categorized	Not implemented
Output format	JSON	JSON (compatible)

---

Next Steps (Tasks B1.3-B1.8)

B1.3: Add PDF Page Detection and Chunking

• Split large PDFs into manageable chunks
• Handle page-spanning code blocks
• Add chapter/section detection

B1.4: Extract Code Blocks from PDFs

• Improve code block detection accuracy
• Add syntax validation
• Better language detection (use tree-sitter?)

B1.5: Add PDF Image Extraction

• Extract diagrams as separate files
• Extract screenshots
• OCR support for code in images

B1.6: Create pdf_scraper.py CLI Tool

• Full-featured CLI like doc_scraper.py
• Config file support
• Category detection
• Multi-PDF support

B1.7: Add MCP Tool scrape_pdf

• Integrate with MCP server
• Add to existing 9 MCP tools
• Test with Claude Code

B1.8: Create PDF Config Format

• Define JSON config for PDF sources
• Similar to web scraper configs
• Support multiple PDFs per skill

---

Testing

Manual Testing

1. Create test PDF (or use existing PDF documentation)
2. Run extractor:

   python3 cli/pdf_extractor_poc.py test.pdf -o test_result.json -v --pretty
   

3. Verify output:
   • Check total_code_blocks > 0
   • Verify languages_detected includes expected languages
   • Inspect code_samples for accuracy

Test with Real Documentation

Recommended test PDFs:

• Python documentation (python.org)
• Django documentation
• PostgreSQL manual
• Any programming language reference

Expected Results

Good PDF (well-formatted with monospace code):

• Detection rate: 80-95%
• Language accuracy: 85-95%
• False positives: < 5%

Poor PDF (scanned or badly formatted):

• Detection rate: 20-50%
• Language accuracy: 60-80%
• False positives: 10-30%

---

Code Examples

Using PDFExtractor Class Directly

from cli.pdf_extractor_poc import PDFExtractor

# Create extractor
extractor = PDFExtractor('docs/manual.pdf', verbose=True)

# Extract all pages
result = extractor.extract_all()

# Access data
print(f"Total pages: {result['total_pages']}")
print(f"Code blocks: {result['total_code_blocks']}")
print(f"Languages: {result['languages_detected']}")

# Iterate pages
for page in result['pages']:
    print(f"\nPage {page['page_number']}:")
    print(f"  Code blocks: {page['code_blocks_count']}")
    for code in page['code_samples']:
        print(f"  - {code['language']}: {len(code['code'])} chars")

Custom Language Detection

from cli.pdf_extractor_poc import PDFExtractor

extractor = PDFExtractor('input.pdf')

# Override language detection
def custom_detect(code):
    if 'SELECT' in code.upper():
        return 'sql'
    return extractor.detect_language_from_code(code)

# Use in extraction
# (requires modifying the class to support custom detection)

---

Contributing

Adding New Languages

To add language detection for a new language, edit detect_language_from_code():

patterns = {
    # ... existing languages ...
    'newlang': [r'pattern1', r'pattern2', r'pattern3'],
}

Adding Detection Methods

To add a new detection method, create a method like:

def detect_code_blocks_by_newmethod(self, page):
    """Detect code using new method"""
    code_blocks = []
    # ... your detection logic ...
    return code_blocks

Then add it to extract_page():

newmethod_code_blocks = self.detect_code_blocks_by_newmethod(page)
all_code_blocks = font_code_blocks + indent_code_blocks + pattern_code_blocks + newmethod_code_blocks

---

Conclusion

This POC successfully demonstrates:

• ✅ PyMuPDF can extract text from PDF documentation
• ✅ Multiple detection methods can identify code blocks
• ✅ Language detection works for common languages
• ✅ JSON output is compatible with existing doc_scraper.py
• ✅ Performance is acceptable for typical documentation PDFs

Ready for B1.3: The foundation is solid. Next step is adding page chunking and handling large PDFs.

---

POC Completed: October 21, 2025 Next Task: B1.3 - Add PDF page detection and chunking