Release v1.8.0: Add transcript-fixer skill
## New Skill: transcript-fixer v1.0.0 Correct speech-to-text (ASR/STT) transcription errors through dictionary-based rules and AI-powered corrections with automatic pattern learning. **Features:** - Two-stage correction pipeline (dictionary + AI) - Automatic pattern detection and learning - Domain-specific dictionaries (general, embodied_ai, finance, medical) - SQLite-based correction repository - Team collaboration with import/export - GLM API integration for AI corrections - Cost optimization through dictionary promotion **Use cases:** - Correcting meeting notes, lecture recordings, or interview transcripts - Fixing Chinese/English homophone errors and technical terminology - Building domain-specific correction dictionaries - Improving transcript accuracy through iterative learning **Documentation:** - Complete workflow guides in references/ - SQL query templates - Troubleshooting guide - Team collaboration patterns - API setup instructions **Marketplace updates:** - Updated marketplace to v1.8.0 - Added transcript-fixer plugin (category: productivity) - Updated README.md with skill description and use cases - Updated CLAUDE.md with skill listing and counts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
daymade
@@ -5,8 +5,8 @@
|
|||||||
"email": "daymadev89@gmail.com"
|
"email": "daymadev89@gmail.com"
|
||||||
},
|
},
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, and secure repomix packaging",
|
"description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, and ASR transcription correction",
|
||||||
"version": "1.7.0",
|
"version": "1.8.0",
|
||||||
"homepage": "https://github.com/daymade/claude-code-skills"
|
"homepage": "https://github.com/daymade/claude-code-skills"
|
||||||
},
|
},
|
||||||
"plugins": [
|
"plugins": [
|
||||||
@@ -149,6 +149,16 @@
|
|||||||
"category": "security",
|
"category": "security",
|
||||||
"keywords": ["repomix", "security", "credentials", "secrets-scanning", "safe-packaging", "secret-detection", "code-security"],
|
"keywords": ["repomix", "security", "credentials", "secrets-scanning", "safe-packaging", "secret-detection", "code-security"],
|
||||||
"skills": ["./repomix-safe-mixer"]
|
"skills": ["./repomix-safe-mixer"]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "transcript-fixer",
|
||||||
|
"description": "Corrects speech-to-text (ASR/STT) transcription errors in meeting notes, lecture recordings, interviews, and voice memos through dictionary-based rules and AI corrections. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files",
|
||||||
|
"source": "./",
|
||||||
|
"strict": false,
|
||||||
|
"version": "1.0.0",
|
||||||
|
"category": "productivity",
|
||||||
|
"keywords": ["transcription", "asr", "stt", "speech-to-text", "correction", "ai", "meeting-notes", "nlp"],
|
||||||
|
"skills": ["./transcript-fixer"]
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
|
|
||||||
## Repository Overview
|
## Repository Overview
|
||||||
|
|
||||||
This is a Claude Code skills marketplace containing 13 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources.
|
This is a Claude Code skills marketplace containing 15 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources.
|
||||||
|
|
||||||
**Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code.
|
**Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code.
|
||||||
|
|
||||||
@@ -118,7 +118,7 @@ Skills for public distribution must NOT contain:
|
|||||||
## Marketplace Configuration
|
## Marketplace Configuration
|
||||||
|
|
||||||
The marketplace is configured in `.claude-plugin/marketplace.json`:
|
The marketplace is configured in `.claude-plugin/marketplace.json`:
|
||||||
- Contains 13 plugins, each mapping to one skill
|
- Contains 15 plugins, each mapping to one skill
|
||||||
- Each plugin has: name, description, version, category, keywords, skills array
|
- Each plugin has: name, description, version, category, keywords, skills array
|
||||||
- Marketplace metadata: name, owner, version, homepage
|
- Marketplace metadata: name, owner, version, homepage
|
||||||
|
|
||||||
@@ -128,7 +128,7 @@ The marketplace is configured in `.claude-plugin/marketplace.json`:
|
|||||||
|
|
||||||
1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`)
|
1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`)
|
||||||
- Tracks the marketplace catalog as a whole
|
- Tracks the marketplace catalog as a whole
|
||||||
- Current: v1.6.0
|
- Current: v1.8.0
|
||||||
- Bump when: Adding/removing skills, major marketplace restructuring
|
- Bump when: Adding/removing skills, major marketplace restructuring
|
||||||
- Semantic versioning: MAJOR.MINOR.PATCH
|
- Semantic versioning: MAJOR.MINOR.PATCH
|
||||||
|
|
||||||
@@ -157,6 +157,8 @@ The marketplace is configured in `.claude-plugin/marketplace.json`:
|
|||||||
11. **ui-designer** - Design system extraction from UI mockups
|
11. **ui-designer** - Design system extraction from UI mockups
|
||||||
12. **ppt-creator** - Professional presentation creation with dual-path PPTX generation
|
12. **ppt-creator** - Professional presentation creation with dual-path PPTX generation
|
||||||
13. **youtube-downloader** - YouTube video and audio downloading with yt-dlp error handling
|
13. **youtube-downloader** - YouTube video and audio downloading with yt-dlp error handling
|
||||||
|
14. **repomix-safe-mixer** - Secure repomix packaging with automatic credential detection
|
||||||
|
15. **transcript-fixer** - ASR/STT transcription error correction with dictionary and AI learning
|
||||||
|
|
||||||
**Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code.
|
**Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code.
|
||||||
|
|
||||||
|
|||||||
53
README.md
53
README.md
@@ -6,15 +6,15 @@
|
|||||||
[](./README.zh-CN.md)
|
[](./README.zh-CN.md)
|
||||||
|
|
||||||
[](https://opensource.org/licenses/MIT)
|
[](https://opensource.org/licenses/MIT)
|
||||||
[](https://github.com/daymade/claude-code-skills)
|
[](https://github.com/daymade/claude-code-skills)
|
||||||
[](https://github.com/daymade/claude-code-skills)
|
[](https://github.com/daymade/claude-code-skills)
|
||||||
[](https://claude.com/code)
|
[](https://claude.com/code)
|
||||||
[](./CONTRIBUTING.md)
|
[](./CONTRIBUTING.md)
|
||||||
[](https://github.com/daymade/claude-code-skills/graphs/commit-activity)
|
[](https://github.com/daymade/claude-code-skills/graphs/commit-activity)
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
Professional Claude Code skills marketplace featuring 14 production-ready skills for enhanced development workflows.
|
Professional Claude Code skills marketplace featuring 15 production-ready skills for enhanced development workflows.
|
||||||
|
|
||||||
## 📑 Table of Contents
|
## 📑 Table of Contents
|
||||||
|
|
||||||
@@ -481,6 +481,49 @@ Safely package codebases with repomix by automatically detecting and removing ha
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### 14. **transcript-fixer** - ASR Transcription Correction
|
||||||
|
|
||||||
|
Correct speech-to-text (ASR/STT) transcription errors through dictionary-based rules and AI-powered corrections with automatic pattern learning.
|
||||||
|
|
||||||
|
**When to use:**
|
||||||
|
- Correcting meeting notes, lecture recordings, or interview transcripts
|
||||||
|
- Fixing Chinese/English homophone errors and technical terminology
|
||||||
|
- Building domain-specific correction dictionaries
|
||||||
|
- Improving transcript accuracy through iterative learning
|
||||||
|
- Collaborating with teams on shared correction knowledge bases
|
||||||
|
|
||||||
|
**Key features:**
|
||||||
|
- Two-stage correction pipeline (dictionary + AI)
|
||||||
|
- Automatic pattern detection and learning
|
||||||
|
- Domain-specific dictionaries (general, embodied_ai, finance, medical)
|
||||||
|
- SQLite-based correction repository
|
||||||
|
- Team collaboration with import/export
|
||||||
|
- GLM API integration for AI corrections
|
||||||
|
- Cost optimization through dictionary promotion
|
||||||
|
|
||||||
|
**Example workflow:**
|
||||||
|
```bash
|
||||||
|
# Initialize and add corrections
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
uv run scripts/fix_transcription.py --add "错误词" "正确词" --domain general
|
||||||
|
|
||||||
|
# Run full correction pipeline
|
||||||
|
uv run scripts/fix_transcription.py --input meeting.md --stage 3
|
||||||
|
|
||||||
|
# Review and approve learned patterns
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
```
|
||||||
|
|
||||||
|
**🎬 Live Demo**
|
||||||
|
|
||||||
|
*Coming soon*
|
||||||
|
|
||||||
|
📚 **Documentation**: See [transcript-fixer/references/](./transcript-fixer/references/) for workflow guides, SQL queries, troubleshooting, best practices, team collaboration, and API setup.
|
||||||
|
|
||||||
|
**Requirements**: Python 3.6+, uv package manager, GLM API key (get from https://open.bigmodel.cn/)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 🎬 Interactive Demo Gallery
|
## 🎬 Interactive Demo Gallery
|
||||||
|
|
||||||
Want to see all demos in one place with click-to-enlarge functionality? Check out our [interactive demo gallery](./demos/index.html) or browse the [demos directory](./demos/).
|
Want to see all demos in one place with click-to-enlarge functionality? Check out our [interactive demo gallery](./demos/index.html) or browse the [demos directory](./demos/).
|
||||||
@@ -508,6 +551,9 @@ Use **ppt-creator** to generate professional slide decks with data visualization
|
|||||||
### For Media & Content Download
|
### For Media & Content Download
|
||||||
Use **youtube-downloader** to download YouTube videos and extract audio from videos with automatic workarounds for common download issues.
|
Use **youtube-downloader** to download YouTube videos and extract audio from videos with automatic workarounds for common download issues.
|
||||||
|
|
||||||
|
### For Transcription & ASR Correction
|
||||||
|
Use **transcript-fixer** to correct speech-to-text errors in meeting notes, lectures, and interviews through dictionary-based rules and AI-powered corrections with automatic learning.
|
||||||
|
|
||||||
## 📚 Documentation
|
## 📚 Documentation
|
||||||
|
|
||||||
Each skill includes:
|
Each skill includes:
|
||||||
@@ -530,6 +576,7 @@ Each skill includes:
|
|||||||
- **ppt-creator**: See `ppt-creator/references/WORKFLOW.md` for 9-stage creation process and `ppt-creator/references/ORCHESTRATION_OVERVIEW.md` for automation
|
- **ppt-creator**: See `ppt-creator/references/WORKFLOW.md` for 9-stage creation process and `ppt-creator/references/ORCHESTRATION_OVERVIEW.md` for automation
|
||||||
- **youtube-downloader**: See `youtube-downloader/SKILL.md` for usage examples and troubleshooting
|
- **youtube-downloader**: See `youtube-downloader/SKILL.md` for usage examples and troubleshooting
|
||||||
- **repomix-safe-mixer**: See `repomix-safe-mixer/references/common_secrets.md` for detected credential patterns
|
- **repomix-safe-mixer**: See `repomix-safe-mixer/references/common_secrets.md` for detected credential patterns
|
||||||
|
- **transcript-fixer**: See `transcript-fixer/references/workflow_guide.md` for step-by-step workflows and `transcript-fixer/references/team_collaboration.md` for collaboration patterns
|
||||||
|
|
||||||
## 🛠️ Requirements
|
## 🛠️ Requirements
|
||||||
|
|
||||||
|
|||||||
14
transcript-fixer/.gitignore
vendored
Normal file
14
transcript-fixer/.gitignore
vendored
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Security scan marker file (generated by security_scan.py)
|
||||||
|
.security-scan-passed
|
||||||
|
|
||||||
|
# Backup files
|
||||||
|
*_backup.py
|
||||||
|
*_old.py
|
||||||
|
*_backup_*.py
|
||||||
|
*.bak
|
||||||
|
|
||||||
|
# Python cache
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
*.pyd
|
||||||
180
transcript-fixer/SKILL.md
Normal file
180
transcript-fixer/SKILL.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
---
|
||||||
|
name: transcript-fixer
|
||||||
|
description: Corrects speech-to-text (ASR/STT) transcription errors in meeting notes, lecture recordings, interviews, and voice memos through dictionary-based rules and AI corrections. This skill should be used when users mention 'transcript', 'ASR errors', 'speech-to-text', 'STT mistakes', 'meeting notes', 'dictation', 'homophone errors', 'voice memo cleanup', or when working with .md/.txt files containing Chinese/English mixed content with obvious transcription errors.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Transcript Fixer
|
||||||
|
|
||||||
|
Correct speech-to-text transcription errors through dictionary-based rules, AI-powered corrections, and automatic pattern detection. Build a personalized knowledge base that learns from each correction.
|
||||||
|
|
||||||
|
## When to Use This Skill
|
||||||
|
|
||||||
|
Activate this skill when:
|
||||||
|
- Correcting speech-to-text (ASR) transcription errors in meeting notes, lectures, or interviews
|
||||||
|
- Building domain-specific correction dictionaries for repeated transcription workflows
|
||||||
|
- Fixing Chinese/English homophone errors, technical terminology, or names
|
||||||
|
- Collaborating with teams on shared correction knowledge bases
|
||||||
|
- Improving transcript accuracy through iterative learning
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
Initialize (first time only):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
export GLM_API_KEY="<api-key>" # Obtain from https://open.bigmodel.cn/
|
||||||
|
```
|
||||||
|
|
||||||
|
Correct a transcript in 3 steps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Add common corrections (5-10 terms)
|
||||||
|
uv run scripts/fix_transcription.py --add "错误词" "正确词" --domain general
|
||||||
|
|
||||||
|
# 2. Run full correction pipeline
|
||||||
|
uv run scripts/fix_transcription.py --input meeting.md --stage 3
|
||||||
|
|
||||||
|
# 3. Review learned patterns after 3-5 runs
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output files**:
|
||||||
|
- `meeting_stage1.md` - Dictionary corrections applied
|
||||||
|
- `meeting_stage2.md` - AI corrections applied (final version)
|
||||||
|
|
||||||
|
## Example Session
|
||||||
|
|
||||||
|
**Input transcript** (`meeting.md`):
|
||||||
|
```
|
||||||
|
今天我们讨论了巨升智能的最新进展。
|
||||||
|
股价系统需要优化，目前性能不够好。
|
||||||
|
```
|
||||||
|
|
||||||
|
**After Stage 1** (`meeting_stage1.md`):
|
||||||
|
```
|
||||||
|
今天我们讨论了具身智能的最新进展。 ← "巨升"→"具身" corrected
|
||||||
|
股价系统需要优化,目前性能不够好。 ← Unchanged (not in dictionary)
|
||||||
|
```
|
||||||
|
|
||||||
|
**After Stage 2** (`meeting_stage2.md`):
|
||||||
|
```
|
||||||
|
今天我们讨论了具身智能的最新进展。
|
||||||
|
框架系统需要优化，目前性能不够好。 ← "股价"→"框架" corrected by AI
|
||||||
|
```
|
||||||
|
|
||||||
|
**Learned pattern detected:**
|
||||||
|
```
|
||||||
|
✓ Detected: "股价" → "框架" (confidence: 85%, count: 1)
|
||||||
|
Run --review-learned after 2 more occurrences to approve
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workflow Checklist
|
||||||
|
|
||||||
|
Copy and customize this checklist for each transcript:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
### Transcript Correction - [FILENAME] - [DATE]
|
||||||
|
- [ ] Validation passed: `uv run scripts/fix_transcription.py --validate`
|
||||||
|
- [ ] GLM_API_KEY verified: `echo $GLM_API_KEY | wc -c` (should be >20)
|
||||||
|
- [ ] Domain selected: [general/embodied_ai/finance/medical]
|
||||||
|
- [ ] Added 5-10 domain-specific corrections to dictionary
|
||||||
|
- [ ] Tested Stage 1 (dictionary only): Output reviewed at [FILENAME]_stage1.md
|
||||||
|
- [ ] Stage 2 (AI) completed: Final output verified at [FILENAME]_stage2.md
|
||||||
|
- [ ] Learned patterns reviewed: `--review-learned`
|
||||||
|
- [ ] High-confidence suggestions approved (if any)
|
||||||
|
- [ ] Team dictionary updated (if applicable): `--export team.json`
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Initialize (first time only)
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
export GLM_API_KEY="<api-key>" # Get from https://open.bigmodel.cn/
|
||||||
|
|
||||||
|
# Add corrections
|
||||||
|
uv run scripts/fix_transcription.py --add "错误词" "正确词" --domain general
|
||||||
|
|
||||||
|
# Run full pipeline (dictionary + AI corrections)
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 3 --domain general
|
||||||
|
|
||||||
|
# Review and approve learned patterns (after 3-5 runs)
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
uv run scripts/fix_transcription.py --approve "错误" "正确"
|
||||||
|
|
||||||
|
# Team collaboration
|
||||||
|
uv run scripts/fix_transcription.py --export team.json --domain <domain>
|
||||||
|
uv run scripts/fix_transcription.py --import team.json --merge
|
||||||
|
|
||||||
|
# Validate setup
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Database**: `~/.transcript-fixer/corrections.db` (SQLite)
|
||||||
|
|
||||||
|
**Stages**:
|
||||||
|
- Stage 1: Dictionary corrections (instant, zero cost)
|
||||||
|
- Stage 2: AI corrections via GLM API (1-2 min per 1000 lines)
|
||||||
|
- Stage 3: Full pipeline (both stages)
|
||||||
|
|
||||||
|
**Domains**: `general`, `embodied_ai`, `finance`, `medical` (prevents cross-domain conflicts)
|
||||||
|
|
||||||
|
**Learning**: Approve patterns appearing ≥3 times with ≥80% confidence to move from expensive AI (Stage 2) to free dictionary (Stage 1).
|
||||||
|
|
||||||
|
See `references/workflow_guide.md` for detailed workflows and `references/team_collaboration.md` for collaboration patterns.
|
||||||
|
|
||||||
|
## Bundled Resources
|
||||||
|
|
||||||
|
### Scripts
|
||||||
|
|
||||||
|
- **`fix_transcription.py`** - Main CLI for all operations
|
||||||
|
- **`examples/bulk_import.py`** - Bulk import example (runnable with `uv run scripts/examples/bulk_import.py`)
|
||||||
|
|
||||||
|
### References
|
||||||
|
|
||||||
|
Load as needed for detailed guidance:
|
||||||
|
|
||||||
|
- **`workflow_guide.md`** - Step-by-step workflows, pre-flight checklist, batch processing
|
||||||
|
- **`quick_reference.md`** - CLI/SQL/Python API quick reference
|
||||||
|
- **`sql_queries.md`** - SQL query templates (copy-paste ready)
|
||||||
|
- **`troubleshooting.md`** - Error resolution, validation
|
||||||
|
- **`best_practices.md`** - Optimization, cost management
|
||||||
|
- **`file_formats.md`** - Complete SQLite schema
|
||||||
|
- **`installation_setup.md`** - Setup and dependencies
|
||||||
|
- **`team_collaboration.md`** - Git workflows, merging
|
||||||
|
- **`glm_api_setup.md`** - API key configuration
|
||||||
|
- **`architecture.md`** - Module structure, extensibility
|
||||||
|
- **`script_parameters.md`** - Complete CLI reference
|
||||||
|
- **`dictionary_guide.md`** - Dictionary strategies
|
||||||
|
|
||||||
|
## Validation and Troubleshooting
|
||||||
|
|
||||||
|
Run validation to check system health:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Healthy output:**
|
||||||
|
```
|
||||||
|
✅ Configuration directory exists: ~/.transcript-fixer
|
||||||
|
✅ Database valid: 4 tables found
|
||||||
|
✅ GLM_API_KEY is set (47 chars)
|
||||||
|
✅ All checks passed
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error recovery:**
|
||||||
|
1. Run validation to identify issue
|
||||||
|
2. Check components:
|
||||||
|
- Database: `sqlite3 ~/.transcript-fixer/corrections.db ".tables"`
|
||||||
|
- API key: `echo $GLM_API_KEY | wc -c` (should be >20)
|
||||||
|
- Permissions: `ls -la ~/.transcript-fixer/`
|
||||||
|
3. Apply fix based on validation output
|
||||||
|
4. Re-validate to confirm
|
||||||
|
|
||||||
|
**Quick fixes:**
|
||||||
|
- Missing database → Run `--init`
|
||||||
|
- Missing API key → `export GLM_API_KEY="<key>"`
|
||||||
|
- Permission errors → Check ownership with `ls -la`
|
||||||
|
|
||||||
|
See `references/troubleshooting.md` for detailed error codes and solutions.
|
||||||
848
transcript-fixer/references/architecture.md
Normal file
848
transcript-fixer/references/architecture.md
Normal file
@@ -0,0 +1,848 @@
|
|||||||
|
# Architecture Reference
|
||||||
|
|
||||||
|
Technical implementation details of the transcript-fixer system.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Module Structure](#module-structure)
|
||||||
|
- [Design Principles](#design-principles)
|
||||||
|
- [SOLID Compliance](#solid-compliance)
|
||||||
|
- [File Length Limits](#file-length-limits)
|
||||||
|
- [Module Architecture](#module-architecture)
|
||||||
|
- [Layer Diagram](#layer-diagram)
|
||||||
|
- [Correction Workflow](#correction-workflow)
|
||||||
|
- [Learning Cycle](#learning-cycle)
|
||||||
|
- [Data Flow](#data-flow)
|
||||||
|
- [SQLite Architecture (v2.0)](#sqlite-architecture-v20)
|
||||||
|
- [Two-Layer Data Access](#two-layer-data-access-simplified)
|
||||||
|
- [Database Schema](#database-schema-schemasql)
|
||||||
|
- [ACID Guarantees](#acid-guarantees)
|
||||||
|
- [Thread Safety](#thread-safety)
|
||||||
|
- [Migration from JSON](#migration-from-json)
|
||||||
|
- [Module Details](#module-details)
|
||||||
|
- [fix_transcription.py](#fix_transcriptionpy-orchestrator)
|
||||||
|
- [correction_repository.py](#correction_repositorypy-data-access-layer)
|
||||||
|
- [correction_service.py](#correction_servicepy-business-logic-layer)
|
||||||
|
- [CLI Integration](#cli-integration-commandspy)
|
||||||
|
- [dictionary_processor.py](#dictionary_processorpy-stage-1)
|
||||||
|
- [ai_processor.py](#ai_processorpy-stage-2)
|
||||||
|
- [learning_engine.py](#learning_enginepy-pattern-detection)
|
||||||
|
- [diff_generator.py](#diff_generatorpy-stage-3)
|
||||||
|
- [State Management](#state-management)
|
||||||
|
- [Database-Backed State](#database-backed-state)
|
||||||
|
- [Thread-Safe Access](#thread-safe-access)
|
||||||
|
- [Error Handling Strategy](#error-handling-strategy)
|
||||||
|
- [Testing Strategy](#testing-strategy)
|
||||||
|
- [Performance Considerations](#performance-considerations)
|
||||||
|
- [Security Architecture](#security-architecture)
|
||||||
|
- [Extensibility Points](#extensibility-points)
|
||||||
|
- [Dependencies](#dependencies)
|
||||||
|
- [Deployment](#deployment)
|
||||||
|
- [Further Reading](#further-reading)
|
||||||
|
|
||||||
|
## Module Structure
|
||||||
|
|
||||||
|
The codebase follows a modular package structure for maintainability:
|
||||||
|
|
||||||
|
```
|
||||||
|
scripts/
|
||||||
|
├── fix_transcription.py # Main entry point (~70 lines)
|
||||||
|
├── core/ # Business logic & data access
|
||||||
|
│ ├── correction_repository.py # Data access layer (466 lines)
|
||||||
|
│ ├── correction_service.py # Business logic layer (525 lines)
|
||||||
|
│ ├── schema.sql # SQLite database schema (216 lines)
|
||||||
|
│ ├── dictionary_processor.py # Stage 1 processor (140 lines)
|
||||||
|
│ ├── ai_processor.py # Stage 2 processor (199 lines)
|
||||||
|
│ └── learning_engine.py # Pattern detection (252 lines)
|
||||||
|
├── cli/ # Command-line interface
|
||||||
|
│ ├── commands.py # Command handlers (180 lines)
|
||||||
|
│ └── argument_parser.py # Argument config (95 lines)
|
||||||
|
└── utils/ # Utility functions
|
||||||
|
├── diff_generator.py # Multi-format diffs (132 lines)
|
||||||
|
├── logging_config.py # Logging configuration (130 lines)
|
||||||
|
└── validation.py # SQLite validation (105 lines)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits of modular structure**:
|
||||||
|
- Clear separation of concerns (business logic / CLI / utilities)
|
||||||
|
- Easy to locate and modify specific functionality
|
||||||
|
- Supports independent testing of modules
|
||||||
|
- Scales well as codebase grows
|
||||||
|
- Follows Python package best practices
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
|
||||||
|
### SOLID Compliance
|
||||||
|
|
||||||
|
Every module follows SOLID principles for maintainability:
|
||||||
|
|
||||||
|
1. **Single Responsibility Principle (SRP)**
|
||||||
|
- Each module has exactly one reason to change
|
||||||
|
- `CorrectionRepository`: Database operations only
|
||||||
|
- `CorrectionService`: Business logic and validation only
|
||||||
|
- `DictionaryProcessor`: Text transformation only
|
||||||
|
- `AIProcessor`: API communication only
|
||||||
|
- `LearningEngine`: Pattern analysis only
|
||||||
|
|
||||||
|
2. **Open/Closed Principle (OCP)**
|
||||||
|
- Open for extension via SQL INSERT
|
||||||
|
- Closed for modification (no code changes needed)
|
||||||
|
- Add corrections via CLI or SQL without editing Python
|
||||||
|
|
||||||
|
3. **Liskov Substitution Principle (LSP)**
|
||||||
|
- All processors implement same interface
|
||||||
|
- Can swap implementations without breaking workflow
|
||||||
|
|
||||||
|
4. **Interface Segregation Principle (ISP)**
|
||||||
|
- Repository, Service, Processor, Engine are independent
|
||||||
|
- No unnecessary dependencies
|
||||||
|
|
||||||
|
5. **Dependency Inversion Principle (DIP)**
|
||||||
|
- Service depends on Repository interface
|
||||||
|
- CLI depends on Service interface
|
||||||
|
- Not tied to concrete implementations
|
||||||
|
|
||||||
|
### File Length Limits
|
||||||
|
|
||||||
|
All files comply with code quality standards:
|
||||||
|
|
||||||
|
| File | Lines | Limit | Status |
|
||||||
|
|------|-------|-------|--------|
|
||||||
|
| `validation.py` | 105 | 200 | ✅ |
|
||||||
|
| `logging_config.py` | 130 | 200 | ✅ |
|
||||||
|
| `diff_generator.py` | 132 | 200 | ✅ |
|
||||||
|
| `dictionary_processor.py` | 140 | 200 | ✅ |
|
||||||
|
| `commands.py` | 180 | 200 | ✅ |
|
||||||
|
| `ai_processor.py` | 199 | 250 | ✅ |
|
||||||
|
| `schema.sql` | 216 | 250 | ✅ |
|
||||||
|
| `learning_engine.py` | 252 | 250 | ✅ |
|
||||||
|
| `correction_repository.py` | 466 | 500 | ✅ |
|
||||||
|
| `correction_service.py` | 525 | 550 | ✅ |
|
||||||
|
|
||||||
|
## Module Architecture
|
||||||
|
|
||||||
|
### Layer Diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ CLI Layer (fix_transcription.py) │
|
||||||
|
│ - Argument parsing │
|
||||||
|
│ - Command routing │
|
||||||
|
│ - User interaction │
|
||||||
|
└───────────────┬─────────────────────────┘
|
||||||
|
│
|
||||||
|
┌───────────────▼─────────────────────────┐
|
||||||
|
│ Business Logic Layer │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────┐ ┌──────────────┐│
|
||||||
|
│ │ Dictionary │ │ AI ││
|
||||||
|
│ │ Processor │ │ Processor ││
|
||||||
|
│ │ (Stage 1) │ │ (Stage 2) ││
|
||||||
|
│ └──────────────────┘ └──────────────┘│
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────┐ ┌──────────────┐│
|
||||||
|
│ │ Learning │ │ Diff ││
|
||||||
|
│ │ Engine │ │ Generator ││
|
||||||
|
│ │ (Pattern detect) │ │ (Stage 3) ││
|
||||||
|
│ └──────────────────┘ └──────────────┘│
|
||||||
|
└───────────────┬─────────────────────────┘
|
||||||
|
│
|
||||||
|
┌───────────────▼─────────────────────────┐
|
||||||
|
│ Data Access Layer (SQLite-based) │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────────────┐ │
|
||||||
|
│ │ CorrectionManager (Facade) │ │
|
||||||
|
│ │ - Backward-compatible API │ │
|
||||||
|
│ └──────────────┬───────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌──────────────▼───────────────────┐ │
|
||||||
|
│ │ CorrectionService │ │
|
||||||
|
│ │ - Business logic │ │
|
||||||
|
│ │ - Validation │ │
|
||||||
|
│ │ - Import/Export │ │
|
||||||
|
│ └──────────────┬───────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌──────────────▼───────────────────┐ │
|
||||||
|
│ │ CorrectionRepository │ │
|
||||||
|
│ │ - ACID transactions │ │
|
||||||
|
│ │ - Thread-safe connections │ │
|
||||||
|
│ │ - Audit logging │ │
|
||||||
|
│ └──────────────────────────────────┘ │
|
||||||
|
└───────────────┬─────────────────────────┘
|
||||||
|
│
|
||||||
|
┌───────────────▼─────────────────────────┐
|
||||||
|
│ Storage Layer │
|
||||||
|
│ ~/.transcript-fixer/corrections.db │
|
||||||
|
│ - SQLite database (ACID compliant) │
|
||||||
|
│ - 8 normalized tables + 3 views │
|
||||||
|
│ - Comprehensive indexes │
|
||||||
|
│ - Foreign key constraints │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
### Correction Workflow
|
||||||
|
|
||||||
|
```
|
||||||
|
1. User Input
|
||||||
|
↓
|
||||||
|
2. fix_transcription.py (Orchestrator)
|
||||||
|
↓
|
||||||
|
3. CorrectionService.get_corrections()
|
||||||
|
← Query from ~/.transcript-fixer/corrections.db
|
||||||
|
↓
|
||||||
|
4. DictionaryProcessor.process()
|
||||||
|
- Apply context rules (regex)
|
||||||
|
- Apply dictionary replacements
|
||||||
|
- Track changes
|
||||||
|
↓
|
||||||
|
5. AIProcessor.process()
|
||||||
|
- Split into chunks
|
||||||
|
- Call GLM-4.6 API
|
||||||
|
- Retry with fallback on error
|
||||||
|
- Track AI changes
|
||||||
|
↓
|
||||||
|
6. CorrectionService.save_history()
|
||||||
|
→ Insert into correction_history table
|
||||||
|
↓
|
||||||
|
7. LearningEngine.analyze_and_suggest()
|
||||||
|
- Query correction_history table
|
||||||
|
- Detect patterns (frequency ≥3, confidence ≥80%)
|
||||||
|
- Generate suggestions
|
||||||
|
→ Insert into learned_suggestions table
|
||||||
|
↓
|
||||||
|
8. Output Files
|
||||||
|
- {filename}_stage1.md
|
||||||
|
- {filename}_stage2.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### Learning Cycle
|
||||||
|
|
||||||
|
```
|
||||||
|
Run 1: meeting1.md
|
||||||
|
AI corrects: "巨升" → "具身"
|
||||||
|
↓
|
||||||
|
INSERT INTO correction_history
|
||||||
|
|
||||||
|
Run 2: meeting2.md
|
||||||
|
AI corrects: "巨升" → "具身"
|
||||||
|
↓
|
||||||
|
INSERT INTO correction_history
|
||||||
|
|
||||||
|
Run 3: meeting3.md
|
||||||
|
AI corrects: "巨升" → "具身"
|
||||||
|
↓
|
||||||
|
INSERT INTO correction_history
|
||||||
|
↓
|
||||||
|
LearningEngine queries patterns:
|
||||||
|
- SELECT ... GROUP BY from_text, to_text
|
||||||
|
- Frequency: 3, Confidence: 100%
|
||||||
|
↓
|
||||||
|
INSERT INTO learned_suggestions (status='pending')
|
||||||
|
↓
|
||||||
|
User reviews: --review-learned
|
||||||
|
↓
|
||||||
|
User approves: --approve "巨升" "具身"
|
||||||
|
↓
|
||||||
|
INSERT INTO corrections (source='learned')
|
||||||
|
UPDATE learned_suggestions (status='approved')
|
||||||
|
↓
|
||||||
|
Future runs query corrections table (Stage 1 - faster!)
|
||||||
|
```
|
||||||
|
|
||||||
|
## SQLite Architecture (v2.0)
|
||||||
|
|
||||||
|
### Two-Layer Data Access (Simplified)
|
||||||
|
|
||||||
|
**Design Principle**: No users = no backward compatibility overhead.
|
||||||
|
|
||||||
|
The system uses a clean 2-layer architecture:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────┐
|
||||||
|
│ CLI Commands (commands.py) │
|
||||||
|
│ - User interaction │
|
||||||
|
│ - Command routing │
|
||||||
|
└──────────────┬───────────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────▼───────────────────────────┐
|
||||||
|
│ CorrectionService (Business Logic) │
|
||||||
|
│ - Input validation & sanitization │
|
||||||
|
│ - Business rules enforcement │
|
||||||
|
│ - Import/export orchestration │
|
||||||
|
│ - Statistics calculation │
|
||||||
|
│ - History tracking │
|
||||||
|
└──────────────┬───────────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────▼───────────────────────────┐
|
||||||
|
│ CorrectionRepository (Data Access) │
|
||||||
|
│ - ACID transactions │
|
||||||
|
│ - Thread-safe connections │
|
||||||
|
│ - SQL query execution │
|
||||||
|
│ - Audit logging │
|
||||||
|
└──────────────┬───────────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────▼───────────────────────────┐
|
||||||
|
│ SQLite Database (corrections.db) │
|
||||||
|
│ - 8 normalized tables │
|
||||||
|
│ - Foreign key constraints │
|
||||||
|
│ - Comprehensive indexes │
|
||||||
|
│ - 3 views for common queries │
|
||||||
|
└───────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Schema (schema.sql)
|
||||||
|
|
||||||
|
**Core Tables**:
|
||||||
|
|
||||||
|
1. **corrections** (main correction storage)
|
||||||
|
- Primary key: id
|
||||||
|
- Unique constraint: (from_text, domain)
|
||||||
|
- Indexes: domain, source, added_at, is_active, from_text
|
||||||
|
- Fields: confidence (0.0-1.0), usage_count, notes
|
||||||
|
|
||||||
|
2. **context_rules** (regex-based rules)
|
||||||
|
- Pattern + replacement with priority ordering
|
||||||
|
- Indexes: priority (DESC), is_active
|
||||||
|
|
||||||
|
3. **correction_history** (audit trail for runs)
|
||||||
|
- Tracks: filename, domain, timestamps, change counts
|
||||||
|
- Links to correction_changes via foreign key
|
||||||
|
- Indexes: run_timestamp, domain, success
|
||||||
|
|
||||||
|
4. **correction_changes** (detailed change log)
|
||||||
|
- Links to history via foreign key (CASCADE delete)
|
||||||
|
- Stores: line_number, from/to text, rule_type, context
|
||||||
|
- Indexes: history_id, rule_type
|
||||||
|
|
||||||
|
5. **learned_suggestions** (AI-detected patterns)
|
||||||
|
- Status: pending → approved/rejected
|
||||||
|
- Unique constraint: (from_text, to_text, domain)
|
||||||
|
- Fields: frequency, confidence, timestamps
|
||||||
|
- Indexes: status, domain, confidence, frequency
|
||||||
|
|
||||||
|
6. **suggestion_examples** (occurrences of patterns)
|
||||||
|
- Links to learned_suggestions via foreign key
|
||||||
|
- Stores context where pattern occurred
|
||||||
|
|
||||||
|
7. **system_config** (configuration storage)
|
||||||
|
- Key-value store with type safety
|
||||||
|
- Stores: API settings, thresholds, defaults
|
||||||
|
|
||||||
|
8. **audit_log** (comprehensive audit trail)
|
||||||
|
- Tracks all database operations
|
||||||
|
- Fields: action, entity_type, entity_id, user, success
|
||||||
|
- Indexes: timestamp, action, entity_type, success
|
||||||
|
|
||||||
|
**Views** (for common queries):
|
||||||
|
- `active_corrections`: Active corrections only
|
||||||
|
- `pending_suggestions`: Suggestions pending review
|
||||||
|
- `correction_statistics`: Statistics per domain
|
||||||
|
|
||||||
|
### ACID Guarantees
|
||||||
|
|
||||||
|
**Atomicity**: All-or-nothing transactions
|
||||||
|
```python
|
||||||
|
with self._transaction() as conn:
|
||||||
|
conn.execute("INSERT ...") # Either all succeed
|
||||||
|
conn.execute("UPDATE ...") # or all rollback
|
||||||
|
```
|
||||||
|
|
||||||
|
**Consistency**: Constraints enforced
|
||||||
|
- Foreign key constraints
|
||||||
|
- Check constraints (confidence 0.0-1.0, usage_count ≥ 0)
|
||||||
|
- Unique constraints
|
||||||
|
|
||||||
|
**Isolation**: Serializable transactions
|
||||||
|
```python
|
||||||
|
conn.execute("BEGIN IMMEDIATE") # Acquire write lock
|
||||||
|
```
|
||||||
|
|
||||||
|
**Durability**: Changes persisted to disk
|
||||||
|
- SQLite guarantees persistence after commit
|
||||||
|
- Backup before migrations
|
||||||
|
|
||||||
|
### Thread Safety
|
||||||
|
|
||||||
|
**Thread-local connections**:
|
||||||
|
```python
|
||||||
|
def _get_connection(self):
|
||||||
|
if not hasattr(self._local, 'connection'):
|
||||||
|
self._local.connection = sqlite3.connect(...)
|
||||||
|
return self._local.connection
|
||||||
|
```
|
||||||
|
|
||||||
|
**Connection pooling**:
|
||||||
|
- One connection per thread
|
||||||
|
- Automatic cleanup on close
|
||||||
|
- Foreign keys enabled per connection
|
||||||
|
|
||||||
|
### Clean Architecture (No Legacy)
|
||||||
|
|
||||||
|
**Design Philosophy**:
|
||||||
|
- Clean 2-layer architecture (Service → Repository)
|
||||||
|
- No backward compatibility overhead
|
||||||
|
- Direct API design without legacy constraints
|
||||||
|
- YAGNI principle: Build for current needs, not hypothetical migrations
|
||||||
|
|
||||||
|
## Module Details
|
||||||
|
|
||||||
|
### fix_transcription.py (Orchestrator)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Parse CLI arguments
|
||||||
|
- Route commands to appropriate handlers
|
||||||
|
- Coordinate workflow between modules
|
||||||
|
- Display user feedback
|
||||||
|
|
||||||
|
**Key Functions**:
|
||||||
|
```python
|
||||||
|
cmd_init() # Initialize ~/.transcript-fixer/
|
||||||
|
cmd_add_correction() # Add single correction
|
||||||
|
cmd_list_corrections() # List corrections
|
||||||
|
cmd_run_correction() # Execute correction workflow
|
||||||
|
cmd_review_learned() # Review AI suggestions
|
||||||
|
cmd_approve() # Approve learned correction
|
||||||
|
```
|
||||||
|
|
||||||
|
**Design Pattern**: Command pattern with function routing
|
||||||
|
|
||||||
|
### correction_repository.py (Data Access Layer)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Execute SQL queries with ACID guarantees
|
||||||
|
- Manage thread-safe database connections
|
||||||
|
- Handle transactions (commit/rollback)
|
||||||
|
- Perform audit logging
|
||||||
|
- Convert between database rows and Python objects
|
||||||
|
|
||||||
|
**Key Methods**:
|
||||||
|
```python
|
||||||
|
add_correction() # INSERT with UNIQUE handling
|
||||||
|
get_correction() # SELECT single correction
|
||||||
|
get_all_corrections() # SELECT with filters
|
||||||
|
get_corrections_dict() # For backward compatibility
|
||||||
|
update_correction() # UPDATE with transaction
|
||||||
|
delete_correction() # Soft delete (is_active=0)
|
||||||
|
increment_usage() # Track usage statistics
|
||||||
|
bulk_import_corrections() # Batch INSERT with conflict resolution
|
||||||
|
```
|
||||||
|
|
||||||
|
**Transaction Management**:
|
||||||
|
```python
|
||||||
|
@contextmanager
|
||||||
|
def _transaction(self):
|
||||||
|
conn = self._get_connection()
|
||||||
|
try:
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
yield conn
|
||||||
|
conn.commit()
|
||||||
|
except Exception:
|
||||||
|
conn.rollback()
|
||||||
|
raise
|
||||||
|
```
|
||||||
|
|
||||||
|
### correction_service.py (Business Logic Layer)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Input validation and sanitization
|
||||||
|
- Business rule enforcement
|
||||||
|
- Orchestrate repository operations
|
||||||
|
- Import/export with conflict detection
|
||||||
|
- Statistics calculation
|
||||||
|
|
||||||
|
**Key Methods**:
|
||||||
|
```python
|
||||||
|
# Validation
|
||||||
|
validate_correction_text() # Check length, control chars, NULL bytes
|
||||||
|
validate_domain_name() # Prevent path traversal, injection
|
||||||
|
validate_confidence() # Range check (0.0-1.0)
|
||||||
|
validate_source() # Enum validation
|
||||||
|
|
||||||
|
# Operations
|
||||||
|
add_correction() # Validate + repository.add
|
||||||
|
get_corrections() # Get corrections for domain
|
||||||
|
remove_correction() # Validate + repository.delete
|
||||||
|
|
||||||
|
# Import/Export
|
||||||
|
import_corrections() # Pre-validate + bulk import + conflict detection
|
||||||
|
export_corrections() # Query + format as JSON
|
||||||
|
|
||||||
|
# Analytics
|
||||||
|
get_statistics() # Calculate metrics per domain
|
||||||
|
```
|
||||||
|
|
||||||
|
**Validation Rules**:
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class ValidationRules:
|
||||||
|
max_text_length: int = 1000
|
||||||
|
min_text_length: int = 1
|
||||||
|
max_domain_length: int = 50
|
||||||
|
allowed_domain_pattern: str = r'^[a-zA-Z0-9_-]+$'
|
||||||
|
```
|
||||||
|
|
||||||
|
### CLI Integration (commands.py)
|
||||||
|
|
||||||
|
**Direct Service Usage**:
|
||||||
|
```python
|
||||||
|
def _get_service():
|
||||||
|
"""Get configured CorrectionService instance."""
|
||||||
|
config_dir = Path.home() / ".transcript-fixer"
|
||||||
|
db_path = config_dir / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
return CorrectionService(repository)
|
||||||
|
|
||||||
|
def cmd_add_correction(args):
|
||||||
|
service = _get_service()
|
||||||
|
service.add_correction(args.from_text, args.to_text, args.domain)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits of Direct Integration**:
|
||||||
|
- No unnecessary abstraction layers
|
||||||
|
- Clear data flow: CLI → Service → Repository
|
||||||
|
- Easy to understand and debug
|
||||||
|
- Performance: One less function call per operation
|
||||||
|
|
||||||
|
### dictionary_processor.py (Stage 1)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Apply context-aware regex rules
|
||||||
|
- Apply simple dictionary replacements
|
||||||
|
- Track all changes with line numbers
|
||||||
|
|
||||||
|
**Processing Order**:
|
||||||
|
1. Context rules first (higher priority)
|
||||||
|
2. Dictionary replacements second
|
||||||
|
|
||||||
|
**Key Methods**:
|
||||||
|
```python
|
||||||
|
process(text) -> (corrected_text, changes)
|
||||||
|
_apply_context_rules()
|
||||||
|
_apply_dictionary()
|
||||||
|
get_summary(changes)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Change Tracking**:
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class Change:
|
||||||
|
line_number: int
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
rule_type: str # "dictionary" or "context_rule"
|
||||||
|
rule_name: str
|
||||||
|
```
|
||||||
|
|
||||||
|
### ai_processor.py (Stage 2)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Split text into API-friendly chunks
|
||||||
|
- Call GLM-4.6 API
|
||||||
|
- Handle retries with fallback model
|
||||||
|
- Track AI-suggested changes
|
||||||
|
|
||||||
|
**Key Methods**:
|
||||||
|
```python
|
||||||
|
process(text, context) -> (corrected_text, changes)
|
||||||
|
_split_into_chunks() # Respect paragraph boundaries
|
||||||
|
_process_chunk() # Single API call
|
||||||
|
_build_prompt() # Construct correction prompt
|
||||||
|
```
|
||||||
|
|
||||||
|
**Chunking Strategy**:
|
||||||
|
- Max 6000 characters per chunk
|
||||||
|
- Split on paragraph boundaries (`\n\n`)
|
||||||
|
- If paragraph too long, split on sentences
|
||||||
|
- Preserve context across chunks
|
||||||
|
|
||||||
|
**Error Handling**:
|
||||||
|
- Retry with fallback model (GLM-4.5-Air)
|
||||||
|
- If both fail, use original text
|
||||||
|
- Never lose user's data
|
||||||
|
|
||||||
|
### learning_engine.py (Pattern Detection)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Analyze correction history
|
||||||
|
- Detect recurring patterns
|
||||||
|
- Calculate confidence scores
|
||||||
|
- Generate suggestions for review
|
||||||
|
- Track rejected suggestions
|
||||||
|
|
||||||
|
**Algorithm**:
|
||||||
|
```python
|
||||||
|
1. Query correction_history table
|
||||||
|
2. Extract stage2 (AI) changes
|
||||||
|
3. Group by pattern (from→to)
|
||||||
|
4. Count frequency
|
||||||
|
5. Calculate confidence
|
||||||
|
6. Filter by thresholds:
|
||||||
|
- frequency ≥ 3
|
||||||
|
- confidence ≥ 0.8
|
||||||
|
7. Save to learned/pending_review.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Confidence Calculation**:
|
||||||
|
```python
|
||||||
|
confidence = (
|
||||||
|
0.5 * frequency_score + # More occurrences = higher
|
||||||
|
0.3 * consistency_score + # Always same correction
|
||||||
|
0.2 * recency_score # Recent = higher
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Methods**:
|
||||||
|
```python
|
||||||
|
analyze_and_suggest() # Main analysis pipeline
|
||||||
|
approve_suggestion() # Move to corrections.json
|
||||||
|
reject_suggestion() # Move to rejected.json
|
||||||
|
list_pending() # Get all suggestions
|
||||||
|
```
|
||||||
|
|
||||||
|
### diff_generator.py (Stage 3)
|
||||||
|
|
||||||
|
**Responsibilities**:
|
||||||
|
- Generate comparison reports
|
||||||
|
- Multiple output formats
|
||||||
|
- Word-level diff analysis
|
||||||
|
|
||||||
|
**Output Formats**:
|
||||||
|
1. Markdown summary (statistics + change list)
|
||||||
|
2. Unified diff (standard diff format)
|
||||||
|
3. HTML side-by-side (visual comparison)
|
||||||
|
4. Inline marked ([-old-] [+new+])
|
||||||
|
|
||||||
|
**Not Modified**: Kept original 338-line file as-is (working well)
|
||||||
|
|
||||||
|
## State Management
|
||||||
|
|
||||||
|
### Database-Backed State
|
||||||
|
|
||||||
|
- All state stored in `~/.transcript-fixer/corrections.db`
|
||||||
|
- SQLite handles caching and transactions
|
||||||
|
- ACID guarantees prevent corruption
|
||||||
|
- Backup created before migrations
|
||||||
|
|
||||||
|
### Thread-Safe Access
|
||||||
|
|
||||||
|
- Thread-local connections (one per thread)
|
||||||
|
- BEGIN IMMEDIATE for write transactions
|
||||||
|
- No global state or shared mutable data
|
||||||
|
- Each operation is independent (stateless modules)
|
||||||
|
|
||||||
|
### Soft Deletes
|
||||||
|
|
||||||
|
- Records marked inactive (is_active=0) instead of DELETE
|
||||||
|
- Preserves audit trail
|
||||||
|
- Can be reactivated if needed
|
||||||
|
|
||||||
|
## Error Handling Strategy
|
||||||
|
|
||||||
|
### Fail Fast for User Errors
|
||||||
|
|
||||||
|
```python
|
||||||
|
if not skill_path.exists():
|
||||||
|
print(f"❌ Error: Skill directory not found")
|
||||||
|
sys.exit(1)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Retry for Transient Errors
|
||||||
|
|
||||||
|
```python
|
||||||
|
try:
|
||||||
|
api_call(model_primary)
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
api_call(model_fallback)
|
||||||
|
except Exception:
|
||||||
|
use_original_text()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Before Destructive Operations
|
||||||
|
|
||||||
|
```python
|
||||||
|
if target_file.exists():
|
||||||
|
shutil.copy2(target_file, backup_file)
|
||||||
|
# Then overwrite target_file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
### Unit Testing (Recommended)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Test dictionary processor
|
||||||
|
def test_dictionary_processor():
|
||||||
|
corrections = {"错误": "正确"}
|
||||||
|
processor = DictionaryProcessor(corrections, [])
|
||||||
|
text = "这是错误的文本"
|
||||||
|
result, changes = processor.process(text)
|
||||||
|
assert result == "这是正确的文本"
|
||||||
|
assert len(changes) == 1
|
||||||
|
|
||||||
|
# Test learning engine thresholds
|
||||||
|
def test_learning_thresholds():
|
||||||
|
engine = LearningEngine(history_dir, learned_dir)
|
||||||
|
# Create mock history with pattern appearing 3+ times
|
||||||
|
suggestions = engine.analyze_and_suggest()
|
||||||
|
assert len(suggestions) > 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# End-to-end test
|
||||||
|
python fix_transcription.py --init
|
||||||
|
python fix_transcription.py --add "test" "TEST"
|
||||||
|
python fix_transcription.py --input test.md --stage 3
|
||||||
|
# Verify output files exist
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Bottlenecks
|
||||||
|
|
||||||
|
1. **AI API calls**: Slowest part (60s timeout per chunk)
|
||||||
|
2. **File I/O**: Negligible (JSON files are small)
|
||||||
|
3. **Pattern matching**: Fast (regex + dict lookups)
|
||||||
|
|
||||||
|
### Optimization Strategies
|
||||||
|
|
||||||
|
1. **Stage 1 First**: Test dictionary corrections before expensive AI calls
|
||||||
|
2. **Chunking**: Process large files in parallel chunks (future enhancement)
|
||||||
|
3. **Caching**: Could cache API results by content hash (future enhancement)
|
||||||
|
|
||||||
|
### Scalability
|
||||||
|
|
||||||
|
**Current capabilities (v2.0 with SQLite)**:
|
||||||
|
- File size: Unlimited (chunks handle large files)
|
||||||
|
- Corrections: Tested up to 100,000 entries (with indexes)
|
||||||
|
- History: Unlimited (database handles efficiently)
|
||||||
|
- Concurrent access: Thread-safe with ACID guarantees
|
||||||
|
- Query performance: O(log n) with B-tree indexes
|
||||||
|
|
||||||
|
**Performance improvements from SQLite**:
|
||||||
|
- Indexed queries (domain, source, added_at)
|
||||||
|
- Views for common aggregations
|
||||||
|
- Batch imports with transactions
|
||||||
|
- Soft deletes (no data loss)
|
||||||
|
|
||||||
|
**Future improvements**:
|
||||||
|
- Parallel chunk processing for AI calls
|
||||||
|
- API response caching
|
||||||
|
- Full-text search for corrections
|
||||||
|
|
||||||
|
## Security Architecture
|
||||||
|
|
||||||
|
### Secret Management
|
||||||
|
|
||||||
|
- API keys via environment variables only
|
||||||
|
- Never hardcode credentials
|
||||||
|
- Security scanner enforces this
|
||||||
|
|
||||||
|
### Backup Security
|
||||||
|
|
||||||
|
- `.bak` files same permissions as originals
|
||||||
|
- No encryption (user's responsibility)
|
||||||
|
- Recommendation: Use encrypted filesystems
|
||||||
|
|
||||||
|
### Git Security
|
||||||
|
|
||||||
|
- `.gitignore` for `.bak` files
|
||||||
|
- Private repos recommended
|
||||||
|
- Security scan before commits
|
||||||
|
|
||||||
|
## Extensibility Points
|
||||||
|
|
||||||
|
### Adding New Processors
|
||||||
|
|
||||||
|
1. Create new processor class
|
||||||
|
2. Implement `process(text) -> (result, changes)` interface
|
||||||
|
3. Add to orchestrator workflow
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```python
|
||||||
|
class SpellCheckProcessor:
|
||||||
|
def process(self, text):
|
||||||
|
# Custom spell checking logic
|
||||||
|
return corrected_text, changes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding New Learning Algorithms
|
||||||
|
|
||||||
|
1. Subclass `LearningEngine`
|
||||||
|
2. Override `_calculate_confidence()`
|
||||||
|
3. Adjust thresholds as needed
|
||||||
|
|
||||||
|
### Adding New Export Formats
|
||||||
|
|
||||||
|
1. Add method to `CorrectionManager`
|
||||||
|
2. Support new file format
|
||||||
|
3. Add CLI command
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
### Required
|
||||||
|
|
||||||
|
- Python 3.8+ (`from __future__ import annotations`)
|
||||||
|
- `httpx` (for API calls)
|
||||||
|
|
||||||
|
### Optional
|
||||||
|
|
||||||
|
- `diff` command (for unified diffs)
|
||||||
|
- Git (for version control)
|
||||||
|
|
||||||
|
### Development
|
||||||
|
|
||||||
|
- `pytest` (for testing)
|
||||||
|
- `black` (for formatting)
|
||||||
|
- `mypy` (for type checking)
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### User Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Clone or download skill to workspace
|
||||||
|
git clone <repo> transcript-fixer
|
||||||
|
cd transcript-fixer
|
||||||
|
|
||||||
|
# 2. Install dependencies
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# 3. Initialize
|
||||||
|
python scripts/fix_transcription.py --init
|
||||||
|
|
||||||
|
# 4. Set API key
|
||||||
|
export GLM_API_KEY="KEY_VALUE"
|
||||||
|
|
||||||
|
# Ready to use!
|
||||||
|
```
|
||||||
|
|
||||||
|
### CI/CD Pipeline (Future)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Potential GitHub Actions workflow
|
||||||
|
test:
|
||||||
|
- Install dependencies
|
||||||
|
- Run unit tests
|
||||||
|
- Run integration tests
|
||||||
|
- Check code style (black, mypy)
|
||||||
|
|
||||||
|
security:
|
||||||
|
- Run security_scan.py
|
||||||
|
- Check for secrets
|
||||||
|
|
||||||
|
deploy:
|
||||||
|
- Package skill
|
||||||
|
- Upload to skill marketplace
|
||||||
|
```
|
||||||
|
|
||||||
|
## Further Reading
|
||||||
|
|
||||||
|
- SOLID Principles: https://en.wikipedia.org/wiki/SOLID
|
||||||
|
- API Patterns: `references/glm_api_setup.md`
|
||||||
|
- File Formats: `references/file_formats.md`
|
||||||
|
- Testing: https://docs.pytest.org/
|
||||||
428
transcript-fixer/references/best_practices.md
Normal file
428
transcript-fixer/references/best_practices.md
Normal file
@@ -0,0 +1,428 @@
|
|||||||
|
# Best Practices
|
||||||
|
|
||||||
|
Recommendations for effective use of transcript-fixer based on production experience.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Getting Started](#getting-started)
|
||||||
|
- [Build Foundation Before Scaling](#build-foundation-before-scaling)
|
||||||
|
- [Review Learned Suggestions Regularly](#review-learned-suggestions-regularly)
|
||||||
|
- [Domain Organization](#domain-organization)
|
||||||
|
- [Use Domain Separation](#use-domain-separation)
|
||||||
|
- [Domain Selection Strategy](#domain-selection-strategy)
|
||||||
|
- [Cost Optimization](#cost-optimization)
|
||||||
|
- [Test Dictionary Changes Before AI Calls](#test-dictionary-changes-before-ai-calls)
|
||||||
|
- [Approve High-Confidence Suggestions](#approve-high-confidence-suggestions)
|
||||||
|
- [Team Collaboration](#team-collaboration)
|
||||||
|
- [Export Corrections for Version Control](#export-corrections-for-version-control)
|
||||||
|
- [Share Corrections via Import/Merge](#share-corrections-via-importmerge)
|
||||||
|
- [Data Management](#data-management)
|
||||||
|
- [Database Backup Strategy](#database-backup-strategy)
|
||||||
|
- [Cleanup Strategy](#cleanup-strategy)
|
||||||
|
- [Workflow Efficiency](#workflow-efficiency)
|
||||||
|
- [File Organization](#file-organization)
|
||||||
|
- [Batch Processing](#batch-processing)
|
||||||
|
- [Context Rules for Edge Cases](#context-rules-for-edge-cases)
|
||||||
|
- [Quality Assurance](#quality-assurance)
|
||||||
|
- [Validate After Manual Changes](#validate-after-manual-changes)
|
||||||
|
- [Monitor Learning Quality](#monitor-learning-quality)
|
||||||
|
- [Production Deployment](#production-deployment)
|
||||||
|
- [Environment Variables](#environment-variables)
|
||||||
|
- [Monitoring](#monitoring)
|
||||||
|
- [Performance](#performance)
|
||||||
|
- [Summary](#summary)
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Build Foundation Before Scaling
|
||||||
|
|
||||||
|
**Start small**: Begin with 5-10 manually-added corrections for the most common errors in your domain.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Example: embodied AI domain
|
||||||
|
uv run scripts/fix_transcription.py --add "巨升智能" "具身智能" --domain embodied_ai
|
||||||
|
uv run scripts/fix_transcription.py --add "巨升" "具身" --domain embodied_ai
|
||||||
|
uv run scripts/fix_transcription.py --add "奇迹创坛" "奇绩创坛" --domain embodied_ai
|
||||||
|
```
|
||||||
|
|
||||||
|
**Let learning discover others**: After 3-5 correction runs, the learning system will suggest additional patterns automatically.
|
||||||
|
|
||||||
|
**Rationale**: Manual corrections provide high-quality training data. Learning amplifies your corrections exponentially.
|
||||||
|
|
||||||
|
### Review Learned Suggestions Regularly
|
||||||
|
|
||||||
|
**Frequency**: Every 3-5 correction runs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why**: Learned corrections move from Stage 2 (AI, expensive) to Stage 1 (dictionary, cheap/instant).
|
||||||
|
|
||||||
|
**Impact**:
|
||||||
|
- 10x faster processing (no API calls)
|
||||||
|
- Zero cost for repeated patterns
|
||||||
|
- Builds domain-specific vocabulary automatically
|
||||||
|
|
||||||
|
## Domain Organization
|
||||||
|
|
||||||
|
### Use Domain Separation
|
||||||
|
|
||||||
|
**Prevent conflicts**: Same phonetic error might have different corrections in different domains.
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
- Finance domain: "股价" (stock price) is correct
|
||||||
|
- General domain: "股价" → "框架" (framework) ASR error
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Domain-specific corrections
|
||||||
|
uv run scripts/fix_transcription.py --add "股价" "框架" --domain general
|
||||||
|
# No correction needed in finance domain - "股价" is correct there
|
||||||
|
```
|
||||||
|
|
||||||
|
**Available domains**:
|
||||||
|
- `general` (default) - General-purpose corrections
|
||||||
|
- `embodied_ai` - Robotics and embodied AI terminology
|
||||||
|
- `finance` - Financial terminology
|
||||||
|
- `medical` - Medical terminology
|
||||||
|
|
||||||
|
**Custom domains**: Any string matching `^[a-z0-9_]+$` (lowercase, numbers, underscore).
|
||||||
|
|
||||||
|
### Domain Selection Strategy
|
||||||
|
|
||||||
|
1. **Default domain** for general corrections (dates, common words)
|
||||||
|
2. **Specialized domains** for technical terminology
|
||||||
|
3. **Project domains** for company/product-specific terms
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Project-specific domain
|
||||||
|
uv run scripts/fix_transcription.py --add "我司" "奇绩创坛" --domain yc_china
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cost Optimization
|
||||||
|
|
||||||
|
### Test Dictionary Changes Before AI Calls
|
||||||
|
|
||||||
|
**Problem**: AI calls (Stage 2) consume API quota and time.
|
||||||
|
|
||||||
|
**Solution**: Test dictionary changes with Stage 1 first.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Add new corrections
|
||||||
|
uv run scripts/fix_transcription.py --add "新错误" "正确词" --domain general
|
||||||
|
|
||||||
|
# 2. Test on small sample (Stage 1 only)
|
||||||
|
uv run scripts/fix_transcription.py --input sample.md --stage 1
|
||||||
|
|
||||||
|
# 3. Review output
|
||||||
|
less sample_stage1.md
|
||||||
|
|
||||||
|
# 4. If satisfied, run full pipeline on large files
|
||||||
|
uv run scripts/fix_transcription.py --input large_file.md --stage 3
|
||||||
|
```
|
||||||
|
|
||||||
|
**Savings**: Avoid wasting API quota on files with dictionary-only corrections.
|
||||||
|
|
||||||
|
### Approve High-Confidence Suggestions
|
||||||
|
|
||||||
|
**Check suggestions regularly**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
```
|
||||||
|
|
||||||
|
**Approve suggestions with**:
|
||||||
|
- Frequency ≥ 5
|
||||||
|
- Confidence ≥ 0.9
|
||||||
|
- Pattern makes semantic sense
|
||||||
|
|
||||||
|
**Impact**: Each approved suggestion saves future API calls.
|
||||||
|
|
||||||
|
## Team Collaboration
|
||||||
|
|
||||||
|
### Export Corrections for Version Control
|
||||||
|
|
||||||
|
**Don't commit** `.db` files to Git:
|
||||||
|
- Binary format causes merge conflicts
|
||||||
|
- Database grows over time (bloats repository)
|
||||||
|
- Not human-reviewable
|
||||||
|
|
||||||
|
**Do commit** JSON exports:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export domain dictionaries
|
||||||
|
uv run scripts/fix_transcription.py --export general_$(date +%Y%m%d).json --domain general
|
||||||
|
uv run scripts/fix_transcription.py --export embodied_ai_$(date +%Y%m%d).json --domain embodied_ai
|
||||||
|
|
||||||
|
# .gitignore
|
||||||
|
*.db
|
||||||
|
*.db-journal
|
||||||
|
*.bak
|
||||||
|
|
||||||
|
# Commit exports
|
||||||
|
git add *_corrections.json
|
||||||
|
git commit -m "Update correction dictionaries"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Share Corrections via Import/Merge
|
||||||
|
|
||||||
|
**Always use `--merge` flag** to combine corrections:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pull latest from team
|
||||||
|
git pull origin main
|
||||||
|
|
||||||
|
# Import new corrections (merge mode)
|
||||||
|
uv run scripts/fix_transcription.py --import general_20250128.json --merge
|
||||||
|
uv run scripts/fix_transcription.py --import embodied_ai_20250128.json --merge
|
||||||
|
```
|
||||||
|
|
||||||
|
**Merge behavior**:
|
||||||
|
- New corrections: inserted
|
||||||
|
- Existing corrections with higher confidence: updated
|
||||||
|
- Existing corrections with lower confidence: skipped
|
||||||
|
- Preserves local customizations
|
||||||
|
|
||||||
|
See `team_collaboration.md` for Git workflows and conflict handling.
|
||||||
|
|
||||||
|
## Data Management
|
||||||
|
|
||||||
|
### Database Backup Strategy
|
||||||
|
|
||||||
|
**Automatic backups**: Database creates timestamped backups before migrations:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.transcript-fixer/
|
||||||
|
├── corrections.db
|
||||||
|
├── corrections.20250128_140532.bak
|
||||||
|
└── corrections.20250127_093021.bak
|
||||||
|
```
|
||||||
|
|
||||||
|
**Manual backups** before bulk changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp ~/.transcript-fixer/corrections.db ~/backups/corrections_$(date +%Y%m%d).db
|
||||||
|
```
|
||||||
|
|
||||||
|
**Or use SQLite backup**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db ".backup ~/backups/corrections.db"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cleanup Strategy
|
||||||
|
|
||||||
|
**History retention**: Keep recent history, archive old entries:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Archive history older than 90 days
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
DELETE FROM correction_history
|
||||||
|
WHERE run_timestamp < datetime('now', '-90 days');
|
||||||
|
"
|
||||||
|
|
||||||
|
# Reclaim space
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "VACUUM;"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Suggestion cleanup**: Reject low-confidence suggestions periodically:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Reject suggestions with frequency < 3
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
UPDATE learned_suggestions
|
||||||
|
SET status = 'rejected'
|
||||||
|
WHERE frequency < 3 AND confidence < 0.7;
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workflow Efficiency
|
||||||
|
|
||||||
|
### File Organization
|
||||||
|
|
||||||
|
**Use consistent naming**:
|
||||||
|
```
|
||||||
|
meeting_20250128.md # Original transcript
|
||||||
|
meeting_20250128_stage1.md # Dictionary corrections
|
||||||
|
meeting_20250128_stage2.md # Final corrected version
|
||||||
|
```
|
||||||
|
|
||||||
|
**Generate diff reports** for review:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/diff_generator.py \
|
||||||
|
meeting_20250128.md \
|
||||||
|
meeting_20250128_stage1.md \
|
||||||
|
meeting_20250128_stage2.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output formats**:
|
||||||
|
- Markdown report (what changed, statistics)
|
||||||
|
- Unified diff (git-style)
|
||||||
|
- HTML side-by-side (visual review)
|
||||||
|
- Inline markers (for direct editing)
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
|
||||||
|
**Process similar files together** to amplify learning:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Day 1: Process 5 similar meetings
|
||||||
|
for file in meeting_*.md; do
|
||||||
|
uv run scripts/fix_transcription.py --input "$file" --stage 3 --domain embodied_ai
|
||||||
|
done
|
||||||
|
|
||||||
|
# Day 2: Review learned patterns
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
|
||||||
|
# Approve good suggestions
|
||||||
|
uv run scripts/fix_transcription.py --approve "常见错误1" "正确词1"
|
||||||
|
uv run scripts/fix_transcription.py --approve "常见错误2" "正确词2"
|
||||||
|
|
||||||
|
# Day 3: Future files benefit from dictionary corrections
|
||||||
|
```
|
||||||
|
|
||||||
|
### Context Rules for Edge Cases
|
||||||
|
|
||||||
|
**Use regex context rules** for:
|
||||||
|
- Positional dependencies (e.g., "的" vs "地" before verbs)
|
||||||
|
- Multi-word patterns
|
||||||
|
- Traditional vs simplified Chinese
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# "的" before verb → "地"
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离的去看', '近距离地去看', '的→地 before verb', 10);
|
||||||
|
|
||||||
|
# Preserve correct usage
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离搏杀', '近距离搏杀', '的 is correct here (noun modifier)', 5);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Priority**: Higher numbers run first (use for exceptions).
|
||||||
|
|
||||||
|
## Quality Assurance
|
||||||
|
|
||||||
|
### Validate After Manual Changes
|
||||||
|
|
||||||
|
**After direct SQL edits**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**After imports**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check statistics
|
||||||
|
uv run scripts/fix_transcription.py --list --domain general | head -20
|
||||||
|
|
||||||
|
# Verify specific corrections
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT from_text, to_text, source, confidence
|
||||||
|
FROM active_corrections
|
||||||
|
WHERE domain = 'general'
|
||||||
|
ORDER BY added_at DESC
|
||||||
|
LIMIT 10;
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monitor Learning Quality
|
||||||
|
|
||||||
|
**Check suggestion confidence distribution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT
|
||||||
|
CASE
|
||||||
|
WHEN confidence >= 0.9 THEN 'high (>=0.9)'
|
||||||
|
WHEN confidence >= 0.8 THEN 'medium (0.8-0.9)'
|
||||||
|
ELSE 'low (<0.8)'
|
||||||
|
END as confidence_level,
|
||||||
|
COUNT(*) as count
|
||||||
|
FROM learned_suggestions
|
||||||
|
WHERE status = 'pending'
|
||||||
|
GROUP BY confidence_level;
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Review examples** for low-confidence suggestions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT s.from_text, s.to_text, s.confidence, e.context
|
||||||
|
FROM learned_suggestions s
|
||||||
|
JOIN suggestion_examples e ON s.id = e.suggestion_id
|
||||||
|
WHERE s.confidence < 0.8 AND s.status = 'pending';
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Deployment
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
**Set permanently** in production:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add to /etc/environment or systemd service
|
||||||
|
GLM_API_KEY=your-production-key
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monitoring
|
||||||
|
|
||||||
|
**Track usage statistics**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Corrections by source
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT source, COUNT(*) as count, SUM(usage_count) as total_usage
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
GROUP BY source;
|
||||||
|
"
|
||||||
|
|
||||||
|
# Success rate
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT
|
||||||
|
COUNT(*) as total_runs,
|
||||||
|
SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END) as successful,
|
||||||
|
ROUND(100.0 * SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END) / COUNT(*), 2) as success_rate
|
||||||
|
FROM correction_history;
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
**Database optimization**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Rebuild indexes periodically
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "REINDEX;"
|
||||||
|
|
||||||
|
# Analyze query patterns
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "ANALYZE;"
|
||||||
|
|
||||||
|
# Vacuum to reclaim space
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "VACUUM;"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
**Key principles**:
|
||||||
|
1. Start small, let learning amplify
|
||||||
|
2. Use domain separation for quality
|
||||||
|
3. Test dictionary changes before AI calls
|
||||||
|
4. Export to JSON for version control
|
||||||
|
5. Review and approve learned suggestions
|
||||||
|
6. Validate after manual changes
|
||||||
|
7. Monitor learning quality
|
||||||
|
8. Backup before bulk operations
|
||||||
|
|
||||||
|
**ROI timeline**:
|
||||||
|
- Week 1: Build foundation (10-20 manual corrections)
|
||||||
|
- Week 2-3: Learning kicks in (20-50 suggestions)
|
||||||
|
- Month 2+: Mature vocabulary (80%+ dictionary coverage, minimal AI calls)
|
||||||
97
transcript-fixer/references/dictionary_guide.md
Normal file
97
transcript-fixer/references/dictionary_guide.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
# 纠错词典配置指南
|
||||||
|
|
||||||
|
## 词典结构
|
||||||
|
|
||||||
|
纠错词典位于 `fix_transcription.py` 中,包含两部分:
|
||||||
|
|
||||||
|
### 1. 上下文规则 (CONTEXT_RULES)
|
||||||
|
|
||||||
|
用于需要结合上下文判断的替换:
|
||||||
|
|
||||||
|
```python
|
||||||
|
CONTEXT_RULES = [
|
||||||
|
{
|
||||||
|
"pattern": r"正则表达式",
|
||||||
|
"replacement": "替换文本",
|
||||||
|
"description": "规则说明"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
**示例:**
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"pattern": r"近距离的去看",
|
||||||
|
"replacement": "近距离地去看",
|
||||||
|
"description": "修正'的'为'地'"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 通用词典 (CORRECTIONS_DICT)
|
||||||
|
|
||||||
|
用于直接字符串替换:
|
||||||
|
|
||||||
|
```python
|
||||||
|
CORRECTIONS_DICT = {
|
||||||
|
"错误词汇": "正确词汇",
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**示例:**
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"巨升智能": "具身智能",
|
||||||
|
"奇迹创坛": "奇绩创坛",
|
||||||
|
"矩阵公司": "初创公司",
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 添加自定义规则
|
||||||
|
|
||||||
|
### 步骤1: 识别错误模式
|
||||||
|
|
||||||
|
从修复报告中识别重复出现的错误。
|
||||||
|
|
||||||
|
### 步骤2: 选择规则类型
|
||||||
|
|
||||||
|
- **简单替换** → 使用 CORRECTIONS_DICT
|
||||||
|
- **需要上下文** → 使用 CONTEXT_RULES
|
||||||
|
|
||||||
|
### 步骤3: 添加到词典
|
||||||
|
|
||||||
|
编辑 `scripts/fix_transcription.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
CORRECTIONS_DICT = {
|
||||||
|
# 现有规则...
|
||||||
|
"你的错误": "正确词汇", # 添加新规则
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 步骤4: 测试
|
||||||
|
|
||||||
|
运行修复脚本测试新规则。
|
||||||
|
|
||||||
|
## 常见错误类型
|
||||||
|
|
||||||
|
### 同音字错误
|
||||||
|
```python
|
||||||
|
"股价": "框架",
|
||||||
|
"三观": "三关",
|
||||||
|
```
|
||||||
|
|
||||||
|
### 专业术语
|
||||||
|
```python
|
||||||
|
"巨升智能": "具身智能",
|
||||||
|
"近距离": "具身", # 某些上下文中
|
||||||
|
```
|
||||||
|
|
||||||
|
### 公司名称
|
||||||
|
```python
|
||||||
|
"奇迹创坛": "奇绩创坛",
|
||||||
|
```
|
||||||
|
|
||||||
|
## 优先级
|
||||||
|
|
||||||
|
1. 先应用 CONTEXT_RULES (精确匹配)
|
||||||
|
2. 再应用 CORRECTIONS_DICT (全局替换)
|
||||||
395
transcript-fixer/references/file_formats.md
Normal file
395
transcript-fixer/references/file_formats.md
Normal file
@@ -0,0 +1,395 @@
|
|||||||
|
# Storage Format Reference
|
||||||
|
|
||||||
|
This document describes the SQLite database format used by transcript-fixer v2.0.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Database Location](#database-location)
|
||||||
|
- [Database Schema](#database-schema)
|
||||||
|
- [Core Tables](#core-tables)
|
||||||
|
- [Views](#views)
|
||||||
|
- [Querying the Database](#querying-the-database)
|
||||||
|
- [Using Python API](#using-python-api)
|
||||||
|
- [Using SQLite CLI](#using-sqlite-cli)
|
||||||
|
- [Import/Export](#importexport)
|
||||||
|
- [Export to JSON](#export-to-json)
|
||||||
|
- [Import from JSON](#import-from-json)
|
||||||
|
- [Backup Strategy](#backup-strategy)
|
||||||
|
- [Automatic Backups](#automatic-backups)
|
||||||
|
- [Manual Backups](#manual-backups)
|
||||||
|
- [Version Control](#version-control)
|
||||||
|
- [Best Practices](#best-practices)
|
||||||
|
- [Troubleshooting](#troubleshooting)
|
||||||
|
- [Database Locked](#database-locked)
|
||||||
|
- [Corrupted Database](#corrupted-database)
|
||||||
|
- [Missing Tables](#missing-tables)
|
||||||
|
|
||||||
|
## Database Location
|
||||||
|
|
||||||
|
**Path**: `~/.transcript-fixer/corrections.db`
|
||||||
|
|
||||||
|
**Type**: SQLite 3 database with ACID guarantees
|
||||||
|
|
||||||
|
## Database Schema
|
||||||
|
|
||||||
|
### Core Tables
|
||||||
|
|
||||||
|
#### corrections
|
||||||
|
|
||||||
|
Main correction dictionary storage.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| from_text | TEXT | NOT NULL | Original (incorrect) text |
|
||||||
|
| to_text | TEXT | NOT NULL | Corrected text |
|
||||||
|
| domain | TEXT | DEFAULT 'general' | Correction domain |
|
||||||
|
| source | TEXT | CHECK IN ('manual', 'learned', 'imported') | Origin of correction |
|
||||||
|
| confidence | REAL | CHECK 0.0-1.0 | Confidence score |
|
||||||
|
| added_by | TEXT | | User who added |
|
||||||
|
| added_at | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | When added |
|
||||||
|
| usage_count | INTEGER | DEFAULT 0, CHECK >= 0 | Times used |
|
||||||
|
| last_used | TIMESTAMP | | Last usage time |
|
||||||
|
| notes | TEXT | | Optional notes |
|
||||||
|
| is_active | BOOLEAN | DEFAULT 1 | Soft delete flag |
|
||||||
|
|
||||||
|
**Unique Constraint**: `(from_text, domain)`
|
||||||
|
|
||||||
|
**Indexes**: domain, source, added_at, is_active, from_text
|
||||||
|
|
||||||
|
#### context_rules
|
||||||
|
|
||||||
|
Regex-based context-aware correction rules.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| pattern | TEXT | NOT NULL, UNIQUE | Regex pattern |
|
||||||
|
| replacement | TEXT | NOT NULL | Replacement text |
|
||||||
|
| description | TEXT | | Rule explanation |
|
||||||
|
| priority | INTEGER | DEFAULT 0 | Higher = applied first |
|
||||||
|
| is_active | BOOLEAN | DEFAULT 1 | Enable/disable |
|
||||||
|
| added_at | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | When added |
|
||||||
|
| added_by | TEXT | | User who added |
|
||||||
|
|
||||||
|
**Indexes**: priority (DESC), is_active
|
||||||
|
|
||||||
|
#### correction_history
|
||||||
|
|
||||||
|
Audit log for all correction runs.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| filename | TEXT | NOT NULL | File corrected |
|
||||||
|
| domain | TEXT | NOT NULL | Domain used |
|
||||||
|
| run_timestamp | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | When run |
|
||||||
|
| original_length | INTEGER | CHECK >= 0 | Original file size |
|
||||||
|
| stage1_changes | INTEGER | CHECK >= 0 | Dictionary changes |
|
||||||
|
| stage2_changes | INTEGER | CHECK >= 0 | AI changes |
|
||||||
|
| model | TEXT | | AI model used |
|
||||||
|
| execution_time_ms | INTEGER | | Runtime in ms |
|
||||||
|
| success | BOOLEAN | DEFAULT 1 | Success flag |
|
||||||
|
| error_message | TEXT | | Error if failed |
|
||||||
|
|
||||||
|
**Indexes**: run_timestamp (DESC), domain, success
|
||||||
|
|
||||||
|
#### correction_changes
|
||||||
|
|
||||||
|
Detailed changes made in each run.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| history_id | INTEGER | FOREIGN KEY → correction_history | Parent run |
|
||||||
|
| line_number | INTEGER | | Line in file |
|
||||||
|
| from_text | TEXT | NOT NULL | Original text |
|
||||||
|
| to_text | TEXT | NOT NULL | Corrected text |
|
||||||
|
| rule_type | TEXT | CHECK IN ('context', 'dictionary', 'ai') | Rule type |
|
||||||
|
| rule_id | INTEGER | | Reference to rule |
|
||||||
|
| context_before | TEXT | | Text before |
|
||||||
|
| context_after | TEXT | | Text after |
|
||||||
|
|
||||||
|
**Foreign Key**: history_id → correction_history.id (CASCADE DELETE)
|
||||||
|
|
||||||
|
**Indexes**: history_id, rule_type
|
||||||
|
|
||||||
|
#### learned_suggestions
|
||||||
|
|
||||||
|
AI-detected patterns pending review.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| from_text | TEXT | NOT NULL | Pattern detected |
|
||||||
|
| to_text | TEXT | NOT NULL | Suggested correction |
|
||||||
|
| domain | TEXT | DEFAULT 'general' | Domain |
|
||||||
|
| frequency | INTEGER | CHECK > 0 | Times seen |
|
||||||
|
| confidence | REAL | CHECK 0.0-1.0 | Confidence score |
|
||||||
|
| first_seen | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | First occurrence |
|
||||||
|
| last_seen | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | Last occurrence |
|
||||||
|
| status | TEXT | CHECK IN ('pending', 'approved', 'rejected') | Review status |
|
||||||
|
| reviewed_at | TIMESTAMP | | When reviewed |
|
||||||
|
| reviewed_by | TEXT | | Who reviewed |
|
||||||
|
|
||||||
|
**Unique Constraint**: `(from_text, to_text, domain)`
|
||||||
|
|
||||||
|
**Indexes**: status, domain, confidence (DESC), frequency (DESC)
|
||||||
|
|
||||||
|
#### suggestion_examples
|
||||||
|
|
||||||
|
Example occurrences of learned patterns.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| suggestion_id | INTEGER | FOREIGN KEY → learned_suggestions | Parent suggestion |
|
||||||
|
| filename | TEXT | NOT NULL | File where found |
|
||||||
|
| line_number | INTEGER | | Line number |
|
||||||
|
| context | TEXT | NOT NULL | Surrounding text |
|
||||||
|
| occurred_at | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | When found |
|
||||||
|
|
||||||
|
**Foreign Key**: suggestion_id → learned_suggestions.id (CASCADE DELETE)
|
||||||
|
|
||||||
|
**Index**: suggestion_id
|
||||||
|
|
||||||
|
#### system_config
|
||||||
|
|
||||||
|
System configuration key-value store.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| key | TEXT | PRIMARY KEY | Config key |
|
||||||
|
| value | TEXT | NOT NULL | Config value |
|
||||||
|
| value_type | TEXT | CHECK IN ('string', 'int', 'float', 'boolean', 'json') | Value type |
|
||||||
|
| description | TEXT | | Config description |
|
||||||
|
| updated_at | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | Last update |
|
||||||
|
|
||||||
|
**Default Values**:
|
||||||
|
- `schema_version`: "2.0"
|
||||||
|
- `api_provider`: "GLM"
|
||||||
|
- `api_model`: "GLM-4.6"
|
||||||
|
- `default_domain`: "general"
|
||||||
|
- `auto_learn_enabled`: "true"
|
||||||
|
- `learning_frequency_threshold`: "3"
|
||||||
|
- `learning_confidence_threshold`: "0.8"
|
||||||
|
|
||||||
|
#### audit_log
|
||||||
|
|
||||||
|
Comprehensive audit trail for all operations.
|
||||||
|
|
||||||
|
| Column | Type | Constraints | Description |
|
||||||
|
|--------|------|-------------|-------------|
|
||||||
|
| id | INTEGER | PRIMARY KEY | Auto-increment ID |
|
||||||
|
| timestamp | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | When occurred |
|
||||||
|
| action | TEXT | NOT NULL | Action type |
|
||||||
|
| entity_type | TEXT | NOT NULL | Entity affected |
|
||||||
|
| entity_id | INTEGER | | Entity ID |
|
||||||
|
| user | TEXT | | User who performed |
|
||||||
|
| details | TEXT | | Action details |
|
||||||
|
| success | BOOLEAN | DEFAULT 1 | Success flag |
|
||||||
|
| error_message | TEXT | | Error if failed |
|
||||||
|
|
||||||
|
**Indexes**: timestamp (DESC), action, entity_type, success
|
||||||
|
|
||||||
|
### Views
|
||||||
|
|
||||||
|
#### active_corrections
|
||||||
|
|
||||||
|
Quick access to active corrections.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT id, from_text, to_text, domain, source, confidence, usage_count, last_used, added_at
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
ORDER BY domain, from_text;
|
||||||
|
```
|
||||||
|
|
||||||
|
#### pending_suggestions
|
||||||
|
|
||||||
|
Suggestions pending review with example count.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT s.id, s.from_text, s.to_text, s.domain, s.frequency, s.confidence,
|
||||||
|
s.first_seen, s.last_seen, COUNT(e.id) as example_count
|
||||||
|
FROM learned_suggestions s
|
||||||
|
LEFT JOIN suggestion_examples e ON s.id = e.suggestion_id
|
||||||
|
WHERE s.status = 'pending'
|
||||||
|
GROUP BY s.id
|
||||||
|
ORDER BY s.confidence DESC, s.frequency DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
#### correction_statistics
|
||||||
|
|
||||||
|
Statistics per domain.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT domain,
|
||||||
|
COUNT(*) as total_corrections,
|
||||||
|
COUNT(CASE WHEN source = 'manual' THEN 1 END) as manual_count,
|
||||||
|
COUNT(CASE WHEN source = 'learned' THEN 1 END) as learned_count,
|
||||||
|
COUNT(CASE WHEN source = 'imported' THEN 1 END) as imported_count,
|
||||||
|
SUM(usage_count) as total_usage,
|
||||||
|
MAX(added_at) as last_updated
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
GROUP BY domain;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Querying the Database
|
||||||
|
|
||||||
|
### Using Python API
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
from core import CorrectionRepository, CorrectionService
|
||||||
|
|
||||||
|
# Initialize
|
||||||
|
db_path = Path.home() / ".transcript-fixer" / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository)
|
||||||
|
|
||||||
|
# Add correction
|
||||||
|
service.add_correction("错误", "正确", domain="general")
|
||||||
|
|
||||||
|
# Get corrections
|
||||||
|
corrections = service.get_corrections(domain="general")
|
||||||
|
|
||||||
|
# Get statistics
|
||||||
|
stats = service.get_statistics(domain="general")
|
||||||
|
print(f"Total: {stats['total_corrections']}")
|
||||||
|
|
||||||
|
# Close
|
||||||
|
service.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using SQLite CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Open database
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# View active corrections
|
||||||
|
SELECT from_text, to_text, domain FROM active_corrections;
|
||||||
|
|
||||||
|
# View statistics
|
||||||
|
SELECT * FROM correction_statistics;
|
||||||
|
|
||||||
|
# View pending suggestions
|
||||||
|
SELECT * FROM pending_suggestions;
|
||||||
|
|
||||||
|
# Check schema version
|
||||||
|
SELECT value FROM system_config WHERE key = 'schema_version';
|
||||||
|
```
|
||||||
|
|
||||||
|
## Import/Export
|
||||||
|
|
||||||
|
### Export to JSON
|
||||||
|
|
||||||
|
```python
|
||||||
|
service = _get_service()
|
||||||
|
corrections = service.export_corrections(domain="general")
|
||||||
|
|
||||||
|
# Write to file
|
||||||
|
import json
|
||||||
|
with open("export.json", "w", encoding="utf-8") as f:
|
||||||
|
json.dump({
|
||||||
|
"version": "2.0",
|
||||||
|
"domain": "general",
|
||||||
|
"corrections": corrections
|
||||||
|
}, f, ensure_ascii=False, indent=2)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Import from JSON
|
||||||
|
|
||||||
|
```python
|
||||||
|
import json
|
||||||
|
|
||||||
|
with open("import.json", "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
service = _get_service()
|
||||||
|
inserted, updated, skipped = service.import_corrections(
|
||||||
|
corrections=data["corrections"],
|
||||||
|
domain=data.get("domain", "general"),
|
||||||
|
merge=True,
|
||||||
|
validate_all=True
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Imported: {inserted} new, {updated} updated, {skipped} skipped")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backup Strategy
|
||||||
|
|
||||||
|
### Automatic Backups
|
||||||
|
|
||||||
|
The system maintains database integrity through SQLite's ACID guarantees and automatic journaling.
|
||||||
|
|
||||||
|
### Manual Backups
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup database
|
||||||
|
cp ~/.transcript-fixer/corrections.db ~/backups/corrections_$(date +%Y%m%d).db
|
||||||
|
|
||||||
|
# Or use SQLite backup
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db ".backup ~/backups/corrections.db"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Version Control
|
||||||
|
|
||||||
|
**Recommended**: Use Git for configuration and export files, but NOT for the database:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# .gitignore
|
||||||
|
*.db
|
||||||
|
*.db-journal
|
||||||
|
*.bak
|
||||||
|
```
|
||||||
|
|
||||||
|
Instead, export corrections periodically:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --export-json corrections_backup.json
|
||||||
|
git add corrections_backup.json
|
||||||
|
git commit -m "Backup corrections"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Regular Exports**: Export to JSON weekly for team sharing
|
||||||
|
2. **Database Backups**: Backup `.db` file before major changes
|
||||||
|
3. **Use Transactions**: All modifications use ACID transactions automatically
|
||||||
|
4. **Soft Deletes**: Records are marked inactive, not deleted (preserves audit trail)
|
||||||
|
5. **Validate**: Run `--validate` after manual database changes
|
||||||
|
6. **Statistics**: Check usage patterns via `correction_statistics` view
|
||||||
|
7. **Cleanup**: Old history can be archived (query by `run_timestamp`)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Database Locked
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check for lingering connections
|
||||||
|
lsof ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# If needed, backup and recreate
|
||||||
|
cp corrections.db corrections_backup.db
|
||||||
|
sqlite3 corrections.db "VACUUM;"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Corrupted Database
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check integrity
|
||||||
|
sqlite3 corrections.db "PRAGMA integrity_check;"
|
||||||
|
|
||||||
|
# Recover if possible
|
||||||
|
sqlite3 corrections.db ".recover" | sqlite3 corrections_new.db
|
||||||
|
```
|
||||||
|
|
||||||
|
### Missing Tables
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Reinitialize schema (safe, uses IF NOT EXISTS)
|
||||||
|
python -c "from core import CorrectionRepository; from pathlib import Path; CorrectionRepository(Path.home() / '.transcript-fixer' / 'corrections.db')"
|
||||||
|
```
|
||||||
116
transcript-fixer/references/glm_api_setup.md
Normal file
116
transcript-fixer/references/glm_api_setup.md
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
# GLM API 配置指南
|
||||||
|
|
||||||
|
## API配置
|
||||||
|
|
||||||
|
### 设置环境变量
|
||||||
|
|
||||||
|
在运行脚本前,设置GLM API密钥环境变量:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux/macOS
|
||||||
|
export GLM_API_KEY="your-api-key-here"
|
||||||
|
|
||||||
|
# Windows (PowerShell)
|
||||||
|
$env:GLM_API_KEY="your-api-key-here"
|
||||||
|
|
||||||
|
# Windows (CMD)
|
||||||
|
set GLM_API_KEY=your-api-key-here
|
||||||
|
```
|
||||||
|
|
||||||
|
**永久设置** (推荐):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux/macOS: 添加到 ~/.bashrc 或 ~/.zshrc
|
||||||
|
echo 'export GLM_API_KEY="your-api-key-here"' >> ~/.bashrc
|
||||||
|
source ~/.bashrc
|
||||||
|
|
||||||
|
# Windows: 在系统环境变量中设置
|
||||||
|
```
|
||||||
|
|
||||||
|
### 脚本配置
|
||||||
|
|
||||||
|
脚本会自动从环境变量读取API密钥:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 脚本会检查环境变量
|
||||||
|
if "GLM_API_KEY" not in os.environ:
|
||||||
|
raise ValueError("请设置 GLM_API_KEY 环境变量")
|
||||||
|
|
||||||
|
os.environ["ANTHROPIC_BASE_URL"] = "https://open.bigmodel.cn/api/anthropic"
|
||||||
|
os.environ["ANTHROPIC_API_KEY"] = os.environ["GLM_API_KEY"]
|
||||||
|
|
||||||
|
# 模型配置
|
||||||
|
GLM_MODEL = "GLM-4.6" # 主力模型
|
||||||
|
GLM_MODEL_FAST = "GLM-4.5-Air" # 快速模型(备用)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 支持的模型
|
||||||
|
|
||||||
|
| 模型名称 | 说明 | 用途 |
|
||||||
|
|---------|------|------|
|
||||||
|
| GLM-4.6 | 最强模型 | 默认使用,精度最高 |
|
||||||
|
| GLM-4.5-Air | 快速模型 | 备用,速度更快 |
|
||||||
|
|
||||||
|
**注意**: 模型名称大小写不敏感。
|
||||||
|
|
||||||
|
## API认证
|
||||||
|
|
||||||
|
智谱GLM使用Anthropic兼容API:
|
||||||
|
|
||||||
|
```python
|
||||||
|
headers = {
|
||||||
|
"anthropic-version": "2023-06-01",
|
||||||
|
"Authorization": f"Bearer {api_key}",
|
||||||
|
"content-type": "application/json"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**关键点:**
|
||||||
|
- 使用 `Authorization: Bearer` 头
|
||||||
|
- 不要使用 `x-api-key` 头
|
||||||
|
|
||||||
|
## API调用示例
|
||||||
|
|
||||||
|
```python
|
||||||
|
def call_glm_api(prompt: str) -> str:
|
||||||
|
url = "https://open.bigmodel.cn/api/anthropic/v1/messages"
|
||||||
|
headers = {
|
||||||
|
"anthropic-version": "2023-06-01",
|
||||||
|
"Authorization": f"Bearer {os.environ.get('ANTHROPIC_API_KEY')}",
|
||||||
|
"content-type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
data = {
|
||||||
|
"model": "GLM-4.6",
|
||||||
|
"max_tokens": 8000,
|
||||||
|
"temperature": 0.3,
|
||||||
|
"messages": [{"role": "user", "content": prompt}]
|
||||||
|
}
|
||||||
|
|
||||||
|
response = httpx.post(url, headers=headers, json=data, timeout=60.0)
|
||||||
|
return response.json()["content"][0]["text"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## 获取API密钥
|
||||||
|
|
||||||
|
1. 访问 https://open.bigmodel.cn/
|
||||||
|
2. 注册/登录账号
|
||||||
|
3. 进入API管理页面
|
||||||
|
4. 创建新的API密钥
|
||||||
|
5. 复制密钥到配置中
|
||||||
|
|
||||||
|
## 费用
|
||||||
|
|
||||||
|
参考智谱AI官方定价:
|
||||||
|
- GLM-4.6: 按token计费
|
||||||
|
- GLM-4.5-Air: 更便宜的选择
|
||||||
|
|
||||||
|
## 故障排查
|
||||||
|
|
||||||
|
### 401错误
|
||||||
|
- 检查API密钥是否正确
|
||||||
|
- 确认使用 `Authorization: Bearer` 头
|
||||||
|
|
||||||
|
### 超时错误
|
||||||
|
- 增加timeout参数
|
||||||
|
- 考虑使用GLM-4.5-Air快速模型
|
||||||
135
transcript-fixer/references/installation_setup.md
Normal file
135
transcript-fixer/references/installation_setup.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
# Setup Guide
|
||||||
|
|
||||||
|
Complete installation and configuration guide for transcript-fixer.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Installation](#installation)
|
||||||
|
- [API Configuration](#api-configuration)
|
||||||
|
- [Environment Setup](#environment-setup)
|
||||||
|
- [Next Steps](#next-steps)
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
|
||||||
|
Install required dependencies using uv:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Or sync the project environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync
|
||||||
|
```
|
||||||
|
|
||||||
|
**Required packages**:
|
||||||
|
- `anthropic` - For Claude API integration (future)
|
||||||
|
- `requests` - For GLM API calls
|
||||||
|
- `difflib` - Standard library for diff generation
|
||||||
|
|
||||||
|
### Database Initialization
|
||||||
|
|
||||||
|
Initialize the SQLite database (first time only):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates `~/.transcript-fixer/corrections.db` with the complete schema:
|
||||||
|
- 8 tables (corrections, context_rules, history, suggestions, etc.)
|
||||||
|
- 3 views (active_corrections, pending_suggestions, statistics)
|
||||||
|
- ACID transactions enabled
|
||||||
|
- Automatic backups before migrations
|
||||||
|
|
||||||
|
See `file_formats.md` for complete database schema.
|
||||||
|
|
||||||
|
## API Configuration
|
||||||
|
|
||||||
|
### GLM API Key (Required for Stage 2)
|
||||||
|
|
||||||
|
Stage 2 AI corrections require a GLM API key.
|
||||||
|
|
||||||
|
1. **Obtain API key**: Visit https://open.bigmodel.cn/
|
||||||
|
2. **Register** for an account
|
||||||
|
3. **Generate** an API key from the dashboard
|
||||||
|
4. **Set environment variable**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export GLM_API_KEY="your-api-key-here"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Persistence**: Add to shell profile for permanent access:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For bash
|
||||||
|
echo 'export GLM_API_KEY="your-key"' >> ~/.bashrc
|
||||||
|
source ~/.bashrc
|
||||||
|
|
||||||
|
# For zsh
|
||||||
|
echo 'export GLM_API_KEY="your-key"' >> ~/.zshrc
|
||||||
|
source ~/.zshrc
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Configuration
|
||||||
|
|
||||||
|
Run validation to check setup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected output**:
|
||||||
|
```
|
||||||
|
🔍 Validating transcript-fixer configuration...
|
||||||
|
|
||||||
|
✅ Configuration directory exists: ~/.transcript-fixer
|
||||||
|
✅ Database valid: 0 corrections
|
||||||
|
✅ All 8 tables present
|
||||||
|
✅ GLM_API_KEY is set
|
||||||
|
|
||||||
|
============================================================
|
||||||
|
✅ All checks passed! Configuration is valid.
|
||||||
|
============================================================
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Setup
|
||||||
|
|
||||||
|
### Python Environment
|
||||||
|
|
||||||
|
**Required**: Python 3.8+
|
||||||
|
|
||||||
|
**Recommended**: Use uv for all Python operations:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Never use system python directly
|
||||||
|
uv run scripts/fix_transcription.py # ✅ Correct
|
||||||
|
|
||||||
|
# Don't use system python
|
||||||
|
python scripts/fix_transcription.py # ❌ Wrong
|
||||||
|
```
|
||||||
|
|
||||||
|
### Directory Structure
|
||||||
|
|
||||||
|
After initialization, the directory structure is:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.transcript-fixer/
|
||||||
|
├── corrections.db # SQLite database
|
||||||
|
├── corrections.YYYYMMDD.bak # Automatic backups
|
||||||
|
└── (migration artifacts)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important**: The `.db` file should NOT be committed to Git. Export corrections to JSON for version control instead.
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
After setup:
|
||||||
|
1. Add initial corrections (5-10 terms)
|
||||||
|
2. Run first correction on a test file
|
||||||
|
3. Review learned suggestions after 3-5 runs
|
||||||
|
4. Build domain-specific dictionaries
|
||||||
|
|
||||||
|
See `workflow_guide.md` for detailed usage instructions.
|
||||||
125
transcript-fixer/references/quick_reference.md
Normal file
125
transcript-fixer/references/quick_reference.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
# Quick Reference
|
||||||
|
|
||||||
|
**Storage**: transcript-fixer uses SQLite database for corrections storage.
|
||||||
|
|
||||||
|
**Database location**: `~/.transcript-fixer/corrections.db`
|
||||||
|
|
||||||
|
## Quick Start Examples
|
||||||
|
|
||||||
|
### Adding Corrections via CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add a simple correction
|
||||||
|
uv run scripts/fix_transcription.py --add "巨升智能" "具身智能" --domain embodied_ai
|
||||||
|
|
||||||
|
# Add corrections for specific domain
|
||||||
|
uv run scripts/fix_transcription.py --add "奇迹创坛" "奇绩创坛" --domain general
|
||||||
|
uv run scripts/fix_transcription.py --add "矩阵公司" "初创公司" --domain general
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding Corrections via SQL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# Insert corrections
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source)
|
||||||
|
VALUES ('巨升智能', '具身智能', 'embodied_ai', 'manual');
|
||||||
|
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source)
|
||||||
|
VALUES ('巨升', '具身', 'embodied_ai', 'manual');
|
||||||
|
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source)
|
||||||
|
VALUES ('奇迹创坛', '奇绩创坛', 'general', 'manual');
|
||||||
|
|
||||||
|
# Exit
|
||||||
|
.quit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding Context Rules via SQL
|
||||||
|
|
||||||
|
Context rules use regex patterns for context-aware corrections:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# Add context-aware rules
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('巨升方向', '具身方向', '巨升→具身', 10);
|
||||||
|
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('巨升现在', '具身现在', '巨升→具身', 10);
|
||||||
|
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离的去看', '近距离地去看', '的→地 副词修饰', 5);
|
||||||
|
|
||||||
|
# Exit
|
||||||
|
.quit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding Corrections via Python API
|
||||||
|
|
||||||
|
Save as `add_corrections.py` and run with `uv run add_corrections.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env -S uv run
|
||||||
|
from pathlib import Path
|
||||||
|
from core import CorrectionRepository, CorrectionService
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
db_path = Path.home() / ".transcript-fixer" / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository)
|
||||||
|
|
||||||
|
# Add corrections
|
||||||
|
corrections = [
|
||||||
|
("巨升智能", "具身智能", "embodied_ai"),
|
||||||
|
("巨升", "具身", "embodied_ai"),
|
||||||
|
("奇迹创坛", "奇绩创坛", "general"),
|
||||||
|
("火星营", "火星营", "general"),
|
||||||
|
("矩阵公司", "初创公司", "general"),
|
||||||
|
("股价", "框架", "general"),
|
||||||
|
("三观", "三关", "general"),
|
||||||
|
]
|
||||||
|
|
||||||
|
for from_text, to_text, domain in corrections:
|
||||||
|
service.add_correction(from_text, to_text, domain)
|
||||||
|
print(f"✅ Added: '{from_text}' → '{to_text}' (domain: {domain})")
|
||||||
|
|
||||||
|
# Close connection
|
||||||
|
service.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Bulk Import Example
|
||||||
|
|
||||||
|
Use the provided bulk import script for importing multiple corrections:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/examples/bulk_import.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Querying the Database
|
||||||
|
|
||||||
|
### View Active Corrections
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT from_text, to_text, domain FROM active_corrections;"
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Statistics
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT * FROM correction_statistics;"
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Context Rules
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT pattern, replacement, priority FROM context_rules WHERE is_active = 1 ORDER BY priority DESC;"
|
||||||
|
```
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
- `references/file_formats.md` - Complete database schema documentation
|
||||||
|
- `references/script_parameters.md` - CLI command reference
|
||||||
|
- `SKILL.md` - Main user documentation
|
||||||
186
transcript-fixer/references/script_parameters.md
Normal file
186
transcript-fixer/references/script_parameters.md
Normal file
@@ -0,0 +1,186 @@
|
|||||||
|
# Script Parameters Reference
|
||||||
|
|
||||||
|
Detailed command-line parameters and usage examples for transcript-fixer Python scripts.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [fix_transcription.py](#fixtranscriptionpy) - Main correction pipeline
|
||||||
|
- [Setup Commands](#setup-commands)
|
||||||
|
- [Correction Management](#correction-management)
|
||||||
|
- [Correction Workflow](#correction-workflow)
|
||||||
|
- [Learning Commands](#learning-commands)
|
||||||
|
- [diff_generator.py](#diffgeneratorpy) - Generate comparison reports
|
||||||
|
- [Common Workflows](#common-workflows)
|
||||||
|
- [Exit Codes](#exit-codes)
|
||||||
|
- [Environment Variables](#environment-variables)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## fix_transcription.py
|
||||||
|
|
||||||
|
Main correction pipeline script supporting three processing stages.
|
||||||
|
|
||||||
|
### Syntax
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --input <file> --stage <1|2|3> [--output <dir>]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parameters
|
||||||
|
|
||||||
|
- `--input, -i` (required): Input Markdown file path
|
||||||
|
- `--stage, -s` (optional): Stage to execute (default: 3)
|
||||||
|
- `1` = Dictionary corrections only
|
||||||
|
- `2` = AI corrections only (requires Stage 1 output file)
|
||||||
|
- `3` = Both stages sequentially
|
||||||
|
- `--output, -o` (optional): Output directory (defaults to input file directory)
|
||||||
|
|
||||||
|
### Usage Examples
|
||||||
|
|
||||||
|
**Run dictionary corrections only:**
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --input meeting.md --stage 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Output: `meeting_阶段1_词典修复.md`
|
||||||
|
|
||||||
|
**Run AI corrections only:**
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --input meeting_阶段1_词典修复.md --stage 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Output: `meeting_阶段2_AI修复.md`
|
||||||
|
|
||||||
|
Note: Requires Stage 1 output file as input.
|
||||||
|
|
||||||
|
**Run complete pipeline:**
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --input meeting.md --stage 3
|
||||||
|
```
|
||||||
|
|
||||||
|
Outputs:
|
||||||
|
- `meeting_阶段1_词典修复.md`
|
||||||
|
- `meeting_阶段2_AI修复.md`
|
||||||
|
|
||||||
|
**Custom output directory:**
|
||||||
|
```bash
|
||||||
|
python scripts/fix_transcription.py --input meeting.md --stage 3 --output ./corrections
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exit Codes
|
||||||
|
|
||||||
|
- `0` - Success
|
||||||
|
- `1` - Missing required parameters or file not found
|
||||||
|
- `2` - GLM_API_KEY environment variable not set (Stage 2 or 3 only)
|
||||||
|
- `3` - API request failed
|
||||||
|
|
||||||
|
## generate_diff_report.py
|
||||||
|
|
||||||
|
Multi-format diff report generator for comparing correction stages.
|
||||||
|
|
||||||
|
### Syntax
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/generate_diff_report.py --original <file> --stage1 <file> --stage2 <file> [--output-dir <dir>]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parameters
|
||||||
|
|
||||||
|
- `--original` (required): Original transcript file path
|
||||||
|
- `--stage1` (required): Stage 1 correction output file path
|
||||||
|
- `--stage2` (required): Stage 2 correction output file path
|
||||||
|
- `--output-dir` (optional): Output directory for diff reports (defaults to original file directory)
|
||||||
|
|
||||||
|
### Usage Examples
|
||||||
|
|
||||||
|
**Basic usage:**
|
||||||
|
```bash
|
||||||
|
python scripts/generate_diff_report.py \
|
||||||
|
--original "meeting.md" \
|
||||||
|
--stage1 "meeting_阶段1_词典修复.md" \
|
||||||
|
--stage2 "meeting_阶段2_AI修复.md"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Custom output directory:**
|
||||||
|
```bash
|
||||||
|
python scripts/generate_diff_report.py \
|
||||||
|
--original "meeting.md" \
|
||||||
|
--stage1 "meeting_阶段1_词典修复.md" \
|
||||||
|
--stage2 "meeting_阶段2_AI修复.md" \
|
||||||
|
--output-dir "./reports"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Output Files
|
||||||
|
|
||||||
|
The script generates four comparison formats:
|
||||||
|
|
||||||
|
1. **Markdown summary** (`*_对比报告.md`)
|
||||||
|
- High-level statistics and change summary
|
||||||
|
- Word count changes per stage
|
||||||
|
- Common error patterns identified
|
||||||
|
|
||||||
|
2. **Unified diff** (`*_unified.diff`)
|
||||||
|
- Traditional Unix diff format
|
||||||
|
- Suitable for command-line review or version control
|
||||||
|
|
||||||
|
3. **HTML side-by-side** (`*_对比.html`)
|
||||||
|
- Visual side-by-side comparison
|
||||||
|
- Color-coded additions/deletions
|
||||||
|
- **Recommended for human review**
|
||||||
|
|
||||||
|
4. **Inline marked** (`*_行内对比.txt`)
|
||||||
|
- Single-column format with inline change markers
|
||||||
|
- Useful for quick text editor review
|
||||||
|
|
||||||
|
### Exit Codes
|
||||||
|
|
||||||
|
- `0` - Success
|
||||||
|
- `1` - Missing required parameters or file not found
|
||||||
|
- `2` - File format error (non-Markdown input)
|
||||||
|
|
||||||
|
## Common Workflows
|
||||||
|
|
||||||
|
### Testing Dictionary Changes
|
||||||
|
|
||||||
|
Test dictionary updates before running expensive AI corrections:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Update CORRECTIONS_DICT in scripts/fix_transcription.py
|
||||||
|
# 2. Run Stage 1 only
|
||||||
|
python scripts/fix_transcription.py --input meeting.md --stage 1
|
||||||
|
|
||||||
|
# 3. Review output
|
||||||
|
cat meeting_阶段1_词典修复.md
|
||||||
|
|
||||||
|
# 4. If satisfied, run Stage 2
|
||||||
|
python scripts/fix_transcription.py --input meeting_阶段1_词典修复.md --stage 2
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
|
||||||
|
Process multiple transcripts in sequence:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for file in transcripts/*.md; do
|
||||||
|
python scripts/fix_transcription.py --input "$file" --stage 3
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quick Review Cycle
|
||||||
|
|
||||||
|
Generate and open comparison report immediately after correction:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run corrections
|
||||||
|
python scripts/fix_transcription.py --input meeting.md --stage 3
|
||||||
|
|
||||||
|
# Generate and open diff report
|
||||||
|
python scripts/generate_diff_report.py \
|
||||||
|
--original "meeting.md" \
|
||||||
|
--stage1 "meeting_阶段1_词典修复.md" \
|
||||||
|
--stage2 "meeting_阶段2_AI修复.md"
|
||||||
|
|
||||||
|
open meeting_对比.html # macOS
|
||||||
|
# xdg-open meeting_对比.html # Linux
|
||||||
|
# start meeting_对比.html # Windows
|
||||||
|
```
|
||||||
188
transcript-fixer/references/sql_queries.md
Normal file
188
transcript-fixer/references/sql_queries.md
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
# SQL Query Reference
|
||||||
|
|
||||||
|
Database location: `~/.transcript-fixer/corrections.db`
|
||||||
|
|
||||||
|
## Basic Operations
|
||||||
|
|
||||||
|
### Add Corrections
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Add a correction
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source)
|
||||||
|
VALUES ('巨升智能', '具身智能', 'embodied_ai', 'manual');
|
||||||
|
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source)
|
||||||
|
VALUES ('奇迹创坛', '奇绩创坛', 'general', 'manual');
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Corrections
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View all active corrections
|
||||||
|
SELECT from_text, to_text, domain, source, usage_count
|
||||||
|
FROM active_corrections
|
||||||
|
ORDER BY domain, from_text;
|
||||||
|
|
||||||
|
-- View corrections for specific domain
|
||||||
|
SELECT from_text, to_text, usage_count, added_at
|
||||||
|
FROM active_corrections
|
||||||
|
WHERE domain = 'embodied_ai';
|
||||||
|
```
|
||||||
|
|
||||||
|
## Context Rules
|
||||||
|
|
||||||
|
### Add Context-Aware Rules
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Add regex-based context rule
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('巨升方向', '具身方向', '巨升→具身', 10);
|
||||||
|
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离的去看', '近距离地去看', '的→地 副词修饰', 5);
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Rules
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View all active context rules (ordered by priority)
|
||||||
|
SELECT pattern, replacement, description, priority
|
||||||
|
FROM context_rules
|
||||||
|
WHERE is_active = 1
|
||||||
|
ORDER BY priority DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Statistics
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View correction statistics by domain
|
||||||
|
SELECT * FROM correction_statistics;
|
||||||
|
|
||||||
|
-- Count corrections by source
|
||||||
|
SELECT source, COUNT(*) as count, SUM(usage_count) as total_usage
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
GROUP BY source;
|
||||||
|
|
||||||
|
-- Most frequently used corrections
|
||||||
|
SELECT from_text, to_text, domain, usage_count, last_used
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1 AND usage_count > 0
|
||||||
|
ORDER BY usage_count DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Learning and Suggestions
|
||||||
|
|
||||||
|
### View Suggestions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View pending suggestions
|
||||||
|
SELECT * FROM pending_suggestions;
|
||||||
|
|
||||||
|
-- View high-confidence suggestions
|
||||||
|
SELECT from_text, to_text, domain, frequency, confidence
|
||||||
|
FROM learned_suggestions
|
||||||
|
WHERE status = 'pending' AND confidence >= 0.8
|
||||||
|
ORDER BY confidence DESC, frequency DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Approve Suggestions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Insert into corrections
|
||||||
|
INSERT INTO corrections (from_text, to_text, domain, source, confidence)
|
||||||
|
SELECT from_text, to_text, domain, 'learned', confidence
|
||||||
|
FROM learned_suggestions
|
||||||
|
WHERE id = 1;
|
||||||
|
|
||||||
|
-- Mark as approved
|
||||||
|
UPDATE learned_suggestions
|
||||||
|
SET status = 'approved', reviewed_at = CURRENT_TIMESTAMP
|
||||||
|
WHERE id = 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
## History and Audit
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View recent correction runs
|
||||||
|
SELECT filename, domain, stage1_changes, stage2_changes, run_timestamp
|
||||||
|
FROM correction_history
|
||||||
|
ORDER BY run_timestamp DESC
|
||||||
|
LIMIT 10;
|
||||||
|
|
||||||
|
-- View detailed changes for a specific run
|
||||||
|
SELECT ch.line_number, ch.from_text, ch.to_text, ch.rule_type
|
||||||
|
FROM correction_changes ch
|
||||||
|
JOIN correction_history h ON ch.history_id = h.id
|
||||||
|
WHERE h.filename = 'meeting.md'
|
||||||
|
ORDER BY ch.line_number;
|
||||||
|
|
||||||
|
-- Calculate success rate
|
||||||
|
SELECT
|
||||||
|
COUNT(*) as total_runs,
|
||||||
|
SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END) as successful,
|
||||||
|
ROUND(100.0 * SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END) / COUNT(*), 2) as success_rate
|
||||||
|
FROM correction_history;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Maintenance
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Deactivate (soft delete) a correction
|
||||||
|
UPDATE corrections
|
||||||
|
SET is_active = 0
|
||||||
|
WHERE from_text = '错误词' AND domain = 'general';
|
||||||
|
|
||||||
|
-- Reactivate a correction
|
||||||
|
UPDATE corrections
|
||||||
|
SET is_active = 1
|
||||||
|
WHERE from_text = '错误词' AND domain = 'general';
|
||||||
|
|
||||||
|
-- Update correction confidence
|
||||||
|
UPDATE corrections
|
||||||
|
SET confidence = 0.95
|
||||||
|
WHERE from_text = '巨升' AND to_text = '具身';
|
||||||
|
|
||||||
|
-- Delete old history (older than 90 days)
|
||||||
|
DELETE FROM correction_history
|
||||||
|
WHERE run_timestamp < datetime('now', '-90 days');
|
||||||
|
|
||||||
|
-- Reclaim space
|
||||||
|
VACUUM;
|
||||||
|
```
|
||||||
|
|
||||||
|
## System Configuration
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View system configuration
|
||||||
|
SELECT key, value, description FROM system_config;
|
||||||
|
|
||||||
|
-- Update configuration
|
||||||
|
UPDATE system_config
|
||||||
|
SET value = '5'
|
||||||
|
WHERE key = 'learning_frequency_threshold';
|
||||||
|
|
||||||
|
-- Check schema version
|
||||||
|
SELECT value FROM system_config WHERE key = 'schema_version';
|
||||||
|
```
|
||||||
|
|
||||||
|
## Export
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Export corrections as CSV
|
||||||
|
.mode csv
|
||||||
|
.headers on
|
||||||
|
.output corrections_export.csv
|
||||||
|
SELECT from_text, to_text, domain, source, confidence, usage_count, added_at
|
||||||
|
FROM active_corrections;
|
||||||
|
.output stdout
|
||||||
|
```
|
||||||
|
|
||||||
|
For JSON export, use Python script with `service.export_corrections()` instead.
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
- `references/file_formats.md` - Complete database schema documentation
|
||||||
|
- `references/quick_reference.md` - CLI command quick reference
|
||||||
|
- `SKILL.md` - Main user documentation
|
||||||
371
transcript-fixer/references/team_collaboration.md
Normal file
371
transcript-fixer/references/team_collaboration.md
Normal file
@@ -0,0 +1,371 @@
|
|||||||
|
# Team Collaboration Guide
|
||||||
|
|
||||||
|
This guide explains how to share correction knowledge across teams using export/import and Git workflows.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Export/Import Workflow](#exportimport-workflow)
|
||||||
|
- [Export Corrections](#export-corrections)
|
||||||
|
- [Import from Teammate](#import-from-teammate)
|
||||||
|
- [Team Workflow Example](#team-workflow-example)
|
||||||
|
- [Git-Based Collaboration](#git-based-collaboration)
|
||||||
|
- [Initial Setup](#initial-setup)
|
||||||
|
- [Team Members Clone](#team-members-clone)
|
||||||
|
- [Ongoing Sync](#ongoing-sync)
|
||||||
|
- [Handling Conflicts](#handling-conflicts)
|
||||||
|
- [Selective Domain Sharing](#selective-domain-sharing)
|
||||||
|
- [Finance Team](#finance-team)
|
||||||
|
- [AI Team](#ai-team)
|
||||||
|
- [Individual imports specific domains](#individual-imports-specific-domains)
|
||||||
|
- [Git Branching Strategy](#git-branching-strategy)
|
||||||
|
- [Feature Branches](#feature-branches)
|
||||||
|
- [Domain Branches (Alternative)](#domain-branches-alternative)
|
||||||
|
- [Automated Sync (Advanced)](#automated-sync-advanced)
|
||||||
|
- [macOS/Linux Cron](#macoslinux-cron)
|
||||||
|
- [Windows Task Scheduler](#windows-task-scheduler)
|
||||||
|
- [Backup and Recovery](#backup-and-recovery)
|
||||||
|
- [Backup Strategy](#backup-strategy)
|
||||||
|
- [Recovery from Backup](#recovery-from-backup)
|
||||||
|
- [Recovery from Git](#recovery-from-git)
|
||||||
|
- [Team Best Practices](#team-best-practices)
|
||||||
|
- [Integration with CI/CD](#integration-with-cicd)
|
||||||
|
- [GitHub Actions Example](#github-actions-example)
|
||||||
|
- [Troubleshooting](#troubleshooting)
|
||||||
|
- [Import Failed](#import-failed)
|
||||||
|
- [Git Sync Failed](#git-sync-failed)
|
||||||
|
- [Merge Conflicts Too Complex](#merge-conflicts-too-complex)
|
||||||
|
- [Security Considerations](#security-considerations)
|
||||||
|
- [Further Reading](#further-reading)
|
||||||
|
|
||||||
|
## Export/Import Workflow
|
||||||
|
|
||||||
|
### Export Corrections
|
||||||
|
|
||||||
|
Share your corrections with team members:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export specific domain
|
||||||
|
python scripts/fix_transcription.py --export team_corrections.json --domain embodied_ai
|
||||||
|
|
||||||
|
# Export general corrections
|
||||||
|
python scripts/fix_transcription.py --export team_corrections.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output**: Creates a standalone JSON file with your corrections.
|
||||||
|
|
||||||
|
### Import from Teammate
|
||||||
|
|
||||||
|
Two modes: **merge** (combine) or **replace** (overwrite):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Merge (recommended) - combines with existing corrections
|
||||||
|
python scripts/fix_transcription.py --import team_corrections.json --merge
|
||||||
|
|
||||||
|
# Replace - overwrites existing corrections (dangerous!)
|
||||||
|
python scripts/fix_transcription.py --import team_corrections.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Merge behavior**:
|
||||||
|
- Adds new corrections
|
||||||
|
- Updates existing corrections with imported values
|
||||||
|
- Preserves corrections not in import file
|
||||||
|
|
||||||
|
### Team Workflow Example
|
||||||
|
|
||||||
|
**Person A (Domain Expert)**:
|
||||||
|
```bash
|
||||||
|
# Build correction dictionary
|
||||||
|
python fix_transcription.py --add "巨升" "具身" --domain embodied_ai
|
||||||
|
python fix_transcription.py --add "奇迹创坛" "奇绩创坛" --domain embodied_ai
|
||||||
|
# ... add 50 more corrections ...
|
||||||
|
|
||||||
|
# Export for team
|
||||||
|
python fix_transcription.py --export ai_corrections.json --domain embodied_ai
|
||||||
|
# Send ai_corrections.json to team via Slack/email
|
||||||
|
```
|
||||||
|
|
||||||
|
**Person B (Team Member)**:
|
||||||
|
```bash
|
||||||
|
# Receive ai_corrections.json
|
||||||
|
# Import and merge with existing corrections
|
||||||
|
python fix_transcription.py --import ai_corrections.json --merge
|
||||||
|
|
||||||
|
# Now Person B has all 50+ corrections!
|
||||||
|
```
|
||||||
|
|
||||||
|
## Git-Based Collaboration
|
||||||
|
|
||||||
|
For teams using Git, version control the entire correction database.
|
||||||
|
|
||||||
|
### Initial Setup
|
||||||
|
|
||||||
|
**Person A (First User)**:
|
||||||
|
```bash
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
git init
|
||||||
|
git add corrections.json context_rules.json config.json
|
||||||
|
git add domains/
|
||||||
|
git commit -m "Initial correction database"
|
||||||
|
|
||||||
|
# Push to shared repo
|
||||||
|
git remote add origin git@github.com:org/transcript-corrections.git
|
||||||
|
git push -u origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Team Members Clone
|
||||||
|
|
||||||
|
**Person B, C, D (Team Members)**:
|
||||||
|
```bash
|
||||||
|
# Clone shared corrections
|
||||||
|
git clone git@github.com:org/transcript-corrections.git ~/.transcript-fixer
|
||||||
|
|
||||||
|
# Now everyone has the same corrections!
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ongoing Sync
|
||||||
|
|
||||||
|
**Daily workflow**:
|
||||||
|
```bash
|
||||||
|
# Morning: Pull team updates
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
git pull origin main
|
||||||
|
|
||||||
|
# During day: Add corrections
|
||||||
|
python fix_transcription.py --add "错误" "正确"
|
||||||
|
|
||||||
|
# Evening: Push your additions
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
git add corrections.json
|
||||||
|
git commit -m "Added 5 new embodied AI corrections"
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Handling Conflicts
|
||||||
|
|
||||||
|
When two people add different corrections to same file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
git pull origin main
|
||||||
|
|
||||||
|
# If conflict occurs:
|
||||||
|
# CONFLICT in corrections.json
|
||||||
|
|
||||||
|
# Option 1: Manual merge (recommended)
|
||||||
|
nano corrections.json # Edit to combine both changes
|
||||||
|
git add corrections.json
|
||||||
|
git commit -m "Merged corrections from teammate"
|
||||||
|
git push
|
||||||
|
|
||||||
|
# Option 2: Keep yours
|
||||||
|
git checkout --ours corrections.json
|
||||||
|
git add corrections.json
|
||||||
|
git commit -m "Kept local corrections"
|
||||||
|
git push
|
||||||
|
|
||||||
|
# Option 3: Keep theirs
|
||||||
|
git checkout --theirs corrections.json
|
||||||
|
git add corrections.json
|
||||||
|
git commit -m "Used teammate's corrections"
|
||||||
|
git push
|
||||||
|
```
|
||||||
|
|
||||||
|
**Best Practice**: JSON merge conflicts are usually easy - just combine the correction entries from both versions.
|
||||||
|
|
||||||
|
## Selective Domain Sharing
|
||||||
|
|
||||||
|
Share only specific domains with different teams:
|
||||||
|
|
||||||
|
### Finance Team
|
||||||
|
```bash
|
||||||
|
# Finance team exports their domain
|
||||||
|
python fix_transcription.py --export finance_corrections.json --domain finance
|
||||||
|
|
||||||
|
# Share finance_corrections.json with finance team only
|
||||||
|
```
|
||||||
|
|
||||||
|
### AI Team
|
||||||
|
```bash
|
||||||
|
# AI team exports their domain
|
||||||
|
python fix_transcription.py --export ai_corrections.json --domain embodied_ai
|
||||||
|
|
||||||
|
# Share ai_corrections.json with AI team only
|
||||||
|
```
|
||||||
|
|
||||||
|
### Individual imports specific domains
|
||||||
|
```bash
|
||||||
|
# Alice works on both finance and AI
|
||||||
|
python fix_transcription.py --import finance_corrections.json --merge
|
||||||
|
python fix_transcription.py --import ai_corrections.json --merge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Git Branching Strategy
|
||||||
|
|
||||||
|
For larger teams, use branches for different domains or workflows:
|
||||||
|
|
||||||
|
### Feature Branches
|
||||||
|
```bash
|
||||||
|
# Create branch for major dictionary additions
|
||||||
|
git checkout -b add-medical-terms
|
||||||
|
python fix_transcription.py --add "医疗术语" "正确术语" --domain medical
|
||||||
|
# ... add 100 medical corrections ...
|
||||||
|
git add domains/medical.json
|
||||||
|
git commit -m "Added 100 medical terminology corrections"
|
||||||
|
git push origin add-medical-terms
|
||||||
|
|
||||||
|
# Create PR for review
|
||||||
|
# After approval, merge to main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Domain Branches (Alternative)
|
||||||
|
```bash
|
||||||
|
# Separate branches per domain
|
||||||
|
git checkout -b domain/embodied-ai
|
||||||
|
# Work on AI corrections
|
||||||
|
git push origin domain/embodied-ai
|
||||||
|
|
||||||
|
git checkout -b domain/finance
|
||||||
|
# Work on finance corrections
|
||||||
|
git push origin domain/finance
|
||||||
|
```
|
||||||
|
|
||||||
|
## Automated Sync (Advanced)
|
||||||
|
|
||||||
|
Set up automatic Git sync using cron/Task Scheduler:
|
||||||
|
|
||||||
|
### macOS/Linux Cron
|
||||||
|
```bash
|
||||||
|
# Edit crontab
|
||||||
|
crontab -e
|
||||||
|
|
||||||
|
# Add daily sync at 9 AM and 6 PM
|
||||||
|
0 9,18 * * * cd ~/.transcript-fixer && git pull origin main && git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Windows Task Scheduler
|
||||||
|
```powershell
|
||||||
|
# Create scheduled task
|
||||||
|
$action = New-ScheduledTaskAction -Execute "git" -Argument "pull origin main" -WorkingDirectory "$env:USERPROFILE\.transcript-fixer"
|
||||||
|
$trigger = New-ScheduledTaskTrigger -Daily -At 9am
|
||||||
|
Register-ScheduledTask -Action $action -Trigger $trigger -TaskName "SyncTranscriptCorrections"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backup and Recovery
|
||||||
|
|
||||||
|
### Backup Strategy
|
||||||
|
```bash
|
||||||
|
# Weekly backup to cloud
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
tar -czf transcript-corrections-$(date +%Y%m%d).tar.gz corrections.json context_rules.json domains/
|
||||||
|
# Upload to Dropbox/Google Drive/S3
|
||||||
|
```
|
||||||
|
|
||||||
|
### Recovery from Backup
|
||||||
|
```bash
|
||||||
|
# Extract backup
|
||||||
|
tar -xzf transcript-corrections-20250127.tar.gz -C ~/.transcript-fixer/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Recovery from Git
|
||||||
|
```bash
|
||||||
|
# View history
|
||||||
|
cd ~/.transcript-fixer
|
||||||
|
git log corrections.json
|
||||||
|
|
||||||
|
# Restore from 3 commits ago
|
||||||
|
git checkout HEAD~3 corrections.json
|
||||||
|
|
||||||
|
# Or restore specific version
|
||||||
|
git checkout abc123def corrections.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Team Best Practices
|
||||||
|
|
||||||
|
1. **Pull Before Push**: Always `git pull` before starting work
|
||||||
|
2. **Commit Often**: Small, frequent commits better than large infrequent ones
|
||||||
|
3. **Descriptive Messages**: "Added 5 finance terms" better than "updates"
|
||||||
|
4. **Review Process**: Use PRs for major dictionary changes (100+ corrections)
|
||||||
|
5. **Domain Ownership**: Assign domain experts as reviewers
|
||||||
|
6. **Weekly Sync**: Schedule team sync meetings to review learned suggestions
|
||||||
|
7. **Backup Policy**: Weekly backups of entire `~/.transcript-fixer/`
|
||||||
|
|
||||||
|
## Integration with CI/CD
|
||||||
|
|
||||||
|
For enterprise teams, integrate validation into CI:
|
||||||
|
|
||||||
|
### GitHub Actions Example
|
||||||
|
```yaml
|
||||||
|
# .github/workflows/validate-corrections.yml
|
||||||
|
name: Validate Corrections
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'corrections.json'
|
||||||
|
- 'domains/*.json'
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
validate:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v2
|
||||||
|
|
||||||
|
- name: Validate JSON
|
||||||
|
run: |
|
||||||
|
python -m json.tool corrections.json > /dev/null
|
||||||
|
for file in domains/*.json; do
|
||||||
|
python -m json.tool "$file" > /dev/null
|
||||||
|
done
|
||||||
|
|
||||||
|
- name: Check for duplicates
|
||||||
|
run: |
|
||||||
|
python scripts/check_duplicates.py corrections.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Import Failed
|
||||||
|
```bash
|
||||||
|
# Check JSON validity
|
||||||
|
python -m json.tool team_corrections.json
|
||||||
|
|
||||||
|
# If invalid, fix JSON syntax errors
|
||||||
|
nano team_corrections.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Git Sync Failed
|
||||||
|
```bash
|
||||||
|
# Check remote connection
|
||||||
|
git remote -v
|
||||||
|
|
||||||
|
# Re-add if needed
|
||||||
|
git remote set-url origin git@github.com:org/corrections.git
|
||||||
|
|
||||||
|
# Verify SSH keys
|
||||||
|
ssh -T git@github.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### Merge Conflicts Too Complex
|
||||||
|
```bash
|
||||||
|
# Nuclear option: Keep one version
|
||||||
|
git checkout --ours corrections.json # Keep yours
|
||||||
|
# OR
|
||||||
|
git checkout --theirs corrections.json # Keep theirs
|
||||||
|
|
||||||
|
# Then re-import the other version
|
||||||
|
python fix_transcription.py --import other_version.json --merge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
1. **Private Repos**: Use private Git repositories for company-specific corrections
|
||||||
|
2. **Access Control**: Limit who can push to main branch
|
||||||
|
3. **Secret Scanning**: Never commit API keys (already handled by security_scan.py)
|
||||||
|
4. **Audit Trail**: Git history provides full audit trail of who changed what
|
||||||
|
5. **Backup Encryption**: Encrypt backups if containing sensitive terminology
|
||||||
|
|
||||||
|
## Further Reading
|
||||||
|
|
||||||
|
- Git workflows: https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows
|
||||||
|
- JSON validation: https://jsonlint.com/
|
||||||
|
- Team Git practices: https://github.com/git-guides
|
||||||
313
transcript-fixer/references/troubleshooting.md
Normal file
313
transcript-fixer/references/troubleshooting.md
Normal file
@@ -0,0 +1,313 @@
|
|||||||
|
# Troubleshooting Guide
|
||||||
|
|
||||||
|
Solutions to common issues and error conditions.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [API Authentication Errors](#api-authentication-errors)
|
||||||
|
- [GLM_API_KEY Not Set](#glm_api_key-not-set)
|
||||||
|
- [Invalid API Key](#invalid-api-key)
|
||||||
|
- [Learning System Issues](#learning-system-issues)
|
||||||
|
- [No Suggestions Generated](#no-suggestions-generated)
|
||||||
|
- [Database Issues](#database-issues)
|
||||||
|
- [Database Not Found](#database-not-found)
|
||||||
|
- [Database Locked](#database-locked)
|
||||||
|
- [Corrupted Database](#corrupted-database)
|
||||||
|
- [Missing Tables](#missing-tables)
|
||||||
|
- [Common Pitfalls](#common-pitfalls)
|
||||||
|
- [1. Stage Order Confusion](#1-stage-order-confusion)
|
||||||
|
- [2. Overwriting Imports](#2-overwriting-imports)
|
||||||
|
- [3. Ignoring Learned Suggestions](#3-ignoring-learned-suggestions)
|
||||||
|
- [4. Testing on Large Files](#4-testing-on-large-files)
|
||||||
|
- [5. Manual Database Edits Without Validation](#5-manual-database-edits-without-validation)
|
||||||
|
- [6. Committing .db Files to Git](#6-committing-db-files-to-git)
|
||||||
|
- [Validation Commands](#validation-commands)
|
||||||
|
- [Quick Health Check](#quick-health-check)
|
||||||
|
- [Detailed Diagnostics](#detailed-diagnostics)
|
||||||
|
- [Getting Help](#getting-help)
|
||||||
|
|
||||||
|
## API Authentication Errors
|
||||||
|
|
||||||
|
### GLM_API_KEY Not Set
|
||||||
|
|
||||||
|
**Symptom**:
|
||||||
|
```
|
||||||
|
❌ Error: GLM_API_KEY environment variable not set
|
||||||
|
Set it with: export GLM_API_KEY='your-key'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
```bash
|
||||||
|
# Check if key is set
|
||||||
|
echo $GLM_API_KEY
|
||||||
|
|
||||||
|
# If empty, export key
|
||||||
|
export GLM_API_KEY="your-api-key-here"
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Persistence**: Add to shell profile (`.bashrc` or `.zshrc`) for permanent access.
|
||||||
|
|
||||||
|
See `glm_api_setup.md` for detailed API key management.
|
||||||
|
|
||||||
|
### Invalid API Key
|
||||||
|
|
||||||
|
**Symptom**: API calls fail with 401/403 errors
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Verify key is correct (copy from https://open.bigmodel.cn/)
|
||||||
|
2. Check for extra spaces or quotes in the key
|
||||||
|
3. Regenerate key if compromised
|
||||||
|
4. Verify API quota hasn't been exceeded
|
||||||
|
|
||||||
|
## Learning System Issues
|
||||||
|
|
||||||
|
### No Suggestions Generated
|
||||||
|
|
||||||
|
**Symptom**: Running `--review-learned` shows no suggestions after multiple corrections.
|
||||||
|
|
||||||
|
**Requirements**:
|
||||||
|
- Minimum 3 correction runs with consistent patterns
|
||||||
|
- Learning frequency threshold ≥3 (default)
|
||||||
|
- Learning confidence threshold ≥0.8 (default)
|
||||||
|
|
||||||
|
**Diagnostic steps**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check correction history count
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT COUNT(*) FROM correction_history;"
|
||||||
|
|
||||||
|
# If 0, no corrections have been run yet
|
||||||
|
# If >0 but <3, run more corrections
|
||||||
|
|
||||||
|
# Check suggestions table
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT * FROM learned_suggestions;"
|
||||||
|
|
||||||
|
# Check system configuration
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT key, value FROM system_config WHERE key LIKE 'learning%';"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Run at least 3 correction sessions
|
||||||
|
2. Ensure patterns repeat (same error → same correction)
|
||||||
|
3. Verify database permissions (should be readable/writable)
|
||||||
|
4. Check `correction_history` table has entries
|
||||||
|
|
||||||
|
## Database Issues
|
||||||
|
|
||||||
|
### Database Not Found
|
||||||
|
|
||||||
|
**Symptom**:
|
||||||
|
```
|
||||||
|
⚠️ Database not found: ~/.transcript-fixer/corrections.db
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates the database with the complete schema.
|
||||||
|
|
||||||
|
### Database Locked
|
||||||
|
|
||||||
|
**Symptom**:
|
||||||
|
```
|
||||||
|
Error: database is locked
|
||||||
|
```
|
||||||
|
|
||||||
|
**Causes**:
|
||||||
|
- Another process is accessing the database
|
||||||
|
- Unfinished transaction from crashed process
|
||||||
|
- File permissions issue
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check for processes using the database
|
||||||
|
lsof ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
# If processes found, kill them or wait for completion
|
||||||
|
|
||||||
|
# If database is corrupted, backup and recreate
|
||||||
|
cp ~/.transcript-fixer/corrections.db ~/.transcript-fixer/corrections_backup.db
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "VACUUM;"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Corrupted Database
|
||||||
|
|
||||||
|
**Symptom**: SQLite errors, integrity check failures
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check integrity
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "PRAGMA integrity_check;"
|
||||||
|
|
||||||
|
# If corrupted, attempt recovery
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db ".recover" | sqlite3 ~/.transcript-fixer/corrections_new.db
|
||||||
|
|
||||||
|
# Replace database with recovered version
|
||||||
|
mv ~/.transcript-fixer/corrections.db ~/.transcript-fixer/corrections_corrupted.db
|
||||||
|
mv ~/.transcript-fixer/corrections_new.db ~/.transcript-fixer/corrections.db
|
||||||
|
```
|
||||||
|
|
||||||
|
### Missing Tables
|
||||||
|
|
||||||
|
**Symptom**:
|
||||||
|
```
|
||||||
|
❌ Database missing tables: ['corrections', ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution**: Reinitialize schema (safe, uses IF NOT EXISTS):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -c "from core import CorrectionRepository; from pathlib import Path; CorrectionRepository(Path.home() / '.transcript-fixer' / 'corrections.db')"
|
||||||
|
```
|
||||||
|
|
||||||
|
Or delete database and reinitialize:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup first
|
||||||
|
cp ~/.transcript-fixer/corrections.db ~/corrections_backup_$(date +%Y%m%d).db
|
||||||
|
|
||||||
|
# Reinitialize
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Pitfalls
|
||||||
|
|
||||||
|
### 1. Stage Order Confusion
|
||||||
|
|
||||||
|
**Problem**: Running Stage 2 without Stage 1 output.
|
||||||
|
|
||||||
|
**Solution**: Use `--stage 3` for full pipeline, or run stages sequentially:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Wrong: Stage 2 on raw file
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 2 # ❌
|
||||||
|
|
||||||
|
# Correct: Full pipeline
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 3 # ✅
|
||||||
|
|
||||||
|
# Or sequential stages
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 1
|
||||||
|
uv run scripts/fix_transcription.py --input file_stage1.md --stage 2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Overwriting Imports
|
||||||
|
|
||||||
|
**Problem**: Using `--import` without `--merge` overwrites existing corrections.
|
||||||
|
|
||||||
|
**Solution**: Always use `--merge` flag:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Wrong: Overwrites existing
|
||||||
|
uv run scripts/fix_transcription.py --import team.json # ❌
|
||||||
|
|
||||||
|
# Correct: Merges with existing
|
||||||
|
uv run scripts/fix_transcription.py --import team.json --merge # ✅
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Ignoring Learned Suggestions
|
||||||
|
|
||||||
|
**Problem**: Not reviewing learned patterns, missing free optimizations.
|
||||||
|
|
||||||
|
**Impact**: Patterns detected by AI remain expensive (Stage 2) instead of cheap (Stage 1).
|
||||||
|
|
||||||
|
**Solution**: Review suggestions every 3-5 runs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
uv run scripts/fix_transcription.py --approve "错误" "正确"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Testing on Large Files
|
||||||
|
|
||||||
|
**Problem**: Testing dictionary changes on large files wastes API quota.
|
||||||
|
|
||||||
|
**Solution**: Start with `--stage 1` on small files (100-500 lines):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test dictionary changes first
|
||||||
|
uv run scripts/fix_transcription.py --input small_sample.md --stage 1
|
||||||
|
|
||||||
|
# Review output, adjust corrections
|
||||||
|
# Then run full pipeline
|
||||||
|
uv run scripts/fix_transcription.py --input large_file.md --stage 3
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Manual Database Edits Without Validation
|
||||||
|
|
||||||
|
**Problem**: Direct SQL edits might violate schema constraints.
|
||||||
|
|
||||||
|
**Solution**: Always validate after manual changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
# ... make changes ...
|
||||||
|
.quit
|
||||||
|
|
||||||
|
# Validate
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Committing .db Files to Git
|
||||||
|
|
||||||
|
**Problem**: Binary database files in Git cause merge conflicts and bloat repository.
|
||||||
|
|
||||||
|
**Solution**: Use JSON exports for version control:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# .gitignore
|
||||||
|
*.db
|
||||||
|
*.db-journal
|
||||||
|
*.bak
|
||||||
|
|
||||||
|
# Export for version control instead
|
||||||
|
uv run scripts/fix_transcription.py --export corrections_$(date +%Y%m%d).json
|
||||||
|
git add corrections_*.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation Commands
|
||||||
|
|
||||||
|
### Quick Health Check
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
### Detailed Diagnostics
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check database integrity
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "PRAGMA integrity_check;"
|
||||||
|
|
||||||
|
# Check table counts
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
SELECT 'corrections' as table_name, COUNT(*) as count FROM corrections
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'context_rules', COUNT(*) FROM context_rules
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'learned_suggestions', COUNT(*) FROM learned_suggestions
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'correction_history', COUNT(*) FROM correction_history;
|
||||||
|
"
|
||||||
|
|
||||||
|
# Check configuration
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT * FROM system_config;"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
If issues persist:
|
||||||
|
|
||||||
|
1. Run `--validate` to collect diagnostic information
|
||||||
|
2. Check `correction_history` and `audit_log` tables for errors
|
||||||
|
3. Review `references/file_formats.md` for schema details
|
||||||
|
4. Check `references/architecture.md` for component details
|
||||||
|
5. Verify Python and uv versions are up to date
|
||||||
|
|
||||||
|
For database corruption, automatic backups are created before migrations. Check for `.bak` files in `~/.transcript-fixer/`.
|
||||||
483
transcript-fixer/references/workflow_guide.md
Normal file
483
transcript-fixer/references/workflow_guide.md
Normal file
@@ -0,0 +1,483 @@
|
|||||||
|
# Workflow Guide
|
||||||
|
|
||||||
|
Detailed step-by-step workflows for transcript correction and management.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Pre-Flight Checklist](#pre-flight-checklist)
|
||||||
|
- [Initial Setup](#initial-setup)
|
||||||
|
- [File Preparation](#file-preparation)
|
||||||
|
- [Execution Parameters](#execution-parameters)
|
||||||
|
- [Environment](#environment)
|
||||||
|
- [Core Workflows](#core-workflows)
|
||||||
|
- [1. First-Time Correction](#1-first-time-correction)
|
||||||
|
- [2. Iterative Improvement](#2-iterative-improvement)
|
||||||
|
- [3. Domain-Specific Corrections](#3-domain-specific-corrections)
|
||||||
|
- [4. Team Collaboration](#4-team-collaboration)
|
||||||
|
- [5. Stage-by-Stage Execution](#5-stage-by-stage-execution)
|
||||||
|
- [6. Context-Aware Rules](#6-context-aware-rules)
|
||||||
|
- [7. Diff Report Generation](#7-diff-report-generation)
|
||||||
|
- [Batch Processing](#batch-processing)
|
||||||
|
- [Process Multiple Files](#process-multiple-files)
|
||||||
|
- [Parallel Processing](#parallel-processing)
|
||||||
|
- [Maintenance Workflows](#maintenance-workflows)
|
||||||
|
- [Weekly: Review Learning](#weekly-review-learning)
|
||||||
|
- [Monthly: Export and Backup](#monthly-export-and-backup)
|
||||||
|
- [Quarterly: Clean Up](#quarterly-clean-up)
|
||||||
|
- [Next Steps](#next-steps)
|
||||||
|
|
||||||
|
## Pre-Flight Checklist
|
||||||
|
|
||||||
|
Before running corrections, verify these prerequisites:
|
||||||
|
|
||||||
|
### Initial Setup
|
||||||
|
- [ ] Initialized with `uv run scripts/fix_transcription.py --init`
|
||||||
|
- [ ] Database exists at `~/.transcript-fixer/corrections.db`
|
||||||
|
- [ ] `GLM_API_KEY` environment variable set (run `echo $GLM_API_KEY`)
|
||||||
|
- [ ] Configuration validated (run `--validate`)
|
||||||
|
|
||||||
|
### File Preparation
|
||||||
|
- [ ] Input file exists and is readable
|
||||||
|
- [ ] File uses supported format (`.md`, `.txt`)
|
||||||
|
- [ ] File encoding is UTF-8
|
||||||
|
- [ ] File size is reasonable (<10MB for first runs)
|
||||||
|
|
||||||
|
### Execution Parameters
|
||||||
|
- [ ] Using `--stage 3` for full pipeline (or specific stage if testing)
|
||||||
|
- [ ] Domain specified with `--domain` if using specialized dictionaries
|
||||||
|
- [ ] Using `--merge` flag when importing team corrections
|
||||||
|
|
||||||
|
### Environment
|
||||||
|
- [ ] Sufficient disk space for output files (~2x input size)
|
||||||
|
- [ ] API quota available for Stage 2 corrections
|
||||||
|
- [ ] Network connectivity for API calls
|
||||||
|
|
||||||
|
**Quick validation**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate && echo $GLM_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Workflows
|
||||||
|
|
||||||
|
### 1. First-Time Correction
|
||||||
|
|
||||||
|
**Goal**: Correct a transcript for the first time.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
1. **Initialize** (if not done):
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --init
|
||||||
|
export GLM_API_KEY="your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Add initial corrections** (5-10 common errors):
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --add "常见错误1" "正确词1" --domain general
|
||||||
|
uv run scripts/fix_transcription.py --add "常见错误2" "正确词2" --domain general
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Test on small sample** (Stage 1 only):
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input sample.md --stage 1
|
||||||
|
less sample_stage1.md # Review output
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Run full pipeline**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input transcript.md --stage 3 --domain general
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Review outputs**:
|
||||||
|
```bash
|
||||||
|
# Stage 1: Dictionary corrections
|
||||||
|
less transcript_stage1.md
|
||||||
|
|
||||||
|
# Stage 2: Final corrected version
|
||||||
|
less transcript_stage2.md
|
||||||
|
|
||||||
|
# Generate diff report
|
||||||
|
uv run scripts/diff_generator.py transcript.md transcript_stage1.md transcript_stage2.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected duration**:
|
||||||
|
- Stage 1: Instant (dictionary lookup)
|
||||||
|
- Stage 2: ~1-2 minutes per 1000 lines (API calls)
|
||||||
|
|
||||||
|
### 2. Iterative Improvement
|
||||||
|
|
||||||
|
**Goal**: Improve correction quality over time through learning.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
1. **Run corrections** on 3-5 similar transcripts:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input day1.md --stage 3 --domain embodied_ai
|
||||||
|
uv run scripts/fix_transcription.py --input day2.md --stage 3 --domain embodied_ai
|
||||||
|
uv run scripts/fix_transcription.py --input day3.md --stage 3 --domain embodied_ai
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Review learned suggestions**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output example**:
|
||||||
|
```
|
||||||
|
📚 Learned Suggestions (Pending Review)
|
||||||
|
========================================
|
||||||
|
|
||||||
|
1. "巨升方向" → "具身方向"
|
||||||
|
Frequency: 5 Confidence: 0.95
|
||||||
|
Examples: day1.md (line 45), day2.md (line 23), ...
|
||||||
|
|
||||||
|
2. "奇迹创坛" → "奇绩创坛"
|
||||||
|
Frequency: 3 Confidence: 0.87
|
||||||
|
Examples: day1.md (line 102), day3.md (line 67)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Approve high-quality suggestions**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --approve "巨升方向" "具身方向"
|
||||||
|
uv run scripts/fix_transcription.py --approve "奇迹创坛" "奇绩创坛"
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Verify approved corrections**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --list --domain embodied_ai | grep "learned"
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Run next batch** (benefits from approved corrections):
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input day4.md --stage 3 --domain embodied_ai
|
||||||
|
```
|
||||||
|
|
||||||
|
**Impact**: Approved corrections move to Stage 1 (instant, free).
|
||||||
|
|
||||||
|
**Cycle**: Repeat every 3-5 transcripts for continuous improvement.
|
||||||
|
|
||||||
|
### 3. Domain-Specific Corrections
|
||||||
|
|
||||||
|
**Goal**: Build specialized dictionaries for different fields.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
1. **Identify domain**:
|
||||||
|
- `embodied_ai` - Robotics, AI terminology
|
||||||
|
- `finance` - Financial terminology
|
||||||
|
- `medical` - Medical terminology
|
||||||
|
- `general` - General-purpose
|
||||||
|
|
||||||
|
2. **Add domain-specific terms**:
|
||||||
|
```bash
|
||||||
|
# Embodied AI domain
|
||||||
|
uv run scripts/fix_transcription.py --add "巨升智能" "具身智能" --domain embodied_ai
|
||||||
|
uv run scripts/fix_transcription.py --add "机器学习" "机器学习" --domain embodied_ai
|
||||||
|
|
||||||
|
# Finance domain
|
||||||
|
uv run scripts/fix_transcription.py --add "股价" "股价" --domain finance # Keep as-is
|
||||||
|
uv run scripts/fix_transcription.py --add "PE比率" "市盈率" --domain finance
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Use appropriate domain** when correcting:
|
||||||
|
```bash
|
||||||
|
# AI meeting transcript
|
||||||
|
uv run scripts/fix_transcription.py --input ai_meeting.md --stage 3 --domain embodied_ai
|
||||||
|
|
||||||
|
# Financial report transcript
|
||||||
|
uv run scripts/fix_transcription.py --input earnings_call.md --stage 3 --domain finance
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Review domain statistics**:
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "SELECT * FROM correction_statistics;"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits**:
|
||||||
|
- Prevents cross-domain conflicts
|
||||||
|
- Higher accuracy per domain
|
||||||
|
- Targeted vocabulary building
|
||||||
|
|
||||||
|
### 4. Team Collaboration
|
||||||
|
|
||||||
|
**Goal**: Share corrections across team members.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
#### Setup (One-time per team)
|
||||||
|
|
||||||
|
1. **Create shared repository**:
|
||||||
|
```bash
|
||||||
|
mkdir transcript-corrections
|
||||||
|
cd transcript-corrections
|
||||||
|
git init
|
||||||
|
|
||||||
|
# .gitignore
|
||||||
|
echo "*.db\n*.db-journal\n*.bak" > .gitignore
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Export initial corrections**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --export general.json --domain general
|
||||||
|
uv run scripts/fix_transcription.py --export embodied_ai.json --domain embodied_ai
|
||||||
|
|
||||||
|
git add *.json
|
||||||
|
git commit -m "Initial correction dictionaries"
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Daily Workflow
|
||||||
|
|
||||||
|
**Team Member A** (adds new corrections):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Run corrections
|
||||||
|
uv run scripts/fix_transcription.py --input transcript.md --stage 3 --domain embodied_ai
|
||||||
|
|
||||||
|
# 2. Review and approve learned suggestions
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
uv run scripts/fix_transcription.py --approve "新错误" "正确词"
|
||||||
|
|
||||||
|
# 3. Export updated corrections
|
||||||
|
uv run scripts/fix_transcription.py --export embodied_ai_$(date +%Y%m%d).json --domain embodied_ai
|
||||||
|
|
||||||
|
# 4. Commit and push
|
||||||
|
git add embodied_ai_*.json
|
||||||
|
git commit -m "Add embodied AI corrections from today's transcripts"
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
**Team Member B** (imports team corrections):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Pull latest corrections
|
||||||
|
git pull origin main
|
||||||
|
|
||||||
|
# 2. Import with merge
|
||||||
|
uv run scripts/fix_transcription.py --import embodied_ai_20250128.json --merge
|
||||||
|
|
||||||
|
# 3. Verify
|
||||||
|
uv run scripts/fix_transcription.py --list --domain embodied_ai | tail -10
|
||||||
|
```
|
||||||
|
|
||||||
|
**Conflict resolution**: See `team_collaboration.md` for handling merge conflicts.
|
||||||
|
|
||||||
|
### 5. Stage-by-Stage Execution
|
||||||
|
|
||||||
|
**Goal**: Test dictionary changes without wasting API quota.
|
||||||
|
|
||||||
|
#### Stage 1 Only (Dictionary)
|
||||||
|
|
||||||
|
**Use when**: Testing new corrections, verifying domain setup.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 1 --domain general
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output**: `file_stage1.md` with dictionary corrections only.
|
||||||
|
|
||||||
|
**Review**: Check if dictionary corrections are sufficient.
|
||||||
|
|
||||||
|
#### Stage 2 Only (AI)
|
||||||
|
|
||||||
|
**Use when**: Running AI corrections on pre-processed file.
|
||||||
|
|
||||||
|
**Prerequisites**: Stage 1 output exists.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stage 1 first
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 1
|
||||||
|
|
||||||
|
# Then Stage 2
|
||||||
|
uv run scripts/fix_transcription.py --input file_stage1.md --stage 2
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output**: `file_stage1_stage2.md` (confusing naming - use Stage 3 instead).
|
||||||
|
|
||||||
|
#### Stage 3 (Full Pipeline)
|
||||||
|
|
||||||
|
**Use when**: Production runs, full correction workflow.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input file.md --stage 3 --domain general
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output**: Both `file_stage1.md` and `file_stage2.md`.
|
||||||
|
|
||||||
|
**Recommended**: Use Stage 3 for most workflows.
|
||||||
|
|
||||||
|
### 6. Context-Aware Rules
|
||||||
|
|
||||||
|
**Goal**: Handle edge cases with regex patterns.
|
||||||
|
|
||||||
|
**Use cases**:
|
||||||
|
- Positional corrections (e.g., "的" vs "地")
|
||||||
|
- Multi-word patterns
|
||||||
|
- Conditional corrections
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
1. **Identify pattern** that simple dictionary can't handle:
|
||||||
|
```
|
||||||
|
Problem: "近距离的去看" (wrong - should be "地")
|
||||||
|
Problem: "近距离搏杀" (correct - should keep "的")
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Add context rules**:
|
||||||
|
```bash
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db
|
||||||
|
|
||||||
|
-- Higher priority for specific context
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离的去看', '近距离地去看', '的→地 before verb', 10);
|
||||||
|
|
||||||
|
-- Lower priority for general pattern
|
||||||
|
INSERT INTO context_rules (pattern, replacement, description, priority)
|
||||||
|
VALUES ('近距离搏杀', '近距离搏杀', 'Keep 的 for noun modifier', 5);
|
||||||
|
|
||||||
|
.quit
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Test context rules**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input test.md --stage 1
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Validate**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Priority**: Higher numbers run first (use for exceptions/edge cases).
|
||||||
|
|
||||||
|
See `file_formats.md` for context_rules schema.
|
||||||
|
|
||||||
|
### 7. Diff Report Generation
|
||||||
|
|
||||||
|
**Goal**: Visualize all changes for review.
|
||||||
|
|
||||||
|
**Use when**:
|
||||||
|
- Reviewing corrections before publishing
|
||||||
|
- Training new team members
|
||||||
|
- Documenting ASR error patterns
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
|
||||||
|
1. **Run corrections**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/fix_transcription.py --input transcript.md --stage 3
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Generate diff reports**:
|
||||||
|
```bash
|
||||||
|
uv run scripts/diff_generator.py \
|
||||||
|
transcript.md \
|
||||||
|
transcript_stage1.md \
|
||||||
|
transcript_stage2.md
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Review outputs**:
|
||||||
|
```bash
|
||||||
|
# Markdown report (statistics + summary)
|
||||||
|
less diff_report.md
|
||||||
|
|
||||||
|
# Unified diff (git-style)
|
||||||
|
less transcript_unified.diff
|
||||||
|
|
||||||
|
# HTML side-by-side (visual review)
|
||||||
|
open transcript_sidebyside.html
|
||||||
|
|
||||||
|
# Inline markers (for editing)
|
||||||
|
less transcript_inline.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**Report contents**:
|
||||||
|
- Total changes count
|
||||||
|
- Stage 1 vs Stage 2 breakdown
|
||||||
|
- Character/word count changes
|
||||||
|
- Side-by-side comparison
|
||||||
|
|
||||||
|
See `script_parameters.md` for advanced diff options.
|
||||||
|
|
||||||
|
## Batch Processing
|
||||||
|
|
||||||
|
### Process Multiple Files
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Simple loop
|
||||||
|
for file in meeting_*.md; do
|
||||||
|
uv run scripts/fix_transcription.py --input "$file" --stage 3 --domain embodied_ai
|
||||||
|
done
|
||||||
|
|
||||||
|
# With error handling
|
||||||
|
for file in meeting_*.md; do
|
||||||
|
echo "Processing $file..."
|
||||||
|
if uv run scripts/fix_transcription.py --input "$file" --stage 3 --domain embodied_ai; then
|
||||||
|
echo "✅ $file completed"
|
||||||
|
else
|
||||||
|
echo "❌ $file failed"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parallel Processing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# GNU parallel (install: brew install parallel)
|
||||||
|
ls meeting_*.md | parallel -j 4 \
|
||||||
|
"uv run scripts/fix_transcription.py --input {} --stage 3 --domain embodied_ai"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Caution**: Monitor API rate limits when processing in parallel.
|
||||||
|
|
||||||
|
## Maintenance Workflows
|
||||||
|
|
||||||
|
### Weekly: Review Learning
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Review suggestions
|
||||||
|
uv run scripts/fix_transcription.py --review-learned
|
||||||
|
|
||||||
|
# Approve high-confidence patterns
|
||||||
|
uv run scripts/fix_transcription.py --approve "错误1" "正确1"
|
||||||
|
uv run scripts/fix_transcription.py --approve "错误2" "正确2"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monthly: Export and Backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export all domains
|
||||||
|
uv run scripts/fix_transcription.py --export general_$(date +%Y%m%d).json --domain general
|
||||||
|
uv run scripts/fix_transcription.py --export embodied_ai_$(date +%Y%m%d).json --domain embodied_ai
|
||||||
|
|
||||||
|
# Backup database
|
||||||
|
cp ~/.transcript-fixer/corrections.db ~/backups/corrections_$(date +%Y%m%d).db
|
||||||
|
|
||||||
|
# Database maintenance
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "VACUUM; REINDEX; ANALYZE;"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quarterly: Clean Up
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Archive old history (> 90 days)
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
DELETE FROM correction_history
|
||||||
|
WHERE run_timestamp < datetime('now', '-90 days');
|
||||||
|
"
|
||||||
|
|
||||||
|
# Reject low-confidence suggestions
|
||||||
|
sqlite3 ~/.transcript-fixer/corrections.db "
|
||||||
|
UPDATE learned_suggestions
|
||||||
|
SET status = 'rejected'
|
||||||
|
WHERE confidence < 0.6 AND frequency < 3;
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
- See `best_practices.md` for optimization tips
|
||||||
|
- See `troubleshooting.md` for error resolution
|
||||||
|
- See `file_formats.md` for database schema
|
||||||
|
- See `script_parameters.md` for advanced CLI options
|
||||||
4
transcript-fixer/requirements.txt
Normal file
4
transcript-fixer/requirements.txt
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
# Transcript Fixer Dependencies
|
||||||
|
|
||||||
|
# HTTP client for GLM API calls
|
||||||
|
httpx>=0.24.0
|
||||||
10
transcript-fixer/scripts/__init__.py
Normal file
10
transcript-fixer/scripts/__init__.py
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
"""
|
||||||
|
Transcript Fixer - Modular Script Package
|
||||||
|
|
||||||
|
Package structure:
|
||||||
|
- core/: Business logic and data access layer
|
||||||
|
- cli/: Command-line interface handlers
|
||||||
|
- utils/: Utility functions and tools
|
||||||
|
"""
|
||||||
|
|
||||||
|
__version__ = "1.0.0"
|
||||||
29
transcript-fixer/scripts/cli/__init__.py
Normal file
29
transcript-fixer/scripts/cli/__init__.py
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
"""
|
||||||
|
CLI Module - Command-Line Interface Handlers
|
||||||
|
|
||||||
|
This module contains command handlers and argument parsing:
|
||||||
|
- commands: Command handler functions (cmd_*)
|
||||||
|
- argument_parser: CLI argument configuration
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .commands import (
|
||||||
|
cmd_init,
|
||||||
|
cmd_add_correction,
|
||||||
|
cmd_list_corrections,
|
||||||
|
cmd_run_correction,
|
||||||
|
cmd_review_learned,
|
||||||
|
cmd_approve,
|
||||||
|
cmd_validate,
|
||||||
|
)
|
||||||
|
from .argument_parser import create_argument_parser
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'cmd_init',
|
||||||
|
'cmd_add_correction',
|
||||||
|
'cmd_list_corrections',
|
||||||
|
'cmd_run_correction',
|
||||||
|
'cmd_review_learned',
|
||||||
|
'cmd_approve',
|
||||||
|
'cmd_validate',
|
||||||
|
'create_argument_parser',
|
||||||
|
]
|
||||||
89
transcript-fixer/scripts/cli/argument_parser.py
Normal file
89
transcript-fixer/scripts/cli/argument_parser.py
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Argument Parser - CLI Argument Configuration
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Configure command-line argument parsing
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
|
||||||
|
def create_argument_parser() -> argparse.ArgumentParser:
|
||||||
|
"""
|
||||||
|
Create and configure the argument parser for transcript-fixer CLI.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Configured ArgumentParser instance
|
||||||
|
"""
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Transcript Fixer - Iterative correction tool",
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter
|
||||||
|
)
|
||||||
|
|
||||||
|
# Setup commands
|
||||||
|
parser.add_argument(
|
||||||
|
"--init",
|
||||||
|
action="store_true",
|
||||||
|
help="Initialize ~/.transcript-fixer/"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Correction management
|
||||||
|
parser.add_argument(
|
||||||
|
"--add",
|
||||||
|
nargs=2,
|
||||||
|
metavar=("FROM", "TO"),
|
||||||
|
dest="add_correction",
|
||||||
|
help="Add correction"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--list",
|
||||||
|
action="store_true",
|
||||||
|
dest="list_corrections",
|
||||||
|
help="List all corrections"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Correction workflow
|
||||||
|
parser.add_argument(
|
||||||
|
"--input", "-i",
|
||||||
|
help="Input file"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output", "-o",
|
||||||
|
help="Output directory"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--stage", "-s",
|
||||||
|
type=int,
|
||||||
|
choices=[1, 2, 3],
|
||||||
|
default=3,
|
||||||
|
help="Run stage (1=dict, 2=AI, 3=full)"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--domain", "-d",
|
||||||
|
default="general",
|
||||||
|
help="Correction domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Learning commands
|
||||||
|
parser.add_argument(
|
||||||
|
"--review-learned",
|
||||||
|
action="store_true",
|
||||||
|
help="Review learned suggestions"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--approve",
|
||||||
|
nargs=2,
|
||||||
|
metavar=("FROM", "TO"),
|
||||||
|
help="Approve suggestion"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Utility commands
|
||||||
|
parser.add_argument(
|
||||||
|
"--validate",
|
||||||
|
action="store_true",
|
||||||
|
help="Validate configuration and JSON files"
|
||||||
|
)
|
||||||
|
|
||||||
|
return parser
|
||||||
181
transcript-fixer/scripts/cli/commands.py
Normal file
181
transcript-fixer/scripts/cli/commands.py
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
CLI Commands - Command Handler Functions
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Handle CLI command execution
|
||||||
|
|
||||||
|
All cmd_* functions take parsed args and execute the requested operation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from core import (
|
||||||
|
CorrectionRepository,
|
||||||
|
CorrectionService,
|
||||||
|
DictionaryProcessor,
|
||||||
|
AIProcessor,
|
||||||
|
LearningEngine,
|
||||||
|
)
|
||||||
|
from utils import validate_configuration, print_validation_summary
|
||||||
|
|
||||||
|
|
||||||
|
def _get_service():
|
||||||
|
"""Get configured CorrectionService instance."""
|
||||||
|
config_dir = Path.home() / ".transcript-fixer"
|
||||||
|
db_path = config_dir / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
return CorrectionService(repository)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_init(args):
|
||||||
|
"""Initialize ~/.transcript-fixer/ directory"""
|
||||||
|
service = _get_service()
|
||||||
|
service.initialize()
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_add_correction(args):
|
||||||
|
"""Add a single correction"""
|
||||||
|
service = _get_service()
|
||||||
|
try:
|
||||||
|
service.add_correction(args.from_text, args.to_text, args.domain)
|
||||||
|
print(f"✅ Added: '{args.from_text}' → '{args.to_text}' (domain: {args.domain})")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error: {e}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_list_corrections(args):
|
||||||
|
"""List all corrections"""
|
||||||
|
service = _get_service()
|
||||||
|
corrections = service.get_corrections(args.domain)
|
||||||
|
|
||||||
|
print(f"\n📋 Corrections (domain: {args.domain})")
|
||||||
|
print("=" * 60)
|
||||||
|
for wrong, correct in sorted(corrections.items()):
|
||||||
|
print(f" '{wrong}' → '{correct}'")
|
||||||
|
print(f"\nTotal: {len(corrections)} corrections\n")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_run_correction(args):
|
||||||
|
"""Run the correction workflow"""
|
||||||
|
# Validate input file
|
||||||
|
input_path = Path(args.input)
|
||||||
|
if not input_path.exists():
|
||||||
|
print(f"❌ Error: File not found: {input_path}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# Setup output directory
|
||||||
|
output_dir = Path(args.output) if args.output else input_path.parent
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
service = _get_service()
|
||||||
|
|
||||||
|
# Load corrections and rules
|
||||||
|
corrections = service.get_corrections(args.domain)
|
||||||
|
context_rules = service.load_context_rules()
|
||||||
|
|
||||||
|
# Read input file
|
||||||
|
print(f"📖 Reading: {input_path.name}")
|
||||||
|
with open(input_path, 'r', encoding='utf-8') as f:
|
||||||
|
original_text = f.read()
|
||||||
|
print(f" File size: {len(original_text):,} characters\n")
|
||||||
|
|
||||||
|
# Stage 1: Dictionary corrections
|
||||||
|
stage1_changes = []
|
||||||
|
stage1_text = original_text
|
||||||
|
if args.stage >= 1:
|
||||||
|
print("=" * 60)
|
||||||
|
print("🔧 Stage 1: Dictionary Corrections")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
processor = DictionaryProcessor(corrections, context_rules)
|
||||||
|
stage1_text, stage1_changes = processor.process(original_text)
|
||||||
|
|
||||||
|
summary = processor.get_summary(stage1_changes)
|
||||||
|
print(f"✓ Applied {summary['total_changes']} corrections")
|
||||||
|
print(f" - Dictionary: {summary['dictionary_changes']}")
|
||||||
|
print(f" - Context rules: {summary['context_rule_changes']}")
|
||||||
|
|
||||||
|
stage1_file = output_dir / f"{input_path.stem}_stage1.md"
|
||||||
|
with open(stage1_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(stage1_text)
|
||||||
|
print(f"💾 Saved: {stage1_file.name}\n")
|
||||||
|
|
||||||
|
# Stage 2: AI corrections
|
||||||
|
stage2_changes = []
|
||||||
|
stage2_text = stage1_text
|
||||||
|
if args.stage >= 2:
|
||||||
|
print("=" * 60)
|
||||||
|
print("🤖 Stage 2: AI Corrections")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
# Check API key
|
||||||
|
api_key = os.environ.get("GLM_API_KEY")
|
||||||
|
if not api_key:
|
||||||
|
print("❌ Error: GLM_API_KEY environment variable not set")
|
||||||
|
print(" Set it with: export GLM_API_KEY='your-key'")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
ai_processor = AIProcessor(api_key)
|
||||||
|
stage2_text, stage2_changes = ai_processor.process(stage1_text)
|
||||||
|
|
||||||
|
print(f"✓ Processed {len(stage2_changes)} chunks\n")
|
||||||
|
|
||||||
|
stage2_file = output_dir / f"{input_path.stem}_stage2.md"
|
||||||
|
with open(stage2_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(stage2_text)
|
||||||
|
print(f"💾 Saved: {stage2_file.name}\n")
|
||||||
|
|
||||||
|
# Save history for learning
|
||||||
|
service.save_history(
|
||||||
|
filename=str(input_path),
|
||||||
|
domain=args.domain,
|
||||||
|
original_length=len(original_text),
|
||||||
|
stage1_changes=len(stage1_changes),
|
||||||
|
stage2_changes=len(stage2_changes),
|
||||||
|
model="GLM-4.6",
|
||||||
|
changes=stage1_changes + stage2_changes
|
||||||
|
)
|
||||||
|
|
||||||
|
# TODO: Run learning engine
|
||||||
|
# learning = LearningEngine(...)
|
||||||
|
# suggestions = learning.analyze_and_suggest()
|
||||||
|
# if suggestions:
|
||||||
|
# print(f"🎓 Learning: Found {len(suggestions)} new correction suggestions")
|
||||||
|
# print(f" Run --review-learned to review them\n")
|
||||||
|
|
||||||
|
# Stage 3: Generate diff report
|
||||||
|
if args.stage >= 3:
|
||||||
|
print("=" * 60)
|
||||||
|
print("📊 Stage 3: Generating Diff Report")
|
||||||
|
print("=" * 60)
|
||||||
|
print(" Use diff_generator.py to create visual comparison\n")
|
||||||
|
|
||||||
|
print("✅ Correction complete!")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_review_learned(args):
|
||||||
|
"""Review learned suggestions"""
|
||||||
|
# TODO: Implement learning engine with SQLite backend
|
||||||
|
print("⚠️ Learning engine not yet implemented with SQLite backend")
|
||||||
|
print(" This feature will be added in a future update")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_approve(args):
|
||||||
|
"""Approve a learned suggestion"""
|
||||||
|
# TODO: Implement learning engine with SQLite backend
|
||||||
|
print("⚠️ Learning engine not yet implemented with SQLite backend")
|
||||||
|
print(" This feature will be added in a future update")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_validate(args):
|
||||||
|
"""Validate configuration and JSON files"""
|
||||||
|
errors, warnings = validate_configuration()
|
||||||
|
exit_code = print_validation_summary(errors, warnings)
|
||||||
|
if exit_code != 0:
|
||||||
|
sys.exit(exit_code)
|
||||||
44
transcript-fixer/scripts/core/__init__.py
Normal file
44
transcript-fixer/scripts/core/__init__.py
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
"""
|
||||||
|
Core Module - Business Logic and Data Access
|
||||||
|
|
||||||
|
This module contains the core business logic for transcript correction:
|
||||||
|
- CorrectionRepository: Data access layer with ACID transactions
|
||||||
|
- CorrectionService: Business logic layer with validation
|
||||||
|
- DictionaryProcessor: Stage 1 dictionary-based corrections
|
||||||
|
- AIProcessor: Stage 2 AI-powered corrections
|
||||||
|
- LearningEngine: Pattern detection and learning
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Core SQLite-based components (always available)
|
||||||
|
from .correction_repository import CorrectionRepository, Correction, DatabaseError, ValidationError
|
||||||
|
from .correction_service import CorrectionService, ValidationRules
|
||||||
|
|
||||||
|
# Processing components (imported lazily to avoid dependency issues)
|
||||||
|
def _lazy_import(name):
|
||||||
|
"""Lazy import to avoid loading heavy dependencies."""
|
||||||
|
if name == 'DictionaryProcessor':
|
||||||
|
from .dictionary_processor import DictionaryProcessor
|
||||||
|
return DictionaryProcessor
|
||||||
|
elif name == 'AIProcessor':
|
||||||
|
from .ai_processor import AIProcessor
|
||||||
|
return AIProcessor
|
||||||
|
elif name == 'LearningEngine':
|
||||||
|
from .learning_engine import LearningEngine
|
||||||
|
return LearningEngine
|
||||||
|
raise ImportError(f"Unknown module: {name}")
|
||||||
|
|
||||||
|
# Export main classes
|
||||||
|
__all__ = [
|
||||||
|
'CorrectionRepository',
|
||||||
|
'CorrectionService',
|
||||||
|
'Correction',
|
||||||
|
'DatabaseError',
|
||||||
|
'ValidationError',
|
||||||
|
'ValidationRules',
|
||||||
|
]
|
||||||
|
|
||||||
|
# Make lazy imports available via __getattr__
|
||||||
|
def __getattr__(name):
|
||||||
|
if name in ['DictionaryProcessor', 'AIProcessor', 'LearningEngine']:
|
||||||
|
return _lazy_import(name)
|
||||||
|
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||||
214
transcript-fixer/scripts/core/ai_processor.py
Normal file
214
transcript-fixer/scripts/core/ai_processor.py
Normal file
@@ -0,0 +1,214 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
AI Processor - Stage 2: AI-powered Text Corrections
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Process text using GLM API for intelligent corrections
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Split text into chunks for API processing
|
||||||
|
- Call GLM-4.6 for context-aware corrections
|
||||||
|
- Track AI-suggested changes
|
||||||
|
- Handle API errors gracefully
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from typing import List, Tuple
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class AIChange:
|
||||||
|
"""Represents an AI-suggested change"""
|
||||||
|
chunk_index: int
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
confidence: float # 0.0 to 1.0
|
||||||
|
|
||||||
|
|
||||||
|
class AIProcessor:
|
||||||
|
"""
|
||||||
|
Stage 2 Processor: AI-powered corrections using GLM-4.6
|
||||||
|
|
||||||
|
Process:
|
||||||
|
1. Split text into chunks (respecting API limits)
|
||||||
|
2. Send each chunk to GLM API
|
||||||
|
3. Track changes for learning engine
|
||||||
|
4. Preserve formatting and structure
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, api_key: str, model: str = "GLM-4.6",
|
||||||
|
base_url: str = "https://open.bigmodel.cn/api/anthropic",
|
||||||
|
fallback_model: str = "GLM-4.5-Air"):
|
||||||
|
"""
|
||||||
|
Initialize AI processor
|
||||||
|
|
||||||
|
Args:
|
||||||
|
api_key: GLM API key
|
||||||
|
model: Model name (default: GLM-4.6)
|
||||||
|
base_url: API base URL
|
||||||
|
fallback_model: Fallback model on primary failure
|
||||||
|
"""
|
||||||
|
self.api_key = api_key
|
||||||
|
self.model = model
|
||||||
|
self.fallback_model = fallback_model
|
||||||
|
self.base_url = base_url
|
||||||
|
self.max_chunk_size = 6000 # Characters per chunk
|
||||||
|
|
||||||
|
def process(self, text: str, context: str = "") -> Tuple[str, List[AIChange]]:
|
||||||
|
"""
|
||||||
|
Process text with AI corrections
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to correct
|
||||||
|
context: Optional domain/meeting context
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(corrected_text, list_of_changes)
|
||||||
|
"""
|
||||||
|
chunks = self._split_into_chunks(text)
|
||||||
|
corrected_chunks = []
|
||||||
|
all_changes = []
|
||||||
|
|
||||||
|
print(f"📝 Processing {len(chunks)} chunks with {self.model}...")
|
||||||
|
|
||||||
|
for i, chunk in enumerate(chunks, 1):
|
||||||
|
print(f" Chunk {i}/{len(chunks)}... ", end="", flush=True)
|
||||||
|
|
||||||
|
try:
|
||||||
|
corrected_chunk = self._process_chunk(chunk, context, self.model)
|
||||||
|
corrected_chunks.append(corrected_chunk)
|
||||||
|
|
||||||
|
# TODO: Extract actual changes for learning
|
||||||
|
# For now, we assume the whole chunk changed
|
||||||
|
if corrected_chunk != chunk:
|
||||||
|
all_changes.append(AIChange(
|
||||||
|
chunk_index=i,
|
||||||
|
from_text=chunk[:50] + "...",
|
||||||
|
to_text=corrected_chunk[:50] + "...",
|
||||||
|
confidence=0.9 # Placeholder
|
||||||
|
))
|
||||||
|
|
||||||
|
print("✓")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"✗ {str(e)[:50]}")
|
||||||
|
|
||||||
|
# Retry with fallback model
|
||||||
|
if self.fallback_model and self.fallback_model != self.model:
|
||||||
|
print(f" Retrying with {self.fallback_model}... ", end="", flush=True)
|
||||||
|
try:
|
||||||
|
corrected_chunk = self._process_chunk(chunk, context, self.fallback_model)
|
||||||
|
corrected_chunks.append(corrected_chunk)
|
||||||
|
print("✓")
|
||||||
|
continue
|
||||||
|
except Exception as e2:
|
||||||
|
print(f"✗ {str(e2)[:50]}")
|
||||||
|
|
||||||
|
print(" Using original text...")
|
||||||
|
corrected_chunks.append(chunk)
|
||||||
|
|
||||||
|
return "\n\n".join(corrected_chunks), all_changes
|
||||||
|
|
||||||
|
def _split_into_chunks(self, text: str) -> List[str]:
|
||||||
|
"""
|
||||||
|
Split text into processable chunks
|
||||||
|
|
||||||
|
Strategy:
|
||||||
|
- Split by double newlines (paragraphs)
|
||||||
|
- Keep chunks under max_chunk_size
|
||||||
|
- Don't split mid-paragraph if possible
|
||||||
|
"""
|
||||||
|
paragraphs = text.split('\n\n')
|
||||||
|
chunks = []
|
||||||
|
current_chunk = []
|
||||||
|
current_length = 0
|
||||||
|
|
||||||
|
for para in paragraphs:
|
||||||
|
para_length = len(para)
|
||||||
|
|
||||||
|
# If single paragraph exceeds limit, force split
|
||||||
|
if para_length > self.max_chunk_size:
|
||||||
|
if current_chunk:
|
||||||
|
chunks.append('\n\n'.join(current_chunk))
|
||||||
|
current_chunk = []
|
||||||
|
current_length = 0
|
||||||
|
|
||||||
|
# Split long paragraph by sentences
|
||||||
|
sentences = re.split(r'([。！？\n])', para)
|
||||||
|
temp_para = ""
|
||||||
|
for i in range(0, len(sentences), 2):
|
||||||
|
sentence = sentences[i] + (sentences[i+1] if i+1 < len(sentences) else "")
|
||||||
|
if len(temp_para) + len(sentence) > self.max_chunk_size:
|
||||||
|
if temp_para:
|
||||||
|
chunks.append(temp_para)
|
||||||
|
temp_para = sentence
|
||||||
|
else:
|
||||||
|
temp_para += sentence
|
||||||
|
if temp_para:
|
||||||
|
chunks.append(temp_para)
|
||||||
|
|
||||||
|
# Normal case: accumulate paragraphs
|
||||||
|
elif current_length + para_length > self.max_chunk_size and current_chunk:
|
||||||
|
chunks.append('\n\n'.join(current_chunk))
|
||||||
|
current_chunk = [para]
|
||||||
|
current_length = para_length
|
||||||
|
else:
|
||||||
|
current_chunk.append(para)
|
||||||
|
current_length += para_length + 2 # +2 for \n\n
|
||||||
|
|
||||||
|
if current_chunk:
|
||||||
|
chunks.append('\n\n'.join(current_chunk))
|
||||||
|
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
def _process_chunk(self, chunk: str, context: str, model: str) -> str:
|
||||||
|
"""Process a single chunk with GLM API"""
|
||||||
|
prompt = self._build_prompt(chunk, context)
|
||||||
|
|
||||||
|
url = f"{self.base_url}/v1/messages"
|
||||||
|
headers = {
|
||||||
|
"anthropic-version": "2023-06-01",
|
||||||
|
"Authorization": f"Bearer {self.api_key}",
|
||||||
|
"content-type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
data = {
|
||||||
|
"model": model,
|
||||||
|
"max_tokens": 8000,
|
||||||
|
"temperature": 0.3,
|
||||||
|
"messages": [{"role": "user", "content": prompt}]
|
||||||
|
}
|
||||||
|
|
||||||
|
with httpx.Client(timeout=60.0) as client:
|
||||||
|
response = client.post(url, headers=headers, json=data)
|
||||||
|
response.raise_for_status()
|
||||||
|
result = response.json()
|
||||||
|
return result["content"][0]["text"]
|
||||||
|
|
||||||
|
def _build_prompt(self, chunk: str, context: str) -> str:
|
||||||
|
"""Build correction prompt for GLM"""
|
||||||
|
base_prompt = """你是专业的会议记录校对专家。请修复以下会议转录中的语音识别错误。
|
||||||
|
|
||||||
|
**修复原则**：
|
||||||
|
1. 严格保留原有格式（时间戳、发言人标识、Markdown标记等）
|
||||||
|
2. 修复明显的同音字错误
|
||||||
|
3. 修复专业术语错误
|
||||||
|
4. 修复语法错误，但保持口语化特征
|
||||||
|
5. 不确定的地方保持原样，不要过度修改
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
if context:
|
||||||
|
base_prompt += f"\n**会议背景**：\n{context}\n"
|
||||||
|
|
||||||
|
base_prompt += f"""
|
||||||
|
**需要修复的内容**：
|
||||||
|
{chunk}
|
||||||
|
|
||||||
|
**请直接输出修复后的文本，不要添加任何解释或标注**："""
|
||||||
|
|
||||||
|
return base_prompt
|
||||||
465
transcript-fixer/scripts/core/correction_repository.py
Normal file
465
transcript-fixer/scripts/core/correction_repository.py
Normal file
@@ -0,0 +1,465 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Correction Repository - SQLite Data Access Layer
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Manage database operations with ACID guarantees
|
||||||
|
|
||||||
|
Thread-safe, transactional, and follows Repository pattern.
|
||||||
|
All database operations are atomic and properly handle errors.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from typing import Dict, List, Optional, Tuple, Any
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from dataclasses import dataclass, asdict
|
||||||
|
import threading
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Correction:
|
||||||
|
"""Correction entity"""
|
||||||
|
id: Optional[int]
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
domain: str
|
||||||
|
source: str # 'manual' | 'learned' | 'imported'
|
||||||
|
confidence: float
|
||||||
|
added_by: Optional[str]
|
||||||
|
added_at: str
|
||||||
|
usage_count: int
|
||||||
|
last_used: Optional[str]
|
||||||
|
notes: Optional[str]
|
||||||
|
is_active: bool
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ContextRule:
|
||||||
|
"""Context-aware rule entity"""
|
||||||
|
id: Optional[int]
|
||||||
|
pattern: str
|
||||||
|
replacement: str
|
||||||
|
description: Optional[str]
|
||||||
|
priority: int
|
||||||
|
is_active: bool
|
||||||
|
added_at: str
|
||||||
|
added_by: Optional[str]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class LearnedSuggestion:
|
||||||
|
"""Learned pattern suggestion"""
|
||||||
|
id: Optional[int]
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
domain: str
|
||||||
|
frequency: int
|
||||||
|
confidence: float
|
||||||
|
first_seen: str
|
||||||
|
last_seen: str
|
||||||
|
status: str # 'pending' | 'approved' | 'rejected'
|
||||||
|
reviewed_at: Optional[str]
|
||||||
|
reviewed_by: Optional[str]
|
||||||
|
|
||||||
|
|
||||||
|
class DatabaseError(Exception):
|
||||||
|
"""Base exception for database errors"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class ValidationError(DatabaseError):
|
||||||
|
"""Data validation error"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class CorrectionRepository:
|
||||||
|
"""
|
||||||
|
Thread-safe repository for correction storage using SQLite.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- ACID transactions
|
||||||
|
- Connection pooling
|
||||||
|
- Prepared statements (SQL injection prevention)
|
||||||
|
- Comprehensive error handling
|
||||||
|
- Audit logging
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, db_path: Path):
|
||||||
|
"""
|
||||||
|
Initialize repository with database path.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
db_path: Path to SQLite database file
|
||||||
|
"""
|
||||||
|
self.db_path = db_path
|
||||||
|
self._local = threading.local()
|
||||||
|
self._ensure_database_exists()
|
||||||
|
|
||||||
|
def _get_connection(self) -> sqlite3.Connection:
|
||||||
|
"""Get thread-local database connection."""
|
||||||
|
if not hasattr(self._local, 'connection'):
|
||||||
|
self._local.connection = sqlite3.connect(
|
||||||
|
self.db_path,
|
||||||
|
isolation_level=None, # Autocommit mode off, manual transactions
|
||||||
|
check_same_thread=False
|
||||||
|
)
|
||||||
|
self._local.connection.row_factory = sqlite3.Row
|
||||||
|
# Enable foreign keys
|
||||||
|
self._local.connection.execute("PRAGMA foreign_keys = ON")
|
||||||
|
return self._local.connection
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def _transaction(self):
|
||||||
|
"""
|
||||||
|
Context manager for database transactions.
|
||||||
|
|
||||||
|
Provides ACID guarantees:
|
||||||
|
- Atomicity: All or nothing
|
||||||
|
- Consistency: Constraints enforced
|
||||||
|
- Isolation: Serializable by default
|
||||||
|
- Durability: Changes persisted to disk
|
||||||
|
"""
|
||||||
|
conn = self._get_connection()
|
||||||
|
try:
|
||||||
|
conn.execute("BEGIN IMMEDIATE") # Acquire write lock immediately
|
||||||
|
yield conn
|
||||||
|
conn.commit()
|
||||||
|
except Exception as e:
|
||||||
|
conn.rollback()
|
||||||
|
logger.error(f"Transaction rolled back: {e}")
|
||||||
|
raise DatabaseError(f"Database operation failed: {e}") from e
|
||||||
|
|
||||||
|
def _ensure_database_exists(self) -> None:
|
||||||
|
"""Create database schema if not exists."""
|
||||||
|
schema_path = Path(__file__).parent / "schema.sql"
|
||||||
|
|
||||||
|
if not schema_path.exists():
|
||||||
|
raise FileNotFoundError(f"Schema file not found: {schema_path}")
|
||||||
|
|
||||||
|
with open(schema_path, 'r', encoding='utf-8') as f:
|
||||||
|
schema_sql = f.read()
|
||||||
|
|
||||||
|
with self._transaction() as conn:
|
||||||
|
conn.executescript(schema_sql)
|
||||||
|
|
||||||
|
logger.info(f"Database initialized: {self.db_path}")
|
||||||
|
|
||||||
|
# ==================== Correction Operations ====================
|
||||||
|
|
||||||
|
def add_correction(
|
||||||
|
self,
|
||||||
|
from_text: str,
|
||||||
|
to_text: str,
|
||||||
|
domain: str = "general",
|
||||||
|
source: str = "manual",
|
||||||
|
confidence: float = 1.0,
|
||||||
|
added_by: Optional[str] = None,
|
||||||
|
notes: Optional[str] = None
|
||||||
|
) -> int:
|
||||||
|
"""
|
||||||
|
Add a new correction with full validation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
from_text: Original (incorrect) text
|
||||||
|
to_text: Corrected text
|
||||||
|
domain: Correction domain
|
||||||
|
source: Origin of correction
|
||||||
|
confidence: Confidence score (0.0-1.0)
|
||||||
|
added_by: User who added it
|
||||||
|
notes: Optional notes
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
ID of inserted correction
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValidationError: If validation fails
|
||||||
|
DatabaseError: If database operation fails
|
||||||
|
"""
|
||||||
|
with self._transaction() as conn:
|
||||||
|
try:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
INSERT INTO corrections
|
||||||
|
(from_text, to_text, domain, source, confidence, added_by, notes)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||||
|
""", (from_text, to_text, domain, source, confidence, added_by, notes))
|
||||||
|
|
||||||
|
correction_id = cursor.lastrowid
|
||||||
|
|
||||||
|
# Audit log
|
||||||
|
self._audit_log(
|
||||||
|
conn,
|
||||||
|
action="add_correction",
|
||||||
|
entity_type="correction",
|
||||||
|
entity_id=correction_id,
|
||||||
|
user=added_by,
|
||||||
|
details=f"Added: '{from_text}' → '{to_text}' (domain: {domain})"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Added correction ID {correction_id}: {from_text} → {to_text}")
|
||||||
|
return correction_id
|
||||||
|
|
||||||
|
except sqlite3.IntegrityError as e:
|
||||||
|
if "UNIQUE constraint failed" in str(e):
|
||||||
|
# Update existing correction instead (within same transaction)
|
||||||
|
logger.warning(f"Correction already exists, updating: {from_text}")
|
||||||
|
cursor = conn.execute("""
|
||||||
|
UPDATE corrections
|
||||||
|
SET to_text = ?, source = ?, confidence = ?,
|
||||||
|
added_by = ?, notes = ?, added_at = CURRENT_TIMESTAMP
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (to_text, source, confidence, added_by, notes, from_text, domain))
|
||||||
|
|
||||||
|
if cursor.rowcount > 0:
|
||||||
|
# Get the ID of the updated row
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT id FROM corrections
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (from_text, domain))
|
||||||
|
correction_id = cursor.fetchone()[0]
|
||||||
|
|
||||||
|
# Audit log
|
||||||
|
self._audit_log(
|
||||||
|
conn,
|
||||||
|
action="update_correction",
|
||||||
|
entity_type="correction",
|
||||||
|
entity_id=correction_id,
|
||||||
|
user=added_by,
|
||||||
|
details=f"Updated: '{from_text}' → '{to_text}' (domain: {domain})"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Updated correction ID {correction_id}: {from_text} → {to_text}")
|
||||||
|
return correction_id
|
||||||
|
else:
|
||||||
|
raise ValidationError(f"Correction not found: {from_text} in domain {domain}")
|
||||||
|
raise ValidationError(f"Integrity constraint violated: {e}") from e
|
||||||
|
|
||||||
|
def get_correction(self, from_text: str, domain: str = "general") -> Optional[Correction]:
|
||||||
|
"""Get a specific correction."""
|
||||||
|
conn = self._get_connection()
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT * FROM corrections
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (from_text, domain))
|
||||||
|
|
||||||
|
row = cursor.fetchone()
|
||||||
|
return self._row_to_correction(row) if row else None
|
||||||
|
|
||||||
|
def get_all_corrections(self, domain: Optional[str] = None, active_only: bool = True) -> List[Correction]:
|
||||||
|
"""Get all corrections, optionally filtered by domain."""
|
||||||
|
conn = self._get_connection()
|
||||||
|
|
||||||
|
if domain:
|
||||||
|
if active_only:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT * FROM corrections
|
||||||
|
WHERE domain = ? AND is_active = 1
|
||||||
|
ORDER BY from_text
|
||||||
|
""", (domain,))
|
||||||
|
else:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT * FROM corrections
|
||||||
|
WHERE domain = ?
|
||||||
|
ORDER BY from_text
|
||||||
|
""", (domain,))
|
||||||
|
else:
|
||||||
|
if active_only:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT * FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
ORDER BY domain, from_text
|
||||||
|
""")
|
||||||
|
else:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT * FROM corrections
|
||||||
|
ORDER BY domain, from_text
|
||||||
|
""")
|
||||||
|
|
||||||
|
return [self._row_to_correction(row) for row in cursor.fetchall()]
|
||||||
|
|
||||||
|
def get_corrections_dict(self, domain: str = "general") -> Dict[str, str]:
|
||||||
|
"""Get corrections as a simple dictionary for processing."""
|
||||||
|
corrections = self.get_all_corrections(domain=domain, active_only=True)
|
||||||
|
return {c.from_text: c.to_text for c in corrections}
|
||||||
|
|
||||||
|
def update_correction(
|
||||||
|
self,
|
||||||
|
from_text: str,
|
||||||
|
to_text: str,
|
||||||
|
domain: str = "general",
|
||||||
|
updated_by: Optional[str] = None
|
||||||
|
) -> int:
|
||||||
|
"""Update an existing correction."""
|
||||||
|
with self._transaction() as conn:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
UPDATE corrections
|
||||||
|
SET to_text = ?, added_at = CURRENT_TIMESTAMP
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (to_text, from_text, domain))
|
||||||
|
|
||||||
|
if cursor.rowcount == 0:
|
||||||
|
raise ValidationError(f"Correction not found: {from_text} in domain {domain}")
|
||||||
|
|
||||||
|
# Audit log
|
||||||
|
self._audit_log(
|
||||||
|
conn,
|
||||||
|
action="update_correction",
|
||||||
|
entity_type="correction",
|
||||||
|
user=updated_by,
|
||||||
|
details=f"Updated: '{from_text}' → '{to_text}' (domain: {domain})"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Updated correction: {from_text} → {to_text}")
|
||||||
|
return cursor.rowcount
|
||||||
|
|
||||||
|
def delete_correction(self, from_text: str, domain: str = "general", deleted_by: Optional[str] = None) -> bool:
|
||||||
|
"""Soft delete a correction (mark as inactive)."""
|
||||||
|
with self._transaction() as conn:
|
||||||
|
cursor = conn.execute("""
|
||||||
|
UPDATE corrections
|
||||||
|
SET is_active = 0
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (from_text, domain))
|
||||||
|
|
||||||
|
if cursor.rowcount > 0:
|
||||||
|
self._audit_log(
|
||||||
|
conn,
|
||||||
|
action="delete_correction",
|
||||||
|
entity_type="correction",
|
||||||
|
user=deleted_by,
|
||||||
|
details=f"Deleted: '{from_text}' (domain: {domain})"
|
||||||
|
)
|
||||||
|
logger.info(f"Deleted correction: {from_text}")
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
def increment_usage(self, from_text: str, domain: str = "general") -> None:
|
||||||
|
"""Increment usage count for a correction."""
|
||||||
|
with self._transaction() as conn:
|
||||||
|
conn.execute("""
|
||||||
|
UPDATE corrections
|
||||||
|
SET usage_count = usage_count + 1,
|
||||||
|
last_used = CURRENT_TIMESTAMP
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (from_text, domain))
|
||||||
|
|
||||||
|
# ==================== Bulk Operations ====================
|
||||||
|
|
||||||
|
def bulk_import_corrections(
|
||||||
|
self,
|
||||||
|
corrections: Dict[str, str],
|
||||||
|
domain: str = "general",
|
||||||
|
source: str = "imported",
|
||||||
|
imported_by: Optional[str] = None,
|
||||||
|
merge: bool = True
|
||||||
|
) -> Tuple[int, int, int]:
|
||||||
|
"""
|
||||||
|
Bulk import corrections with conflict resolution.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (inserted_count, updated_count, skipped_count)
|
||||||
|
"""
|
||||||
|
inserted, updated, skipped = 0, 0, 0
|
||||||
|
|
||||||
|
with self._transaction() as conn:
|
||||||
|
for from_text, to_text in corrections.items():
|
||||||
|
try:
|
||||||
|
if merge:
|
||||||
|
# Check if exists
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT id, to_text FROM corrections
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (from_text, domain))
|
||||||
|
existing = cursor.fetchone()
|
||||||
|
|
||||||
|
if existing:
|
||||||
|
if existing['to_text'] != to_text:
|
||||||
|
# Update
|
||||||
|
conn.execute("""
|
||||||
|
UPDATE corrections
|
||||||
|
SET to_text = ?, source = ?, added_at = CURRENT_TIMESTAMP
|
||||||
|
WHERE from_text = ? AND domain = ? AND is_active = 1
|
||||||
|
""", (to_text, source, from_text, domain))
|
||||||
|
updated += 1
|
||||||
|
else:
|
||||||
|
skipped += 1
|
||||||
|
else:
|
||||||
|
# Insert
|
||||||
|
conn.execute("""
|
||||||
|
INSERT INTO corrections
|
||||||
|
(from_text, to_text, domain, source, confidence, added_by)
|
||||||
|
VALUES (?, ?, ?, ?, 1.0, ?)
|
||||||
|
""", (from_text, to_text, domain, source, imported_by))
|
||||||
|
inserted += 1
|
||||||
|
else:
|
||||||
|
# Replace mode: just insert
|
||||||
|
conn.execute("""
|
||||||
|
INSERT OR REPLACE INTO corrections
|
||||||
|
(from_text, to_text, domain, source, confidence, added_by)
|
||||||
|
VALUES (?, ?, ?, ?, 1.0, ?)
|
||||||
|
""", (from_text, to_text, domain, source, imported_by))
|
||||||
|
inserted += 1
|
||||||
|
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
logger.warning(f"Failed to import '{from_text}': {e}")
|
||||||
|
skipped += 1
|
||||||
|
|
||||||
|
# Audit log
|
||||||
|
self._audit_log(
|
||||||
|
conn,
|
||||||
|
action="bulk_import",
|
||||||
|
entity_type="correction",
|
||||||
|
user=imported_by,
|
||||||
|
details=f"Imported {inserted} new, updated {updated}, skipped {skipped} (domain: {domain})"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Bulk import: {inserted} inserted, {updated} updated, {skipped} skipped")
|
||||||
|
return (inserted, updated, skipped)
|
||||||
|
|
||||||
|
# ==================== Helper Methods ====================
|
||||||
|
|
||||||
|
def _row_to_correction(self, row: sqlite3.Row) -> Correction:
|
||||||
|
"""Convert database row to Correction object."""
|
||||||
|
return Correction(
|
||||||
|
id=row['id'],
|
||||||
|
from_text=row['from_text'],
|
||||||
|
to_text=row['to_text'],
|
||||||
|
domain=row['domain'],
|
||||||
|
source=row['source'],
|
||||||
|
confidence=row['confidence'],
|
||||||
|
added_by=row['added_by'],
|
||||||
|
added_at=row['added_at'],
|
||||||
|
usage_count=row['usage_count'],
|
||||||
|
last_used=row['last_used'],
|
||||||
|
notes=row['notes'],
|
||||||
|
is_active=bool(row['is_active'])
|
||||||
|
)
|
||||||
|
|
||||||
|
def _audit_log(
|
||||||
|
self,
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
action: str,
|
||||||
|
entity_type: str,
|
||||||
|
entity_id: Optional[int] = None,
|
||||||
|
user: Optional[str] = None,
|
||||||
|
details: Optional[str] = None,
|
||||||
|
success: bool = True,
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
) -> None:
|
||||||
|
"""Write audit log entry."""
|
||||||
|
conn.execute("""
|
||||||
|
INSERT INTO audit_log (action, entity_type, entity_id, user, details, success, error_message)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||||
|
""", (action, entity_type, entity_id, user, details, success, error_message))
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Close database connection."""
|
||||||
|
if hasattr(self._local, 'connection'):
|
||||||
|
self._local.connection.close()
|
||||||
|
delattr(self._local, 'connection')
|
||||||
|
logger.info("Database connection closed")
|
||||||
524
transcript-fixer/scripts/core/correction_service.py
Normal file
524
transcript-fixer/scripts/core/correction_service.py
Normal file
@@ -0,0 +1,524 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Correction Service - Business Logic Layer
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Implement business rules and validation
|
||||||
|
|
||||||
|
Orchestrates repository operations with comprehensive validation,
|
||||||
|
error handling, and business logic enforcement.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List, Optional, Tuple
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from .correction_repository import (
|
||||||
|
CorrectionRepository,
|
||||||
|
ValidationError,
|
||||||
|
DatabaseError
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ValidationRules:
|
||||||
|
"""Validation rules configuration"""
|
||||||
|
max_text_length: int = 1000
|
||||||
|
min_text_length: int = 1
|
||||||
|
max_domain_length: int = 50
|
||||||
|
allowed_domain_pattern: str = r'^[a-zA-Z0-9_-]+$'
|
||||||
|
max_confidence: float = 1.0
|
||||||
|
min_confidence: float = 0.0
|
||||||
|
|
||||||
|
|
||||||
|
class CorrectionService:
|
||||||
|
"""
|
||||||
|
Service layer for correction management.
|
||||||
|
|
||||||
|
Responsibilities:
|
||||||
|
- Input validation and sanitization
|
||||||
|
- Business rule enforcement
|
||||||
|
- Conflict detection and resolution
|
||||||
|
- Statistics and reporting
|
||||||
|
- Integration with repository layer
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, repository: CorrectionRepository, rules: Optional[ValidationRules] = None):
|
||||||
|
"""
|
||||||
|
Initialize service with repository.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
repository: Data access layer
|
||||||
|
rules: Validation rules (uses defaults if None)
|
||||||
|
"""
|
||||||
|
self.repository = repository
|
||||||
|
self.rules = rules or ValidationRules()
|
||||||
|
self.db_path = repository.db_path
|
||||||
|
logger.info("CorrectionService initialized")
|
||||||
|
|
||||||
|
def initialize(self) -> None:
|
||||||
|
"""
|
||||||
|
Initialize database (already done by repository, kept for API compatibility).
|
||||||
|
"""
|
||||||
|
# Database is auto-initialized by repository on first access
|
||||||
|
logger.info(f"✅ Database ready: {self.db_path}")
|
||||||
|
|
||||||
|
# ==================== Validation Methods ====================
|
||||||
|
|
||||||
|
def validate_correction_text(self, text: str, field_name: str = "text") -> None:
|
||||||
|
"""
|
||||||
|
Validate correction text with comprehensive checks.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to validate
|
||||||
|
field_name: Field name for error messages
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValidationError: If validation fails
|
||||||
|
"""
|
||||||
|
# Check not None or empty
|
||||||
|
if not text:
|
||||||
|
raise ValidationError(f"{field_name} cannot be None or empty")
|
||||||
|
|
||||||
|
# Check not only whitespace
|
||||||
|
if not text.strip():
|
||||||
|
raise ValidationError(f"{field_name} cannot be only whitespace")
|
||||||
|
|
||||||
|
# Check length constraints
|
||||||
|
if len(text) < self.rules.min_text_length:
|
||||||
|
raise ValidationError(
|
||||||
|
f"{field_name} too short: {len(text)} chars (min: {self.rules.min_text_length})"
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(text) > self.rules.max_text_length:
|
||||||
|
raise ValidationError(
|
||||||
|
f"{field_name} too long: {len(text)} chars (max: {self.rules.max_text_length})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check for control characters (except newline and tab)
|
||||||
|
invalid_chars = [c for c in text if ord(c) < 32 and c not in '\n\t']
|
||||||
|
if invalid_chars:
|
||||||
|
raise ValidationError(
|
||||||
|
f"{field_name} contains invalid control characters: {invalid_chars}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check for NULL bytes
|
||||||
|
if '\x00' in text:
|
||||||
|
raise ValidationError(f"{field_name} contains NULL bytes")
|
||||||
|
|
||||||
|
def validate_domain_name(self, domain: str) -> None:
|
||||||
|
"""
|
||||||
|
Validate domain name to prevent path traversal and injection.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
domain: Domain name to validate
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValidationError: If validation fails
|
||||||
|
"""
|
||||||
|
if not domain:
|
||||||
|
raise ValidationError("Domain name cannot be empty")
|
||||||
|
|
||||||
|
if len(domain) > self.rules.max_domain_length:
|
||||||
|
raise ValidationError(
|
||||||
|
f"Domain name too long: {len(domain)} chars (max: {self.rules.max_domain_length})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check pattern: only alphanumeric, underscore, hyphen
|
||||||
|
if not re.match(self.rules.allowed_domain_pattern, domain):
|
||||||
|
raise ValidationError(
|
||||||
|
f"Domain name contains invalid characters: {domain}. "
|
||||||
|
f"Allowed pattern: {self.rules.allowed_domain_pattern}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check for path traversal attempts
|
||||||
|
if '..' in domain or '/' in domain or '\\' in domain:
|
||||||
|
raise ValidationError(f"Domain name contains path traversal: {domain}")
|
||||||
|
|
||||||
|
# Reserved names
|
||||||
|
reserved = ['con', 'prn', 'aux', 'nul', 'com1', 'lpt1'] # Windows reserved
|
||||||
|
if domain.lower() in reserved:
|
||||||
|
raise ValidationError(f"Domain name is reserved: {domain}")
|
||||||
|
|
||||||
|
def validate_confidence(self, confidence: float) -> None:
|
||||||
|
"""Validate confidence score."""
|
||||||
|
if not isinstance(confidence, (int, float)):
|
||||||
|
raise ValidationError(f"Confidence must be numeric, got {type(confidence)}")
|
||||||
|
|
||||||
|
if not (self.rules.min_confidence <= confidence <= self.rules.max_confidence):
|
||||||
|
raise ValidationError(
|
||||||
|
f"Confidence must be between {self.rules.min_confidence} "
|
||||||
|
f"and {self.rules.max_confidence}, got {confidence}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def validate_source(self, source: str) -> None:
|
||||||
|
"""Validate correction source."""
|
||||||
|
valid_sources = ['manual', 'learned', 'imported']
|
||||||
|
if source not in valid_sources:
|
||||||
|
raise ValidationError(
|
||||||
|
f"Invalid source: {source}. Must be one of: {valid_sources}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# ==================== Correction Operations ====================
|
||||||
|
|
||||||
|
def add_correction(
|
||||||
|
self,
|
||||||
|
from_text: str,
|
||||||
|
to_text: str,
|
||||||
|
domain: str = "general",
|
||||||
|
source: str = "manual",
|
||||||
|
confidence: float = 1.0,
|
||||||
|
notes: Optional[str] = None
|
||||||
|
) -> int:
|
||||||
|
"""
|
||||||
|
Add a correction with full validation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
from_text: Original (incorrect) text
|
||||||
|
to_text: Corrected text
|
||||||
|
domain: Correction domain
|
||||||
|
source: Origin of correction
|
||||||
|
confidence: Confidence score
|
||||||
|
notes: Optional notes
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
ID of inserted correction
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValidationError: If validation fails
|
||||||
|
"""
|
||||||
|
# Comprehensive validation
|
||||||
|
self.validate_correction_text(from_text, "from_text")
|
||||||
|
self.validate_correction_text(to_text, "to_text")
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
self.validate_source(source)
|
||||||
|
self.validate_confidence(confidence)
|
||||||
|
|
||||||
|
# Business rule: from_text and to_text should be different
|
||||||
|
if from_text.strip() == to_text.strip():
|
||||||
|
raise ValidationError(
|
||||||
|
f"from_text and to_text are identical: '{from_text}'"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get current user
|
||||||
|
added_by = os.getenv("USER") or os.getenv("USERNAME") or "unknown"
|
||||||
|
|
||||||
|
try:
|
||||||
|
correction_id = self.repository.add_correction(
|
||||||
|
from_text=from_text,
|
||||||
|
to_text=to_text,
|
||||||
|
domain=domain,
|
||||||
|
source=source,
|
||||||
|
confidence=confidence,
|
||||||
|
added_by=added_by,
|
||||||
|
notes=notes
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Successfully added correction ID {correction_id}: "
|
||||||
|
f"'{from_text}' → '{to_text}' (domain: {domain})"
|
||||||
|
)
|
||||||
|
return correction_id
|
||||||
|
|
||||||
|
except DatabaseError as e:
|
||||||
|
logger.error(f"Failed to add correction: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def get_corrections(self, domain: Optional[str] = None) -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
Get corrections as a dictionary for processing.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
domain: Optional domain filter
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary of corrections {from_text: to_text}
|
||||||
|
"""
|
||||||
|
if domain:
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
return self.repository.get_corrections_dict(domain)
|
||||||
|
else:
|
||||||
|
# Get all domains
|
||||||
|
all_corrections = self.repository.get_all_corrections(active_only=True)
|
||||||
|
return {c.from_text: c.to_text for c in all_corrections}
|
||||||
|
|
||||||
|
def remove_correction(
|
||||||
|
self,
|
||||||
|
from_text: str,
|
||||||
|
domain: str = "general"
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Remove a correction (soft delete).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
from_text: Text to remove
|
||||||
|
domain: Domain
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if removed, False if not found
|
||||||
|
"""
|
||||||
|
self.validate_correction_text(from_text, "from_text")
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
|
||||||
|
deleted_by = os.getenv("USER") or os.getenv("USERNAME") or "unknown"
|
||||||
|
|
||||||
|
success = self.repository.delete_correction(from_text, domain, deleted_by)
|
||||||
|
|
||||||
|
if success:
|
||||||
|
logger.info(f"Removed correction: '{from_text}' (domain: {domain})")
|
||||||
|
else:
|
||||||
|
logger.warning(f"Correction not found: '{from_text}' (domain: {domain})")
|
||||||
|
|
||||||
|
return success
|
||||||
|
|
||||||
|
# ==================== Import/Export Operations ====================
|
||||||
|
|
||||||
|
def import_corrections(
|
||||||
|
self,
|
||||||
|
corrections: Dict[str, str],
|
||||||
|
domain: str = "general",
|
||||||
|
merge: bool = True,
|
||||||
|
validate_all: bool = True
|
||||||
|
) -> Tuple[int, int, int]:
|
||||||
|
"""
|
||||||
|
Import corrections with validation and conflict resolution.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
corrections: Dictionary of corrections to import
|
||||||
|
domain: Target domain
|
||||||
|
merge: If True, merge with existing; if False, replace
|
||||||
|
validate_all: If True, validate all before import (safer but slower)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (inserted_count, updated_count, skipped_count)
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValidationError: If validation fails (when validate_all=True)
|
||||||
|
"""
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
|
||||||
|
if not corrections:
|
||||||
|
raise ValidationError("Cannot import empty corrections dictionary")
|
||||||
|
|
||||||
|
# Pre-validation (if requested)
|
||||||
|
if validate_all:
|
||||||
|
logger.info(f"Pre-validating {len(corrections)} corrections...")
|
||||||
|
invalid_count = 0
|
||||||
|
for from_text, to_text in corrections.items():
|
||||||
|
try:
|
||||||
|
self.validate_correction_text(from_text, "from_text")
|
||||||
|
self.validate_correction_text(to_text, "to_text")
|
||||||
|
except ValidationError as e:
|
||||||
|
logger.error(f"Validation failed for '{from_text}' → '{to_text}': {e}")
|
||||||
|
invalid_count += 1
|
||||||
|
|
||||||
|
if invalid_count > 0:
|
||||||
|
raise ValidationError(
|
||||||
|
f"Pre-validation failed: {invalid_count}/{len(corrections)} corrections invalid"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Detect conflicts if merge mode
|
||||||
|
if merge:
|
||||||
|
existing = self.repository.get_corrections_dict(domain)
|
||||||
|
conflicts = self._detect_conflicts(corrections, existing)
|
||||||
|
|
||||||
|
if conflicts:
|
||||||
|
logger.warning(
|
||||||
|
f"Found {len(conflicts)} conflicts that will be overwritten"
|
||||||
|
)
|
||||||
|
for from_text, (old_val, new_val) in conflicts.items():
|
||||||
|
logger.debug(f"Conflict: '{from_text}': '{old_val}' → '{new_val}'")
|
||||||
|
|
||||||
|
# Perform import
|
||||||
|
imported_by = os.getenv("USER") or os.getenv("USERNAME") or "unknown"
|
||||||
|
|
||||||
|
try:
|
||||||
|
inserted, updated, skipped = self.repository.bulk_import_corrections(
|
||||||
|
corrections=corrections,
|
||||||
|
domain=domain,
|
||||||
|
source="imported",
|
||||||
|
imported_by=imported_by,
|
||||||
|
merge=merge
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Import complete: {inserted} inserted, {updated} updated, "
|
||||||
|
f"{skipped} skipped (domain: {domain})"
|
||||||
|
)
|
||||||
|
|
||||||
|
return (inserted, updated, skipped)
|
||||||
|
|
||||||
|
except DatabaseError as e:
|
||||||
|
logger.error(f"Import failed: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def export_corrections(self, domain: str = "general") -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
Export corrections for sharing.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
domain: Domain to export
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary of corrections
|
||||||
|
"""
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
|
||||||
|
corrections = self.repository.get_corrections_dict(domain)
|
||||||
|
|
||||||
|
logger.info(f"Exported {len(corrections)} corrections (domain: {domain})")
|
||||||
|
|
||||||
|
return corrections
|
||||||
|
|
||||||
|
# ==================== Statistics and Reporting ====================
|
||||||
|
|
||||||
|
def get_statistics(self, domain: Optional[str] = None) -> Dict[str, any]:
|
||||||
|
"""
|
||||||
|
Get correction statistics.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
domain: Optional domain filter
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary of statistics
|
||||||
|
"""
|
||||||
|
if domain:
|
||||||
|
self.validate_domain_name(domain)
|
||||||
|
corrections = self.repository.get_all_corrections(domain=domain, active_only=True)
|
||||||
|
else:
|
||||||
|
corrections = self.repository.get_all_corrections(active_only=True)
|
||||||
|
|
||||||
|
# Calculate statistics
|
||||||
|
total = len(corrections)
|
||||||
|
by_source = {'manual': 0, 'learned': 0, 'imported': 0}
|
||||||
|
total_usage = 0
|
||||||
|
high_confidence = 0
|
||||||
|
|
||||||
|
for c in corrections:
|
||||||
|
by_source[c.source] = by_source.get(c.source, 0) + 1
|
||||||
|
total_usage += c.usage_count
|
||||||
|
if c.confidence >= 0.9:
|
||||||
|
high_confidence += 1
|
||||||
|
|
||||||
|
stats = {
|
||||||
|
'total_corrections': total,
|
||||||
|
'by_source': by_source,
|
||||||
|
'total_usage': total_usage,
|
||||||
|
'average_usage': total_usage / total if total > 0 else 0,
|
||||||
|
'high_confidence_count': high_confidence,
|
||||||
|
'high_confidence_ratio': high_confidence / total if total > 0 else 0
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.debug(f"Statistics for domain '{domain}': {stats}")
|
||||||
|
|
||||||
|
return stats
|
||||||
|
|
||||||
|
# ==================== Helper Methods ====================
|
||||||
|
|
||||||
|
def _detect_conflicts(
|
||||||
|
self,
|
||||||
|
incoming: Dict[str, str],
|
||||||
|
existing: Dict[str, str]
|
||||||
|
) -> Dict[str, Tuple[str, str]]:
|
||||||
|
"""
|
||||||
|
Detect conflicts between incoming and existing corrections.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary of conflicts {from_text: (existing_to, incoming_to)}
|
||||||
|
"""
|
||||||
|
conflicts = {}
|
||||||
|
|
||||||
|
for from_text in set(incoming.keys()) & set(existing.keys()):
|
||||||
|
if existing[from_text] != incoming[from_text]:
|
||||||
|
conflicts[from_text] = (existing[from_text], incoming[from_text])
|
||||||
|
|
||||||
|
return conflicts
|
||||||
|
|
||||||
|
def load_context_rules(self) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Load active context-aware regex rules.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of rule dictionaries with pattern, replacement, description
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
conn = self.repository._get_connection()
|
||||||
|
cursor = conn.execute("""
|
||||||
|
SELECT pattern, replacement, description
|
||||||
|
FROM context_rules
|
||||||
|
WHERE is_active = 1
|
||||||
|
ORDER BY priority DESC
|
||||||
|
""")
|
||||||
|
|
||||||
|
rules = []
|
||||||
|
for row in cursor.fetchall():
|
||||||
|
rules.append({
|
||||||
|
"pattern": row[0],
|
||||||
|
"replacement": row[1],
|
||||||
|
"description": row[2]
|
||||||
|
})
|
||||||
|
|
||||||
|
logger.debug(f"Loaded {len(rules)} context rules")
|
||||||
|
return rules
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to load context rules: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def save_history(self, filename: str, domain: str, original_length: int,
|
||||||
|
stage1_changes: int, stage2_changes: int, model: str,
|
||||||
|
changes: List[Dict]) -> None:
|
||||||
|
"""
|
||||||
|
Save correction run history for learning.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
filename: File that was corrected
|
||||||
|
domain: Correction domain
|
||||||
|
original_length: Original file length
|
||||||
|
stage1_changes: Number of Stage 1 changes
|
||||||
|
stage2_changes: Number of Stage 2 changes
|
||||||
|
model: AI model used
|
||||||
|
changes: List of individual changes
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with self.repository._transaction() as conn:
|
||||||
|
# Insert history record
|
||||||
|
cursor = conn.execute("""
|
||||||
|
INSERT INTO correction_history
|
||||||
|
(filename, domain, original_length, stage1_changes, stage2_changes, model)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?)
|
||||||
|
""", (filename, domain, original_length, stage1_changes, stage2_changes, model))
|
||||||
|
|
||||||
|
history_id = cursor.lastrowid
|
||||||
|
|
||||||
|
# Insert individual changes
|
||||||
|
for change in changes:
|
||||||
|
conn.execute("""
|
||||||
|
INSERT INTO correction_changes
|
||||||
|
(history_id, line_number, from_text, to_text, rule_type, context_before, context_after)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||||
|
""", (
|
||||||
|
history_id,
|
||||||
|
change.get("line_number"),
|
||||||
|
change.get("from_text", ""),
|
||||||
|
change.get("to_text", ""),
|
||||||
|
change.get("rule_type", "dictionary"),
|
||||||
|
change.get("context_before"),
|
||||||
|
change.get("context_after")
|
||||||
|
))
|
||||||
|
|
||||||
|
logger.info(f"Saved correction history for {filename}: {stage1_changes + stage2_changes} total changes")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to save history: {e}")
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Close underlying repository."""
|
||||||
|
self.repository.close()
|
||||||
|
logger.info("CorrectionService closed")
|
||||||
140
transcript-fixer/scripts/core/dictionary_processor.py
Normal file
140
transcript-fixer/scripts/core/dictionary_processor.py
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Dictionary Processor - Stage 1: Dictionary-based Text Corrections
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Apply dictionary and regex-based corrections to text
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Apply simple dictionary replacements
|
||||||
|
- Apply context-aware regex rules
|
||||||
|
- Track all changes for history
|
||||||
|
- Case-sensitive and insensitive matching
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List, Tuple
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Change:
|
||||||
|
"""Represents a single text change"""
|
||||||
|
line_number: int
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
rule_type: str # "dictionary" or "context_rule"
|
||||||
|
rule_name: str
|
||||||
|
|
||||||
|
|
||||||
|
class DictionaryProcessor:
|
||||||
|
"""
|
||||||
|
Stage 1 Processor: Apply dictionary-based corrections
|
||||||
|
|
||||||
|
Process:
|
||||||
|
1. Apply context-aware regex rules first (more specific)
|
||||||
|
2. Apply simple dictionary replacements (more general)
|
||||||
|
3. Track all changes for learning
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, corrections: Dict[str, str], context_rules: List[Dict]):
|
||||||
|
"""
|
||||||
|
Initialize processor with corrections and rules
|
||||||
|
|
||||||
|
Args:
|
||||||
|
corrections: Dictionary of {wrong: correct} pairs
|
||||||
|
context_rules: List of context-aware regex rules
|
||||||
|
"""
|
||||||
|
self.corrections = corrections
|
||||||
|
self.context_rules = context_rules
|
||||||
|
|
||||||
|
def process(self, text: str) -> Tuple[str, List[Change]]:
|
||||||
|
"""
|
||||||
|
Apply all corrections to text
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(corrected_text, list_of_changes)
|
||||||
|
"""
|
||||||
|
corrected_text = text
|
||||||
|
all_changes = []
|
||||||
|
|
||||||
|
# Step 1: Apply context rules (more specific, higher priority)
|
||||||
|
corrected_text, context_changes = self._apply_context_rules(corrected_text)
|
||||||
|
all_changes.extend(context_changes)
|
||||||
|
|
||||||
|
# Step 2: Apply dictionary replacements (more general)
|
||||||
|
corrected_text, dict_changes = self._apply_dictionary(corrected_text)
|
||||||
|
all_changes.extend(dict_changes)
|
||||||
|
|
||||||
|
return corrected_text, all_changes
|
||||||
|
|
||||||
|
def _apply_context_rules(self, text: str) -> Tuple[str, List[Change]]:
|
||||||
|
"""Apply context-aware regex rules"""
|
||||||
|
changes = []
|
||||||
|
corrected = text
|
||||||
|
|
||||||
|
for rule in self.context_rules:
|
||||||
|
pattern = rule["pattern"]
|
||||||
|
replacement = rule["replacement"]
|
||||||
|
description = rule.get("description", "")
|
||||||
|
|
||||||
|
# Find all matches with their positions
|
||||||
|
for match in re.finditer(pattern, corrected):
|
||||||
|
line_num = corrected[:match.start()].count('\n') + 1
|
||||||
|
changes.append(Change(
|
||||||
|
line_number=line_num,
|
||||||
|
from_text=match.group(0),
|
||||||
|
to_text=replacement,
|
||||||
|
rule_type="context_rule",
|
||||||
|
rule_name=description or pattern
|
||||||
|
))
|
||||||
|
|
||||||
|
# Apply replacement
|
||||||
|
corrected = re.sub(pattern, replacement, corrected)
|
||||||
|
|
||||||
|
return corrected, changes
|
||||||
|
|
||||||
|
def _apply_dictionary(self, text: str) -> Tuple[str, List[Change]]:
|
||||||
|
"""Apply simple dictionary replacements"""
|
||||||
|
changes = []
|
||||||
|
corrected = text
|
||||||
|
|
||||||
|
for wrong, correct in self.corrections.items():
|
||||||
|
if wrong not in corrected:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Find all occurrences
|
||||||
|
occurrences = []
|
||||||
|
start = 0
|
||||||
|
while True:
|
||||||
|
pos = corrected.find(wrong, start)
|
||||||
|
if pos == -1:
|
||||||
|
break
|
||||||
|
line_num = corrected[:pos].count('\n') + 1
|
||||||
|
occurrences.append(line_num)
|
||||||
|
start = pos + len(wrong)
|
||||||
|
|
||||||
|
# Track changes
|
||||||
|
for line_num in occurrences:
|
||||||
|
changes.append(Change(
|
||||||
|
line_number=line_num,
|
||||||
|
from_text=wrong,
|
||||||
|
to_text=correct,
|
||||||
|
rule_type="dictionary",
|
||||||
|
rule_name="corrections_dict"
|
||||||
|
))
|
||||||
|
|
||||||
|
# Apply replacement
|
||||||
|
corrected = corrected.replace(wrong, correct)
|
||||||
|
|
||||||
|
return corrected, changes
|
||||||
|
|
||||||
|
def get_summary(self, changes: List[Change]) -> Dict[str, int]:
|
||||||
|
"""Generate summary statistics"""
|
||||||
|
summary = {
|
||||||
|
"total_changes": len(changes),
|
||||||
|
"dictionary_changes": sum(1 for c in changes if c.rule_type == "dictionary"),
|
||||||
|
"context_rule_changes": sum(1 for c in changes if c.rule_type == "context_rule")
|
||||||
|
}
|
||||||
|
return summary
|
||||||
252
transcript-fixer/scripts/core/learning_engine.py
Normal file
252
transcript-fixer/scripts/core/learning_engine.py
Normal file
@@ -0,0 +1,252 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Learning Engine - Pattern Detection from Correction History
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Analyze history and suggest new corrections
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Analyze correction history for patterns
|
||||||
|
- Detect frequently occurring corrections
|
||||||
|
- Calculate confidence scores
|
||||||
|
- Generate suggestions for user review
|
||||||
|
- Track rejected suggestions to avoid re-suggesting
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Dict
|
||||||
|
from dataclasses import dataclass, asdict
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Suggestion:
|
||||||
|
"""Represents a learned correction suggestion"""
|
||||||
|
from_text: str
|
||||||
|
to_text: str
|
||||||
|
frequency: int
|
||||||
|
confidence: float
|
||||||
|
examples: List[Dict] # List of {file, line, context}
|
||||||
|
first_seen: str
|
||||||
|
last_seen: str
|
||||||
|
status: str # "pending", "approved", "rejected"
|
||||||
|
|
||||||
|
|
||||||
|
class LearningEngine:
|
||||||
|
"""
|
||||||
|
Analyzes correction history to suggest new corrections
|
||||||
|
|
||||||
|
Algorithm:
|
||||||
|
1. Load all history files
|
||||||
|
2. Extract stage2 (AI) changes
|
||||||
|
3. Group by pattern (from_text → to_text)
|
||||||
|
4. Calculate frequency and confidence
|
||||||
|
5. Filter by thresholds
|
||||||
|
6. Save suggestions for user review
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Thresholds for suggesting corrections
|
||||||
|
MIN_FREQUENCY = 3 # Must appear at least 3 times
|
||||||
|
MIN_CONFIDENCE = 0.8 # Must have 80%+ confidence
|
||||||
|
|
||||||
|
def __init__(self, history_dir: Path, learned_dir: Path):
|
||||||
|
"""
|
||||||
|
Initialize learning engine
|
||||||
|
|
||||||
|
Args:
|
||||||
|
history_dir: Directory containing correction history
|
||||||
|
learned_dir: Directory for learned suggestions
|
||||||
|
"""
|
||||||
|
self.history_dir = history_dir
|
||||||
|
self.learned_dir = learned_dir
|
||||||
|
self.pending_file = learned_dir / "pending_review.json"
|
||||||
|
self.rejected_file = learned_dir / "rejected.json"
|
||||||
|
|
||||||
|
def analyze_and_suggest(self) -> List[Suggestion]:
|
||||||
|
"""
|
||||||
|
Analyze history and generate suggestions
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of new suggestions for user review
|
||||||
|
"""
|
||||||
|
# Load all history
|
||||||
|
patterns = self._extract_patterns()
|
||||||
|
|
||||||
|
# Filter rejected patterns
|
||||||
|
rejected = self._load_rejected()
|
||||||
|
patterns = {k: v for k, v in patterns.items()
|
||||||
|
if k not in rejected}
|
||||||
|
|
||||||
|
# Generate suggestions
|
||||||
|
suggestions = []
|
||||||
|
for (from_text, to_text), occurrences in patterns.items():
|
||||||
|
frequency = len(occurrences)
|
||||||
|
|
||||||
|
if frequency < self.MIN_FREQUENCY:
|
||||||
|
continue
|
||||||
|
|
||||||
|
confidence = self._calculate_confidence(occurrences)
|
||||||
|
|
||||||
|
if confidence < self.MIN_CONFIDENCE:
|
||||||
|
continue
|
||||||
|
|
||||||
|
suggestion = Suggestion(
|
||||||
|
from_text=from_text,
|
||||||
|
to_text=to_text,
|
||||||
|
frequency=frequency,
|
||||||
|
confidence=confidence,
|
||||||
|
examples=occurrences[:5], # Top 5 examples
|
||||||
|
first_seen=occurrences[0]["timestamp"],
|
||||||
|
last_seen=occurrences[-1]["timestamp"],
|
||||||
|
status="pending"
|
||||||
|
)
|
||||||
|
|
||||||
|
suggestions.append(suggestion)
|
||||||
|
|
||||||
|
# Save new suggestions
|
||||||
|
if suggestions:
|
||||||
|
self._save_pending_suggestions(suggestions)
|
||||||
|
|
||||||
|
return suggestions
|
||||||
|
|
||||||
|
def approve_suggestion(self, from_text: str) -> bool:
|
||||||
|
"""
|
||||||
|
Approve a suggestion (remove from pending)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if approved, False if not found
|
||||||
|
"""
|
||||||
|
pending = self._load_pending_suggestions()
|
||||||
|
|
||||||
|
for suggestion in pending:
|
||||||
|
if suggestion["from_text"] == from_text:
|
||||||
|
pending.remove(suggestion)
|
||||||
|
self._save_suggestions(pending, self.pending_file)
|
||||||
|
return True
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
def reject_suggestion(self, from_text: str, to_text: str) -> None:
|
||||||
|
"""
|
||||||
|
Reject a suggestion (move to rejected list)
|
||||||
|
"""
|
||||||
|
# Remove from pending
|
||||||
|
pending = self._load_pending_suggestions()
|
||||||
|
pending = [s for s in pending
|
||||||
|
if not (s["from_text"] == from_text and s["to_text"] == to_text)]
|
||||||
|
self._save_suggestions(pending, self.pending_file)
|
||||||
|
|
||||||
|
# Add to rejected
|
||||||
|
rejected = self._load_rejected()
|
||||||
|
rejected.add((from_text, to_text))
|
||||||
|
self._save_rejected(rejected)
|
||||||
|
|
||||||
|
def list_pending(self) -> List[Dict]:
|
||||||
|
"""List all pending suggestions"""
|
||||||
|
return self._load_pending_suggestions()
|
||||||
|
|
||||||
|
def _extract_patterns(self) -> Dict[tuple, List[Dict]]:
|
||||||
|
"""Extract all correction patterns from history"""
|
||||||
|
patterns = defaultdict(list)
|
||||||
|
|
||||||
|
if not self.history_dir.exists():
|
||||||
|
return patterns
|
||||||
|
|
||||||
|
for history_file in self.history_dir.glob("*.json"):
|
||||||
|
with open(history_file, 'r', encoding='utf-8') as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
# Extract stage2 changes (AI corrections)
|
||||||
|
if "stages" in data and "stage2" in data["stages"]:
|
||||||
|
changes = data["stages"]["stage2"].get("changes", [])
|
||||||
|
|
||||||
|
for change in changes:
|
||||||
|
key = (change["from"], change["to"])
|
||||||
|
patterns[key].append({
|
||||||
|
"file": data["filename"],
|
||||||
|
"line": change.get("line", 0),
|
||||||
|
"context": change.get("context", ""),
|
||||||
|
"timestamp": data["timestamp"]
|
||||||
|
})
|
||||||
|
|
||||||
|
return patterns
|
||||||
|
|
||||||
|
def _calculate_confidence(self, occurrences: List[Dict]) -> float:
|
||||||
|
"""
|
||||||
|
Calculate confidence score for a pattern
|
||||||
|
|
||||||
|
Factors:
|
||||||
|
- Frequency (more = higher)
|
||||||
|
- Consistency (always same correction = higher)
|
||||||
|
- Recency (recent occurrences = higher)
|
||||||
|
"""
|
||||||
|
# Base confidence from frequency
|
||||||
|
frequency_score = min(len(occurrences) / 10.0, 1.0)
|
||||||
|
|
||||||
|
# Consistency: always the same from→to mapping
|
||||||
|
consistency_score = 1.0 # Already consistent by grouping
|
||||||
|
|
||||||
|
# Recency: more recent = higher
|
||||||
|
# (Simplified: assume chronological order)
|
||||||
|
recency_score = 0.9 if len(occurrences) > 1 else 0.8
|
||||||
|
|
||||||
|
# Weighted average
|
||||||
|
confidence = (
|
||||||
|
0.5 * frequency_score +
|
||||||
|
0.3 * consistency_score +
|
||||||
|
0.2 * recency_score
|
||||||
|
)
|
||||||
|
|
||||||
|
return confidence
|
||||||
|
|
||||||
|
def _load_pending_suggestions(self) -> List[Dict]:
|
||||||
|
"""Load pending suggestions from file"""
|
||||||
|
if not self.pending_file.exists():
|
||||||
|
return []
|
||||||
|
|
||||||
|
with open(self.pending_file, 'r', encoding='utf-8') as f:
|
||||||
|
content = f.read().strip()
|
||||||
|
if not content:
|
||||||
|
return []
|
||||||
|
return json.loads(content).get("suggestions", [])
|
||||||
|
|
||||||
|
def _save_pending_suggestions(self, suggestions: List[Suggestion]) -> None:
|
||||||
|
"""Save pending suggestions to file"""
|
||||||
|
existing = self._load_pending_suggestions()
|
||||||
|
|
||||||
|
# Convert to dict and append
|
||||||
|
new_suggestions = [asdict(s) for s in suggestions]
|
||||||
|
all_suggestions = existing + new_suggestions
|
||||||
|
|
||||||
|
self._save_suggestions(all_suggestions, self.pending_file)
|
||||||
|
|
||||||
|
def _save_suggestions(self, suggestions: List[Dict], filepath: Path) -> None:
|
||||||
|
"""Save suggestions to file"""
|
||||||
|
data = {"suggestions": suggestions}
|
||||||
|
with open(filepath, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
def _load_rejected(self) -> set:
|
||||||
|
"""Load rejected patterns"""
|
||||||
|
if not self.rejected_file.exists():
|
||||||
|
return set()
|
||||||
|
|
||||||
|
with open(self.rejected_file, 'r', encoding='utf-8') as f:
|
||||||
|
content = f.read().strip()
|
||||||
|
if not content:
|
||||||
|
return set()
|
||||||
|
data = json.loads(content)
|
||||||
|
return {(r["from"], r["to"]) for r in data.get("rejected", [])}
|
||||||
|
|
||||||
|
def _save_rejected(self, rejected: set) -> None:
|
||||||
|
"""Save rejected patterns"""
|
||||||
|
data = {
|
||||||
|
"rejected": [
|
||||||
|
{"from": from_text, "to": to_text}
|
||||||
|
for from_text, to_text in rejected
|
||||||
|
]
|
||||||
|
}
|
||||||
|
with open(self.rejected_file, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||||
215
transcript-fixer/scripts/core/schema.sql
Normal file
215
transcript-fixer/scripts/core/schema.sql
Normal file
@@ -0,0 +1,215 @@
|
|||||||
|
-- Transcript Fixer Database Schema v2.0
|
||||||
|
-- Migration from JSON to SQLite for ACID compliance and scalability
|
||||||
|
-- Author: ISTJ Chief Engineer
|
||||||
|
-- Date: 2025-01-28
|
||||||
|
|
||||||
|
-- Enable foreign keys
|
||||||
|
PRAGMA foreign_keys = ON;
|
||||||
|
|
||||||
|
-- Table: corrections
|
||||||
|
-- Stores all correction mappings with metadata
|
||||||
|
CREATE TABLE IF NOT EXISTS corrections (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
from_text TEXT NOT NULL,
|
||||||
|
to_text TEXT NOT NULL,
|
||||||
|
domain TEXT NOT NULL DEFAULT 'general',
|
||||||
|
source TEXT NOT NULL CHECK(source IN ('manual', 'learned', 'imported')),
|
||||||
|
confidence REAL NOT NULL DEFAULT 1.0 CHECK(confidence >= 0.0 AND confidence <= 1.0),
|
||||||
|
added_by TEXT,
|
||||||
|
added_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
usage_count INTEGER NOT NULL DEFAULT 0 CHECK(usage_count >= 0),
|
||||||
|
last_used TIMESTAMP,
|
||||||
|
notes TEXT,
|
||||||
|
is_active BOOLEAN NOT NULL DEFAULT 1,
|
||||||
|
UNIQUE(from_text, domain)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_corrections_domain ON corrections(domain);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_corrections_source ON corrections(source);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_corrections_added_at ON corrections(added_at);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_corrections_is_active ON corrections(is_active);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_corrections_from_text ON corrections(from_text);
|
||||||
|
|
||||||
|
-- Table: context_rules
|
||||||
|
-- Regex-based context-aware correction rules
|
||||||
|
CREATE TABLE IF NOT EXISTS context_rules (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
pattern TEXT NOT NULL UNIQUE,
|
||||||
|
replacement TEXT NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
priority INTEGER NOT NULL DEFAULT 0,
|
||||||
|
is_active BOOLEAN NOT NULL DEFAULT 1,
|
||||||
|
added_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
added_by TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_context_rules_priority ON context_rules(priority DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_context_rules_is_active ON context_rules(is_active);
|
||||||
|
|
||||||
|
-- Table: correction_history
|
||||||
|
-- Audit log for all correction runs
|
||||||
|
CREATE TABLE IF NOT EXISTS correction_history (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
filename TEXT NOT NULL,
|
||||||
|
domain TEXT NOT NULL,
|
||||||
|
run_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
original_length INTEGER NOT NULL CHECK(original_length >= 0),
|
||||||
|
stage1_changes INTEGER NOT NULL DEFAULT 0 CHECK(stage1_changes >= 0),
|
||||||
|
stage2_changes INTEGER NOT NULL DEFAULT 0 CHECK(stage2_changes >= 0),
|
||||||
|
model TEXT,
|
||||||
|
execution_time_ms INTEGER CHECK(execution_time_ms >= 0),
|
||||||
|
success BOOLEAN NOT NULL DEFAULT 1,
|
||||||
|
error_message TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_history_run_timestamp ON correction_history(run_timestamp DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_history_domain ON correction_history(domain);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_history_success ON correction_history(success);
|
||||||
|
|
||||||
|
-- Table: correction_changes
|
||||||
|
-- Detailed changes made in each correction run
|
||||||
|
CREATE TABLE IF NOT EXISTS correction_changes (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
history_id INTEGER NOT NULL,
|
||||||
|
line_number INTEGER,
|
||||||
|
from_text TEXT NOT NULL,
|
||||||
|
to_text TEXT NOT NULL,
|
||||||
|
rule_type TEXT NOT NULL CHECK(rule_type IN ('context', 'dictionary', 'ai')),
|
||||||
|
rule_id INTEGER,
|
||||||
|
context_before TEXT,
|
||||||
|
context_after TEXT,
|
||||||
|
FOREIGN KEY (history_id) REFERENCES correction_history(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_changes_history_id ON correction_changes(history_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_changes_rule_type ON correction_changes(rule_type);
|
||||||
|
|
||||||
|
-- Table: learned_suggestions
|
||||||
|
-- AI-learned patterns pending user review
|
||||||
|
CREATE TABLE IF NOT EXISTS learned_suggestions (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
from_text TEXT NOT NULL,
|
||||||
|
to_text TEXT NOT NULL,
|
||||||
|
domain TEXT NOT NULL DEFAULT 'general',
|
||||||
|
frequency INTEGER NOT NULL DEFAULT 1 CHECK(frequency > 0),
|
||||||
|
confidence REAL NOT NULL CHECK(confidence >= 0.0 AND confidence <= 1.0),
|
||||||
|
first_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
last_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'approved', 'rejected')),
|
||||||
|
reviewed_at TIMESTAMP,
|
||||||
|
reviewed_by TEXT,
|
||||||
|
UNIQUE(from_text, to_text, domain)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_suggestions_status ON learned_suggestions(status);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_suggestions_domain ON learned_suggestions(domain);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_suggestions_confidence ON learned_suggestions(confidence DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_suggestions_frequency ON learned_suggestions(frequency DESC);
|
||||||
|
|
||||||
|
-- Table: suggestion_examples
|
||||||
|
-- Example occurrences of learned patterns
|
||||||
|
CREATE TABLE IF NOT EXISTS suggestion_examples (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
suggestion_id INTEGER NOT NULL,
|
||||||
|
filename TEXT NOT NULL,
|
||||||
|
line_number INTEGER,
|
||||||
|
context TEXT NOT NULL,
|
||||||
|
occurred_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (suggestion_id) REFERENCES learned_suggestions(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_examples_suggestion_id ON suggestion_examples(suggestion_id);
|
||||||
|
|
||||||
|
-- Table: system_config
|
||||||
|
-- System configuration and preferences
|
||||||
|
CREATE TABLE IF NOT EXISTS system_config (
|
||||||
|
key TEXT PRIMARY KEY,
|
||||||
|
value TEXT NOT NULL,
|
||||||
|
value_type TEXT NOT NULL CHECK(value_type IN ('string', 'int', 'float', 'boolean', 'json')),
|
||||||
|
description TEXT,
|
||||||
|
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Insert default configuration
|
||||||
|
INSERT OR IGNORE INTO system_config (key, value, value_type, description) VALUES
|
||||||
|
('schema_version', '2.0', 'string', 'Database schema version'),
|
||||||
|
('api_provider', 'GLM', 'string', 'API provider name'),
|
||||||
|
('api_model', 'GLM-4.6', 'string', 'Default AI model'),
|
||||||
|
('api_base_url', 'https://open.bigmodel.cn/api/anthropic', 'string', 'API endpoint URL'),
|
||||||
|
('default_domain', 'general', 'string', 'Default correction domain'),
|
||||||
|
('auto_learn_enabled', 'true', 'boolean', 'Enable automatic pattern learning'),
|
||||||
|
('backup_enabled', 'true', 'boolean', 'Create backups before operations'),
|
||||||
|
('learning_frequency_threshold', '3', 'int', 'Min frequency for learned suggestions'),
|
||||||
|
('learning_confidence_threshold', '0.8', 'float', 'Min confidence for learned suggestions'),
|
||||||
|
('history_retention_days', '90', 'int', 'Days to retain correction history'),
|
||||||
|
('max_correction_length', '1000', 'int', 'Maximum length for correction text');
|
||||||
|
|
||||||
|
-- Table: audit_log
|
||||||
|
-- Comprehensive audit trail for all operations
|
||||||
|
CREATE TABLE IF NOT EXISTS audit_log (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
action TEXT NOT NULL,
|
||||||
|
entity_type TEXT NOT NULL,
|
||||||
|
entity_id INTEGER,
|
||||||
|
user TEXT,
|
||||||
|
details TEXT,
|
||||||
|
success BOOLEAN NOT NULL DEFAULT 1,
|
||||||
|
error_message TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_audit_timestamp ON audit_log(timestamp DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_log(action);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_audit_entity_type ON audit_log(entity_type);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_audit_success ON audit_log(success);
|
||||||
|
|
||||||
|
-- View: active_corrections
|
||||||
|
-- Quick access to active corrections
|
||||||
|
CREATE VIEW IF NOT EXISTS active_corrections AS
|
||||||
|
SELECT
|
||||||
|
id,
|
||||||
|
from_text,
|
||||||
|
to_text,
|
||||||
|
domain,
|
||||||
|
source,
|
||||||
|
confidence,
|
||||||
|
usage_count,
|
||||||
|
last_used,
|
||||||
|
added_at
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
ORDER BY domain, from_text;
|
||||||
|
|
||||||
|
-- View: pending_suggestions
|
||||||
|
-- Quick access to suggestions pending review
|
||||||
|
CREATE VIEW IF NOT EXISTS pending_suggestions AS
|
||||||
|
SELECT
|
||||||
|
s.id,
|
||||||
|
s.from_text,
|
||||||
|
s.to_text,
|
||||||
|
s.domain,
|
||||||
|
s.frequency,
|
||||||
|
s.confidence,
|
||||||
|
s.first_seen,
|
||||||
|
s.last_seen,
|
||||||
|
COUNT(e.id) as example_count
|
||||||
|
FROM learned_suggestions s
|
||||||
|
LEFT JOIN suggestion_examples e ON s.id = e.suggestion_id
|
||||||
|
WHERE s.status = 'pending'
|
||||||
|
GROUP BY s.id
|
||||||
|
ORDER BY s.confidence DESC, s.frequency DESC;
|
||||||
|
|
||||||
|
-- View: correction_statistics
|
||||||
|
-- Statistics per domain
|
||||||
|
CREATE VIEW IF NOT EXISTS correction_statistics AS
|
||||||
|
SELECT
|
||||||
|
domain,
|
||||||
|
COUNT(*) as total_corrections,
|
||||||
|
COUNT(CASE WHEN source = 'manual' THEN 1 END) as manual_count,
|
||||||
|
COUNT(CASE WHEN source = 'learned' THEN 1 END) as learned_count,
|
||||||
|
COUNT(CASE WHEN source = 'imported' THEN 1 END) as imported_count,
|
||||||
|
SUM(usage_count) as total_usage,
|
||||||
|
MAX(added_at) as last_updated
|
||||||
|
FROM corrections
|
||||||
|
WHERE is_active = 1
|
||||||
|
GROUP BY domain;
|
||||||
153
transcript-fixer/scripts/examples/bulk_import.py
Normal file
153
transcript-fixer/scripts/examples/bulk_import.py
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Example: Bulk Import Corrections to SQLite Database
|
||||||
|
|
||||||
|
This script demonstrates how to import corrections from various sources
|
||||||
|
into the transcript-fixer SQLite database.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run scripts/examples/bulk_import.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from core import CorrectionRepository, CorrectionService
|
||||||
|
|
||||||
|
|
||||||
|
def import_from_dict():
|
||||||
|
"""Example: Import corrections from Python dictionary"""
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
db_path = Path.home() / ".transcript-fixer" / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository)
|
||||||
|
|
||||||
|
# Define corrections as dictionary
|
||||||
|
corrections_dict = {
|
||||||
|
"巨升智能": "具身智能",
|
||||||
|
"巨升": "具身",
|
||||||
|
"奇迹创坛": "奇绩创坛",
|
||||||
|
"火星营": "火星营",
|
||||||
|
"矩阵公司": "初创公司",
|
||||||
|
"股价": "框架",
|
||||||
|
"三观": "三关"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Convert to list format for import
|
||||||
|
corrections_list = []
|
||||||
|
for from_text, to_text in corrections_dict.items():
|
||||||
|
corrections_list.append({
|
||||||
|
"from_text": from_text,
|
||||||
|
"to_text": to_text,
|
||||||
|
"domain": "embodied_ai",
|
||||||
|
"source": "imported",
|
||||||
|
"confidence": 1.0
|
||||||
|
})
|
||||||
|
|
||||||
|
# Import
|
||||||
|
inserted, updated, skipped = service.import_corrections(
|
||||||
|
corrections=corrections_list,
|
||||||
|
merge=True
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"✅ Import complete:")
|
||||||
|
print(f" - Inserted: {inserted}")
|
||||||
|
print(f" - Updated: {updated}")
|
||||||
|
print(f" - Skipped: {skipped}")
|
||||||
|
|
||||||
|
service.close()
|
||||||
|
|
||||||
|
|
||||||
|
def import_from_json_file():
|
||||||
|
"""Example: Import from old JSON format file"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
# Sample JSON structure (v1.0 format)
|
||||||
|
sample_json = {
|
||||||
|
"metadata": {
|
||||||
|
"version": "1.0",
|
||||||
|
"domains": ["embodied_ai"],
|
||||||
|
},
|
||||||
|
"corrections": {
|
||||||
|
"巨升智能": "具身智能",
|
||||||
|
"巨升": "具身",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Initialize service
|
||||||
|
db_path = Path.home() / ".transcript-fixer" / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository)
|
||||||
|
|
||||||
|
# Convert JSON to import format
|
||||||
|
domain = sample_json["metadata"].get("domains", ["general"])[0]
|
||||||
|
corrections_list = []
|
||||||
|
|
||||||
|
for from_text, to_text in sample_json["corrections"].items():
|
||||||
|
corrections_list.append({
|
||||||
|
"from_text": from_text,
|
||||||
|
"to_text": to_text,
|
||||||
|
"domain": domain,
|
||||||
|
"source": "imported",
|
||||||
|
"confidence": 1.0
|
||||||
|
})
|
||||||
|
|
||||||
|
# Import
|
||||||
|
inserted, updated, skipped = service.import_corrections(
|
||||||
|
corrections=corrections_list,
|
||||||
|
merge=True
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"✅ JSON import complete:")
|
||||||
|
print(f" - Inserted: {inserted}")
|
||||||
|
print(f" - Updated: {updated}")
|
||||||
|
print(f" - Skipped: {skipped}")
|
||||||
|
|
||||||
|
service.close()
|
||||||
|
|
||||||
|
|
||||||
|
def add_context_rules():
|
||||||
|
"""Example: Add context-aware regex rules directly"""
|
||||||
|
|
||||||
|
db_path = Path.home() / ".transcript-fixer" / "corrections.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
|
||||||
|
# Add context rules via SQL
|
||||||
|
with repository._transaction() as conn:
|
||||||
|
rules = [
|
||||||
|
("巨升方向", "具身方向", "巨升→具身", 10),
|
||||||
|
("巨升现在", "具身现在", "巨升→具身", 10),
|
||||||
|
("近距离的去看", "近距离地去看", "的→地 副词修饰", 5),
|
||||||
|
("近距离搏杀", "近距离搏杀", "这里的'近距离'是正确的", 5),
|
||||||
|
]
|
||||||
|
|
||||||
|
for pattern, replacement, description, priority in rules:
|
||||||
|
conn.execute("""
|
||||||
|
INSERT OR IGNORE INTO context_rules
|
||||||
|
(pattern, replacement, description, priority)
|
||||||
|
VALUES (?, ?, ?, ?)
|
||||||
|
""", (pattern, replacement, description, priority))
|
||||||
|
|
||||||
|
print("✅ Context rules added successfully")
|
||||||
|
repository.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print("Transcript-Fixer Bulk Import Examples\n")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
# Example 1: Import from dictionary
|
||||||
|
print("\n1. Importing from Python dictionary...")
|
||||||
|
import_from_dict()
|
||||||
|
|
||||||
|
# Example 2: Import from JSON file
|
||||||
|
print("\n2. Importing from JSON format...")
|
||||||
|
import_from_json_file()
|
||||||
|
|
||||||
|
# Example 3: Add context rules
|
||||||
|
print("\n3. Adding context rules...")
|
||||||
|
add_context_rules()
|
||||||
|
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
print("✅ All examples completed!")
|
||||||
|
print("\nVerify with:")
|
||||||
|
print(" sqlite3 ~/.transcript-fixer/corrections.db 'SELECT COUNT(*) FROM active_corrections;'")
|
||||||
70
transcript-fixer/scripts/fix_transcription.py
Executable file
70
transcript-fixer/scripts/fix_transcription.py
Executable file
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Transcript Fixer - Main Entry Point
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Route CLI commands to handlers
|
||||||
|
|
||||||
|
This is the main entry point for the transcript-fixer tool.
|
||||||
|
It parses arguments and dispatches to appropriate command handlers.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
# Setup
|
||||||
|
python fix_transcription.py --init
|
||||||
|
|
||||||
|
# Correction workflow
|
||||||
|
python fix_transcription.py --input file.md --stage 3
|
||||||
|
|
||||||
|
# Manage corrections
|
||||||
|
python fix_transcription.py --add "错误" "正确"
|
||||||
|
python fix_transcription.py --list
|
||||||
|
|
||||||
|
# Review learned suggestions
|
||||||
|
python fix_transcription.py --review-learned
|
||||||
|
python fix_transcription.py --approve "错误" "正确"
|
||||||
|
|
||||||
|
# Validate configuration
|
||||||
|
python fix_transcription.py --validate
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from cli import (
|
||||||
|
cmd_init,
|
||||||
|
cmd_add_correction,
|
||||||
|
cmd_list_corrections,
|
||||||
|
cmd_run_correction,
|
||||||
|
cmd_review_learned,
|
||||||
|
cmd_approve,
|
||||||
|
cmd_validate,
|
||||||
|
create_argument_parser,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point - parse arguments and dispatch to commands"""
|
||||||
|
parser = create_argument_parser()
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Dispatch commands
|
||||||
|
if args.init:
|
||||||
|
cmd_init(args)
|
||||||
|
elif args.validate:
|
||||||
|
cmd_validate(args)
|
||||||
|
elif args.add_correction:
|
||||||
|
args.from_text, args.to_text = args.add_correction
|
||||||
|
cmd_add_correction(args)
|
||||||
|
elif args.list_corrections:
|
||||||
|
cmd_list_corrections(args)
|
||||||
|
elif args.review_learned:
|
||||||
|
cmd_review_learned(args)
|
||||||
|
elif args.approve:
|
||||||
|
args.from_text, args.to_text = args.approve
|
||||||
|
cmd_approve(args)
|
||||||
|
elif args.input:
|
||||||
|
cmd_run_correction(args)
|
||||||
|
else:
|
||||||
|
parser.print_help()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
3
transcript-fixer/scripts/tests/__init__.py
Normal file
3
transcript-fixer/scripts/tests/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
"""
|
||||||
|
Test suite for transcript-fixer
|
||||||
|
"""
|
||||||
272
transcript-fixer/scripts/tests/test_correction_service.py
Normal file
272
transcript-fixer/scripts/tests/test_correction_service.py
Normal file
@@ -0,0 +1,272 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Unit Tests for Correction Service
|
||||||
|
|
||||||
|
Tests business logic, validation, and service layer functionality.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
import tempfile
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# Add parent directory to path
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from core.correction_repository import CorrectionRepository
|
||||||
|
from core.correction_service import CorrectionService, ValidationError
|
||||||
|
|
||||||
|
|
||||||
|
class TestCorrectionService(unittest.TestCase):
|
||||||
|
"""Test suite for CorrectionService"""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
"""Create temporary database for each test."""
|
||||||
|
self.test_dir = Path(tempfile.mkdtemp())
|
||||||
|
self.db_path = self.test_dir / "test.db"
|
||||||
|
self.repository = CorrectionRepository(self.db_path)
|
||||||
|
self.service = CorrectionService(self.repository)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
"""Clean up temporary files."""
|
||||||
|
self.service.close()
|
||||||
|
shutil.rmtree(self.test_dir)
|
||||||
|
|
||||||
|
# ==================== Validation Tests ====================
|
||||||
|
|
||||||
|
def test_validate_empty_text(self):
|
||||||
|
"""Test rejection of empty text."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_correction_text("", "test_field")
|
||||||
|
|
||||||
|
def test_validate_whitespace_only(self):
|
||||||
|
"""Test rejection of whitespace-only text."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_correction_text(" ", "test_field")
|
||||||
|
|
||||||
|
def test_validate_too_long(self):
|
||||||
|
"""Test rejection of text exceeding max length."""
|
||||||
|
long_text = "A" * 1001
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_correction_text(long_text, "test_field")
|
||||||
|
|
||||||
|
def test_validate_control_characters(self):
|
||||||
|
"""Test rejection of control characters."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_correction_text("test\x00text", "test_field")
|
||||||
|
|
||||||
|
def test_validate_valid_text(self):
|
||||||
|
"""Test acceptance of valid text."""
|
||||||
|
# Should not raise
|
||||||
|
self.service.validate_correction_text("valid text", "test_field")
|
||||||
|
self.service.validate_correction_text("有效文本", "test_field")
|
||||||
|
|
||||||
|
def test_validate_domain_path_traversal(self):
|
||||||
|
"""Test rejection of path traversal in domain."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_domain_name("../etc/passwd")
|
||||||
|
|
||||||
|
def test_validate_domain_invalid_chars(self):
|
||||||
|
"""Test rejection of invalid characters in domain."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_domain_name("invalid/domain")
|
||||||
|
|
||||||
|
def test_validate_domain_reserved(self):
|
||||||
|
"""Test rejection of reserved domain names."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.validate_domain_name("con") # Windows reserved
|
||||||
|
|
||||||
|
def test_validate_valid_domain(self):
|
||||||
|
"""Test acceptance of valid domain."""
|
||||||
|
# Should not raise
|
||||||
|
self.service.validate_domain_name("general")
|
||||||
|
self.service.validate_domain_name("embodied_ai")
|
||||||
|
self.service.validate_domain_name("test-domain-123")
|
||||||
|
|
||||||
|
# ==================== Correction Operations Tests ====================
|
||||||
|
|
||||||
|
def test_add_correction(self):
|
||||||
|
"""Test adding a correction."""
|
||||||
|
correction_id = self.service.add_correction(
|
||||||
|
from_text="错误",
|
||||||
|
to_text="正确",
|
||||||
|
domain="general"
|
||||||
|
)
|
||||||
|
self.assertIsInstance(correction_id, int)
|
||||||
|
self.assertGreater(correction_id, 0)
|
||||||
|
|
||||||
|
# Verify it was added
|
||||||
|
corrections = self.service.get_corrections("general")
|
||||||
|
self.assertEqual(corrections["错误"], "正确")
|
||||||
|
|
||||||
|
def test_add_identical_correction_rejected(self):
|
||||||
|
"""Test rejection of from_text == to_text."""
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
self.service.add_correction(
|
||||||
|
from_text="same",
|
||||||
|
to_text="same",
|
||||||
|
domain="general"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_add_duplicate_correction_updates(self):
|
||||||
|
"""Test that duplicate from_text updates existing."""
|
||||||
|
# Add first
|
||||||
|
self.service.add_correction("错误", "正确A", "general")
|
||||||
|
|
||||||
|
# Add duplicate (should update)
|
||||||
|
self.service.add_correction("错误", "正确B", "general")
|
||||||
|
|
||||||
|
# Verify updated
|
||||||
|
corrections = self.service.get_corrections("general")
|
||||||
|
self.assertEqual(corrections["错误"], "正确B")
|
||||||
|
|
||||||
|
def test_get_corrections_multiple_domains(self):
|
||||||
|
"""Test getting corrections from different domains."""
|
||||||
|
self.service.add_correction("test1", "result1", "domain1")
|
||||||
|
self.service.add_correction("test2", "result2", "domain2")
|
||||||
|
|
||||||
|
domain1_corr = self.service.get_corrections("domain1")
|
||||||
|
domain2_corr = self.service.get_corrections("domain2")
|
||||||
|
|
||||||
|
self.assertEqual(len(domain1_corr), 1)
|
||||||
|
self.assertEqual(len(domain2_corr), 1)
|
||||||
|
self.assertEqual(domain1_corr["test1"], "result1")
|
||||||
|
self.assertEqual(domain2_corr["test2"], "result2")
|
||||||
|
|
||||||
|
def test_remove_correction(self):
|
||||||
|
"""Test removing a correction."""
|
||||||
|
# Add correction
|
||||||
|
self.service.add_correction("错误", "正确", "general")
|
||||||
|
|
||||||
|
# Remove it
|
||||||
|
success = self.service.remove_correction("错误", "general")
|
||||||
|
self.assertTrue(success)
|
||||||
|
|
||||||
|
# Verify removed
|
||||||
|
corrections = self.service.get_corrections("general")
|
||||||
|
self.assertNotIn("错误", corrections)
|
||||||
|
|
||||||
|
def test_remove_nonexistent_correction(self):
|
||||||
|
"""Test removing non-existent correction."""
|
||||||
|
success = self.service.remove_correction("nonexistent", "general")
|
||||||
|
self.assertFalse(success)
|
||||||
|
|
||||||
|
# ==================== Import/Export Tests ====================
|
||||||
|
|
||||||
|
def test_import_corrections(self):
|
||||||
|
"""Test importing corrections."""
|
||||||
|
import_data = {
|
||||||
|
"错误1": "正确1",
|
||||||
|
"错误2": "正确2",
|
||||||
|
"错误3": "正确3"
|
||||||
|
}
|
||||||
|
|
||||||
|
inserted, updated, skipped = self.service.import_corrections(
|
||||||
|
corrections=import_data,
|
||||||
|
domain="test_domain",
|
||||||
|
merge=True
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(inserted, 3)
|
||||||
|
self.assertEqual(updated, 0)
|
||||||
|
self.assertEqual(skipped, 0)
|
||||||
|
|
||||||
|
# Verify imported
|
||||||
|
corrections = self.service.get_corrections("test_domain")
|
||||||
|
self.assertEqual(len(corrections), 3)
|
||||||
|
|
||||||
|
def test_import_merge_with_conflicts(self):
|
||||||
|
"""Test import with merge mode and conflicts."""
|
||||||
|
# Add existing correction
|
||||||
|
self.service.add_correction("错误", "旧值", "test_domain")
|
||||||
|
|
||||||
|
# Import with conflict
|
||||||
|
import_data = {
|
||||||
|
"错误": "新值",
|
||||||
|
"新错误": "新正确"
|
||||||
|
}
|
||||||
|
|
||||||
|
inserted, updated, skipped = self.service.import_corrections(
|
||||||
|
corrections=import_data,
|
||||||
|
domain="test_domain",
|
||||||
|
merge=True
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(inserted, 1) # "新错误"
|
||||||
|
self.assertEqual(updated, 1) # "错误" updated
|
||||||
|
|
||||||
|
# Verify updated
|
||||||
|
corrections = self.service.get_corrections("test_domain")
|
||||||
|
self.assertEqual(corrections["错误"], "新值")
|
||||||
|
self.assertEqual(corrections["新错误"], "新正确")
|
||||||
|
|
||||||
|
def test_export_corrections(self):
|
||||||
|
"""Test exporting corrections."""
|
||||||
|
# Add some corrections
|
||||||
|
self.service.add_correction("错误1", "正确1", "export_test")
|
||||||
|
self.service.add_correction("错误2", "正确2", "export_test")
|
||||||
|
|
||||||
|
# Export
|
||||||
|
exported = self.service.export_corrections("export_test")
|
||||||
|
|
||||||
|
self.assertEqual(len(exported), 2)
|
||||||
|
self.assertEqual(exported["错误1"], "正确1")
|
||||||
|
self.assertEqual(exported["错误2"], "正确2")
|
||||||
|
|
||||||
|
# ==================== Statistics Tests ====================
|
||||||
|
|
||||||
|
def test_get_statistics_empty(self):
|
||||||
|
"""Test statistics for empty domain."""
|
||||||
|
stats = self.service.get_statistics("empty_domain")
|
||||||
|
|
||||||
|
self.assertEqual(stats['total_corrections'], 0)
|
||||||
|
self.assertEqual(stats['total_usage'], 0)
|
||||||
|
|
||||||
|
def test_get_statistics(self):
|
||||||
|
"""Test statistics calculation."""
|
||||||
|
# Add corrections with different sources
|
||||||
|
self.service.add_correction("test1", "result1", "stats_test", source="manual")
|
||||||
|
self.service.add_correction("test2", "result2", "stats_test", source="learned")
|
||||||
|
self.service.add_correction("test3", "result3", "stats_test", source="imported")
|
||||||
|
|
||||||
|
stats = self.service.get_statistics("stats_test")
|
||||||
|
|
||||||
|
self.assertEqual(stats['total_corrections'], 3)
|
||||||
|
self.assertEqual(stats['by_source']['manual'], 1)
|
||||||
|
self.assertEqual(stats['by_source']['learned'], 1)
|
||||||
|
self.assertEqual(stats['by_source']['imported'], 1)
|
||||||
|
|
||||||
|
|
||||||
|
class TestValidationRules(unittest.TestCase):
|
||||||
|
"""Test validation rules configuration."""
|
||||||
|
|
||||||
|
def test_custom_validation_rules(self):
|
||||||
|
"""Test service with custom validation rules."""
|
||||||
|
from core.correction_service import ValidationRules
|
||||||
|
|
||||||
|
custom_rules = ValidationRules(
|
||||||
|
max_text_length=100,
|
||||||
|
min_text_length=3
|
||||||
|
)
|
||||||
|
|
||||||
|
test_dir = Path(tempfile.mkdtemp())
|
||||||
|
db_path = test_dir / "test.db"
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository, rules=custom_rules)
|
||||||
|
|
||||||
|
# Should reject short text
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
service.validate_correction_text("ab", "test") # Too short
|
||||||
|
|
||||||
|
# Should reject long text
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
service.validate_correction_text("A" * 101, "test") # Too long
|
||||||
|
|
||||||
|
# Clean up
|
||||||
|
service.close()
|
||||||
|
shutil.rmtree(test_dir)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
16
transcript-fixer/scripts/utils/__init__.py
Normal file
16
transcript-fixer/scripts/utils/__init__.py
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
"""
|
||||||
|
Utils Module - Utility Functions and Tools
|
||||||
|
|
||||||
|
This module contains utility functions:
|
||||||
|
- diff_generator: Multi-format diff report generation
|
||||||
|
- validation: Configuration validation
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .diff_generator import generate_full_report
|
||||||
|
from .validation import validate_configuration, print_validation_summary
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'generate_full_report',
|
||||||
|
'validate_configuration',
|
||||||
|
'print_validation_summary',
|
||||||
|
]
|
||||||
18
transcript-fixer/scripts/utils/diff_formats/__init__.py
Normal file
18
transcript-fixer/scripts/utils/diff_formats/__init__.py
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
"""
|
||||||
|
Diff format generators for transcript comparison
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .unified_format import generate_unified_diff
|
||||||
|
from .html_format import generate_html_diff
|
||||||
|
from .inline_format import generate_inline_diff
|
||||||
|
from .markdown_format import generate_markdown_report
|
||||||
|
from .change_extractor import extract_changes, generate_change_summary
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'generate_unified_diff',
|
||||||
|
'generate_html_diff',
|
||||||
|
'generate_inline_diff',
|
||||||
|
'generate_markdown_report',
|
||||||
|
'extract_changes',
|
||||||
|
'generate_change_summary',
|
||||||
|
]
|
||||||
102
transcript-fixer/scripts/utils/diff_formats/change_extractor.py
Normal file
102
transcript-fixer/scripts/utils/diff_formats/change_extractor.py
Normal file
@@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Change extraction and summarization
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Extract and summarize changes between text versions
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import difflib
|
||||||
|
|
||||||
|
from .text_splitter import split_into_words
|
||||||
|
|
||||||
|
|
||||||
|
def extract_changes(original: str, fixed: str) -> list[dict]:
|
||||||
|
"""
|
||||||
|
Extract all changes and return change list
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original: Original text
|
||||||
|
fixed: Fixed text
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of change dictionaries with type, context, and content
|
||||||
|
"""
|
||||||
|
original_words = split_into_words(original)
|
||||||
|
fixed_words = split_into_words(fixed)
|
||||||
|
|
||||||
|
diff = difflib.SequenceMatcher(None, original_words, fixed_words)
|
||||||
|
changes = []
|
||||||
|
|
||||||
|
for tag, i1, i2, j1, j2 in diff.get_opcodes():
|
||||||
|
if tag == 'replace':
|
||||||
|
original_text = ''.join(original_words[i1:i2])
|
||||||
|
fixed_text = ''.join(fixed_words[j1:j2])
|
||||||
|
changes.append({
|
||||||
|
'type': 'replace',
|
||||||
|
'original': original_text,
|
||||||
|
'fixed': fixed_text,
|
||||||
|
'context_before': ''.join(original_words[max(0, i1-5):i1]),
|
||||||
|
'context_after': ''.join(original_words[i2:min(len(original_words), i2+5)])
|
||||||
|
})
|
||||||
|
elif tag == 'delete':
|
||||||
|
original_text = ''.join(original_words[i1:i2])
|
||||||
|
changes.append({
|
||||||
|
'type': 'delete',
|
||||||
|
'original': original_text,
|
||||||
|
'fixed': '',
|
||||||
|
'context_before': ''.join(original_words[max(0, i1-5):i1]),
|
||||||
|
'context_after': ''.join(original_words[i2:min(len(original_words), i2+5)])
|
||||||
|
})
|
||||||
|
elif tag == 'insert':
|
||||||
|
fixed_text = ''.join(fixed_words[j1:j2])
|
||||||
|
changes.append({
|
||||||
|
'type': 'insert',
|
||||||
|
'original': '',
|
||||||
|
'fixed': fixed_text,
|
||||||
|
'context_before': ''.join(fixed_words[max(0, j1-5):j1]) if j1 > 0 else '',
|
||||||
|
'context_after': ''.join(fixed_words[j2:min(len(fixed_words), j2+5)])
|
||||||
|
})
|
||||||
|
|
||||||
|
return changes
|
||||||
|
|
||||||
|
|
||||||
|
def generate_change_summary(changes: list[dict]) -> str:
|
||||||
|
"""
|
||||||
|
Generate change summary
|
||||||
|
|
||||||
|
Args:
|
||||||
|
changes: List of change dictionaries
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted summary string
|
||||||
|
"""
|
||||||
|
result = []
|
||||||
|
result.append("=" * 80)
|
||||||
|
result.append(f"修改摘要 (共 {len(changes)} 处修改)")
|
||||||
|
result.append("=" * 80)
|
||||||
|
result.append("")
|
||||||
|
|
||||||
|
for i, change in enumerate(changes, 1):
|
||||||
|
change_type = {
|
||||||
|
'replace': '替换',
|
||||||
|
'delete': '删除',
|
||||||
|
'insert': '添加'
|
||||||
|
}[change['type']]
|
||||||
|
|
||||||
|
result.append(f"[{i}] {change_type}")
|
||||||
|
|
||||||
|
if change['original']:
|
||||||
|
result.append(f" 原文: {change['original']}")
|
||||||
|
if change['fixed']:
|
||||||
|
result.append(f" 修复: {change['fixed']}")
|
||||||
|
|
||||||
|
# Show context
|
||||||
|
context = change['context_before'] + "【修改处】" + change['context_after']
|
||||||
|
if context.strip():
|
||||||
|
result.append(f" 上下文: ...{context}...")
|
||||||
|
|
||||||
|
result.append("")
|
||||||
|
|
||||||
|
return '\n'.join(result)
|
||||||
37
transcript-fixer/scripts/utils/diff_formats/html_format.py
Normal file
37
transcript-fixer/scripts/utils/diff_formats/html_format.py
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
HTML diff format generator
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Generate HTML side-by-side comparison
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import difflib
|
||||||
|
|
||||||
|
|
||||||
|
def generate_html_diff(original: str, fixed: str) -> str:
|
||||||
|
"""
|
||||||
|
Generate HTML format comparison report (side-by-side)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original: Original text
|
||||||
|
fixed: Fixed text
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
HTML format string with side-by-side comparison
|
||||||
|
"""
|
||||||
|
original_lines = original.splitlines(keepends=True)
|
||||||
|
fixed_lines = fixed.splitlines(keepends=True)
|
||||||
|
|
||||||
|
differ = difflib.HtmlDiff(wrapcolumn=80)
|
||||||
|
html = differ.make_file(
|
||||||
|
original_lines,
|
||||||
|
fixed_lines,
|
||||||
|
fromdesc='原始版本',
|
||||||
|
todesc='修复版本',
|
||||||
|
context=True,
|
||||||
|
numlines=3
|
||||||
|
)
|
||||||
|
|
||||||
|
return html
|
||||||
65
transcript-fixer/scripts/utils/diff_formats/inline_format.py
Normal file
65
transcript-fixer/scripts/utils/diff_formats/inline_format.py
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Inline diff format generator
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Generate inline diff with change markers
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import difflib
|
||||||
|
|
||||||
|
from .text_splitter import split_into_words
|
||||||
|
|
||||||
|
|
||||||
|
def generate_inline_diff(original: str, fixed: str) -> str:
|
||||||
|
"""
|
||||||
|
Generate inline diff marking deletions and additions
|
||||||
|
|
||||||
|
Format:
|
||||||
|
- Normal words: unchanged
|
||||||
|
- Deletions: [-word-]
|
||||||
|
- Additions: [+word+]
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original: Original text
|
||||||
|
fixed: Fixed text
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Inline diff string with markers
|
||||||
|
"""
|
||||||
|
original_words = split_into_words(original)
|
||||||
|
fixed_words = split_into_words(fixed)
|
||||||
|
|
||||||
|
diff = difflib.ndiff(original_words, fixed_words)
|
||||||
|
|
||||||
|
result = []
|
||||||
|
result.append("=" * 80)
|
||||||
|
result.append("行内词语级别对比 (- 删除, + 添加, ? 修改标记)")
|
||||||
|
result.append("=" * 80)
|
||||||
|
result.append("")
|
||||||
|
|
||||||
|
current_line = []
|
||||||
|
for item in diff:
|
||||||
|
marker = item[0]
|
||||||
|
word = item[2:]
|
||||||
|
|
||||||
|
if marker == ' ':
|
||||||
|
current_line.append(word)
|
||||||
|
elif marker == '-':
|
||||||
|
current_line.append(f"[-{word}-]")
|
||||||
|
elif marker == '+':
|
||||||
|
current_line.append(f"[+{word}+]")
|
||||||
|
elif marker == '?':
|
||||||
|
# Skip change marker lines
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Wrap at 80 characters
|
||||||
|
if len(''.join(current_line)) > 80:
|
||||||
|
result.append(''.join(current_line))
|
||||||
|
current_line = []
|
||||||
|
|
||||||
|
if current_line:
|
||||||
|
result.append(''.join(current_line))
|
||||||
|
|
||||||
|
return '\n'.join(result)
|
||||||
104
transcript-fixer/scripts/utils/diff_formats/markdown_format.py
Normal file
104
transcript-fixer/scripts/utils/diff_formats/markdown_format.py
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Markdown report generator
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Generate detailed Markdown comparison report
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .change_extractor import extract_changes, generate_change_summary
|
||||||
|
|
||||||
|
|
||||||
|
def generate_markdown_report(
|
||||||
|
original_file: str,
|
||||||
|
stage1_file: str,
|
||||||
|
stage2_file: str,
|
||||||
|
original: str,
|
||||||
|
stage1: str,
|
||||||
|
stage2: str
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Generate comprehensive Markdown comparison report
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original_file: Original file path
|
||||||
|
stage1_file: Stage 1 file path
|
||||||
|
stage2_file: Stage 2 file path
|
||||||
|
original: Original text content
|
||||||
|
stage1: Stage 1 text content
|
||||||
|
stage2: Stage 2 text content
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted Markdown report string
|
||||||
|
"""
|
||||||
|
original_path = Path(original_file)
|
||||||
|
stage1_path = Path(stage1_file)
|
||||||
|
stage2_path = Path(stage2_file)
|
||||||
|
|
||||||
|
# Extract changes for each stage
|
||||||
|
changes_stage1 = extract_changes(original, stage1)
|
||||||
|
changes_stage2 = extract_changes(stage1, stage2)
|
||||||
|
changes_total = extract_changes(original, stage2)
|
||||||
|
|
||||||
|
# Generate summaries
|
||||||
|
summary_stage1 = generate_change_summary(changes_stage1)
|
||||||
|
summary_stage2 = generate_change_summary(changes_stage2)
|
||||||
|
summary_total = generate_change_summary(changes_total)
|
||||||
|
|
||||||
|
# Build report
|
||||||
|
report = f"""# 会议记录修复对比报告
|
||||||
|
|
||||||
|
## 文件信息
|
||||||
|
|
||||||
|
- **原始文件**: {original_path.name}
|
||||||
|
- **阶段1修复**: {stage1_path.name}
|
||||||
|
- **阶段2修复**: {stage2_path.name}
|
||||||
|
- **生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
|
||||||
|
|
||||||
|
## 修改统计
|
||||||
|
|
||||||
|
| 阶段 | 修改数量 | 说明 |
|
||||||
|
|------|---------|------|
|
||||||
|
| 阶段1: 词典修复 | {len(changes_stage1)} | 基于预定义词典的批量替换 |
|
||||||
|
| 阶段2: AI修复 | {len(changes_stage2)} | GLM-4.6智能纠错 |
|
||||||
|
| **总计** | **{len(changes_total)}** | **原始→最终版本** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 阶段1: 词典修复详情
|
||||||
|
|
||||||
|
{summary_stage1}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 阶段2: AI智能修复详情
|
||||||
|
|
||||||
|
{summary_stage2}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 总体修改详情 (原始→最终)
|
||||||
|
|
||||||
|
{summary_total}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 使用说明
|
||||||
|
|
||||||
|
1. **查看修改**: 每处修改都包含上下文,便于理解修改原因
|
||||||
|
2. **人工审核**: 重点审核标记为"替换"的修改
|
||||||
|
3. **专业术语**: 特别注意公司名、人名、技术术语的修改
|
||||||
|
|
||||||
|
## 建议审核重点
|
||||||
|
|
||||||
|
- [ ] 专业术语(具身智能、机器人等)
|
||||||
|
- [ ] 人名和公司名
|
||||||
|
- [ ] 数字(金额、时间等)
|
||||||
|
- [ ] 上下文是否通顺
|
||||||
|
"""
|
||||||
|
|
||||||
|
return report
|
||||||
33
transcript-fixer/scripts/utils/diff_formats/text_splitter.py
Normal file
33
transcript-fixer/scripts/utils/diff_formats/text_splitter.py
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Text splitter utility for word-level diff generation
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Split text into words while preserving structure
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
def split_into_words(text: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Split text into words, preserving whitespace and punctuation
|
||||||
|
|
||||||
|
This enables word-level diff generation for Chinese and English text
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Input text to split
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of word tokens (Chinese words, English words, numbers, punctuation)
|
||||||
|
"""
|
||||||
|
# Pattern: Chinese chars, English words, numbers, non-alphanumeric chars
|
||||||
|
pattern = r'[\u4e00-\u9fff]+|[a-zA-Z]+|[0-9]+|[^\u4e00-\u9fffa-zA-Z0-9]'
|
||||||
|
return re.findall(pattern, text)
|
||||||
|
|
||||||
|
|
||||||
|
def read_file(file_path: str) -> str:
|
||||||
|
"""Read file contents"""
|
||||||
|
with open(file_path, 'r', encoding='utf-8') as f:
|
||||||
|
return f.read()
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Unified diff format generator
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Generate unified diff format output
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import difflib
|
||||||
|
|
||||||
|
from .text_splitter import split_into_words
|
||||||
|
|
||||||
|
|
||||||
|
def generate_unified_diff(
|
||||||
|
original: str,
|
||||||
|
fixed: str,
|
||||||
|
original_label: str = "原始版本",
|
||||||
|
fixed_label: str = "修复版本"
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Generate unified format diff report
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original: Original text
|
||||||
|
fixed: Fixed text
|
||||||
|
original_label: Label for original version
|
||||||
|
fixed_label: Label for fixed version
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Unified diff format string
|
||||||
|
"""
|
||||||
|
original_words = split_into_words(original)
|
||||||
|
fixed_words = split_into_words(fixed)
|
||||||
|
|
||||||
|
diff = difflib.unified_diff(
|
||||||
|
original_words,
|
||||||
|
fixed_words,
|
||||||
|
fromfile=original_label,
|
||||||
|
tofile=fixed_label,
|
||||||
|
lineterm=''
|
||||||
|
)
|
||||||
|
|
||||||
|
return '\n'.join(diff)
|
||||||
132
transcript-fixer/scripts/utils/diff_generator.py
Normal file
132
transcript-fixer/scripts/utils/diff_generator.py
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Generate word-level correction comparison reports
|
||||||
|
Orchestrates multiple diff formats for visualization
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Coordinate diff generation workflow
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .diff_formats import (
|
||||||
|
generate_unified_diff,
|
||||||
|
generate_html_diff,
|
||||||
|
generate_inline_diff,
|
||||||
|
generate_markdown_report,
|
||||||
|
)
|
||||||
|
from .diff_formats.text_splitter import read_file
|
||||||
|
|
||||||
|
|
||||||
|
def generate_full_report(
|
||||||
|
original_file: str,
|
||||||
|
stage1_file: str,
|
||||||
|
stage2_file: str,
|
||||||
|
output_dir: str = None
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Generate comprehensive comparison report
|
||||||
|
|
||||||
|
Creates 4 output files:
|
||||||
|
1. Markdown format detailed report
|
||||||
|
2. Unified diff format
|
||||||
|
3. HTML side-by-side comparison
|
||||||
|
4. Inline marked comparison
|
||||||
|
|
||||||
|
Args:
|
||||||
|
original_file: Path to original transcript
|
||||||
|
stage1_file: Path to stage 1 (dictionary) corrected version
|
||||||
|
stage2_file: Path to stage 2 (AI) corrected version
|
||||||
|
output_dir: Optional output directory (defaults to original file location)
|
||||||
|
"""
|
||||||
|
original_path = Path(original_file)
|
||||||
|
stage1_path = Path(stage1_file)
|
||||||
|
stage2_path = Path(stage2_file)
|
||||||
|
|
||||||
|
# Determine output directory
|
||||||
|
if output_dir:
|
||||||
|
output_path = Path(output_dir)
|
||||||
|
output_path.mkdir(parents=True, exist_ok=True)
|
||||||
|
else:
|
||||||
|
output_path = original_path.parent
|
||||||
|
|
||||||
|
base_name = original_path.stem
|
||||||
|
|
||||||
|
# Read files
|
||||||
|
print(f"📖 读取文件...")
|
||||||
|
original = read_file(original_file)
|
||||||
|
stage1 = read_file(stage1_file)
|
||||||
|
stage2 = read_file(stage2_file)
|
||||||
|
|
||||||
|
# Generate reports
|
||||||
|
print(f"📝 生成对比报告...")
|
||||||
|
|
||||||
|
# 1. Markdown report
|
||||||
|
print(f" 生成Markdown报告...")
|
||||||
|
md_report = generate_markdown_report(
|
||||||
|
original_file, stage1_file, stage2_file,
|
||||||
|
original, stage1, stage2
|
||||||
|
)
|
||||||
|
md_file = output_path / f"{base_name}_对比报告.md"
|
||||||
|
with open(md_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(md_report)
|
||||||
|
print(f" ✓ Markdown报告: {md_file.name}")
|
||||||
|
|
||||||
|
# 2. Unified Diff
|
||||||
|
print(f" 生成Unified Diff...")
|
||||||
|
unified_diff = generate_unified_diff(original, stage2)
|
||||||
|
diff_file = output_path / f"{base_name}_unified.diff"
|
||||||
|
with open(diff_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(unified_diff)
|
||||||
|
print(f" ✓ Unified Diff: {diff_file.name}")
|
||||||
|
|
||||||
|
# 3. HTML comparison
|
||||||
|
print(f" 生成HTML对比...")
|
||||||
|
html_diff = generate_html_diff(original, stage2)
|
||||||
|
html_file = output_path / f"{base_name}_对比.html"
|
||||||
|
with open(html_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(html_diff)
|
||||||
|
print(f" ✓ HTML对比: {html_file.name}")
|
||||||
|
|
||||||
|
# 4. Inline diff
|
||||||
|
print(f" 生成行内diff...")
|
||||||
|
inline_diff = generate_inline_diff(original, stage2)
|
||||||
|
inline_file = output_path / f"{base_name}_行内对比.txt"
|
||||||
|
with open(inline_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(inline_diff)
|
||||||
|
print(f" ✓ 行内对比: {inline_file.name}")
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
print(f"\n✅ 对比报告生成完成!")
|
||||||
|
print(f"📂 输出目录: {output_path}")
|
||||||
|
print(f"\n生成的文件:")
|
||||||
|
print(f" 1. {md_file.name} - Markdown格式详细报告")
|
||||||
|
print(f" 2. {diff_file.name} - Unified Diff格式")
|
||||||
|
print(f" 3. {html_file.name} - HTML并排对比")
|
||||||
|
print(f" 4. {inline_file.name} - 行内标记对比")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""CLI entry point"""
|
||||||
|
if len(sys.argv) < 4:
|
||||||
|
print("用法: python generate_diff_report.py <原始文件> <阶段1文件> <阶段2文件> [输出目录]")
|
||||||
|
print()
|
||||||
|
print("示例:")
|
||||||
|
print(" python generate_diff_report.py \\")
|
||||||
|
print(" 原始.md \\")
|
||||||
|
print(" 原始_阶段1_词典修复.md \\")
|
||||||
|
print(" 原始_阶段2_AI修复.md")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
original_file = sys.argv[1]
|
||||||
|
stage1_file = sys.argv[2]
|
||||||
|
stage2_file = sys.argv[3]
|
||||||
|
output_dir = sys.argv[4] if len(sys.argv) > 4 else None
|
||||||
|
|
||||||
|
generate_full_report(original_file, stage1_file, stage2_file, output_dir)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
129
transcript-fixer/scripts/utils/logging_config.py
Normal file
129
transcript-fixer/scripts/utils/logging_config.py
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Logging Configuration for Transcript Fixer
|
||||||
|
|
||||||
|
Provides structured logging with rotation, levels, and audit trails.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import logging.handlers
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(
|
||||||
|
log_dir: Optional[Path] = None,
|
||||||
|
level: str = "INFO",
|
||||||
|
enable_console: bool = True,
|
||||||
|
enable_file: bool = True,
|
||||||
|
enable_audit: bool = True
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Configure logging for the application.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
log_dir: Directory for log files (default: ~/.transcript-fixer/logs)
|
||||||
|
level: Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
|
||||||
|
enable_console: Enable console output
|
||||||
|
enable_file: Enable file logging
|
||||||
|
enable_audit: Enable audit logging
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> setup_logging(level="DEBUG")
|
||||||
|
>>> logger = logging.getLogger(__name__)
|
||||||
|
>>> logger.info("Application started")
|
||||||
|
"""
|
||||||
|
# Default log directory
|
||||||
|
if log_dir is None:
|
||||||
|
log_dir = Path.home() / ".transcript-fixer" / "logs"
|
||||||
|
|
||||||
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Root logger configuration
|
||||||
|
root_logger = logging.getLogger()
|
||||||
|
root_logger.setLevel(logging.DEBUG) # Capture all, filter by handler
|
||||||
|
|
||||||
|
# Clear existing handlers
|
||||||
|
root_logger.handlers.clear()
|
||||||
|
|
||||||
|
# Formatters
|
||||||
|
detailed_formatter = logging.Formatter(
|
||||||
|
fmt='%(asctime)s - %(name)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s',
|
||||||
|
datefmt='%Y-%m-%d %H:%M:%S'
|
||||||
|
)
|
||||||
|
|
||||||
|
simple_formatter = logging.Formatter(
|
||||||
|
fmt='%(asctime)s - %(levelname)s - %(message)s',
|
||||||
|
datefmt='%Y-%m-%d %H:%M:%S'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Console handler
|
||||||
|
if enable_console:
|
||||||
|
console_handler = logging.StreamHandler(sys.stdout)
|
||||||
|
console_handler.setLevel(getattr(logging, level.upper()))
|
||||||
|
console_handler.setFormatter(simple_formatter)
|
||||||
|
root_logger.addHandler(console_handler)
|
||||||
|
|
||||||
|
# File handler (rotating)
|
||||||
|
if enable_file:
|
||||||
|
file_handler = logging.handlers.RotatingFileHandler(
|
||||||
|
filename=log_dir / "transcript-fixer.log",
|
||||||
|
maxBytes=10 * 1024 * 1024, # 10MB
|
||||||
|
backupCount=5,
|
||||||
|
encoding='utf-8'
|
||||||
|
)
|
||||||
|
file_handler.setLevel(logging.DEBUG)
|
||||||
|
file_handler.setFormatter(detailed_formatter)
|
||||||
|
root_logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
# Error file handler (only errors)
|
||||||
|
if enable_file:
|
||||||
|
error_handler = logging.handlers.RotatingFileHandler(
|
||||||
|
filename=log_dir / "errors.log",
|
||||||
|
maxBytes=10 * 1024 * 1024, # 10MB
|
||||||
|
backupCount=3,
|
||||||
|
encoding='utf-8'
|
||||||
|
)
|
||||||
|
error_handler.setLevel(logging.ERROR)
|
||||||
|
error_handler.setFormatter(detailed_formatter)
|
||||||
|
root_logger.addHandler(error_handler)
|
||||||
|
|
||||||
|
# Audit handler (separate audit trail)
|
||||||
|
if enable_audit:
|
||||||
|
audit_handler = logging.handlers.RotatingFileHandler(
|
||||||
|
filename=log_dir / "audit.log",
|
||||||
|
maxBytes=50 * 1024 * 1024, # 50MB
|
||||||
|
backupCount=10,
|
||||||
|
encoding='utf-8'
|
||||||
|
)
|
||||||
|
audit_handler.setLevel(logging.INFO)
|
||||||
|
audit_handler.setFormatter(detailed_formatter)
|
||||||
|
|
||||||
|
# Create audit logger
|
||||||
|
audit_logger = logging.getLogger('audit')
|
||||||
|
audit_logger.setLevel(logging.INFO)
|
||||||
|
audit_logger.addHandler(audit_handler)
|
||||||
|
audit_logger.propagate = False # Don't propagate to root
|
||||||
|
|
||||||
|
logging.info(f"Logging configured: level={level}, log_dir={log_dir}")
|
||||||
|
|
||||||
|
|
||||||
|
def get_audit_logger() -> logging.Logger:
|
||||||
|
"""Get the dedicated audit logger."""
|
||||||
|
return logging.getLogger('audit')
|
||||||
|
|
||||||
|
|
||||||
|
# Example usage
|
||||||
|
if __name__ == "__main__":
|
||||||
|
setup_logging(level="DEBUG")
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
logger.debug("Debug message")
|
||||||
|
logger.info("Info message")
|
||||||
|
logger.warning("Warning message")
|
||||||
|
logger.error("Error message")
|
||||||
|
logger.critical("Critical message")
|
||||||
|
|
||||||
|
audit_logger = get_audit_logger()
|
||||||
|
audit_logger.info("User 'admin' added correction: '错误' → '正确'")
|
||||||
141
transcript-fixer/scripts/utils/validation.py
Normal file
141
transcript-fixer/scripts/utils/validation.py
Normal file
@@ -0,0 +1,141 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Validation Utility - Configuration Health Checker
|
||||||
|
|
||||||
|
SINGLE RESPONSIBILITY: Validate transcript-fixer configuration and JSON files
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Check directory structure
|
||||||
|
- Validate JSON syntax in all config files
|
||||||
|
- Check environment variables
|
||||||
|
- Report statistics and health status
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Handle imports for both standalone and package usage
|
||||||
|
try:
|
||||||
|
from core import CorrectionRepository, CorrectionService
|
||||||
|
except ImportError:
|
||||||
|
# Fallback for when run from scripts directory directly
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
from core import CorrectionRepository, CorrectionService
|
||||||
|
|
||||||
|
|
||||||
|
def validate_configuration() -> tuple[list[str], list[str]]:
|
||||||
|
"""
|
||||||
|
Validate transcript-fixer configuration.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (errors, warnings) as string lists
|
||||||
|
"""
|
||||||
|
config_dir = Path.home() / ".transcript-fixer"
|
||||||
|
db_path = config_dir / "corrections.db"
|
||||||
|
|
||||||
|
errors = []
|
||||||
|
warnings = []
|
||||||
|
|
||||||
|
print("🔍 Validating transcript-fixer configuration...\n")
|
||||||
|
|
||||||
|
# Check directory exists
|
||||||
|
if not config_dir.exists():
|
||||||
|
errors.append(f"Configuration directory not found: {config_dir}")
|
||||||
|
print(f"❌ {errors[-1]}")
|
||||||
|
print("\n💡 Run: python fix_transcription.py --init")
|
||||||
|
return errors, warnings
|
||||||
|
|
||||||
|
print(f"✅ Configuration directory exists: {config_dir}")
|
||||||
|
|
||||||
|
# Validate SQLite database
|
||||||
|
if db_path.exists():
|
||||||
|
try:
|
||||||
|
repository = CorrectionRepository(db_path)
|
||||||
|
service = CorrectionService(repository)
|
||||||
|
|
||||||
|
# Query basic stats
|
||||||
|
stats = service.get_statistics()
|
||||||
|
print(f"✅ Database valid: {stats['total_corrections']} corrections")
|
||||||
|
|
||||||
|
# Check tables exist
|
||||||
|
conn = repository._get_connection()
|
||||||
|
cursor = conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
|
||||||
|
tables = [row[0] for row in cursor.fetchall()]
|
||||||
|
|
||||||
|
expected_tables = [
|
||||||
|
'corrections', 'context_rules', 'correction_history',
|
||||||
|
'correction_changes', 'learned_suggestions', 'suggestion_examples',
|
||||||
|
'system_config', 'audit_log'
|
||||||
|
]
|
||||||
|
|
||||||
|
missing_tables = [t for t in expected_tables if t not in tables]
|
||||||
|
if missing_tables:
|
||||||
|
errors.append(f"Database missing tables: {missing_tables}")
|
||||||
|
print(f"❌ {errors[-1]}")
|
||||||
|
else:
|
||||||
|
print(f"✅ All {len(expected_tables)} tables present")
|
||||||
|
|
||||||
|
service.close()
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(f"Database validation failed: {e}")
|
||||||
|
print(f"❌ {errors[-1]}")
|
||||||
|
else:
|
||||||
|
warnings.append("Database not found (will be created on first use)")
|
||||||
|
print(f"⚠️ Database not found: {db_path}")
|
||||||
|
|
||||||
|
# Check API key
|
||||||
|
api_key = os.getenv("GLM_API_KEY")
|
||||||
|
if not api_key:
|
||||||
|
warnings.append("GLM_API_KEY environment variable not set")
|
||||||
|
print("⚠️ GLM_API_KEY not set (required for Stage 2 AI corrections)")
|
||||||
|
else:
|
||||||
|
print("✅ GLM_API_KEY is set")
|
||||||
|
|
||||||
|
return errors, warnings
|
||||||
|
|
||||||
|
|
||||||
|
def print_validation_summary(errors: list[str], warnings: list[str]) -> int:
|
||||||
|
"""
|
||||||
|
Print validation summary and return exit code.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
0 if valid, 1 if errors found
|
||||||
|
"""
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
print(f"❌ {len(errors)} error(s) found:")
|
||||||
|
for err in errors:
|
||||||
|
print(f" - {err}")
|
||||||
|
print("\n💡 Fix errors and run --validate again")
|
||||||
|
print("=" * 60)
|
||||||
|
return 1
|
||||||
|
elif warnings:
|
||||||
|
print(f"⚠️ {len(warnings)} warning(s):")
|
||||||
|
for warn in warnings:
|
||||||
|
print(f" - {warn}")
|
||||||
|
print("\n✅ Configuration is valid (with warnings)")
|
||||||
|
print("=" * 60)
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
print("✅ All checks passed! Configuration is valid.")
|
||||||
|
print("=" * 60)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Run validation as standalone script"""
|
||||||
|
errors, warnings = validate_configuration()
|
||||||
|
exit_code = print_validation_summary(errors, warnings)
|
||||||
|
sys.exit(exit_code)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user