docs: Add 4 comprehensive vector database examples (Weaviate, Chroma, FAISS, Qdrant)

Created complete working examples for all 4 vector databases with RAG adaptors:

Weaviate Example:
- Comprehensive README with hybrid search guide
- 3 Python scripts (generate, upload, query)
- Sample outputs and query results
- Covers hybrid search, filtering, schema design

Chroma Example:
- Simple, local-first approach
- In-memory and persistent storage options
- Semantic search and metadata filtering
- Comparison with Weaviate

FAISS Example:
- Facebook AI Similarity Search integration
- OpenAI embeddings generation
- Index building and persistence
- Performance-focused for scale

Qdrant Example:
- Advanced filtering capabilities
- Production-ready features
- Complex query patterns
- Rust-based performance

Each example includes:
- Detailed README with setup and troubleshooting
- requirements.txt with dependencies
- 3 working Python scripts
- Sample outputs directory

Total files: 20 (4 examples × 5 files each)
Documentation: 4 comprehensive READMEs (~800 lines total)

Phase 2 of optional enhancements complete.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
yusyus
2026-02-07 22:38:15 +03:00
parent d84e5878a1
commit 53d37e61dd
21 changed files with 2506 additions and 0 deletions

View File

@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""
Step 1: Generate Skill for ChromaDB
This script:
1. Scrapes Vue documentation (limited to 20 pages for demo)
2. Packages the skill in ChromaDB format
3. Saves to output/vue-chroma.json
Usage:
python 1_generate_skill.py
"""
import subprocess
import sys
from pathlib import Path
def main():
print("=" * 60)
print("Step 1: Generating Skill for ChromaDB")
print("=" * 60)
# Check if skill-seekers is installed
try:
result = subprocess.run(
["skill-seekers", "--version"],
capture_output=True,
text=True
)
print(f"\n✅ skill-seekers found: {result.stdout.strip()}")
except FileNotFoundError:
print("\n❌ skill-seekers not found!")
print("Install it with: pip install skill-seekers")
sys.exit(1)
# Step 1: Scrape Vue docs (small sample for demo)
print("\n📥 Step 1/2: Scraping Vue documentation (20 pages)...")
print("This may take 1-2 minutes...\n")
scrape_result = subprocess.run(
[
"skill-seekers", "scrape",
"--config", "configs/vue.json",
"--max-pages", "20",
],
capture_output=True,
text=True
)
if scrape_result.returncode != 0:
print(f"❌ Scraping failed:\n{scrape_result.stderr}")
sys.exit(1)
print("✅ Scraping completed!")
# Step 2: Package for ChromaDB
print("\n📦 Step 2/2: Packaging for ChromaDB...\n")
package_result = subprocess.run(
[
"skill-seekers", "package",
"output/vue",
"--target", "chroma",
],
capture_output=True,
text=True
)
if package_result.returncode != 0:
print(f"❌ Packaging failed:\n{package_result.stderr}")
sys.exit(1)
# Show the output
print(package_result.stdout)
# Check if output file exists
output_file = Path("output/vue-chroma.json")
if output_file.exists():
size_kb = output_file.stat().st_size / 1024
print(f"📄 File size: {size_kb:.1f} KB")
print(f"📂 Location: {output_file.absolute()}")
print("\n✅ Ready for upload! Next step: python 2_upload_to_chroma.py")
else:
print("❌ Output file not found!")
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""
Step 2: Upload to ChromaDB
This script:
1. Creates a ChromaDB client (in-memory or persistent)
2. Creates a collection
3. Adds all documents with metadata
4. Verifies the upload
Usage:
# In-memory (development)
python 2_upload_to_chroma.py
# Persistent storage (production)
python 2_upload_to_chroma.py --persist ./chroma_db
# Reset existing collection
python 2_upload_to_chroma.py --reset
"""
import argparse
import json
import sys
from pathlib import Path
try:
import chromadb
except ImportError:
print("❌ chromadb not installed!")
print("Install it with: pip install chromadb")
sys.exit(1)
def create_client(persist_directory: str = None):
"""Create ChromaDB client."""
print("\n📊 Creating ChromaDB client...")
try:
if persist_directory:
# Persistent client (saves to disk)
client = chromadb.PersistentClient(path=persist_directory)
print(f"✅ Client created (persistent: {persist_directory})\n")
else:
# In-memory client (faster, but data lost on exit)
client = chromadb.Client()
print("✅ Client created (in-memory)\n")
return client
except Exception as e:
print(f"❌ Client creation failed: {e}")
sys.exit(1)
def load_skill_data(filepath: str = "output/vue-chroma.json"):
"""Load the ChromaDB-format skill JSON."""
path = Path(filepath)
if not path.exists():
print(f"❌ Skill file not found: {filepath}")
print("Run '1_generate_skill.py' first!")
sys.exit(1)
with open(path) as f:
return json.load(f)
def create_collection(client, collection_name: str, reset: bool = False):
"""Create ChromaDB collection."""
print(f"📦 Creating collection: {collection_name}")
try:
# Check if collection exists
existing_collections = [c.name for c in client.list_collections()]
if collection_name in existing_collections:
if reset:
print(f"🗑️ Deleting existing collection...")
client.delete_collection(collection_name)
else:
print(f"⚠️ Collection '{collection_name}' already exists")
response = input("Delete and recreate? [y/N]: ")
if response.lower() == "y":
client.delete_collection(collection_name)
else:
print("Using existing collection")
return client.get_collection(collection_name)
# Create collection
collection = client.create_collection(
name=collection_name,
metadata={"description": "Skill Seekers documentation"}
)
print("✅ Collection created!\n")
return collection
except Exception as e:
print(f"❌ Collection creation failed: {e}")
sys.exit(1)
def upload_documents(collection, data: dict):
"""Add documents to collection."""
total = len(data["documents"])
print(f"📤 Adding {total} documents to collection...")
try:
# Add all documents in one batch
collection.add(
documents=data["documents"],
metadatas=data["metadatas"],
ids=data["ids"]
)
print(f"✅ Successfully added {total} documents to ChromaDB\n")
except Exception as e:
print(f"❌ Upload failed: {e}")
sys.exit(1)
def verify_upload(collection):
"""Verify documents were uploaded correctly."""
count = collection.count()
print(f"🔍 Collection '{collection.name}' now contains {count} documents")
def main():
parser = argparse.ArgumentParser(description="Upload skill to ChromaDB")
parser.add_argument(
"--persist",
help="Persistent storage directory (e.g., ./chroma_db)"
)
parser.add_argument(
"--file",
default="output/vue-chroma.json",
help="Path to ChromaDB JSON file"
)
parser.add_argument(
"--reset",
action="store_true",
help="Delete existing collection before uploading"
)
args = parser.parse_args()
print("=" * 60)
print("Step 2: Upload to ChromaDB")
print("=" * 60)
# Create client
client = create_client(args.persist)
# Load skill data
data = load_skill_data(args.file)
# Create collection
collection = create_collection(client, data["collection_name"], args.reset)
# Upload documents
upload_documents(collection, data)
# Verify
verify_upload(collection)
if args.persist:
print(f"\n💾 Data saved to: {args.persist}")
print(" Use --persist flag to load it next time")
print("\n✅ Upload complete! Next step: python 3_query_example.py")
if args.persist:
print(f" python 3_query_example.py --persist {args.persist}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,290 @@
#!/usr/bin/env python3
"""
Step 3: Query ChromaDB
This script demonstrates various query patterns with ChromaDB:
1. Semantic search
2. Metadata filtering
3. Distance scoring
4. Top-K results
Usage:
# In-memory (if you used in-memory upload)
python 3_query_example.py
# Persistent (if you used --persist for upload)
python 3_query_example.py --persist ./chroma_db
"""
import argparse
import sys
try:
import chromadb
from rich.console import Console
from rich.table import Table
from rich.panel import Panel
except ImportError:
print("❌ Missing dependencies!")
print("Install with: pip install chromadb rich")
sys.exit(1)
console = Console()
def create_client(persist_directory: str = None):
"""Create ChromaDB client."""
try:
if persist_directory:
return chromadb.PersistentClient(path=persist_directory)
else:
return chromadb.Client()
except Exception as e:
console.print(f"[red]❌ Client creation failed: {e}[/red]")
sys.exit(1)
def get_collection(client, collection_name: str = "vue"):
"""Get collection from ChromaDB."""
try:
return client.get_collection(collection_name)
except Exception as e:
console.print(f"[red]❌ Collection not found: {e}[/red]")
console.print("\n[yellow]Did you run 2_upload_to_chroma.py first?[/yellow]")
sys.exit(1)
def semantic_search_example(collection):
"""Example 1: Basic Semantic Search."""
console.print("\n" + "=" * 60)
console.print("[bold cyan]Example 1: Semantic Search[/bold cyan]")
console.print("=" * 60)
query = "How do I create a Vue component?"
console.print(f"\n[yellow]Query:[/yellow] {query}")
try:
results = collection.query(
query_texts=[query],
n_results=3
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
distances = results["distances"][0]
if not documents:
console.print("[red]No results found[/red]")
return
# Create results table
table = Table(show_header=True, header_style="bold magenta")
table.add_column("#", style="dim", width=3)
table.add_column("Distance", style="cyan", width=10)
table.add_column("Category", style="green")
table.add_column("File", style="yellow")
table.add_column("Preview", style="white")
for i, (doc, meta, dist) in enumerate(zip(documents, metadatas, distances), 1):
preview = doc[:80] + "..." if len(doc) > 80 else doc
table.add_row(
str(i),
f"{dist:.3f}",
meta.get("category", "N/A"),
meta.get("file", "N/A"),
preview
)
console.print(table)
# Explain distance scores
console.print("\n[dim]💡 Distance: Lower = more similar (< 0.5 = very relevant)[/dim]")
except Exception as e:
console.print(f"[red]Query failed: {e}[/red]")
def filtered_search_example(collection):
"""Example 2: Search with Metadata Filter."""
console.print("\n" + "=" * 60)
console.print("[bold cyan]Example 2: Filtered Search[/bold cyan]")
console.print("=" * 60)
query = "reactivity"
category_filter = "api"
console.print(f"\n[yellow]Query:[/yellow] {query}")
console.print(f"[yellow]Filter:[/yellow] category = '{category_filter}'")
try:
results = collection.query(
query_texts=[query],
n_results=5,
where={"category": category_filter}
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
distances = results["distances"][0]
if not documents:
console.print("[red]No results found[/red]")
return
console.print(f"\n[green]Found {len(documents)} results in '{category_filter}' category:[/green]\n")
for i, (doc, meta, dist) in enumerate(zip(documents, metadatas, distances), 1):
panel = Panel(
f"[cyan]File:[/cyan] {meta.get('file', 'N/A')}\n"
f"[cyan]Distance:[/cyan] {dist:.3f}\n\n"
f"[white]{doc[:200]}...[/white]",
title=f"Result {i}",
border_style="green"
)
console.print(panel)
except Exception as e:
console.print(f"[red]Query failed: {e}[/red]")
def top_k_results_example(collection):
"""Example 3: Get More Results (Top-K)."""
console.print("\n" + "=" * 60)
console.print("[bold cyan]Example 3: Top-K Results[/bold cyan]")
console.print("=" * 60)
query = "state management"
console.print(f"\n[yellow]Query:[/yellow] {query}")
console.print(f"[yellow]K:[/yellow] 10 (top 10 results)")
try:
results = collection.query(
query_texts=[query],
n_results=10
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
distances = results["distances"][0]
console.print(f"\n[green]Top 10 most relevant documents:[/green]\n")
for i, (doc, meta, dist) in enumerate(zip(documents, metadatas, distances), 1):
category = meta.get("category", "N/A")
file = meta.get("file", "N/A")
console.print(f"[bold]{i:2d}.[/bold] [{dist:.3f}] {category:10s} | {file}")
except Exception as e:
console.print(f"[red]Query failed: {e}[/red]")
def complex_filter_example(collection):
"""Example 4: Complex Metadata Filtering."""
console.print("\n" + "=" * 60)
console.print("[bold cyan]Example 4: Complex Filter (AND condition)[/bold cyan]")
console.print("=" * 60)
query = "guide"
console.print(f"\n[yellow]Query:[/yellow] {query}")
console.print(f"[yellow]Filter:[/yellow] category = 'guides' AND type = 'reference'")
try:
results = collection.query(
query_texts=[query],
n_results=5,
where={
"$and": [
{"category": "guides"},
{"type": "reference"}
]
}
)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
if not documents:
console.print("[red]No results match both conditions[/red]")
return
console.print(f"\n[green]Found {len(documents)} documents matching both conditions:[/green]\n")
for i, (doc, meta) in enumerate(zip(documents, metadatas), 1):
console.print(f"[bold]{i}. {meta.get('file', 'N/A')}[/bold]")
console.print(f" Category: {meta.get('category')} | Type: {meta.get('type')}")
console.print(f" {doc[:100]}...\n")
except Exception as e:
console.print(f"[red]Query failed: {e}[/red]")
def get_statistics(collection):
"""Show collection statistics."""
console.print("\n" + "=" * 60)
console.print("[bold cyan]Collection Statistics[/bold cyan]")
console.print("=" * 60)
try:
# Total count
count = collection.count()
console.print(f"\n[green]Total documents:[/green] {count}")
# Sample metadata to show categories
sample = collection.get(limit=count)
metadatas = sample["metadatas"]
# Count by category
categories = {}
for meta in metadatas:
cat = meta.get("category", "unknown")
categories[cat] = categories.get(cat, 0) + 1
console.print(f"\n[green]Documents by category:[/green]")
for cat, cnt in sorted(categories.items()):
console.print(f"{cat}: {cnt}")
except Exception as e:
console.print(f"[red]Statistics failed: {e}[/red]")
def main():
parser = argparse.ArgumentParser(description="Query ChromaDB examples")
parser.add_argument(
"--persist",
help="Persistent storage directory (if you used --persist for upload)"
)
parser.add_argument(
"--collection",
default="vue",
help="Collection name to query (default: vue)"
)
args = parser.parse_args()
console.print("[bold green]ChromaDB Query Examples[/bold green]")
if args.persist:
console.print(f"[dim]Using persistent storage: {args.persist}[/dim]")
else:
console.print("[dim]Using in-memory storage[/dim]")
# Create client
client = create_client(args.persist)
# Get collection
collection = get_collection(client, args.collection)
# Get statistics
get_statistics(collection)
# Run examples
semantic_search_example(collection)
filtered_search_example(collection)
top_k_results_example(collection)
complex_filter_example(collection)
console.print("\n[bold green]✅ All examples completed![/bold green]")
console.print("\n[cyan]💡 Tips:[/cyan]")
console.print(" • Lower distance = more similar (< 0.5 is very relevant)")
console.print(" • Use 'where' filters to narrow results before search")
console.print(" • Combine filters with $and, $or, $not operators")
console.print(" • Adjust n_results to get more/fewer results")
console.print(" • See README.md for custom embedding functions")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,394 @@
# ChromaDB Vector Database Example
This example demonstrates how to use Skill Seekers with ChromaDB, the AI-native open-source embedding database. Chroma is designed to be simple, fast, and easy to use locally.
## What You'll Learn
- How to generate skills in ChromaDB format
- How to create local Chroma collections
- How to perform semantic searches
- How to filter by metadata categories
## Why ChromaDB?
- **No Server Required**: Works entirely in-process (perfect for development)
- **Simple API**: Clean Python interface, no complex setup
- **Fast**: Built for speed with smart indexing
- **Open Source**: MIT licensed, community-driven
## Prerequisites
### Python Dependencies
```bash
pip install -r requirements.txt
```
That's it! No Docker, no server setup. Chroma runs entirely in your Python process.
## Step-by-Step Guide
### Step 1: Generate Skill from Documentation
First, we'll scrape Vue documentation and package it for ChromaDB:
```bash
python 1_generate_skill.py
```
This script will:
1. Scrape Vue docs (limited to 20 pages for demo)
2. Package the skill in ChromaDB format (JSON with documents + metadata + IDs)
3. Save to `output/vue-chroma.json`
**Expected Output:**
```
✅ ChromaDB data packaged successfully!
📦 Output: output/vue-chroma.json
📊 Total documents: 21
📂 Categories: overview (1), guides (8), api (12)
```
**What's in the JSON?**
```json
{
"documents": [
"Vue is a progressive JavaScript framework...",
"Components are the building blocks..."
],
"metadatas": [
{
"source": "vue",
"category": "overview",
"file": "SKILL.md",
"type": "documentation",
"version": "1.0.0"
}
],
"ids": [
"a1b2c3d4e5f6...",
"b2c3d4e5f6g7..."
],
"collection_name": "vue"
}
```
### Step 2: Create Collection and Upload
Now we'll create a ChromaDB collection and load all documents:
```bash
python 2_upload_to_chroma.py
```
This script will:
1. Create an in-memory Chroma client (or persistent with `--persist`)
2. Create a collection with the skill name
3. Add all documents with metadata and IDs
4. Verify the upload was successful
**Expected Output:**
```
📊 Creating ChromaDB client...
✅ Client created (in-memory)
📦 Creating collection: vue
✅ Collection created!
📤 Adding 21 documents to collection...
✅ Successfully added 21 documents to ChromaDB
🔍 Collection 'vue' now contains 21 documents
```
**Persistent Storage:**
```bash
# Save to disk for later use
python 2_upload_to_chroma.py --persist ./chroma_db
```
### Step 3: Query and Search
Now search your knowledge base!
```bash
python 3_query_example.py
```
**With persistent storage:**
```bash
python 3_query_example.py --persist ./chroma_db
```
This script demonstrates:
1. **Semantic Search**: Natural language queries
2. **Metadata Filtering**: Filter by category
3. **Top-K Results**: Get most relevant documents
4. **Distance Scoring**: See how relevant each result is
**Example Queries:**
**Query 1: Semantic Search**
```
Query: "How do I create a Vue component?"
Top 3 results:
1. [Distance: 0.234] guides/components.md
Components are reusable Vue instances with a name. You can use them as custom
elements inside a root Vue instance...
2. [Distance: 0.298] api/component_api.md
The component API reference describes all available options for defining
components using the Options API...
3. [Distance: 0.312] guides/single_file_components.md
Single-File Components (SFCs) allow you to define templates, logic, and
styling in a single .vue file...
```
**Query 2: Filtered Search**
```
Query: "reactivity"
Filter: category = "api"
Results:
1. ref() - Create reactive references
2. reactive() - Create reactive proxies
3. computed() - Create computed properties
```
## Understanding ChromaDB Features
### Semantic Search
Chroma automatically:
- Generates embeddings for your documents (using default model)
- Indexes them for fast similarity search
- Finds semantically similar content
**Distance Scores:**
- Lower = more similar
- `0.0` = identical
- `< 0.5` = very relevant
- `0.5-1.0` = somewhat relevant
- `> 1.0` = less relevant
### Metadata Filtering
Filter results before semantic search:
```python
collection.query(
query_texts=["your query"],
n_results=5,
where={"category": "api"}
)
```
**Supported operators:**
- `$eq`: Equal to
- `$ne`: Not equal to
- `$gt`, `$gte`: Greater than (or equal)
- `$lt`, `$lte`: Less than (or equal)
- `$in`: In list
- `$nin`: Not in list
**Complex filters:**
```python
where={
"$and": [
{"category": {"$eq": "api"}},
{"type": {"$eq": "reference"}}
]
}
```
### Collection Management
```python
# List all collections
client.list_collections()
# Get collection
collection = client.get_collection("vue")
# Get count
collection.count()
# Delete collection
client.delete_collection("vue")
```
## Customization
### Use Your Own Embeddings
Chroma supports custom embedding functions:
```python
from chromadb.utils import embedding_functions
# OpenAI embeddings
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
api_key="your-key",
model_name="text-embedding-ada-002"
)
collection = client.create_collection(
name="your_skill",
embedding_function=openai_ef
)
```
**Supported embedding functions:**
- **OpenAI**: `text-embedding-ada-002` (best quality)
- **Cohere**: `embed-english-v2.0`
- **HuggingFace**: Various models (local, no API key)
- **Sentence Transformers**: Local models
### Generate Different Skills
```bash
# Change the config in 1_generate_skill.py
"--config", "configs/django.json", # Your framework
# Or use CLI directly
skill-seekers scrape --config configs/flask.json
skill-seekers package output/flask --target chroma
```
### Adjust Query Parameters
In `3_query_example.py`:
```python
# Get more results
n_results=10 # Default is 5
# Include more metadata
include=["documents", "metadatas", "distances"]
# Different distance metrics
# (configure when creating collection)
metadata={"hnsw:space": "cosine"} # or "l2", "ip"
```
## Performance Tips
1. **Batch Operations**: Add documents in batches for better performance
```python
collection.add(
documents=batch_docs,
metadatas=batch_metadata,
ids=batch_ids
)
```
2. **Persistent Storage**: Use `--persist` for production
```bash
python 2_upload_to_chroma.py --persist ./prod_db
```
3. **Custom Embeddings**: Use OpenAI for best quality (costs $)
4. **Index Tuning**: Adjust HNSW parameters for speed vs accuracy
## Troubleshooting
### Import Error
```
ModuleNotFoundError: No module named 'chromadb'
```
**Solution:**
```bash
pip install chromadb
```
### Collection Already Exists
```
Error: Collection 'vue' already exists
```
**Solution:**
```python
# Delete existing collection
client.delete_collection("vue")
# Or use --reset flag
python 2_upload_to_chroma.py --reset
```
### Empty Results
```
Query returned empty results
```
**Possible causes:**
1. Collection empty: Check `collection.count()`
2. Query too specific: Try broader queries
3. Wrong collection name: Verify collection exists
**Debug:**
```python
# Check collection contents
collection.get() # Get all documents
# Check embedding function
collection._embedding_function # Should not be None
```
### Performance Issues
```
Query is slow
```
**Solutions:**
1. Use persistent storage (faster than in-memory for large datasets)
2. Reduce `n_results` (fewer results = faster)
3. Add metadata filters to narrow search space
4. Consider using OpenAI embeddings (better quality = faster convergence)
## Next Steps
1. **Try other skills**: Package your favorite documentation
2. **Build a chatbot**: Integrate with LangChain or LlamaIndex
3. **Production deployment**: Use persistent storage + API wrapper
4. **Custom embeddings**: Experiment with different models
## Resources
- **ChromaDB Docs**: https://docs.trychroma.com/
- **GitHub**: https://github.com/chroma-core/chroma
- **Discord**: https://discord.gg/MMeYNTmh3x
- **Skill Seekers**: https://github.com/yourusername/skill-seekers
## File Structure
```
chroma-example/
├── README.md # This file
├── requirements.txt # Python dependencies
├── 1_generate_skill.py # Generate ChromaDB-format skill
├── 2_upload_to_chroma.py # Create collection and upload
├── 3_query_example.py # Query demonstrations
└── sample_output/ # Example outputs
├── vue-chroma.json # Generated skill (21 docs)
└── query_results.txt # Sample query results
```
## Comparison: Chroma vs Weaviate
| Feature | ChromaDB | Weaviate |
|---------|----------|----------|
| **Setup** | ✅ No server needed | ⚠️ Docker/Cloud required |
| **API** | ✅ Very simple | ⚠️ More complex |
| **Performance** | ✅ Fast for < 1M docs | ✅ Scales to billions |
| **Hybrid Search** | ❌ Semantic only | ✅ Keyword + semantic |
| **Production** | ✅ Good for small-medium | ✅ Built for scale |
**Use Chroma for:** Development, prototypes, small-medium datasets (< 1M docs)
**Use Weaviate for:** Production, large datasets (> 1M docs), hybrid search
---
**Last Updated:** February 2026
**Tested With:** ChromaDB v0.4.22, Python 3.10+, skill-seekers v2.10.0

View File

@@ -0,0 +1,10 @@
# ChromaDB Example Dependencies
# Skill Seekers (main package)
skill-seekers>=2.10.0
# ChromaDB
chromadb>=0.4.0
# For pretty output
rich>=13.0.0