117 lines
2.7 KiB
Markdown
117 lines
2.7 KiB
Markdown
# Knowledge Base RAG System
|
|
|
|
A self-hosted RAG (Retrieval Augmented Generation) system for your Obsidian vault with MCP server integration.
|
|
|
|
## Features
|
|
|
|
- **Semantic Search**: Find relevant content using embeddings, not just keywords
|
|
- **MCP Server**: Exposes search, indexing, and stats tools via MCP protocol
|
|
- **Local-first**: No external APIs - everything runs locally
|
|
- **Obsidian Compatible**: Works with your existing markdown vault
|
|
|
|
## Requirements
|
|
|
|
- Python 3.11+
|
|
- ~2GB disk space for embeddings model
|
|
|
|
## Quick Start
|
|
|
|
### 1. Install uv (if not already)
|
|
|
|
```bash
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
source ~/.local/bin/env
|
|
```
|
|
|
|
### 2. Clone and setup
|
|
|
|
```bash
|
|
cd ~/knowledge-base
|
|
cp .env.example .env
|
|
```
|
|
|
|
### 3. Configure
|
|
|
|
Edit `.env` to set your vault path:
|
|
|
|
```bash
|
|
VAULT_PATH=/path/to/your/obsidian-vault
|
|
EMBEDDING_MODEL=all-MiniLM-L6-v2 # optional
|
|
```
|
|
|
|
### 4. Install dependencies
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
### 5. Run the server
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
VAULT_PATH=./knowledge python -m knowledge_rag.server
|
|
```
|
|
|
|
The server will:
|
|
- Auto-index your vault on startup
|
|
- Listen for MCP requests via stdio
|
|
|
|
## MCP Tools
|
|
|
|
Once running, these tools are available:
|
|
|
|
| Tool | Description |
|
|
|------|-------------|
|
|
| `search_knowledge` | Semantic search across your vault |
|
|
| `index_knowledge` | Re-index the vault (use after adding files) |
|
|
| `get_knowledge_stats` | View indexing statistics |
|
|
|
|
## Usage Example
|
|
|
|
```python
|
|
# Example: Searching the knowledge base
|
|
# (via MCP client or Claude Desktop integration)
|
|
|
|
await search_knowledge({
|
|
"query": "how does the RAG system work",
|
|
"top_k": 5
|
|
})
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
knowledge-base/
|
|
├── src/knowledge_rag/ # Source code
|
|
│ ├── server.py # MCP server
|
|
│ ├── chunker.py # Markdown chunking
|
|
│ ├── embeddings.py # Sentence-transformers wrapper
|
|
│ └── vector_store.py # ChromaDB wrapper
|
|
├── knowledge/ # Your Obsidian vault (gitignored)
|
|
├── pyproject.toml # Project config
|
|
└── .env.example # Environment template
|
|
```
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `VAULT_PATH` | `/data/vault` | Path to your Obsidian vault |
|
|
| `EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Sentence-transformers model |
|
|
| `EMBEDDINGS_CACHE_DIR` | `/data/embeddings_cache` | Model cache location |
|
|
|
|
## Troubleshooting
|
|
|
|
### First run is slow
|
|
The embedding model (~90MB) downloads on first run. Subsequent runs are faster.
|
|
|
|
### No search results
|
|
Run `index_knowledge` tool to index your vault, or restart the server.
|
|
|
|
### Out of memory
|
|
The default model is lightweight. For even smaller models, try `paraphrase-MiniLM-L3-v2`.
|
|
|
|
## License
|
|
|
|
MIT
|