A production RAG architecture for deep technical document analysis — built on open-source models, deployed fully on-premise.
Drop in one or more documents (PDF, DOCX, TXT, images), let the pipeline index them, then ask questions answered only from their contents — with source passages and a full compliance audit. This is the whole RAG loop, end to end.
Your conversation will appear here.
Index one or more documents on the left, then ask a question below.
Answers are grounded only in the indexed documents. Full pipeline: NV-EmbedQA-E5-v5 embeddings · BM25 keyword search · RRF fusion · Nemotron reranking · Nemotron-Super deep inference · compliance audit log.
Each stage is handled by the model best suited for that task. Click any stage to see what happens inside.
Model: Mistral 7B · Always-on · ~5GB VRAM · <50ms latency
Is this a simple lookup, a multi-doc analysis, or a classification task? Each route activates a different pipeline path.
Runs continuously at under 50ms latency. No chain-of-thought overhead needed for a routing classification — speed is the only priority here.
{ "type": "deep_analysis", "priority": "high", "pipeline": "full" } — pushed onto Redis Streams queue.
Sits before Layer 1 (Ingestion). Partially built into your existing Layer 1 health monitor HTTP endpoints.
Model: Qwen2.5-32B (embedding mode) · Batch worker · ~18GB VRAM
Token-aware splits — not character-based. 300–500 tokens with 50-token overlap. Preserves table and formula boundaries in engineering specs.
Training on multilingual technical text and code means embedding quality is high for engineering specs, tables, schematics, and JSON structures.
ChromaDB for development, FAISS for production scale. Metadata stored: doc_id, page, chunk_index, doc_type, timestamp.
This IS your Layer 1 Ingestion Pipeline — already built with OpenCV + Redis Streams. Add Qwen embedding call after frame capture.
Model: Qwen2.5-32B (query embedding) + BM25 (keyword) · Fused via RRF
Semantic search finds meaning. Keyword search finds exact part numbers, spec codes, and model IDs. Engineering documents need both simultaneously.
Score(D) = Σ 1/(k + rᵢ(D)) across all retrievers. A chunk ranking 1st in both vector and keyword gets the highest fused score.
Retrieve top 10 total, pass top 5 to inference. More chunks = richer context but higher token cost for R1's reasoning trace output.
Layer 2 (Classification) — retrieval feeds the classification decision that was previously just a rule-based system in your pipeline.
Model: Mistral 7B (cross-encoder) · Fast precision pass · Same Ollama instance
Initial retrieval is recall-focused (cast wide). Reranking is precision-focused — scores each candidate against the specific query and drops noise before R1 sees it.
Reranking is 10 fast single-passage comparisons. Mistral's speed advantage matters. Qwen's size advantage is wasted on this shallow task.
Top 5 chunks + original query + system prompt = the augmented prompt passed to DeepSeek-R1. Target: fits within 8K–16K tokens.
New sub-layer inside your Layer 2/3 boundary. Protects R1's context window from low-relevance chunks that inflate token cost and degrade reasoning.
Model: DeepSeek-R1 70B · ~40GB VRAM Q4 · Chain-of-thought reasoning · RLVR-trained
R1 outputs a hidden <think> block first. It cross-references chunks, flags contradictions, then produces a grounded, sourced answer. Visible for audit.
{ "finding": "...", "confidence": 0.92, "source_chunks": [...], "reasoning_summary": "..." } — parse-ready for Layer 5 audit log.
R1 produces 2–4× more tokens per query than a standard model due to the reasoning trace. Budget ~2K–4K output tokens per engineering document query.
Layer 3 (Inference) in the 5-layer spec. Video anomaly + retrieved context → engineering judgment → structured output lives entirely in this stage.
No model needed · SurrealDB structured logging · API response layer
query · route decision · retrieved_chunks[] · reranked_chunks[] · r1_thinking_trace · final_answer · confidence · latency_ms · timestamp
Enterprise clients require explainability. The audit log lets engineers trace exactly why a finding was flagged — chunk by chunk, step by step.
If R1 confidence < 0.70, flag for human review. High-stakes findings (safety alerts) always route to human queue regardless of confidence score.
Layer 4 (Response Aggregation) + Layer 5 (Audit) in your spec. SurrealDB is already in your planned stack — zero additional tooling needed.
Each model is ranked #1 for its specific role — not because it's universally "best," but because it's optimised for that exact moment in the pipeline.
Each product is a self-contained reference architecture we deploy at client sites. Click a tab to dive into the design.
Combining private on-premise documents with public real-time data — without leaking either way.
| Boundary | What can leak | Mitigation |
|---|---|---|
| Documents → embeddings | Embedding vectors can be inverted back to source text via known attacks. | Run nomic-embed-text locally inside the Docker network. Never call cloud embedding APIs (OpenAI /embeddings, etc.) on private data. |
| Vector DB → retriever | Retrieval patterns reveal what users are investigating. | Keep SurrealDB on-prem. For high-threat workloads, add Gumbel noise to similarity scores (differentially private retrieval). |
| Query → public API | The query itself — "our AR exposure to ACME Corp" sent to a search API leaks the relationship. | PII classifier rewrites the public sub-query as an abstract question ("current weather in Houston") before egress. The proper-noun + the figure stay home. |
| LLM inference | Cloud LLMs see the entire fused context, including private chunks. | Local Ollama inference. The only model that sees both worlds runs inside the client perimeter. |
This is the orchestration layer in plain Python. It sits between Open Notebook and Ollama. Anyone with basic Python can read it.
def hybrid_rag(user_query: str) -> str:
# 1. Decide what is safe to send outside.
public_query, redactions = pii_classifier.sanitise(user_query)
# 2. Run both retrievers in parallel.
private_chunks = private_retriever.search(user_query, k=5) # stays local
public_chunks = public_retriever.search(public_query, k=3) # web/API
# 3. Tag each chunk so the LLM and the user can tell them apart.
context = (
[f"[PRIVATE] {c.text} (source: {c.source})" for c in private_chunks] +
[f"[PUBLIC] {c.text} (source: {c.url})" for c in public_chunks]
)
# 4. Build the prompt with explicit citation rules.
prompt = PROMPT_TEMPLATE.format(
question = user_query,
context = "\n\n".join(context),
)
# 5. Local LLM generates the answer.
return ollama.generate(model="mistral:7b", prompt=prompt)| Component | Already in stack? | What to add |
|---|---|---|
| Private retriever | ✓ Open Notebook + SurrealDB + nomic-embed-text | nothing |
| Local LLM | ✓ Ollama (Mistral 7B) | nothing |
| Public retriever | — | Thin wrapper around Tavily or Brave API ($5–20/month for low volume) |
| PII classifier | — | Presidio (open-source, runs locally) or a small fine-tuned classifier |
| Orchestrator | — | ~150 lines of Python; LangChain's MultiRetriever works, or hand-roll for fewer moving parts |
Send one prompt to three open-weight models side by side. Watch how a fast router model, a deep reasoner, and a lightweight Google model each handle the same question.
Leave blank to stay in demo mode. When set, the page POSTs { prompt, model } to this URL and expects { text } back. Run the included playground-server (see setup instructions below) to get a working endpoint.
Outputs are generated independently per model with no shared context. In demo mode, responses are pre-written illustrations of each model's characteristic style — not live inference.
We design and implement on-premise RAG pipelines tailored to your engineering domain. Full data sovereignty. No cloud dependency.
james@10xlab.org →