In today's cloud-dominant AI landscape, data privacy has become the primary concern for developers, enterprises, and researchers. Sharing sensitive business files, medical reports, legal briefs, or proprietary codebases with commercial cloud APIs (like OpenAI GPT or Anthropic Claude) presents continuous security risks and unpredictable token billing.

LocalMind AI solves this by providing a developer-friendly monorepo template for a fully offline, air-gapped AI chatbot. By leveraging Ollama for model inference and Qdrant (embedded mode) as a zero-configuration local vector database, the application performs text extraction, embedding vectorization, similarity search, and answer generation completely on localhost.

The system runs in two primary operational modes:

General Offline Chat Mode: Converse directly with locally serving LLMs (such as qwen3:8b or gemma3:4b), providing general-purpose AI assistance.
Local PDF RAG Mode: Allows users to ingest a private PDF library, chunk pages, create vector embeddings, perform vector searches, and obtain answers with precise, page-specific inline source citations (e.g., [Corporate Rules, p. 14]).

🛠️ Technology Stack & Architecture

Layer	Technology	Developer Role / Utility
Frontend	React, Vite, TypeScript, Zustand, Tailwind CSS, PDF.js	Handles client-side state, markdown rendering, dynamic UI layout, and embedded PDF visualization.
Backend	Python, FastAPI, Uvicorn, SQLAlchemy, Alembic	Serves the REST API, manages background file ingestion, executes chunking logic, and coordinates model queries.
Database	SQLite, Alembic Migrations	Persists user settings, document collection structures, chat logs, message histories, and page mapping details.
Vector Engine	Qdrant Client (Embedded Mode)	Performs high-speed local vector storage and semantic search directly inside the Python process (no Docker required).
Local LLM Host	Ollama Engine (Localhost)	Hosts the offline LLM (chat) and dense vectorizer (embeddings) on GPU/CPU hardware.

🚀 Key Features of the Offline Chatbot

Zero-Cloud Privacy: No cloud database connections, external analytics, remote tracking telemetry, or remote CDN resources. Operates strictly on 127.0.0.1.
Built-in PDF Viewer & Interactive Citations: When the local LLM answers a document query, it outputs source references. Clicking a citation jumps to the exact PDF page inside the app.
Granular Folder & Collection Management: Categorize files into custom folders (e.g., Legal Documents, Reference Manuals) and selectively query individual files or entire collections.
Streaming SSE Tokens: Utilizing Server-Sent Events (SSE), response tokens stream in real-time, matching the performance feel of commercial web clients.
SHA-256 Duplicate Protection: Files are hashed upon upload to prevent duplicate processing and redundant vector indexing.
Real-time Diagnostics Dashboard: A system health check panel tracks CPU/GPU directory paths, DB connectivity, local storage directories, Qdrant status, and Ollama server readiness.

📐 Systems & RAG Pipeline Architecture

graph TD
    User([User UI / Web Browser]) <-->|HTTP / SSE Streams| Frontend[React / Vite Frontend]
    
    subgraph Local Environment (Air-Gapped)
        Frontend <-->|REST API / Uvicorn| Backend[FastAPI Backend]
        
        Backend <-->|SQL Schemas| DB[(SQLite Database)]
        Backend <-->|PDF Storage| Storage[(Local File Storage)]
        
        subgraph Local RAG Pipeline
            Backend -->|Read Text| PyMuPDF[PyMuPDF Extractor]
            PyMuPDF -->|Segment Text| Chunker[Custom Chunker: 800t size / 120t overlap]
            Chunker -->|Vectorize Chunks| OllamaEmbed[Ollama Embeddings API: nomic-embed-text]
        end
        
        OllamaEmbed -->|Dense Vectors| Qdrant[(Qdrant Vector DB - Local Embedded)]
        Backend <-->|Similarity Search| Qdrant
        Backend <-->|Local Inference| OllamaLLM[Ollama LLM Engine: qwen3 / gemma3]
    end

👥 Target Audience: Who Can Use LocalMind AI?

🎓 Final Year CS & IT Students

Perfect Capstone/Thesis Topic: Demonstrate full-stack engineering competency, database normalization (SQLite), local vector spaces (Qdrant), and advanced prompt formatting.
Hands-on AI Experience: Understand the mathematical fundamentals of semantic embeddings, retrieval mechanics, overlapping chunk parameters, and context engineering.

💼 Software Engineers & AI Developers

Offline Enterprise Boilerplate: Use this as a starter repo to build on-premise document search solutions for compliance-heavy departments.
Zero-Cost Prototyping: Avoid pay-per-token API fees while evaluating system behavior, system prompts, and ingestion performance.

🔒 Researchers, Legal, & Medical Professionals

Secure Document Analysis: Search and summarize highly classified client briefs, research publications, or patient records without violating confidentiality agreements or privacy compliance guidelines.

🛠️ Ingestion & In-depth RAG Design Parameters

To achieve optimal retrieval precision, LocalMind AI uses highly customized text processing metrics:

Document Reader: Powered by PyMuPDF for lightweight, rapid text extraction from scanned and text-based PDFs.
Chunk Size: Approximately 800 characters per chunk, avoiding sentence splitting where possible.
Chunk Overlap: 120 characters to ensure semantic continuity between adjacent text segments.
Embedding Model: nomic-embed-text (768-dimension vectors) run locally.
Vector Metric: Cosine similarity for accurate semantic search scores.

🏁 How to Setup and Run Locally

📋 Prerequisites

Operating System: Windows 10/11, macOS, or Linux
Python: Version 3.11+
Node.js: Version 20+
Package Managers: uv (Fast Python package manager) and npm
Inference Engine: Ollama installed and running

Step 1: Initialize Local Models

Start the Ollama daemon and pull the model weights:

# Pull the default RAG chat model
ollama pull qwen3:8b

# Pull the default text vectorization model
ollama pull nomic-embed-text

Step 2: Run the Automatic Setup Script

LocalMind AI features automated setup scripts to configure virtual environments, database migrations, and folders.

On Windows (PowerShell):

.\scripts\setup-windows.ps1

To download default models during setup:

.\scripts\setup-windows.ps1 -PullModels

Step 3: Launch Local Servers

Run the development servers:

.\scripts\start-dev.ps1

Interactive Web UI: http://localhost:5173
API Swagger Documentation: http://127.0.0.1:8000/docs
Backend JSON Health Check: http://127.0.0.1:8000/api/v1/health

❓ Frequently Asked Questions (FAQ) — GEO Optimization

Q1: Why is a local RAG pipeline better than using OpenAI APIs?

A: Building an offline RAG pipeline ensures absolute data privacy and zero ongoing token costs. By hosting models like qwen3 and nomic-embed-text locally, sensitive PDFs and private chat histories never travel to external cloud environments. Additionally, local operations remain functional even when disconnected from the internet.

Q2: What is the purpose of running Qdrant in "Embedded Mode"?

A: Running Qdrant in local embedded mode means the vector database initializes directly inside your Python FastAPI process. It saves vectors directly to local disk folders under ./data/qdrant without requiring developers to manage, install, or run Docker containers.

Q3: How does the citation mapping prevent AI hallucinations?

A: In LocalMind AI, the model is strictly instructed to only answer based on the retrieved snippets. If a text chunk is retrieved, the backend maps it to a unique index key. The prompt formats these contexts into strict templates. This prevents the LLM from fabricating outside facts, and forces it to cite exact file names and page numbers (e.g., [S1, Page 4]).

Q4: Can I run this offline RAG project on a standard laptop?

A: Yes. The backend is compatible with CPU-only inference, but running a GPU (e.g., NVIDIA RTX series) significantly improves token generation speeds. For laptops with lower RAM or no dedicated GPU, lightweight models such as gemma3:4b are recommended fallback options.

Local-First Offline AI Chatbot with PDF RAG

Project Screenshots

Project Description