How to Set Up a Local AI Stack With Ollama, Open WebUI, and Qdrant

The era of renting your AI brain by the token is coming to a close. For anyone who values privacy, wants to avoid monthly API bills, or needs to experiment without data leaving their machine, self-hosting a local AI stack is no longer a pipe dream; it is a practical reality. This guide walks you through setting up a complete, production-ready local AI infrastructure using three open‑source powerhouses: Ollama for streamlined LLM serving, Open WebUI for a polished chat interface with built‑in RAG (Retrieval-Augmented Generation) support, and Qdrant as the high‑performance vector database that makes your documents searchable without sending them to a cloud. The stack runs comfortably on consumer hardware—a modern laptop with 16 GB of RAM and a decent GPU will suffice. By the end of this tutorial, you will have your own private AI assistant that can search, summarize, and reason over your local files, all running on your terms.

Why This Stack? – Ollama, Open WebUI, and Qdrant Defined

Before diving into installation, it is worth understanding how these three tools complement each other. Ollama acts as the LLM server: it downloads, manages, and runs quantised models (like Llama 3.2, Mistral, Gemma) with minimal fuss. Its API is compatible with OpenAI’s, meaning any client that speaks that dialect can plug right in. Open WebUI (formerly Ollama WebUI) is the front‑end that wraps Ollama’s API in a clean, ChatGPT‑like interface. It supports multiple models, conversation history, markdown rendering, and—critically—a built‑in RAG pipeline that can connect to external vector databases. Qdrant is the vector database of choice here: it is written in Rust, offers extremely fast ANN (Approximate Nearest Neighbour) search, and runs equally well as a single binary or a Docker container. Together, they deliver a self‑contained loop: you chat in Open WebUI, which queries Ollama for completions, and when you ask about your documents, Open WebUI embeds the query, searches Qdrant for relevant chunks, and injects the context into the prompt.

The beauty of this stack is its modularity. You can swap out the model in Ollama without touching the WebUI or the vector store. You can scale Qdrant to a cluster later, but for a personal setup, a single instance is plenty. And because everything runs locally, latency is near zero and your data never leaves your network. The only trade‑off is hardware—larger models require more VRAM—but with the right quantisation, even a 7‑billion‑parameter model runs fluidly on a consumer GPU.

Setting Up Ollama – The LLM Server

Ollama’s charm is its simplicity. On Linux or macOS, a single install command from the official site fetches the binary. Windows users should use WSL2, though a native Windows client is in beta. After installation, verify that the Ollama service is running by opening a terminal and typing ollama --version. Then pull your first model: ollama pull llama3.2:3b downloads a 3‑billion‑parameter instruction‑tuned model that runs comfortably on 8 GB of RAM. For deeper reasoning tasks, try ollama pull mistral:7b or llama3.2:7b. The list of supported models is at the Ollama library; you can also import custom GGUF files.

Once a model is pulled, test it directly: ollama run llama3.2:3b "Explain how a transformer works." You should see a streaming response. By default, Ollama exposes an HTTP API on port 11434. Confirm with curl http://localhost:11434/api/tags. This endpoint is what Open WebUI will connect to. For production, you may want to bind Ollama to 0.0.0.0 and enable authentication, but on a local machine the default is fine. One opinionated tip: use OLLAMA_KEEP_ALIVE=0 in your environment to unload models from VRAM when idle—saves memory for Qdrant.

Installing Open WebUI – The Chat Interface

Open WebUI can be installed via Python’s pip or, more reliably, Docker. The Docker method is cross‑platform and ensures all dependencies (like the embedding model) are self‑contained. Run:

docker run -d -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Access the interface at http://localhost:3000. The first setup prompts you to create an admin account. Once inside, navigate to “Admin Settings” → “Connections” and set the Ollama base URL to http://host.docker.internal:11434 on Docker for Windows/Mac, or http://localhost:11434 if you run the WebUI natively. You will instantly see the available models and can start chatting. Open WebUI also supports image generation via another backend, but that is optional.

To enable RAG later, you need to configure an embedding model. In the same Admin Settings, go to “RAG” and set the “Embedding Model” to something like nomic-embed-text (you will pull it via Ollama: ollama pull nomic-embed-text). This model converts text chunks into vectors. For the vector database backend, select “Qdrant” and provide the connection details—we will set that up next. The WebUI also supports local document upload (PDF, TXT, MD, etc.) and will automatically chunk, embed, and store them into Qdrant after ingestion.

Deploying Qdrant as the Vector Database

Qdrant is exceptionally easy to launch. The recommended method is Docker, which gives you a persistent database and a neat web UI for debugging:

docker run -d -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  --name qdrant \
  qdrant/qdrant

Port 6333 is the gRPC/HTTP API, and 6334 serves the web UI at http://localhost:6334/dashboard. Verify the API works: curl http://localhost:6333/collections should return an empty list. For a local personal setup, no authentication is needed; later you can add an API key if you expose the stack to a network. Qdrant stores vectors in collections; the WebUI will automatically create the collection named documents when you first upload a file, so you do not need to manually define schemas.

If you prefer a lighter footprint, Qdrant also ships as a single binary. Download it, run ./qdrant, and it works identically. The vector index uses HNSW by default, which gives high recall on modest hardware. One important config: set the segment size to auto or a value that fits your memory (e.g., 2 GB). For a stack running on 16 GB RAM, Qdrant typically uses under 1 GB for a few thousand documents. Ensure the Ollama embedding model (nomic-embed-text) dimension is 768, which Qdrant handles easily.

Connecting Everything for Full RAG Capabilities

With all three services running, the true power emerges when you feed your own documents into the loop. In Open WebUI, click on the “Workspace” icon (the document/vector icon) and upload a PDF or text file. The WebUI will split it into chunks (default 1000 characters, 200 overlap), embed each chunk using the Ollama embedding model, and store the vectors in the Qdrant collection you specified. Once indexed, any chat that includes “#” followed by the document name will trigger RAG: Open WebUI searches Qdrant for the top‑K relevant chunks, prepends them to your question, and forwards the enriched prompt to Ollama.

You can test this by starting a conversation and referencing an uploaded document. For example, after uploading a technical manual, ask “What are the safety warnings in the manual?” The response will cite specific sections. Under the hood, Open WebUI uses the /v1/embeddings endpoint of Ollama to create vectors and Qdrant’s search endpoint to retrieve results. The latency is surprisingly low—typically under a second for a 10‑document corpus. If you want to ingest entire folders, use a small script that calls the Open WebUI API (it exposes a Swagger interface at /docs). The entire stack can run on a single machine as long as you allocate enough RAM to hold the model, the vector index, and the WebUI process.

Optimising for Consumer Hardware – Tips and Trade‑Offs

Running three services on a laptop means being mindful of resources. Here are hard‑won tips:

• Choose quantised models. Ollama defaults to Q4_K_M quantisation, which compresses a 7 GB model to about 4.5 GB. For a 4 GB VRAM card, stick with 3‑billion‑parameter models like llama3.2:3b. They are surprisingly capable for summarisation and Q&A.
• Tweak Ollama’s concurrency. By default, Ollama only processes one request at a time. If you want to embed documents while chatting, set OLLAMA_NUM_PARALLEL=2 (though this increases VRAM pressure). Alternatively, batch your embeddings offline.
• Use a separate embedding model for RAG. Ollama loads embedding models differently from generative models—they are much smaller. nomic-embed-text is about 130 MB, so it can stay in memory alongside your chat model without significant overhead.
• Monitor with htop and nvidia-smi. If Qdrant eats too much RAM, reduce the optimizers_segment_number config to 1 or set storage.performance.max_search_threads to 2. For disk space, Qdrant stores vectors in /qdrant/storage; a thousand documents with 768‑dimensional vectors take about 100 MB.
• Consider a lightweight WebUI alternative if Open WebUI feels heavy. But for most users, the feature set (RAG, model switching, conversation folders) justifies the memory footprint of about 200 MB.

One trade‑off to accept: consumer hardware cannot run 70‑billion‑parameter models. But for personal knowledge management and code assistance, the 7B class models are more than adequate. The local stack also gives you the freedom to experiment with fine‑tuning later, which cloud APIs cannot easily replicate without expensive custom instances.

Conclusion

You now have a fully self‑hosted AI stack that respects your privacy, eliminates per‑query token costs, and lets you iterate without bureaucratic approval. The combination of Ollama, Open WebUI, and Qdrant provides a foundation that can scale from a single laptop to a small server cluster, yet remains manageable for a single developer or knowledge worker. Start by uploading a handful of documents—your notes, white papers, or manuals—and test the RAG pipeline. The sense of ownership and control you gain is not just philosophical; it translates into faster experiment cycles and deeper understanding of how retrieval‑augmented generation actually works. Dive in, break things, and join the growing community of self‑hosters who are proving that local AI is not just a hobby—it is the future of practical machine intelligence.

Frequently Asked Questions

Can I run this stack on a laptop without a dedicated GPU?

Yes, but with caveats. Ollama can fall back to CPU inference, though performance will be significantly slower—a 7B model might generate only a few tokens per second. For acceptable speeds, use a 3B‑parameter model or enable GPU offloading if you have an integrated GPU with enough shared memory. Qdrant runs fine on CPU because indexing is already optimised for x86. The WebUI itself is lightweight. Expect response times of 5–15 seconds for chat completions on a CPU‑only machine.

What models work best for RAG on a 16 GB RAM machine?

The sweet spot is a 7B model using 4‑