RAG and Embeddings: How to Give Your LLM Access to Your Company’s Private Knowledge

Enterprise clients exploring AI always land on one question: "How do I get the LLM to use MY data instead of generic web results?" The short answer is RAG (Retrieval-Augmented Generation). Let’s break it down.

What RAG Actually Means

RAG stands for Retrieval-Augmented Generation. In plain English: you fetch the most relevant documents from your knowledge base before asking the LLM, then feed those documents—along with the user’s question—as context. The LLM responds using only the information you provided.

Key point: The LLM doesn’t "learn" your data. It reads it on demand. That means whenever you update your documents, the system automatically reflects those changes—no retraining required.

Why Embeddings Are Non-Negotiable

If your knowledge base is just five documents, you could technically pass them all to the LLM with every query. But when you scale to 500 or 5,000 documents? It becomes expensive, slow, and often impossible (context windows are limited). You need a way to retrieve only the relevant chunks of text.

Keyword searches (like a traditional database) won’t cut it. If a user asks, "What’s the cost of our consulting services?" but your document says "Professional fee structure," a literal match won’t connect the dots.

Enter embeddings: numerical vectors that capture the meaning of text. Documents with similar meaning end up close together in vector space. This is semantic search—finding answers based on relevance, not just keywords.

The Critical Infrastructure Decision: Where to Store Vectors

Historically, teams used specialized vector databases like Pinecone, Weaviate, Qdrant, or Chroma. These work well but add complexity: another database to maintain, another vendor to pay, another potential failure point.

Our approach at Tontin was different. We run pgvector inside Postgres via Supabase. This PostgreSQL extension adds vector data types and lets you run semantic queries using plain SQL. The benefits:

• Single source of truth: Store both transactional data and embeddings in one database.
• Atomic consistency: When a document is updated, its embeddings refresh in the same transaction.
• Zero extra cost: No need for a separate vector DB service.
• Enterprise-grade performance: Handles millions of vectors efficiently with proper indexing (HNSW, IVFFlat).

Specialized vector databases still shine at massive scale (tens of millions of vectors, sub-100ms latency requirements), but for 90% of business use cases, pgvector is more than enough.

End-to-End RAG Workflow in Production

A functional RAG pipeline has five core components:

1. Ingestion: Load documents, split them into logical chunks (paragraphs, sections), generate embeddings for each chunk, and store them.
2. Vector Index: An optimized data structure for fast similarity searches (e.g., HNSW, IVFFlat).
3. Retrieval: Convert the user’s query into an embedding, then fetch the N most relevant chunks.
4. Prompt Engineering: Assemble the final prompt with the user’s question + retrieved chunks + instructions.
5. Generation: Send the prompt to the LLM and return the response.

Each step has hidden complexities:

• How to chunk documents effectively
• How many chunks to retrieve
• How to filter irrelevant or low-quality chunks
• How to prevent hallucinations when context is insufficient

Where RAG Delivers Real Business Value

We’ve seen RAG transform workflows in these areas:

• Internal Support: Teams query internal documentation and get answers grounded in real company policies.

• Customer-Facing Assistants: Chatbots that answer product questions using official documentation.

• Onboarding: New hires search internal wikis and process guides in natural language.

• Research & Analysis: Legal, medical, and finance teams query large document sets semantically.

The Bottom Line on RAG

For companies with structured knowledge in PDFs, wikis, emails, or file systems, RAG offers the best ROI in AI today. Compared to fine-tuning:

• No retraining required
• Lower cost (no expensive GPU clusters)
• Real-time updates (changes reflect instantly)

If your team struggles to find answers in scattered documents and wants a natural-language search experience, let’s talk. In 30 minutes, we can assess if your use case is a great fit for RAG.

---

By Esteban Aleart, Founder & Lead Engineer at Pair Programming.