# How to build an MCP server for RAG

- Tool: MCP
- Difficulty: Advanced
- Time required: 40-60 min
- Compatibility: MCP TypeScript SDK v1.x, Node.js 18+, OpenAI API, pgvector or Pinecone
- Last updated: March 2026

## TL;DR

Build an MCP server that exposes vector search as a tool, letting AI assistants perform RAG (Retrieval-Augmented Generation) on your documents. The server embeds documents using OpenAI's embedding API, stores vectors in pgvector or Pinecone, and provides a search_documents tool that returns relevant chunks with similarity scores. This gives any MCP client grounded, factual answers from your knowledge base.

## Building a RAG-Powered MCP Server for Document Search

Retrieval-Augmented Generation (RAG) lets AI assistants answer questions using your actual documents instead of relying solely on training data. This tutorial builds an MCP server that ingests documents, generates embeddings, stores them in a vector database, and exposes a search_documents tool that any MCP client (Claude Desktop, Cursor, Windsurf) can call. The AI sends a natural language query, your server returns the most relevant document chunks, and the AI uses those chunks to generate grounded answers.

## Before you start

- Node.js 18+ and npm installed
- An OpenAI API key for generating embeddings
- Either PostgreSQL with pgvector extension or a Pinecone account
- Basic understanding of vector embeddings and similarity search
- A collection of documents (Markdown, text, or PDF) to index

## Step-by-step guide

### 1. Set up the project and install dependencies

Create the project and install the MCP SDK, OpenAI client for embeddings, and your chosen vector database client. The openai package generates text embeddings. Use pg with pgvector for self-hosted PostgreSQL, or @pinecone-database/pinecone for managed Pinecone. Also install zod for input validation and a text splitter for chunking documents.

```
mkdir mcp-rag-server && cd mcp-rag-server
npm init -y
npm install @modelcontextprotocol/sdk zod openai
npm install @pinecone-database/pinecone  # or: npm install pg pgvector
npm install -D typescript @types/node
npx tsc --init
```

**Expected result:** Project initialized with all dependencies for MCP server, embeddings, and vector storage.

### 2. Build the document chunking and embedding pipeline

Documents must be split into chunks before embedding because embedding models have token limits and smaller chunks produce more precise search results. Split documents into overlapping chunks of 500-1000 tokens. Then generate embeddings for each chunk using OpenAI's text-embedding-3-small model, which produces 1536-dimensional vectors. Store the chunk text, embedding vector, and metadata (source file, position) together.

```
// src/embeddings.ts
import OpenAI from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export interface DocumentChunk {
  id: string;
  text: string;
  source: string;
  chunkIndex: number;
  embedding?: number[];
}

export function chunkText(
  text: string,
  source: string,
  chunkSize: number = 800,
  overlap: number = 200
): DocumentChunk[] {
  const chunks: DocumentChunk[] = [];
  let start = 0;
  let index = 0;

  while (start < text.length) {
    const end = Math.min(start + chunkSize, text.length);
    chunks.push({
      id: `${source}-chunk-${index}`,
      text: text.slice(start, end),
      source,
      chunkIndex: index,
    });
    start += chunkSize - overlap;
    index++;
  }
  return chunks;
}

export async function embedTexts(texts: string[]): Promise<number[][]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: texts,
  });
  return response.data.map(d => d.embedding);
}

export async function embedQuery(query: string): Promise<number[]> {
  const [embedding] = await embedTexts([query]);
  return embedding;
}
```

**Expected result:** Functions that chunk documents and generate embeddings using OpenAI's API.

### 3. Create the vector store integration with pgvector

Set up a PostgreSQL table with a vector column using the pgvector extension. The table stores chunk text, source metadata, and the embedding vector. Create an insert function for indexing and a search function that uses cosine distance for similarity queries. The search function takes a query embedding and returns the top-k most similar chunks with their similarity scores.

```
// src/vector-store.ts
import pg from "pg";

const pool = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
});

export async function initializeStore(): Promise<void> {
  await pool.query(`CREATE EXTENSION IF NOT EXISTS vector`);
  await pool.query(`
    CREATE TABLE IF NOT EXISTS document_chunks (
      id TEXT PRIMARY KEY,
      text TEXT NOT NULL,
      source TEXT NOT NULL,
      chunk_index INTEGER NOT NULL,
      embedding vector(1536) NOT NULL,
      created_at TIMESTAMP DEFAULT NOW()
    )
  `);
  await pool.query(`
    CREATE INDEX IF NOT EXISTS chunks_embedding_idx
    ON document_chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100)
  `);
}

export async function insertChunks(
  chunks: { id: string; text: string; source: string; chunkIndex: number; embedding: number[] }[]
): Promise<void> {
  for (const chunk of chunks) {
    await pool.query(
      `INSERT INTO document_chunks (id, text, source, chunk_index, embedding)
       VALUES ($1, $2, $3, $4, $5)
       ON CONFLICT (id) DO UPDATE SET text = $2, embedding = $5`,
      [chunk.id, chunk.text, chunk.source, chunk.chunkIndex, JSON.stringify(chunk.embedding)]
    );
  }
}

export async function searchSimilar(
  queryEmbedding: number[],
  topK: number = 5,
  sourceFilter?: string
): Promise<{ text: string; source: string; score: number }[]> {
  const filterClause = sourceFilter ? `WHERE source = $3` : "";
  const params = sourceFilter
    ? [JSON.stringify(queryEmbedding), topK, sourceFilter]
    : [JSON.stringify(queryEmbedding), topK];

  const result = await pool.query(
    `SELECT text, source, 1 - (embedding <=> $1::vector) AS score
     FROM document_chunks ${filterClause}
     ORDER BY embedding <=> $1::vector
     LIMIT $2`,
    params
  );
  return result.rows;
}
```

**Expected result:** A vector store module that can insert document chunks and search by similarity using pgvector.

### 4. Register the search_documents MCP tool

Create the main MCP tool that clients will call. The search_documents tool takes a natural language query, an optional number of results, and an optional source filter. It embeds the query, searches the vector store, and returns the top results formatted as text with source citations and similarity scores. The tool description is critical — it tells the AI what the tool does and when to use it.

```
// src/index.ts
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { embedQuery } from "./embeddings.js";
import { searchSimilar, initializeStore } from "./vector-store.js";

const server = new McpServer({
  name: "rag-document-server",
  version: "1.0.0",
});

server.tool(
  "search_documents",
  "Search the knowledge base using natural language. Returns the most relevant document chunks with source citations and similarity scores. Use this to answer questions about the indexed documents.",
  {
    query: z.string().describe("Natural language search query"),
    topK: z.number().min(1).max(20).default(5).describe("Number of results to return"),
    source: z.string().optional().describe("Filter by source filename"),
  },
  async ({ query, topK, source }) => {
    try {
      const queryEmbedding = await embedQuery(query);
      const results = await searchSimilar(queryEmbedding, topK, source);

      if (results.length === 0) {
        return { content: [{ type: "text", text: "No relevant documents found for this query." }] };
      }

      const formatted = results
        .map((r, i) => `[${i + 1}] (score: ${r.score.toFixed(3)}) [Source: ${r.source}]\n${r.text}`)
        .join("\n\n---\n\n");

      return { content: [{ type: "text", text: formatted }] };
    } catch (error) {
      return {
        content: [{ type: "text", text: `Error: ${error instanceof Error ? error.message : String(error)}` }],
        isError: true,
      };
    }
  }
);
```

**Expected result:** A search_documents tool that accepts natural language queries and returns relevant document chunks.

### 5. Add a document ingestion tool for indexing new content

Expose an ingest_document tool that accepts document text and source metadata, chunks it, generates embeddings, and stores the vectors. This lets the AI (or an admin) add new documents to the knowledge base without restarting the server. Include a count of chunks created so the caller knows the operation succeeded.

```
server.tool(
  "ingest_document",
  "Add a document to the knowledge base for future searches. Chunks the text, generates embeddings, and stores vectors.",
  {
    text: z.string().describe("Full document text to index"),
    source: z.string().describe("Source identifier (e.g., filename or URL)"),
    chunkSize: z.number().default(800).describe("Characters per chunk"),
  },
  async ({ text, source, chunkSize }) => {
    try {
      const chunks = chunkText(text, source, chunkSize);
      const texts = chunks.map(c => c.text);
      const embeddings = await embedTexts(texts);

      const chunksWithEmbeddings = chunks.map((c, i) => ({
        ...c,
        embedding: embeddings[i],
      }));

      await insertChunks(chunksWithEmbeddings);

      return {
        content: [{ type: "text", text: `Ingested "${source}": ${chunks.length} chunks indexed.` }],
      };
    } catch (error) {
      return {
        content: [{ type: "text", text: `Error: ${error instanceof Error ? error.message : String(error)}` }],
        isError: true,
      };
    }
  }
);
```

**Expected result:** An ingest_document tool that chunks, embeds, and stores documents in the vector database.

### 6. Configure Claude Desktop or Cursor to use the RAG server

Add the server to your MCP client configuration. For Claude Desktop, edit claude_desktop_config.json. For Cursor, edit .cursor/mcp.json. Set the OPENAI_API_KEY and DATABASE_URL environment variables so the server can connect to the embedding API and vector database at startup. Teams building complex RAG pipelines often work with RapidDev to optimize chunking strategies and embedding model selection for their specific document types.

```
// Claude Desktop: ~/Library/Application Support/Claude/claude_desktop_config.json
// Cursor: .cursor/mcp.json
{
  "mcpServers": {
    "rag-documents": {
      "command": "node",
      "args": ["dist/index.js"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DATABASE_URL": "postgresql://user:pass@localhost:5432/ragdb"
      }
    }
  }
}
```

**Expected result:** The RAG server appears in your MCP client's tool list and responds to search queries with relevant document chunks.

## Complete code example

File: `src/index.ts`

```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import OpenAI from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// --- Chunking ---
function chunkText(text: string, source: string, size = 800, overlap = 200) {
  const chunks: { id: string; text: string; source: string; index: number }[] = [];
  let start = 0, i = 0;
  while (start < text.length) {
    chunks.push({ id: `${source}-${i}`, text: text.slice(start, start + size), source, index: i });
    start += size - overlap;
    i++;
  }
  return chunks;
}

// --- Embeddings ---
async function embed(texts: string[]): Promise<number[][]> {
  const res = await openai.embeddings.create({ model: "text-embedding-3-small", input: texts });
  return res.data.map(d => d.embedding);
}

// --- In-memory vector store (replace with pgvector/Pinecone in production) ---
const store: { id: string; text: string; source: string; embedding: number[] }[] = [];

function cosineSim(a: number[], b: number[]): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i]*b[i]; na += a[i]*a[i]; nb += b[i]*b[i]; }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

function search(queryEmb: number[], topK: number, source?: string) {
  let items = source ? store.filter(s => s.source === source) : store;
  return items
    .map(item => ({ ...item, score: cosineSim(queryEmb, item.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

// --- MCP Server ---
const server = new McpServer({ name: "rag-server", version: "1.0.0" });

server.tool("search_documents", "Search knowledge base with natural language", {
  query: z.string().describe("Natural language query"),
  topK: z.number().min(1).max(20).default(5),
  source: z.string().optional(),
}, async ({ query, topK, source }) => {
  const [qEmb] = await embed([query]);
  const results = search(qEmb, topK, source);
  if (!results.length) return { content: [{ type: "text", text: "No results found." }] };
  const text = results.map((r, i) =>
    `[${i+1}] (${r.score.toFixed(3)}) [${r.source}]\n${r.text}`
  ).join("\n---\n");
  return { content: [{ type: "text", text }] };
});

server.tool("ingest_document", "Add a document to the knowledge base", {
  text: z.string(), source: z.string(), chunkSize: z.number().default(800),
}, async ({ text, source, chunkSize }) => {
  const chunks = chunkText(text, source, chunkSize);
  const embeddings = await embed(chunks.map(c => c.text));
  chunks.forEach((c, i) => store.push({ ...c, embedding: embeddings[i] }));
  return { content: [{ type: "text", text: `Indexed ${chunks.length} chunks from ${source}` }] };
});

server.tool("list_sources", "List all indexed document sources", {}, async () => {
  const sources = [...new Set(store.map(s => s.source))];
  return { content: [{ type: "text", text: JSON.stringify(sources, null, 2) }] };
});

async function main() {
  const transport = new StdioServerTransport();
  await server.connect(transport);
  console.error("RAG MCP server running");
}
main().catch(e => { console.error(e); process.exit(1); });
```

## Common mistakes

- **Using chunks that are too large (2000+ tokens), resulting in poor search precision** — undefined Fix: Keep chunks between 500-1000 characters with 100-200 character overlap for the best balance of precision and context.
- **Not including overlap between chunks, causing information at chunk boundaries to be lost** — undefined Fix: Use 20-25% overlap (e.g., 200 characters overlap for 800 character chunks) to ensure boundary content is captured.
- **Embedding the query with a different model than the documents** — undefined Fix: Always use the same embedding model for both document indexing and query embedding. Mixing models produces meaningless similarity scores.
- **Not handling empty search results gracefully** — undefined Fix: Return a clear message like 'No relevant documents found' instead of an empty array or error when no results match.

## Best practices

- Use the same embedding model for both indexing and querying — never mix models
- Chunk documents at 500-1000 characters with 20% overlap for optimal retrieval
- Include source citations and similarity scores in search results so the AI can assess relevance
- Filter by source metadata to narrow search scope when the user specifies a document
- Store raw text alongside vectors so you can re-embed when upgrading to a new model
- Use IVFFlat or HNSW indexes in pgvector for sub-100ms search on large collections
- Batch embedding requests to reduce API calls and latency
- Log query terms and result counts to stderr for monitoring search quality

## Frequently asked questions

### What embedding model should I use for RAG with MCP?

OpenAI's text-embedding-3-small is the best balance of cost and quality for most use cases at $0.02 per million tokens. For higher accuracy on technical content, use text-embedding-3-large at $0.13 per million tokens.

### How many documents can I index in a single MCP RAG server?

With pgvector and proper indexing (IVFFlat or HNSW), you can search millions of chunks with sub-100ms latency. The bottleneck is usually embedding generation speed, not search speed.

### Should I use pgvector or Pinecone for MCP RAG?

Use pgvector if you already have PostgreSQL or want to self-host. Use Pinecone if you want a fully managed service with built-in scaling. Both work well as MCP tool backends.

### How do I update documents that have already been indexed?

Delete the old chunks by source identifier, then re-ingest the updated document. Use ON CONFLICT DO UPDATE in PostgreSQL to handle upserts automatically.

### Can I use local embedding models instead of OpenAI?

Yes. Use Ollama with a model like nomic-embed-text for fully local embeddings. Replace the OpenAI embed function with an HTTP call to Ollama's embedding endpoint.

### What chunk size works best for code documentation?

For code and technical documentation, use smaller chunks (400-600 characters) with higher overlap (30%). Code has more information density per character than prose, so smaller chunks improve precision.

---

Source: https://www.rapidevelopers.com/mcp-tutorial/how-to-build-mcp-server-for-rag
© RapidDev — https://www.rapidevelopers.com/mcp-tutorial/how-to-build-mcp-server-for-rag