The Problem

A client came to me with a deceptively simple request: "We want our team to be able to query our internal data using plain English." What started as a weekend prototype turned into a three-month production build processing 10,000+ daily queries.

Here's what I learned.

Architecture Overview

The stack I settled on after a few wrong turns:

• Next.js 14 App Router — React Server Components handle auth and data fetching without exposing secrets to the client
• OpenAI GPT-4o — for query understanding and response generation
• MongoDB Atlas — primary store + vector search for semantic retrieval
• Vercel Edge Functions — streaming responses with global sub-50ms cold starts
• Clerk — role-based authentication without the boilerplate

Streaming Is Non-Negotiable

The biggest UX lesson: users will not wait for a 4-second static response. Streaming changes everything.

// app/api/chat/route.ts
import { OpenAIStream, StreamingTextResponse } from "ai"
import OpenAI from "openai"

const openai = new OpenAI()

export const runtime = "edge"

export async function POST(req: Request) {
  const { messages } = await req.json()

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    stream: true,
    messages,
  })

  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}

On the client, Vercel's useChat hook handles the stream automatically — tokens appear as they're generated, making the experience feel instant even on slow connections.

Role-Based Access Done Right

Different user roles need different data access. A sales rep shouldn't be able to query engineering incident logs. I solved this by injecting a system prompt tailored to the user's role:

const ROLE_PROMPTS: Record<string, string> = {
  sales: "You have access to CRM data, deal stages, and revenue metrics only.",
  engineering: "You have access to deployment logs, error rates, and system metrics.",
  admin: "You have full access to all company data.",
}

const systemPrompt = ROLE_PROMPTS[user.role] ?? ROLE_PROMPTS.sales

Combined with Clerk's auth() helper in Server Components, this ensures the role check happens server-side — never trust the client.

Vector Search for Context Retrieval

Raw GPT-4o doesn't know your internal data. The trick is retrieval-augmented generation (RAG): embed the user's query, find the most semantically similar documents in your database, and inject them as context.

MongoDB Atlas Vector Search made this surprisingly straightforward:

const embedding = await openai.embeddings.create({
  model: "text-embedding-3-small",
  input: userQuery,
})

const results = await collection.aggregate([
  {
    $vectorSearch: {
      index: "vector_index",
      path: "embedding",
      queryVector: embedding.data[0].embedding,
      numCandidates: 100,
      limit: 5,
    },
  },
]).toArray()

The top 5 chunks land in the system prompt. Response quality improved dramatically compared to naive prompting.

What I'd Do Differently

Rate limiting earlier. We hit OpenAI's limits on day two after a demo went viral internally. Add per-user rate limits from day one.

Cost tracking from the start. Token usage adds up fast. I added a tokensUsed field to every request log in week three — should have been week one.

Evals before you ship. Setting up a lightweight eval suite (even just 20 golden Q&A pairs) would have caught two regressions before users did.

The Result

The dashboard now handles 10,000+ queries daily, stays under $200/month in API costs, and has a 94% user satisfaction score in our internal survey. The client has since expanded the project to three more departments.

The whole thing took 12 weeks solo. If you're building something similar and want to talk architecture, get in touch.