Building Jarvis, a Multi-User Voice-First AI Agent Platform

Jarvis started as a simple question: what would it take to build a personal AI assistant that doesn't just answer questions, but can actually do things — open a tab, fill a form, remember what I told it last week — safely, and for more than one user at a time?

This post walks through what I ended up building: the services, how a request actually moves through the system, the security model behind letting an AI click things in a real browser, and the tech stack and folder structure underneath it all.

The JARVIS on github is Jarvis.

What Jarvis actually does

At its core, Jarvis is a voice-first AI assistant with two clients — a React web dashboard and a Chrome extension — that share one backend. A user can:

• Register and log in with a normal email/password flow
• Start a conversation and talk to the agent by voice or text
• Get a streamed response instead of waiting for the whole answer at once
• Save and semantically search long-term memories ("remember that I prefer concise answers")
• Ask the agent to take real browser actions — open a tab, click something, fill a form — with sensitive actions gated behind a confirmation prompt

That last point is the part that makes this more than a chatbot wrapper: Jarvis can act inside your browser, not just talk.

High-level architecture

Everything funnels through one public entry point — a stateless API gateway — which then talks to a set of internal services over gRPC:

Web app / Chrome extension
          |
          | HTTP JSON, NDJSON, or WebSocket
          v
Nginx (optional) -> API Gateway
                        |
                        | gRPC
       +----------------+----------------+
       v                v                v
 Auth service     Agent service     Worker service
       |                |                |
       +--------+-------+                |
                v                        v
        PostgreSQL/Prisma       Redis + BullMQ
                v                        |
          Qdrant vectors <---------------+
                v
             Groq LLM

The gateway is deliberately "dumb" — it authenticates requests, routes them, and enforces rate limits, but it never touches business data directly. That responsibility is split across three internal services:

None of these three services are reachable from the public internet directly — only the gateway is. Every internal call carries a RequestContext with a userId that the gateway itself derived from a verified JWT, never one supplied by the client. That single design decision is what makes the whole system safe for multiple users: every downstream query — conversations, messages, memories — is scoped by that server-verified ID, so one user's data is structurally unreachable from another user's session.

Walking through a real request

Registering and logging in

Client → POST /auth/register → Gateway → gRPC → Auth service
                                                     |
                                              bcrypt hash password
                                              create User row
                                              issue access + refresh JWT
                                                     |
Client ← session { accessToken, refreshToken, userId } ←

Access tokens are short-lived (15 minutes); refresh tokens last 7 days and rotate on every use, with only their hash stored server-side. From here on, every request carries Authorization: Bearer <accessToken>.

Sending a voice command

This is the flow that matters most, because it's where latency, memory, and tool-calling all intersect:

1. The client sends POST /agent/voice-command with a transcript and optional browser context (current tab URL, title, selected text).
2. The gateway verifies the token, builds a RequestContext, and opens a server-streaming gRPC call to the agent service.
3. Agent service loads the last ~30 messages of conversation history (from Redis cache, falling back to PostgreSQL on a miss).
4. It builds an AgentState, hands the LLM the conversation, the browser context, and a registry of available tools.
5. If the model wants to call a tool — calculator, weather, memory search, or a browser action — the tool registry executes it, results go back into the state, and the loop continues (capped at four steps).
6. The final answer streams back token by token as AgentResponseChunk messages.
7. The gateway converts each chunk into one line of NDJSON, and the client renders it as it arrives.
8. Once the stream finishes, agent service enqueues a conversation-persist job — it does not wait for the database write before finishing the response, which is what keeps time-to-first-token low.
9. The worker service picks up that job, writes the user and assistant messages inside a single Postgres transaction, and invalidates the relevant cache keys.

That last step is a deliberate tradeoff worth naming: the response feels instant because writing to Postgres happens after the user has already seen the answer. The cost is that a queue outage could make a turn look successful in the UI while it was never durably saved — which is why queue health monitoring belongs on the roadmap for a system built this way.

Memory as retrieval-augmented generation

Jarvis's memory system is a real, working RAG pipeline, just scoped to personal memory rather than general documents:

Save:   content → embed (384-dim) → Qdrant upsert (payload: userId, content, category)
                                  → Postgres row (durable copy, vectorId link)
 
Search: query → embed → Qdrant nearest-neighbor search, filtered by userId
              → results above score threshold → merged into agent context

PostgreSQL is the durable source of truth; Qdrant is purely a search index over the same content. If the two ever disagree, Postgres wins — which is why a production version of this needs an outbox or reconciliation process to keep them in sync after partial failures.

Browser automation, and why it needs a safety layer

Letting an LLM click things in a real browser is the single riskiest capability in the whole system, so it gets the most deliberate design of anything here.

LLM proposes an action
        |
        v
actionClassifier tags it: safe | sensitive | blocked
        |
   +----+-----------------+------------------+
   |                       |                  |
 safe                  sensitive           blocked
   |                       |                  |
dispatch            confirmation UI      rejected, never
immediately          waits for user       leaves the server
   |                  CONFIRM/CANCEL
   v                       |
Redis Pub/Sub  <-----------+
   |
Gateway WebSocket (/ws/extension)
   |
Chrome extension → dom-actions.js executes it → ACTION_RESULT

A few rules make this actually trustworthy rather than just theatrical:

• The LLM never gets to self-certify risk. The classifier runs independently on the server and inspects selectors, labels, and page metadata — patterns like password, cvv, card-number, or private-key get blocked before the request ever reaches the extension.
• The extension enforces the denylist a second time. Defense in depth: even a compromised or buggy server-side classifier can't get the extension to fill a password field.
• Every action is server-generated, user-bound, and time-boxed. An actionId is minted server-side, tied to the authenticated user's socket, and expires after 30–60 seconds — so a stale or replayed confirmation can't execute an action the user no longer intends.
• The agent never assumes success. It waits for an explicit ACTION_RESULT from the extension before continuing, rather than optimistically assuming a dispatched action worked.

Every action — safe, sensitive, or blocked — gets written to an audit table with its risk tier, sanitized payload, and final status. That audit trail is as much a security feature as a debugging one.

Tech stack

Folder structure

The repository is organized around clear service boundaries — each backend service is independently deployable, and the gateway is the only one with public ingress:

jarvis/
├── shared/proto/                 # gRPC contracts shared across services
├── database/prisma/schema.prisma # Canonical relational schema
├── apps/
│   ├── web-app/                  # React dashboard
│   └── chrome-extension/         # Manifest V3 extension
├── backend/
│   ├── gateway/                  # Public HTTP + WebSocket entrypoint
│   └── services/
│       ├── auth-service/         # Identity, JWTs, sessions
│       ├── agent-service/        # LangGraph loop, tools, memory, browser dispatch
│       └── worker-service/       # BullMQ consumers, async persistence
└── compose.yaml                  # Local dev topology

Inside agent-service, the structure mirrors the request lifecycle: langgraph/ holds the reasoning loop and tool registry, memory/ owns the Qdrant read/write path, browser/ holds the risk classifier and action dispatcher, and cache/ wraps the Redis read-through logic — each concern gets its own folder rather than living in one large controller file.

What I'd still call unfinished

Being honest about the gaps matters as much as describing what works:

• Health checks currently confirm a process is running, not that its dependencies (Postgres, Redis, Qdrant) are actually reachable.
• The worker's Redis fallback path can report success without a durable write — a fail-open behavior that's fine for a demo and wrong for production.
• CORS is intentionally broad in development and needs tightening before a public deployment.
• Postgres and Qdrant can drift out of sync if one write succeeds and the other fails, with no reconciliation process yet.

None of these are architecture problems — they're the normal, honest list of what's next once the core loop works end to end.

Closing thoughts

The interesting part of building Jarvis wasn't wiring up an LLM to answer questions — that part is almost commoditized at this point. It was everything around it: making multi-tenancy actually safe rather than a userId column and a prayer, deciding what an AI agent should be allowed to do autonomously versus what needs a human in the loop, and being deliberate about where latency matters (streaming the first token fast) versus where correctness matters more (never letting the agent assume an action succeeded before it's confirmed).

If there's one takeaway from this build, it's that the hard part of an "AI agent" isn't the AI — it's the agent part: the boundaries, the confirmations, and the audit trail that make it trustworthy enough to actually use.