AI SaaS Architecture Guide

The Architecture, Upfront

Before the breakdown: here is what a production-ready AI SaaS looks like when you get the decisions right.

┌─────────────────────────────────────────────────────────────────┐
│                         Client Layer                            │
│           (SSE stream / polling / webhook callback)             │
└───────────────────────────┬─────────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                      API Gateway / Edge                         │
│              Auth (JWT) → Rate Limit → Tenant ID                │
└───────────────────────────┬─────────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                    AI Request Pipeline                          │
│   Prompt Construction → Model Abstraction → Output Validation   │
│              ↕ Usage Meter  ↕ Cost Attribution                  │
└──────────┬──────────────────────────────────────┬───────────────┘
           │ (sync, <10s)                          │ (async)
┌──────────▼──────────┐               ┌────────────▼──────────────┐
│   LLM Provider(s)   │               │     Job Queue (BullMQ /    │
│  OpenAI / Anthropic │               │     Inngest / Temporal)    │
│  / Gemini / local   │               └────────────┬──────────────┘
└─────────────────────┘                            │
                                       ┌────────────▼──────────────┐
                                       │   Worker Fleet             │
                                       │  (long-running AI tasks)  │
                                       └────────────┬──────────────┘
                                                    │
┌───────────────────────────────────────────────────▼──────────────┐
│                         Data Layer                               │
│  Postgres (tenant rows) │ Redis (quotas) │ S3 (artifacts/logs)   │
│  Vector DB (per-tenant context) │ ClickHouse (usage analytics)   │
└──────────────────────────────────────────────────────────────────┘

Key decisions baked into this architecture:

• Tenant isolation at the data layer, not the application layer. Every row carries a tenant_id. Row-level security in Postgres (or equivalent) enforces it.
• A model abstraction layer between your business logic and any specific LLM provider. You never call openai.chat.completions.create() directly from feature code.
• Async by default for anything over ~8 seconds. Synchronous LLM calls are a liability at scale.
• Usage is metered at the pipeline layer, before the response is returned, so it cannot be bypassed.
• Feedback is a first-class data type — not an afterthought, not a dashboard nobody checks.

The rest of this article explains why each decision is made this way and how to implement it.

1. Multi-Tenant Data Architecture for AI

AI SaaS has a harder multi-tenancy problem than traditional SaaS because the “data” that customizes the product is not just rows in a table — it is prompt templates, embedded context, fine-tuned adapters, and interaction history, all of which need per-tenant isolation and lifecycle management.

Tenant context lives in the database, not the codebase. Every tenant should have a row (or namespace) that controls:

• Their system prompt template (customizable if your product supports it)
• Feature flags that gate which AI capabilities they have access to
• Model preferences (if you offer model selection)
• Domain-specific terminology or instructions passed in every request

Use Postgres with tenant_id as a mandatory column on every AI-related table. Enable Row Level Security and set a session-level variable (SET app.tenant_id = $1) at connection time. This makes it structurally impossible to accidentally cross-contaminate tenant data. Supabase’s RLS model is a reasonable reference implementation.

Vector stores are per-tenant. If your product does retrieval-augmented generation, each tenant’s embedded corpus must be isolated. In Pinecone, use namespaces keyed to tenant_id. In pgvector, the same row-level security model applies. Do not share a single embedding namespace across tenants — the security and relevance problems compound.

Fine-tuned models require a registry. If you fine-tune models per tenant (Replicate, Together AI, and OpenAI all support this), you need a model_registry table that maps (tenant_id, task_type, provider) to a specific model ID. Your model abstraction layer queries this table at inference time. When a fine-tune is deprecated or a better one is deployed, you update one row, not your codebase.

2. The AI Request Pipeline

Every AI request in your system should pass through the same pipeline. This is not a microservice — it is a function call chain or middleware stack, depending on your architecture. The stages:

Request → [Auth] → [Rate Limit] → [Quota Check] → [Prompt Construction]
       → [Model Abstraction] → [LLM Call] → [Output Validation]
       → [Usage Recording] → [Logging] → Response

Auth resolves the tenant and user from the request token. In a JWT-based system, the tenant ID and user ID should be in the token claims. Do not do a database lookup here if you can avoid it — keep this path as fast as possible.

Rate limiting protects you from runaway clients and denial-of-wallet attacks. Use Redis with a sliding window counter keyed to (tenant_id, endpoint). Upstash is a managed Redis that works well for this at serverless scale. Set separate limits for tokens-per-minute and requests-per-minute — token limits matter more for cost, but request limits catch loops.

Quota check is distinct from rate limiting. Rate limits are time-windowed; quotas are billing-period caps. A tenant on your Growth plan gets 500k tokens/month. When they hit that, the request fails with a 402, not a 429. This check hits Redis (for speed) against a counter that your billing reconciliation job keeps synchronized with your source-of-truth (Stripe usage records or your own ledger in Postgres).

Prompt construction is where you assemble the actual LLM input. Retrieve the tenant’s system prompt template, inject relevant context from the vector store (if applicable), append the conversation history within your context window budget, and add the user message. This step should be unit-testable in isolation — your prompt construction logic should be a pure function that takes inputs and returns a message array.

Output validation catches the cases where the LLM returns structurally invalid output. If you’re expecting JSON, use zod or pydantic to validate the schema. If the model returns malformed output, retry once with a stricter instruction before surfacing an error. Instructor (Python) and zod with structured output mode (TypeScript) are the right tools here.

Logging is non-negotiable. Log the full request and response — prompt, completion, model, latency, token counts, tenant ID, user ID, request ID — to an append-only store (S3 or your data warehouse). This powers debugging, evaluation, and the data flywheel. Use LangSmith, Braintrust, or a homegrown ClickHouse table. Do not rely on provider dashboards.

3. Async Job Architecture for Long-Running AI Tasks

Any AI operation that can take more than 8–10 seconds should be async. This includes multi-step agent loops, document processing pipelines, batch generation jobs, and anything that chains more than two LLM calls.

The pattern:

1. Client submits a job → you return a job_id immediately (HTTP 202)
2. Job is enqueued to your task queue
3. Worker picks it up and processes it
4. On completion, either: push a webhook to the client’s configured URL, or let the client poll GET /jobs/{job_id}/status
5. Artifacts (generated files, structured results) are stored in S3; the status response includes signed URLs

Queue choices: BullMQ (Redis-backed, Node.js) is the right default for most teams. Inngest is worth considering if you want built-in retry logic, fan-out, and observability without running your own Redis. Temporal is the right choice if your workflows are long-lived (minutes to hours), need durability across restarts, or have complex branching — it is more operational overhead but the correct tool for agent orchestration.

Worker scaling: Workers should be separate processes from your API servers. In Kubernetes, this means a separate Deployment. On Railway or Render, a separate service. Workers should be horizontally scalable — the queue is the backpressure mechanism, not the worker count. Size workers for the memory footprint of your LLM calls (context window × model overhead), not CPU.

Progress reporting: For UX, you often need to report progress on long jobs, not just “running” / “done”. Store a progress field on the job record (e.g., { step: 3, total: 7, message: "Generating section 2..." }). Clients can poll this on a 2-second interval. Do not use WebSockets for this — polling is simpler, more reliable, and sufficient for task progress.

4. Per-User AI Usage Tracking and Quota Enforcement

Token costs are your largest variable expense. You need to know, per user and per feature, exactly what you are spending — and you need to be able to cap it.

Token counting at the pipeline layer. Every LLM response includes token usage in the response metadata (usage.prompt_tokens, usage.completion_tokens). Record these synchronously, in the same transaction or atomic operation that returns the response. Do not fire-and-forget to a logging queue — if the log is lost, your quota state is wrong.

Redis for real-time quota enforcement. Use an atomic Redis INCR on a key like quota:{tenant_id}:{billing_period}:{unit}. Check and increment in a single Lua script to avoid race conditions. The increment happens at the pipeline layer, before you return the response, so quota enforcement is synchronous with usage.

Postgres as source of truth. Periodically (every 5 minutes or on significant increments), flush Redis counters to a usage_events table in Postgres. This table is the audit trail. Your billing system reads from here, not Redis. The schema should include: tenant_id, user_id, feature_name, model, input_tokens, output_tokens, cost_usd, created_at.

Quota tiers. Your pricing tiers should map directly to quota limits in a tenant_quotas table. When a tenant upgrades, you update their quota row. When they downgrade or churn, their quota is reduced at the next billing period reset. Do not hardcode quota limits in application code.

5. Streaming in a SaaS Context

Streaming is a UX requirement for any generation that takes more than a second. Nobody waits for a 3-second completion to appear all at once.

Use SSE (Server-Sent Events) over WebSockets for LLM streaming in a SaaS context. SSE is simpler: it is a unidirectional HTTP stream, it works through load balancers without configuration, it has built-in browser reconnection, and it does not require a persistent socket upgrade. WebSockets are appropriate when you need bidirectional real-time communication (e.g., a collaborative editor) — for LLM streaming, they are overhead.

SSE in Node.js is straightforward: set Content-Type: text/event-stream, write data: {chunk}\n\n for each token, and pipe the OpenAI or Anthropic stream through. In Express, Hono, or Fastify, this is a dozen lines of code.

Reconnection handling. SSE has a built-in Last-Event-ID mechanism for resuming after disconnection. Use it. Assign an event ID to each streamed chunk (monotonically incrementing within the request). On reconnect, the client sends Last-Event-ID, and your server resumes from that point if the stream is still in progress (reading from a buffer you wrote to Redis). If the stream is complete, return the full result.

Streaming through an API. If your product has an API that customers consume programmatically, offer both a streaming and non-streaming endpoint. The streaming endpoint uses SSE. The non-streaming endpoint should proxy through the async job architecture — do not make your API customers wait on synchronous long completions.

6. Cost Attribution

You need to know which features and which customers are costing you money. Without this, your unit economics are invisible.

Tag every LLM call with a feature name. Your model abstraction layer should accept a feature parameter — "document_summary", "chat_reply", "code_generation" — and include it in the usage log. This is the most important single thing you can add. It turns a lump sum bill into a breakdown that tells you which features are economically viable.

Cost per unit by feature. Your usage_events table, joined against a model_pricing table (input cost per 1k tokens, output cost per 1k tokens, by model), gives you cost per feature invocation. If document summarization costs $0.04 per run and you charge $10/month, you have a unit economics problem at any usage level above 250 runs/month per user.

Cost alerts. Implement a background job that runs every hour and compares actual spend against projected spend for the billing period. Alert on Slack when a tenant’s daily spend exceeds their expected daily average by 3x. This catches runaway usage, bugs in your prompt construction, and customers who are abusing your product before your bill does.

Tools. ClickHouse is the right backend for usage analytics at scale — it is columnar, fast on aggregation queries, and cheap for append-only workloads. For smaller teams, Postgres with pg_partman (partition by month) works until you hit a few hundred million rows. Metabase or Grafana on top of either gives you the dashboards.

7. Model Abstraction Layer

You will change models. The model you build on today will be superseded, deprecated, or undercut by a competitor within 12 months. If your business logic is coupled to a specific provider’s SDK, every model change is a refactor.

The interface. Define a ModelClient interface with a small surface area:

interface ModelClient {
  complete(params: CompletionParams): Promise<CompletionResult>;
  stream(params: CompletionParams): AsyncIterable<CompletionChunk>;
  countTokens(messages: Message[]): number;
}

CompletionParams includes your normalized message format, model name (looked up from your registry, not hardcoded), max tokens, temperature, and response format hints. CompletionResult includes the generated text, token usage, stop reason, and latency.

Concrete implementations. Wrap each provider SDK in a class that implements this interface: OpenAIClient, AnthropicClient, GeminiClient, OllamaClient (for local/on-prem). The wrapping is mostly boilerplate — the value is that your feature code never imports openai or @anthropic-ai/sdk directly. LiteLLM (Python) or ai (Vercel’s TypeScript SDK) can handle some of this if you prefer a library over rolling your own.

Model registry. A models table maps logical model names ("gpt-5.4", "claude-opus-4-8") to provider, actual model ID, pricing, and capability flags. Your pipeline layer looks up the model from this table at runtime. When a new model generation ships, you add a row and update the default — no code change.

8. Evaluation and Regression Testing in CI

AI features break silently. The code deploys fine. The tests pass. And the model starts returning outputs that are worse in ways that no unit test catches. You need an evaluation layer.

Eval datasets, not just unit tests. For each AI feature, maintain a dataset of (input, expected_output_or_criteria) pairs. These are not unit tests — they do not check for exact string matches. They check for criteria: “the summary mentions the key decision”, “the generated SQL is syntactically valid”, “the tone is not aggressive”. Start with 20–50 examples per feature and grow the dataset from real user interactions.

LLM-as-judge. For criteria that are hard to encode as heuristics, use a second (cheaper, faster) LLM call to evaluate the output. “Does this response answer the user’s question? Score 1–5.” Store the scores alongside the outputs. This is not a perfect signal, but it is much better than no signal.

CI integration. Run your eval suite in CI on every PR that touches prompt templates, model configuration, or prompt construction logic. Gate merges on an eval score threshold — if average quality drops more than 10% from baseline, the PR fails. Braintrust and Promptfoo both have CI integrations that make this straightforward.

Regression detection. When you rotate models or modify prompts, run your eval suite against both the old and new configuration in parallel. A/B test at the eval level before you A/B test in production. The eval run should take under 5 minutes in CI — keep the dataset small enough to be fast, and use gpt-5-nano or claude-haiku-4-5 as the judge to keep costs low.

9. The Data Flywheel

The only durable moat in AI SaaS is data. Not the model (you do not own it), not the prompts (they are easy to replicate), not the infrastructure (it is a commodity). The moat is the feedback data from your users that lets you improve faster than anyone starting from scratch.

Feedback is a first-class data type. Every AI output in your product should have a feedback path — a thumbs up/down at minimum, a freeform correction field ideally. This is not a nice-to-have. It is the mechanism by which your product gets better over time. Design it into the UI from day one, not as a feature request in month six.

Log everything, always. The full prompt, the full response, the model, the latency, the tenant, the user, the feature. Store it in S3 in newline-delimited JSON. It costs almost nothing. You cannot go back and regenerate this data. Six months from now, when you want to fine-tune or build evals, you will be grateful you have it.

Closing the loop. Feedback data feeds three systems:

1. Eval datasets — flag bad outputs as examples for your regression suite
2. Fine-tuning — once you have enough high-quality (prompt, preferred_completion) pairs (typically 500–1000), you can fine-tune a smaller model that matches your use case better than the general-purpose frontier model
3. Prompt iteration — patterns in negative feedback tell you where your prompt is failing. Cluster the thumbs-down responses weekly and look for common failure modes.

Product analytics on AI quality. Track downstream metrics — not just “did the user rate the output” but “did the user use the output.” Did they copy it? Did they edit it significantly before using it? Did they regenerate? These implicit signals, collected from your application event stream, are often more reliable than explicit ratings.

The Default Stack

For a team of 2–5 engineers building a new AI SaaS, this is the concrete stack that fits the architecture above without overengineering:

The architecture does not change when you outgrow any individual tool. That is the point of the abstraction layers.

What Goes Wrong When You Skip This

Teams that skip this architecture end up in the same place: a working prototype that cannot be extended safely. The prompt is hardcoded. The model is called directly from the controller. There is no usage tracking. Tenant B can accidentally see tenant A’s context. A model deprecation breaks production. A single user runs up $400 in compute over a weekend and nobody notices until the bill arrives.

None of these are exotic problems. They are the default outcomes when the architecture is assembled ad hoc under shipping pressure. The blueprint above is not premature optimization — it is the minimum viable architecture for a product you intend to operate and improve at scale.

Build it right the first time. The refactor cost is much higher than the upfront cost.