Building AI Apps on AWS

The Bottom Line First

AWS is the right choice for AI apps when: you’re already deep in the AWS ecosystem, you need fine-grained IAM control over who touches your model endpoints, you’re running regulated workloads (HIPAA, SOC 2, FedRAMP) where AWS’s compliance portfolio is hard to match, or you need GPU capacity at a scale that smaller clouds can’t reliably provide.

The services that actually matter, in rough order of importance for most LLM-powered SaaS products:

• Bedrock — managed access to foundation models (Claude, Llama, Mistral, Titan) with no infrastructure to run. Start here.
• Lambda — serverless inference endpoints for low-to-medium traffic, glue logic, and async processing.
• ECS/Fargate — containerized inference for consistent, warm workloads where Lambda cold starts hurt.
• SageMaker — when you need to fine-tune, deploy custom models, or want a managed ML platform end-to-end.
• EC2 with GPU instances — last resort for maximum control or workloads that don’t fit anything managed.
• S3 + DynamoDB — storage backbone for everything: embeddings, conversation history, document stores.
• CDK — the only sane way to define all of the above as code.

Now let’s go deeper on each.

Bedrock: Your Default Starting Point

Amazon Bedrock gives you API access to a curated set of foundation models — Anthropic’s Claude family, Meta’s Llama, Mistral, Cohere, AI21, and Amazon’s own Titan — without provisioning a single GPU instance. You pay per token and get the same IAM, VPC, and CloudTrail integration you’d expect from any AWS service.

When to use it: Any time you’re calling a hosted frontier model. If you’d otherwise be routing traffic to the Anthropic API or OpenAI API, Bedrock is worth evaluating. The primary advantages over going direct to the model provider are:

• Traffic stays inside AWS (no egress to a third-party API if you’re VPC-constrained)
• IAM policies control which roles can call which models — useful for multi-tenant apps
• CloudTrail logs every invocation for audit purposes
• You can use AWS PrivateLink to keep requests entirely off the public internet

Cost reality: Bedrock charges the same per-token rates as the model providers’ public APIs, sometimes slightly higher for on-demand access. You can reduce cost with Provisioned Throughput (committing to a specific throughput tier for a flat hourly rate), which makes sense once you’re doing consistent high volume. For sporadic traffic, on-demand pricing is fine.

What Bedrock doesn’t solve: Model availability lag. When Anthropic ships a new Claude version, it can take days to appear in Bedrock. If you need day-one access to the latest model, you’re calling the Anthropic API directly.

Bedrock Knowledge Bases is the native RAG offering — it connects to S3, indexes your documents, and handles retrieval. It works, but it’s opinionated about chunking and retrieval strategies. If you have custom retrieval logic, you’re better off building your own pipeline with OpenSearch or a dedicated vector DB like Pinecone or Weaviate.

Lambda: Serverless Inference Endpoints

Lambda works well as the entry point for LLM requests — it handles the HTTP plumbing, auth, rate limiting, and fan-out to model APIs, then returns results. What it does not work well for is running inference itself on large models.

Practical Lambda limits for AI apps:

Cold starts are the main pain point. A containerized Lambda function calling Bedrock or an external API will cold-start in 1–3 seconds. For a chat interface, users notice. Mitigations:

• Provisioned Concurrency — keeps N instances warm at a fixed hourly cost (~$0.015/GB-hour). Run the math: for a small function with 512 MB memory, keeping 5 instances warm costs roughly $18/month. Probably worth it.
• Optimize your init code — defer heavy imports, cache SDK clients outside the handler, use Lambda SnapStart if you’re on Java.
• Move to Fargate — if you’re hitting cold start problems consistently, containerized services with ECS give you always-warm processes.

Lambda as orchestration layer: Lambda shines for async workflows. A user uploads a document → S3 event triggers Lambda → Lambda calls Textract for extraction → pushes to SQS → another Lambda calls Bedrock for summarization → stores result in DynamoDB. This fan-out pattern is where Lambda genuinely earns its place in AI architectures.

ECS/Fargate: Containerized AI Services

When you need always-on inference endpoints with no cold starts, Fargate is the right layer. You define a container image, declare CPU/memory, and ECS runs it behind an Application Load Balancer. No EC2 management, no patching.

Fargate CPU-only inference is suitable for:

• Serving embedding models (small transformers, e.g., sentence-transformers/all-MiniLM-L6-v2 runs comfortably on 2 vCPU / 4 GB RAM)
• Proxies and routing layers in front of Bedrock
• LangChain/LlamaIndex orchestration servers
• Streaming proxy services that need persistent connections

Fargate GPU is still limited — Fargate doesn’t support GPU instances natively. If you need GPU compute in a container, you’re on ECS with EC2 launch type or going to SageMaker endpoints.

Cost comparison, Lambda vs. Fargate:

For a service handling 1,000 requests/day at ~2 seconds per request:

• Lambda on-demand: roughly $0.20/month (essentially free at this scale)
• Fargate (0.25 vCPU / 0.5 GB, always on): ~$10/month

The break-even point where Fargate becomes cheaper than Lambda + Provisioned Concurrency is usually around a few million requests per month at medium memory sizes. Below that, Lambda wins on cost.

SageMaker: When You’ve Outgrown Managed APIs

SageMaker is a full ML platform — training, fine-tuning, experiment tracking, model registry, and endpoint deployment. It’s heavy. Don’t reach for it unless you have a specific reason.

Legitimate reasons to use SageMaker:

• Fine-tuning your own model. SageMaker Training Jobs give you managed GPU clusters for fine-tuning. You define a training script, point it at S3 data, pick an instance type, and SageMaker handles distributed training orchestration.
• Deploying open-source models at scale. SageMaker Endpoints run your containerized model on managed infrastructure with autoscaling, health checks, and blue/green deploys. For serving Llama 4 or Mistral on your own infrastructure, this is cleaner than rolling EC2 yourself.
• Inference at high throughput. SageMaker supports multi-model endpoints (one endpoint, many models) and async inference (queue-backed processing). These are genuinely useful patterns for batch workloads.

SageMaker’s friction is real. The console UX is notoriously clunky. The SDK abstractions are inconsistent between the classic SDK and the newer sagemaker Python library. Debugging failing training jobs or endpoints requires spelunking through CloudWatch logs. Budget time for this.

SageMaker JumpStart provides one-click deployment of popular open-source models including Llama, Mistral, and others. It’s useful for evaluation, but the generated CloudFormation is often over-provisioned and expensive for production use.

Rough SageMaker endpoint costs:

Keep these running 24/7 and the costs become significant fast. SageMaker Serverless Inference (scales to zero) is an option for bursty, latency-tolerant workloads, but cold starts can hit 30+ seconds for large models.

EC2 with GPU Instances: Maximum Control, Maximum Pain

EC2 GPU instances (the g4dn, g5, p3, p4d families) give you raw compute. You manage everything: the OS, CUDA drivers, model serving framework (vLLM, TGI, Triton), scaling, health checks, and on-call rotation.

When EC2 makes sense:

• You need a specific GPU/memory configuration that SageMaker doesn’t offer
• You’re running inference software with requirements that don’t fit in a SageMaker container
• You’ve done the math and self-managed is significantly cheaper at your scale
• You need Spot Instance pricing for batch/async workloads (up to 70% cheaper, but handle interruptions)

vLLM on EC2 is the most common pattern for teams self-hosting open-source models. It handles continuous batching, paged attention, and quantization, and it’s meaningfully faster than naive Hugging Face generate() calls. Run it behind an ALB, use an Auto Scaling Group with a launch template, and you have a production-grade serving layer.

For most teams, SageMaker or Bedrock is a better choice than raw EC2. The operational overhead of managing GPU instances rarely pays off until you’re spending thousands of dollars per month on inference and have an ML infra team to own it.

S3 + DynamoDB: The Storage Backbone

Every serious AI app on AWS eventually leans on the same storage primitives:

S3:

• Raw document storage (PDFs, text, audio before transcription)
• Prompt/response logging for observability
• Model artifacts and fine-tuning datasets
• Embedding snapshots for offline reprocessing

DynamoDB:

• Conversation history (session ID → message list)
• User state and preferences
• Request/response metadata with TTL for automatic expiry
• Feature flags and model routing config

A note on vector search: Neither S3 nor DynamoDB does vector search natively. Your options on AWS are:

• Amazon OpenSearch Service with the k-NN plugin — full-featured, expensive, operationally complex
• pgvector on RDS/Aurora — great if you’re already on Postgres; latency is fine for most apps
• External hosted vector DB (Pinecone, Weaviate, Qdrant) — often the simplest choice despite leaving AWS

Don’t let AWS-native loyalty push you toward OpenSearch if pgvector or an external service meets your requirements. Retrieval quality matters more than staying on one cloud.

CDK: Infrastructure as Code That Doesn’t Fight You

Use AWS CDK (TypeScript preferred). CloudFormation is verbose and error-prone to write directly. Terraform is a solid alternative if you’re multi-cloud, but for AWS-native stacks, CDK’s typed constructs and L2/L3 abstractions save significant time.

Key CDK patterns for AI apps:

// Lambda function calling Bedrock
const inferenceFunction = new lambda.Function(this, 'InferenceFn', {
  runtime: lambda.Runtime.NODEJS_22_X,
  code: lambda.Code.fromAsset('lambda/inference'),
  handler: 'index.handler',
  timeout: Duration.minutes(5),
  memorySize: 1024,
  environment: {
    BEDROCK_MODEL_ID: 'anthropic.claude-sonnet-4-6',
    DYNAMODB_TABLE: conversationTable.tableName,
  },
});

// Grant Bedrock invoke permissions
inferenceFunction.addToRolePolicy(new iam.PolicyStatement({
  actions: ['bedrock:InvokeModel', 'bedrock:InvokeModelWithResponseStream'],
  resources: ['arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-6'],
}));

conversationTable.grantReadWriteData(inferenceFunction);

Defining Bedrock permissions via CDK IAM policies is one area where the AWS AI experience is genuinely better than alternatives — you get fine-grained control over which IAM roles can invoke which models, enforced at the AWS API layer.

Recommended Architecture: LLM-Powered SaaS on AWS

Here’s a practical starting architecture for a chat/document-processing SaaS product:

User → API Gateway → Lambda (auth + routing)
                         │
              ┌──────────┴──────────┐
              │                     │
         Bedrock (Claude)     SQS Queue
         [sync, chat]              │
                              Lambda (async)
                              [doc processing]
                                   │
                            Bedrock + S3
                            [summarization]
                                   │
                              DynamoDB
                              [results store]

Supporting services:
- S3: document uploads, logs
- DynamoDB: sessions, user data
- CloudWatch: logs, metrics, alarms
- Secrets Manager: API keys, model IDs
- CDK: all of the above as code

Why this shape:

• API Gateway + Lambda handles the synchronous chat path with per-request scaling and no baseline cost
• SQS decouples async document processing from the synchronous API, preventing timeouts and enabling retries
• Bedrock is the model layer for both paths — no GPU instances to manage
• DynamoDB holds conversation history with a TTL attribute so old sessions age out automatically
• Everything is defined in CDK, reviewable in PRs, deployable to multiple environments

Add Fargate to this architecture when cold starts become unacceptable. Add SageMaker when you have a fine-tuned model that outperforms Bedrock’s hosted options on your specific task. Add EC2 when the SageMaker cost at your scale makes raw instances worth the operational overhead.

AWS vs. the Alternatives

If you’re starting from zero and don’t have an existing AWS dependency, GCP’s Vertex AI is worth a serious look — the tooling UX is cleaner and Gemini integration is tighter. But if your team already ships on AWS and your data is in S3 and DynamoDB, the cost of switching clouds is rarely worth it.

Vendor Lock-In: What You’re Actually Signing Up For

Using Bedrock ties you to AWS’s model availability and pricing decisions. If AWS lags on a model release or raises per-token prices, you’re migrating API calls — not a huge lift. The deeper lock-in is your IAM policies, VPC configuration, CDK stacks, and DynamoDB schemas. That’s the real switching cost.

Mitigation strategies:

• Abstract model calls behind an internal service interface — swap Bedrock for direct Anthropic calls without touching application code
• Keep your data schemas model-agnostic; don’t store raw embeddings from one model provider mixed with another without versioning
• Use CDK/Terraform so your infrastructure is auditable and portable to the extent any cloud infra can be

Lock-in is not a reason to avoid AWS. It’s a reason to be intentional about which parts of your stack are AWS-specific and which aren’t.

The One-Sentence Summary

Use Bedrock + Lambda + DynamoDB as your default AWS AI stack, reach for SageMaker only when you need to own the model, and define everything in CDK so future-you can understand what past-you built.