Split out from the original flat backend-learning.md (moved to learning/archive/). See also
every other file in learning/backend/ — this one is the synthesis layer that assumes them, and
learning/frontend/08-rebuild-and-reference.md (the matching frontend rebuild path).
This file covers: the architecture decision records for the backend (what was chosen, what else was
considered, why), a step-by-step path to replicating the entire backend from scratch, and the
expert-level internals — the parts of Node, Express and Prisma worth understanding once the rest is
comfortable.
19. Architecture Decision Records
This section explains the “why” behind each major decision.
Why Module Pattern (MVC-like)?
Alternative: Flat files — one big routes.ts that contains all routes, models, and logic.
Problem with flat files: At 10 routes it is manageable. At 50 routes, nobody can find anything. Adding a feature means reading 2,000 lines to find the right place. Testing requires a running HTTP server.
Module pattern benefits:
- Predictable file locations — new developer can find
review.service.tswithout searching. - Separation of concerns — the controller does not know about Redis, the model does not know about HTTP.
- Testability —
PostService.schedulePostcan be unit-tested without Express. - Scalability — adding a new feature means adding a new folder, not modifying existing files.
Why Prisma Instead of Raw SQL?
Alternative: pg pool with raw SQL strings.
Raw SQL problems:
- SQL strings have no type safety — typos in column names fail at runtime.
- No autocomplete for table names or fields.
- Manual result mapping —
rows[0].user_idvsuser.userId. - No built-in migration system.
Prisma benefits:
- The schema is the single source of truth — change the schema, run migrate, get updated types.
- Every query is type-checked —
prisma.post.findMany({ where: { userId: 'string' } })is a compile error. - Relations are first-class — joining tables is Prisma query syntax, not SQL.
prisma studiogives a visual database browser for free.
Why BullMQ for Scheduling Instead of setTimeout or Cron?
setTimeout problems: Lost on restart. No retry on failure. No visibility. Can’t run multiple instances.
Cron problems: Runs at wall-clock times, not relative to when a post was created. Requires checking the database every minute to find posts due. Does not scale.
BullMQ benefits:
- Jobs persist in Redis — survive restarts.
- Built-in retry with backoff — if posting to Twitter fails, retry 3 times with exponential backoff.
- Exact delay — job fires at
scheduledAttime, not when a cron next runs. - Concurrency control — run 5 jobs in parallel, not more.
- Monitoring — BullMQ UI (Bull Board) shows queue status.
- Separate process — a slow job does not block HTTP responses.
Why Redis for the Queue?
Alternative: PostgreSQL-backed queue (PGMQ, pg-boss), or in-memory queue.
Redis is the right choice because:
- BullMQ is purpose-built for Redis and uses Redis data structures (sorted sets) optimally.
- Redis is extremely fast for queue operations — pushing a job takes microseconds.
- Redis Pub/Sub can be used for real-time notifications (future feature).
- In-memory queues do not persist. PostgreSQL queues are possible but slower and more complex.
Why AI Provider Abstraction?
Alternative: Call anthropic.messages.create() directly in each service.
Problem: If you later want to try GPT-4o for a specific task, or OpenRouter for cost reasons, you change 8 files. If Anthropic has an outage, you cannot switch providers without a deploy.
Abstraction benefits:
- Change
AI_PROVIDERin.envand redeploy — done. - Add a third provider (Gemini, Ollama for local inference) by adding one function to
ai.ts. - Prompt engineering stays in the services, provider details stay in
ai.ts.
Why express-rate-limit on AI Routes Specifically?
The AI routes (/reviews/generate, /discovery/search, /discovery/generate-response) each trigger one or more Anthropic API calls. These calls:
- Cost money (billed per token)
- Take 2–15 seconds each
- Can be expensive if called thousands of times
The global 200/15min limit applies to all routes. The AI limiter (15/min) is an additional layer on just the expensive routes. If a user has 200 requests in their global budget, they still cannot use more than 15 AI calls per minute. This protects against:
- Abusive users running automated scripts
- A compromised JWT being used to drain the AI budget
Why Helmet for Security Headers?
Helmet is a collection of middleware functions, each setting one security-relevant HTTP header. Without Helmet, a browser vulnerability that triggers XSS might allow attacker scripts to run in the victim’s browser. Helmet headers tell the browser to:
- Refuse to render the page in an
<iframe>(prevents clickjacking) - Not guess content types (prevents MIME confusion attacks)
- Only load resources from trusted origins (CSP)
- Force HTTPS (HSTS)
These are browser-enforced protections. Setting them costs nothing and provides meaningful defense in depth.
Why TypeScript Path Aliases?
Without aliases:
import prisma from '../../../config/prisma';
import { PostModel } from '../modules/posts/post.model';With aliases (@/* → src/*):
import prisma from '@/config/prisma';
import { PostModel } from '@/api/v1/modules/posts/post.model';Benefits:
- Moving files does not break imports.
- Imports are readable — you can tell at a glance what
@/config/prismais. - Refactoring is easier — search for
@/config/prismainstead of../../../config/prisma.
Why Separate Docker Stages for Dev vs Production?
The dev stage runs ts-node — it compiles and runs TypeScript in one step, making the development feedback loop fast. But ts-node is slow for production (it compiles on every startup) and adds ~50MB of devDependencies.
The production stage pre-compiles with tsc, copies only the compiled JavaScript and production dependencies, and produces an image that starts in milliseconds. The security posture is better too — TypeScript source code, development tools, and test infrastructure are not in the production image.
20. Replicating the Backend From Scratch
Here is the exact sequence to build this from zero.
1. Initialize the Project
mkdir socioboom-backend && cd socioboom-backend
git init
echo "node_modules\ndist\n.env" >> .gitignore
pnpm init
pnpm add express dotenv helmet cors express-rate-limit passport passport-jwt \
@prisma/client bullmq ioredis axios @anthropic-ai/sdk pg
pnpm add -D typescript ts-node tsconfig-paths nodemon prisma \
@types/express @types/node @types/cors @types/pg \
@types/passport @types/passport-jwt2. Create tsconfig.json
{
"compilerOptions": {
"target": "es2016",
"module": "commonjs",
"paths": { "@/*": ["./src/*"] },
"esModuleInterop": true,
"strict": true,
"skipLibCheck": true,
"outDir": "./dist"
}
}3. Initialize Prisma
npx prisma initThis creates prisma/schema.prisma and a .env with DATABASE_URL. Edit the schema to define your models, then:
npx prisma migrate dev --name init
npx prisma generate4. Create the Directory Structure
mkdir -p src/app src/config src/api/v1/shared/services src/api/v1/shared/jobs
mkdir -p src/api/v1/modules/{posts,analytics,notifications,accounts,teams,users,reviews,discovery}5. Create Config Files
Create src/config/prisma.ts, src/config/queue.ts, src/config/auth.ts, and src/express.d.ts as shown in this guide.
6. Create Each Module
For each module, create all five files in order: type → model → service → controller → routes.
7. Create src/app/main.ts
Wire all routes into the Express app with the full middleware chain.
8. Create src/worker.ts
Set up BullMQ workers for all three queues.
9. Add Scripts to package.json
{
"scripts": {
"dev": "ts-node -r tsconfig-paths/register src/app/main.ts",
"worker": "ts-node -r tsconfig-paths/register src/worker.ts",
"build": "tsc",
"start": "node dist/app/main.js"
}
}10. Create .env
Fill in all required environment variables (see Section 17).
11. Start Development
# Terminal 1: Start Redis
docker run -p 6379:6379 redis:alpine
# Terminal 2: Start the API server
pnpm dev
# Terminal 3: Start the worker
pnpm worker21. Expert-Level Internals
BullMQ’s Delayed Job Implementation
When you call postQueue.add('schedulePost', data, { delay: 5000 }), BullMQ:
- Assigns the job a UUID.
- Stores the job data in a Redis hash:
bull:postQueue:job:<id>. - Adds the job ID to the sorted set
bull:postQueue:delayedwith scoreDate.now() + 5000.
The worker process polls this sorted set every 1 second (configurable) using:
ZRANGEBYSCORE bull:postQueue:delayed 0 <currentTimestamp> LIMIT 0 1
When the current timestamp is past the job’s score, it atomically moves the job to bull:postQueue:wait using a Lua script. The script ensures this operation is atomic — no two workers can claim the same job.
Why maxRetriesPerRequest: null
ioredis by default retries failed Redis commands a limited number of times before throwing. BullMQ workers run long-lived blocking commands (BLPOP, blocking pop from a list) to wait for new jobs. If ioredis’s retry limit is reached during this blocking wait, it throws an error that crashes the worker. Setting maxRetriesPerRequest: null disables this limit, allowing BullMQ to manage its own retry logic for the worker’s connection.
Prisma’s Query Engine
When you run npx prisma generate, Prisma downloads a query engine binary — a Rust-compiled executable that handles the actual SQL generation and database connection pooling. The TypeScript client you import is a thin wrapper that sends type-safe queries to this binary via inter-process communication. This is why npx prisma generate is needed in the Dockerfile — the binary is platform-specific and must be generated for the target OS (Linux Alpine in Docker).
TypeScript Declaration Merging in Depth
The express.d.ts file uses two TypeScript features:
- Declaration merging: TypeScript merges multiple declarations with the same name. When you declare
namespace Express { interface Request { user?: IUser } }, TypeScript merges this with Express’s existingRequestinterface. - Ambient declarations:
declare globalmakes the declaration globally available without importing it, which is what you want for augmenting third-party types.
This is a standard pattern called “module augmentation” and is the correct TypeScript way to extend third-party types.
OAuth Token Expiry Buffer Explained
DiscoveryService.redditTokenExpiry = Date.now() + (data.expires_in as number) * 1000 - 60_000;Why subtract 60 seconds? Imagine this sequence without the buffer:
- Token expires at T=3600s.
- At T=3599s,
Date.now() < redditTokenExpiryis true, so we use the cached token. - The network request takes 2 seconds.
- At T=3601s, Reddit receives the request with an expired token → 401 error.
With a 60-second buffer:
- Token is considered expired at T=3540s.
- At T=3539s, we use the cached token.
- If the request takes up to 59 seconds, it still arrives before Reddit’s actual expiry.
- At T=3540s, we fetch a new token before the real expiry.
This is a standard pattern in OAuth token management. The buffer size depends on your expected worst-case network latency.
AI Streaming vs Non-Streaming
SocioBoom uses anthropic.messages.stream() but immediately calls stream.finalMessage(), so it behaves like a non-streaming call. Why use the streaming API at all?
-
Timeout handling: The Anthropic SDK’s non-streaming
create()waits for the entire response before returning. For long responses (8,000 tokens with extended thinking), this can take 60+ seconds and may hit HTTP timeout limits. The streaming API receives chunks as they arrive, so the connection stays alive. -
Future flexibility: If you want to stream the AI response to the frontend in real-time (showing the post as it’s being written), you already have the streaming infrastructure. You just need to pipe chunks to the response instead of awaiting
finalMessage(). -
Extended thinking compatibility: The
thinkingparameter only works with the streaming API in some SDK versions.
Redis Key Naming in BullMQ
BullMQ’s Redis keys follow the pattern bull:<queueName>:<type>:<id>. In SocioBoom:
bull:postQueue:job:1— job data hashbull:postQueue:delayed— sorted set of delayed jobsbull:postQueue:wait— list of jobs ready to processbull:postQueue:active— sorted set of jobs being processedbull:postQueue:completed— sorted set of completed jobsbull:postQueue:failed— sorted set of failed jobsbull:postQueue:repeat:schedulePost:every:...— repeat job configuration
You can inspect these directly with:
redis-cli
KEYS bull:postQueue:*
HGETALL bull:postQueue:job:1
ZRANGEBYSCORE bull:postQueue:delayed -inf +inf WITHSCORESThe as const Assertion
In the review service:
return rawReviews.map((r) => ({
source: 'google' as const, // Type is 'google', not string
reviewText: r.text ?? '',
}));Without as const, TypeScript infers the type of 'google' as string. But the Review interface requires source: ReviewSource where ReviewSource = 'google' | 'yelp' | 'twitter' | 'reddit' | 'manual'. A string does not satisfy ReviewSource. as const narrows the type to the literal 'google', which satisfies the union type.
Compound WHERE OR with Prisma Relations
prisma.team.findMany({
where: {
OR: [
{ ownerId: userId },
{ members: { some: { userId } } },
],
},
});The SQL Prisma generates for this:
SELECT t.*
FROM "teams" t
WHERE t.owner_id = $1
OR EXISTS (
SELECT 1
FROM "team_members" tm
WHERE tm.team_id = t.id
AND tm.user_id = $1
){ members: { some: { userId } } } is Prisma’s way of expressing “at least one related TeamMember where userId matches.” some becomes EXISTS (SELECT 1 FROM ... WHERE ...). The equivalent operators are every (ALL related records match) and none (NO related records match).
Software engineering
Infrastructure & security
Computer science
Hardware & engineering