Back

System Design

Published: 14/04/26

Back to top

Overview

System design is deciding how the parts of an app fit together as it grows. Where data lives, how requests flow, what happens when something breaks.

There are no perfect designs, only trade-offs:

Most apps need none of this. One server and one database goes a long way. Add a piece when you've measured that you need it.

Scaling

Two directions when one machine isn't enough:

Horizontal scaling needs stateless servers. Any server must handle any request:

// Bad: state in memory ties the user to one machine
const sessions = new Map<string, Session>();
const session = sessions.get(req.cookies.sid);

// Good: shared store, any machine can serve
const session = await redis.get(`session:${req.cookies.sid}`);

Load balancing

Sits in front of your servers and picks one per request.

Algorithms:

It also runs health checks and pulls dead servers out of rotation. Without this a crashed server keeps getting traffic.

A good health check verifies the server can reach its database, not just that the process is alive.

Caching

Store the result so you skip the work next time. The fastest query is the one you never send.

Caches live at every layer:

Cache aside

Check the cache, fall back to the source, write it back:

async function getUser(id: string): Promise<User> {
  const cached = await redis.get(`user:${id}`);
  if (cached) return JSON.parse(cached);

  const user = await db.users.findById(id);
  await redis.set(`user:${id}`, JSON.stringify(user), { EX: 300 });

  return user;
}

Only requested data gets cached. First request pays full cost.

Eviction

Caches fill up, so something has to go:

Invalidation

The moment data changes, every cached copy is wrong. Options:

There are only two hard things in computer science: cache invalidation and naming things.

Two failure modes:

SQL vs NoSQL

SQL gives you ACID transactions, joins and constraints. Correctness by default.

NoSQL gives you horizontal scale and a flexible schema, and gives up transactions and joins.

Default to SQL. Move when you have a specific problem it can't solve.

Replication

Copies of the same data on multiple machines. Writes go to the leader, reads can go to any follower.

Buys you read capacity, availability and durability. Costs you replication lag.

That lag causes a specific bug: a user writes, then reads from a follower that hasn't caught up, and their own change is missing.

Fix: route reads to the leader straight after a write. Known as read your own writes.

Sharding

Replication copies everything to every machine. Sharding gives each machine a slice.

The shard key decides which machine holds what:

function shardFor(key: string, shardCount: number): number {
  let hash = 0;
  for (let i = 0; i < key.length; i++) {
    hash = (hash * 31 + key.charCodeAt(i)) | 0;
  }
  return Math.abs(hash) % shardCount;
}

Change shardCount and nearly every key moves. Consistent hashing fixes this by putting keys and shards on a ring, so adding a shard only moves its neighbours.

Sharding kills joins and cross-shard transactions. Last resort, after caching and replication.

CAP theorem

During a network partition you pick two of three: consistency, availability, partition tolerance.

Partitions aren't optional, so the real choice is what to do when one happens:

PACELC is the useful version: during a Partition choose A or C, Else choose Latency or Consistency. That second half applies to every request, not just the rare failure.

Consistency is a spectrum, not a switch:

Queues

Not everything needs to happen while the user waits.

app.post("/videos", async (req, res) => {
  const video = await db.videos.create({ status: "processing" });
  await queue.publish("video.transcode", { videoId: video.id });
  res.status(202).json({ id: video.id });
});

What you get:

Most queues deliver at least once, so a message can arrive twice. Make handlers idempotent - running twice does the same as running once.

Send repeatedly failing jobs to a dead letter queue, or they block everything behind them.

Rate limiting

Caps how many requests a client can make. Protects against abuse and buggy clients.

Token bucket is the usual choice. Tokens refill at a fixed rate, each request spends one, empty bucket means rejected:

tryConsume(): boolean {
  const now = Date.now();
  const elapsed = (now - this.lastRefill) / 1000;

  this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
  this.lastRefill = now;

  if (this.tokens < 1) return false;

  this.tokens -= 1;
  return true;
}

It allows bursts, which matches how clients actually behave. A fixed window counter lets a client send double its limit across a window boundary.

Counters go in Redis so limits work across servers. Reject with 429 and a Retry-After header.

Observability

Three signals:

Watch percentiles, not averages. A 200ms average hides the 1% taking eight seconds, and those are usually your heaviest users.

Alert on symptoms, not causes. High CPU might be fine. Users getting errors never is.

Putting it together

A read-heavy request, end to end:

  1. Browser cache answers, or
  2. CDN serves it near the user, or
  3. Load balancer picks a healthy server
  4. Server checks the cache
  5. On a miss, read from a replica and write back to cache
  6. Writes go to the leader, slow work goes on the queue

Every piece above exists because of a specific pressure. No pressure, no piece.

The best system isn't the cleverest one. It's the one you can still reason about at 3am.

System Design