Skip to content
Available for work
Adam
←  All posts

Designing for Scale - Statelessness, Load Balancing, and the CAP Theorem in Practice

The fundamentals of horizontal scaling: why stateless services are the key, how load balancing and replication actually work, and the trade-offs CAP forces you to make.

4 min readsystem-design · scalability · distributed-systems

TL;DR

  • Vertical scaling (bigger box) hits a ceiling; horizontal scaling (more boxes) doesn't.
  • Horizontal scaling requires stateless services - push state to a database, cache, or object store.
  • Load balancers spread traffic; read replicas spread database reads.
  • CAP: under a network partition you choose Consistency or Availability - you don't get to dodge it.

Why I studied this

I'd built the pieces - containers, a database, a cache, an event bus, observability. But there was a question I couldn't answer honestly: if traffic suddenly 10x'd tomorrow, what would break first, and what would I actually do about it? Not knowing bugged me. So I stopped adding features for a bit and went back to fundamentals.

Vertical vs. horizontal scaling

  • Vertical: give one server more CPU/RAM. Simple, but there's a hard ceiling and a single point of failure.
  • Horizontal: run many smaller servers behind a load balancer. Near-limitless, fault-tolerant - but only if your service can run as many identical copies.

That "if" is the whole lesson.

The key that unlocks horizontal scaling: statelessness

A stateless service keeps no client-specific data in its own memory between requests. Any instance can handle any request. The instant you store a user's session in local memory, you've pinned that user to one server - and now you can't freely add, remove, or replace instances.

So state gets externalized:

  • Sessions → Redis or signed JWTs.
  • Uploaded files → object storage (S3), not local disk.
  • Application data → the database.
  • Background work → a queue/event log (Kafka).

Once every instance is interchangeable, scaling becomes "run more copies," and deploys become "replace copies one at a time" with zero downtime.

Load balancing

A load balancer sits in front of your instances and distributes requests. The strategies I learned to distinguish:

  • Round-robin - simple rotation. Fine when requests are uniform.
  • Least-connections - send to the least-busy instance. Better for uneven workloads.
  • Consistent hashing - route by a key (e.g. user ID) so the same user tends to hit the same instance. Crucial for cache locality and sticky scenarios.

The LB also does health checks - it stops routing to an instance that fails its /health probe, which is how rolling deploys and crash recovery stay invisible to users.

Scaling the database

Stateless app servers are easy to multiply. The database is where it gets real, because state has to live somewhere.

  • Read replicas: a primary handles writes and streams changes to replicas that serve reads. Most apps are read-heavy, so this buys a lot. The catch is replication lag - a replica can be milliseconds behind, so a user might not immediately see their own write.
  • Sharding (partitioning): split data across databases by a shard key (e.g. user_id). This scales writes too, but cross-shard queries and rebalancing get hard fast. You reach for it only when a single primary genuinely can't keep up.

CAP - the trade-off you can't escape

The CAP theorem: in the presence of a network Partition, a distributed system must choose between Consistency (every read sees the latest write) and Availability (every request gets a response).

The nuance I had wrong at first: partitions aren't optional. Networks will fail. So CAP is really "when (not if) a partition happens, do you reject requests to stay consistent (CP), or keep serving possibly-stale data to stay available (AP)?"

  • A bank balance → CP. Better to error than show wrong money.
  • A social feed → AP. Better to show a slightly stale feed than show nothing.

There's no universally right answer - it's a product decision encoded in infrastructure.

How I'd handle a 10x traffic spike now

  1. App tier: it's stateless, so autoscale more instances behind the load balancer.
  2. Reads: add read replicas; serve cacheable reads from Redis.
  3. Writes: absorb spikes by writing events to Kafka and processing asynchronously.
  4. Static assets / uploads: serve from a CDN + object storage, off the app servers entirely.
  5. Watch: lean on the metrics and traces from the last post to see what saturates first.

I can now actually narrate that, and back each step with why.

Takeaways

  • Statelessness is the prerequisite for everything else in scaling.
  • Reads and writes scale differently - replicas for reads, sharding/queues for writes.
  • CAP is a decision, not a problem to solve - pick per use case.

Next

I've been deploying by hand this whole time, which doesn't scale either - for the system or for me. Next I'm automating everything with a proper CI/CD pipeline.