Jamie Turner

3 months ago

The Six Systems Design Principles That Agents Don't Understand

Jamie and James have spent their careers building the kind of systems that store enormous amounts of data and can't afford to fall over: billion-dollar infrastructure at Dropbox and now the guts of Convex. So when they sat down to talk about what decades of building systems at scale actually teaches you, they opened with a joke about training their replacements. "I feel like in the spirit of giving all our wisdom to the Borg for free to train on it and obviate our value, it would be good to share some elements of counterintuitive systems wisdom." The consolation is that this is an auditory medium, so the robots might be a little slower to pick it up than they'd like.

That's the premise of this one: six hard-won, counterintuitive system design principles for building at scale, the kind you don't pick up unless you've been burned by them first.

1. You can't test your way to correctness

The first target is a claim that's gone mainstream in the age of AI coding assistants. Give a model the right test suite and enough tokens, the story goes, and it'll build the right thing. That claim is ridiculous, and tests being useless has nothing to do with why. It bundles two separate questions together, and the answer to both is no. Can you just write tests? And once you've written them, is your system correct?

The difficulty of testing has little to do with the writing. It comes from everything you have to already understand about the system before a test means anything. Early software education teaches unit tests that check trivial things. Those are rarely useful once you're working in a type-safe language on a system anyone would call sophisticated. The tests that matter are tests of state and functionality under real conditions, and those are a different kind of hard.

Take a specific example. Could you replicate SQLite with a sufficiently good test suite? Maybe, though the hard part of building SQLite was always the tests rather than the database. The tests are a conceptual statement of what a database should do, and that statement is hard to pin down in code.

Fully describing what a system does in practice is incredibly hard. Unit tests reason about a small, easy-to-isolate component, and that's usually not the part that matters. Systems design is about invariants: correct state, consistent behavior under concurrency.

Could you write a test for a database running in read-committed versus repeatable-read isolation? A database researcher who has spent a career thinking about transaction anomalies probably can. Most engineers who haven't built a database can't. They can't yet picture the system as a tight state machine. Without that, a test that validates correctness is close to impossible.

The lesson that comes out of this is that a poorly specified system is untestable. A simple app with a clean specification escapes this. Anything like an eventually consistent storage system, or a database with an open-ended query language, doesn't. There the combinatorial state space is so large that nobody can enumerate every edge case in advance.

This is the reasoning behind deterministic simulation testing. It's the approach the Convex team favors, and the one projects like TigerBeetle are known for. The philosophy admits that you can't anticipate every combinatorial state a real system might reach. So instead of trying, you state invariants, things that should always be true, like "a record that was inserted should never go missing." Then you run the system many times over with randomized inputs and watch whether any of those invariants break. That process surfaces edge cases nobody would have thought to test for directly.

Even a well-tested, formally specified system doesn't guarantee correctness in production. Dropbox's sync protocol was formally specified and tested extensively, and the code working and the system working turned out to be different claims.

Real failures they've seen:

• A disk that reports a successful fsync and a clean SMART status, but the data was never actually written, because the hardware lied.
• CPUs that branched incorrectly, confirmed by vendors flown out specifically to investigate.
• Error-correcting memory that corrupted memory anyway.
• Operators making unexpected changes.

And a more common failure than any of those: a misinterpretation of an API. The team tested against the spec they thought was right, and it meant something else.

James once deleted every block of data in a storage system during development. The cause was node names specified by IP address instead of hostname. It never reached users. Dropbox ran verification systems in production that continuously checked whether the data on disk matched what the system believed was there. That's the actual lesson: your job isn't finished once the code is written. You roll out, watch, and have a plan to roll back.

There's a subtler trap in who writes the tests. Say the same engineer who misunderstood how a system should behave also writes the test for it. Now the test just encodes and validates that same broken assumption. One mitigation is to have someone who didn't write the implementation write the validation test. That way a broken assumption doesn't travel silently from code to test.

The same logic explains the skepticism about mocks. The argument for mocks is coverage. Without them you'd need slower, more painful integration tests, so a simplified stub (an in-memory S3, say) gets you more tests written faster. The problem is that mocks almost never model the real system correctly. A mock that diverges from reality gives you false confidence, right up until it doesn't.

In one real case, an internal test harness didn't model a service accurately, and the team got surprised later than they wanted to be. Convex has tests because people want them, though the two of them are probably the least enthusiastic advocates for those tests internally. Testing code on real systems earns more trust. Good engineers rarely make off-by-one errors. They get tripped up at boundaries, and in places where their understanding of a system's semantics doesn't match reality. Mocks are exactly what hides that mismatch.

None of this is an argument against testing. The real lesson is to spend time thinking about the states your system can reach, lean toward integration testing over unit testing, and build on a platform with genuine guarantees. Narrowing the space of things that can go wrong is what makes any test suite meaningful in the first place. That's part of why Convex's design philosophy leans toward strong guarantees over flexible-but-untestable primitives. It's the same reasoning behind building durable workflows with strong guarantees for anything long-running.

2. Slow is worse than broken

The instinct when something starts going wrong is to do everything possible to keep making progress. That instinct is almost always wrong. It's better for a system to break than to slow down.

Here's the thought experiment. Imagine a service with a 10ms response time that handles 100 requests per second on a single thread. If latency jumps to 5 seconds, the load needed to serve the same requests-per-second jumps 500x. You now need 500 threads doing what one thread did before.

Most systems haven't over-provisioned memory 500x, so something has to give. The choice is between 0% uptime for everyone or 100% uptime for whichever requests can still complete. Responsible systems typically run at 50-80% capacity, not with 500x of headroom sitting idle. So when everything slows down and starts timing out, you fail 100% of requests instead of most of them. Slowing everything down uniformly is the hardest state to recover from.

This is also why you can't run analytics on the same Postgres instance handling low-latency production traffic. A big analytics query doing a full table scan can lock the table while ordinary requests queue up behind it. The system tips into congestion collapse: repeated failure that won't resolve without someone stepping in by hand.

They've watched this happen from the inside. A dashboard spikes CPU from a comfortable 6% to 25%, six people open that same dashboard at once, indexed queries start to time out and retry, and the whole system falls over because of a browser tab. Heterogeneous load is dangerous. Homogeneous load is what lets a system stay predictable.

If the system is going to break regardless, the question becomes what breaking well looks like. Shedding load deliberately means some requests get handled and the rest get a clean, fast rejection instead of a slow death. A layer that returns "I can't handle this right now" gives the layer above it something to react to. A layer that locks up gives nobody anything to work with.

The concrete architectural response has two parts. First, use short timeouts for live traffic. Then structurally separate live-site work from background activity. Anything that can tolerate delay (analytics, batch jobs, anything mirrored out to a warehouse) shouldn't be able to starve the requests that can't wait.

Second, on the database side, transactional writes that need to avoid races belong on the primary. Traffic without that requirement can run against replicas. Add more replicas and background work never competes with the transactional core.

3. Steady state should be worst state

This one runs against the instinct to optimize for the common case. Take a Dropbox storage design where a read is normally served from a single disk. If that disk fails, the system falls back to reading from seven other disks and reconstructing the data. On paper, that's a reasonable trade: keep the common path cheap and accept a rare, expensive fallback.

The problem shows up under load, not under failure. If demand on that disk gets high enough, the system can switch into reconstruction mode even without a hardware failure. Reconstruction turns 1x load into 7x load, exactly when the system can least afford it. A cheap steady state paired with an expensive failure mode is a recipe for cascading collapse the moment you cross a threshold. The system does more work precisely when it's already under duress.

The alternative is treating your worst day as your only day. kill -9 is the only kill signal a system needs to handle gracefully. If a system can go down hard at any moment, the day it happens isn't special. You just restart.

PlanetScale is built this way, failing over routinely rather than treating failover as an exceptional event. If your average operation looks like your disaster operation, disaster stops being disastrous.

There's an obvious objection. Doesn't running the expensive path constantly waste resources? On a fully elastic platform, maybe you can scale around it. But companies like Convex aren't running on infinitely elastic hardware. They provision for worst-case load, so that capacity is already paid for whether or not it's exercised regularly.

The operational takeaway is to design for the worst case first. Then spend your effort narrowing the gap between average and worst-case cost, instead of optimizing the common case and hoping the rare case doesn't matter. A good operational culture accepts that bad things will happen regardless, and its job is to keep them from becoming catastrophic. You get there by training the system, and the team, to operate near worst-case conditions as routine rather than exception.

4. Simple is better than sophisticated

Complexity is inevitable in anything real. The corollary is that if you don't keep the individual pieces as simple as possible, the combinatorial complexity of the whole system will get you. Almost nobody needs to write their own consensus protocol. Very few companies genuinely do. The right default is to make systems and code as simple as they can possibly be. Succeed at the actual problem and they'll get complicated on their own soon enough.

A recurring failure mode is reaching for a fancy scheme, like a distributed hash table, when a simple mapping would do. Fancy schemes are hard to get right, hard to verify, and hard to reason about. A simple database that maps a filename to its location can be checked and reasoned about directly. That legibility is what saves you when something breaks.

There's a second dimension that's easy to miss. Writing simple systems is itself hard, and simple-looking output gets mistaken for effortless output. A senior engineer whose code looks like a toddler wrote it is getting a compliment. Solving a genuinely hard problem and still landing on code that reads as obvious is rare.

There's a parallel in art and music. Great performers often do simple things with real precision, and the simple decisions turn out to be the hard ones to make well. If code full of decorators and side-effecting operator overloading feels impressive, that's usually a sign you're not at the top of the mountain yet.

There's a note specific to the current moment. Fully AI-generated codebases tend not to be simple. Language models often don't eliminate dead code. They leave in branches that are never true, and they don't converge on the smallest correct solution. Nothing in how they generate code selects for that. Good constraints, guardrails, and architecture give you a head start against that tendency. Which is a decent argument for building on a platform designed around those guarantees in the first place.

5. Architecture matters more than raw performance

Is Oracle the fastest database in the world? That's the wrong question to be asking. Being fast enough matters. Being fastest on a specific benchmark usually doesn't. Performance is often close to binary in practice: either a system is fast enough to solve the customer's problem, or it isn't. Architecture is what determines which capabilities are on the table at all.

Caching is the clearest illustration. A system that serves a request from cache and one that makes several backend round trips per request are in different architectural leagues. At that scale, benchmark comparisons between the two stop meaning anything. What matters is whether requests are small and fast, the system supports real parallelism, locking stays minimal, caching is used well, and the API itself nudges developers toward simple designs. Those are architecture decisions, not tuning decisions.

Take a chat app. You can build it on naive polling, which works but is expensive, or on subscriptions that wake a client only when there's something new to deliver. That single architectural choice can be a thousand times more efficient than the alternative. It's part of why Convex is built around end-to-end data sync rather than treating real-time delivery as an add-on.

A CMS is the inverse case. You could spend years optimizing update throughput. But if posts are updated rarely and read far more often than written, static generation behind a CDN is the right architecture, and a faster write path nobody needed is wasted work.

The same principle is why Convex tries to get faster through architectural change, not through buying faster disks. There's a hard ceiling on how much raw performance work can buy you, too. Amdahl's Law says you can't speed up a system's overall throughput past whatever fraction of the work can't be parallelized.

If committing a transaction sits on the serial critical path, making everything else in the system faster won't move the number that matters. It's a good discipline for knowing what's worth optimizing and what's a maze with no exit.

This is also the honest counterpoint to a lot of database marketing, roughly the argument Convex makes in its own take on competitive benchmarks. A bar chart comparing throughput across systems with different architectures usually isn't measuring the same thing twice, no matter how it's presented.

6. Queues are usually bad

This last one is the hot take, flagged as a favorite up front. Queues and retries have a reputation as reliability tools, and most of the time they make a system worse.

The mechanism traces back to network research and queuing theory, specifically bufferbloat. Making a router's queue bigger looks like it should help, right up until the queue fills. Then it adds latency instead of preventing loss. Network researchers responded by designing algorithms that proactively drop packets as queues grow, rather than letting them fill and stall everything behind them. Bufferbloat is really just a specific case of "slow is worse than broken."

When a system receives more work than it can process, there are two honest options: refuse the excess work outright (backpressure, drop requests), or queue it and convert overload into latency. The second option is the trap, because that latency is what causes systems to grind to a halt.

A well-intentioned engineer sees a system under strain and adds a queue in front of it, expecting it to smooth things out. Once that queue grows past a threshold, it tends to make failure worse instead of preventing it. The summary is blunt. Queues are bad in general, and putting one in front of something you don't want to admit is overloaded is about the most overapplied pattern in infrastructure of the last few decades. The right question to ask before adding one is what the queue is actually doing besides adding backpressure and bloat.

That's not a blanket argument against queues everywhere. Background jobs are a legitimately good use case. Reindexing a mailbox over the course of a week is fine sitting in a queue, because nothing downstream is waiting on it in real time. The distinction is queues in the critical path versus queues behind something. "Behind" means work waiting to be picked up asynchronously, not a gate blocking a live request.

For teams that need queues at scale anyway, there are techniques borrowed from networking. Adaptive LIFO processes the newest requests first under load, so you're serving requests that haven't already timed out instead of ones that have. Proactive dropping drains the queue and signals the layers above to back off before things get worse. If those techniques sound like they require some careful navigation, that's a sign you probably shouldn't be putting a queue in front of things at all.

This is also, unsurprisingly, why scheduled functions and retries in Convex are designed to separate durable background work from anything sitting on the live request path. It's why automatically retrying actions is treated as a distinct concern from queuing them.

A story from the Dropbox days shows how counterintuitive this gets in practice. Moving a large volume of data from S3 to Dropbox, close to a terabit per second, the team used 40 Gbit connections. They kept dropping to a stable 10 Gbit instead.

The cause turned out to be that those 40 Gbit links were actually four bundled 10 Gbit circuits sitting behind a connection pool. When one of the underlying links slowed down, requests on it returned more slowly. The fast links kept returning quickly and, because they sat idle more often, became eligible for removal during the pool's routine cleanup.

Over time, the pool pruned away the fast connections and left the slow one carrying everything. All the traffic ended up funneled through the single worst circuit. Connection pools and queues both have this kind of emergent behavior. Even mature systems like MySQL and Postgres handle connection pooling badly. Reason enough to treat it as a genuinely hard problem rather than a solved one.

Before and after: a connection pool with four bundled 10 Gbit links prunes its idle fast links during routine cleanup, leaving all traffic funneled through the single slow circuit.

The recap

None of this came from a class. It came from building things, watching them fail, and having to explain why. Six things to hold onto:

1. You can't test your way to correctness, because testing is hard exactly to the degree that it requires understanding a system you haven't fully specified.
2. Slow is worse than broken.
3. Steady state should be your worst state.
4. Simple is better than sophisticated, even though simple is the harder thing to write.
5. Architecture beats raw performance.
6. And queues, despite their reputation as a reliability tool, are usually a sign that a system is overloaded in a way nobody wants to admit yet.

Systems tend toward complexity and, left alone, they tend to fail badly rather than fail a little. Working against that tendency on purpose, toward simplicity and away from complexity, is what ties these six system design principles together.