The Six Systems Design Principles That Agents Don't Understand

[music] Good day, Jamie. How is it going? It's going pretty well. It's going all right. How are you doing? I'm all right. I'm doing pretty well. I've been thinking about how am I different to a robot? You know, I'm thinking about do my as of now several decades of experience contribute anything beyond what the robot's going to tell you. Do they contribute anything? They might contribute a few things. The robots are rapidly integrating. They're rapidly integrating. Uh so I feel like in the spirit of of giving all of our wisdom to the Borg for free so we can train on it and and um and obviate our value. Um I I think it'd be good to share some elements of counterintuitive systems wisdom. counterintuitive wisdom of of um wisdom about designing and building systems that you probably wouldn't have unless you've built systems before that seems unintuitive. Yeah, I that may be the kind of thing that the robots haven't yet mastered, although they're I'm sure they're rapidly working on it. In addition, because this is a auditory medium instead of a textual one, they might just be a little slower on picking this all up. No one typed this down, right? This is just between us. uh we're going to share our advice. So yeah, Jamie and I have um built a lot of large systems, you know, like um billion dollar systems, the the the real big ones that you know store very large data and amounts of data and you learn some things in the process about how to design systems that scale even if you're not building large systems, even if you're building something small. I think this wisdom is uh fairly important. And so I thought today we can go through uh six topics um of of um maybe hot takes in systems design. And they are just to just to give them up front. You can't test your way to correctness. Testing is not going to save you. Slow is worse than broken. Steady state should be worse state. Simple is better than sophisticated. Architecture is more important than performance. And Q's are normally bad. Q's are normally bad. Um, I love that Q's are clearly so bad. They just get the most straightforward. Like everything else is a little more philosophical and then Q's are just like Q's are [ __ ] and don't use them. That's the that's going to [laughter] be take at the end. That's save it for the end, right? But like people think you put Q's in your systems and retries in your systems and that's going to make your systems more reliable and most of the time it makes them worse. Um, that probably just means you haven't tried you haven't packed enough cues yet, James. You got to keep going. We'll get there. We'll get there. [laughter] So, so, so, um, you ready to kick off? Yeah. Yeah, let's do it. What's the first one? Cool. So, you can't test your way to correctness. I think this is a interesting topic right now because there's there's a few things going on. One is people are saying, I think, rather silly things. Like, for example, um, LM can build anything. You've just got to give them the right test suite. like if you just give them the right tests, they'll that with enough tokens they'll build the right thing which is a ridiculous statement we'll talk about in a second. Um but not only that even if you do write perfect tests um your system still will not be reliable. Um so two topics can you can you just write tests and secondly once you have just written your tests is your system correct? Uh the answer to both is no is no. What do you think J? I think if uh I think you're just not trying hard enough, James. I think if you were able to write tests sufficient to model the behavior of the actual universe and the emotional response of customers to every inherent edge case, then that test would be this is like the um you know could go live could make something so heavy even he can't lift it kind of thing. [laughter] All right. Well, let's let's get started on on the first one. Can you just write tests and have your LLM build a correct system? Yeah. The challenge is I think when people hear testing, they think back to like university or like early college early early software education where you do unit tests and frankly rather useless things that test like in this function I put two values in and it adds the numbers together and were the numbers added together correctly. Those tests are rarely useful if you are a sophisticated developer, especially if you're a sophisticated developer using a type safe language. Um, the tests that are interesting are tests of state and functionality in a program, right? For example, if you had a really good test suite, could you replicate SQLite? Maybe. But how damn hard is it to write the SQLite test suite? That's the thing. The really hard thing about building SQLite is not building the database. It's writing the tests because within the tests are a a clear conceptual statement of what a database should do, which is very very difficult to articulate in a programmatic way. Yeah. Yeah. I think the more time goes on, the more I'm finding myself in conversations where I'm reminded of the the old Hitchhiker's Guide to the Galaxy 42 uh thing. So, if you if you haven't read Hitchhiker's Guide to the Galaxy, there's this old joke. It's made its way on the internet that [clears throat] humanity finally developed an infinitely powerful computer and they asked the computer, could it calculate the meaning of life? And the computer said sure. And it worked for a thousand years and generations waited for it to finish and then it said 42. And then everybody was completely confused at that and they all debated. They're like 42 that can't be right. But the machine was sure it was right. But then when they they suddenly realized they didn't know what the question was. And so they asked the machine, "Well, could you tell us what we meant by the question?" And the machine thought for a while and went, "No." Right? So [laughter] this this going this philosophical question going all the way back in this like science fiction comedy in the 70s is like it's re it's it's it's reverberating over and over again, right? it's rhyming these days, which is, you know, actually describing comprehensively what a system is in practice is actually an incredibly difficult thing to do. And so, and tests, especially things as simple as unit tests, are normally reasoning about some zoomed in small component of the system that's that in some ways is almost defined by how easy it is to test it, right? And that's not actually the part that is very meaningful at the end of the day. Now, this is particularly hard in systems design because systems design involves invariance. Like you want your systems to be correct. It's very important that a database stores the right state and it's very important that if you're running two concurrent transactions in a database, something sensical happens when they both run at the same time. But you know the challenge I would pose to the audience is can you write a test for running a database in uh read committed mode or repeatable read? I think I can, but I'm a database researcher who spent their whole life thinking about um an anomalies in transaction, you know, coordination. Um I think I could write a test for the difference between snapshot isolation and serializable. But I would say if you haven't written a database, you probably can't. Um you pro and and so I think in some this is a tricky thing about systems. I think in many respects like if you haven't if you can't conceptualize a system in terms of a very tight state machine, it is extraordinarily difficult to write a test that actually validates the system is correct. And so my kind of hot take here is that a poorly specified system is unt untestable. Yeah, that's almost certainly true. Yeah. And this is not the case for like an app like some apps have a very very simple interface to them and so the the specification is easy, right? like you click the button and something happens, right? That's an easy specific so you can test that. But a system that's poorly specified like an eventually consistent storage system or or a database that runs SQL that is extraordinarily complex open-ended query language. Yeah. Almost impossible to test. Yeah. Yeah. And I think in many cases one of the reasons for that is the kind of combinatorial state space of all the things that could happen in a real system is so large it's very very difficult to a priority reason about every edge case that should be tested. And so certainly you know our team likes things like deterministic simulation testing the tiger beetle team does and and that's just one facet too of testing. But one of the things that's interesting about that philosophy, if you think about it in terms of this conversation we're having around how limited is testing really, is it's almost throwing your hands up in the air and saying it's impossible to anticipate every sort of combinatorial state in the system. And so what we're going to do instead is state some things that should be true, some invariance, like there should never be a record missing that was inserted, for example, and then just run the system a bajillion times randomly and see if that that case is ever not true. And so that will often surface an edge case that you never would have anticipated that this ran before that, then this system had this one small bug and normally you don't notice that, but then it interacted with this other system at this later time. And so um but if you somehow tried to anticipate ahead of time to test that case, you just never would have thought of that. So so that yeah the complexity of um real systems and uh the the sort of effects it could have the external effects a system has when the entirety runs in the real world. So it's so large you just can't design tests ahead of time. I just see so many people saying this really reductive takes like oh just write a test just write tests if if if you were good at testing you wouldn't have this problem testing is is testing is harder than writing code because testing requires you to think conceptually how how your system works but let's say and again this is all about systems it's not about building a web app right um let's say that you really can test a system because you and I have both worked um in deterministic simulation testing. We've written systems that are formally specified. I've written systems that are formally specified where you really get pretty confident about the tests like you think you wrote the sync protocol at Dropbox where you're like tested this very extensively and so you're like cool I think this code works. Does your system work? I mean not not completely. [laughter] No nope. Yeah. I think any Yeah. Why? Why not? Like, why can't you just test your way? Why can't you just keep adding more tests and and make sure it works? Um, I mean, I think there's a lot of reasons why it's pretty tough to like, uh, first of all, I mean, SQLite, you mentioned SQLite before. SQLite's a fascinating example, right? So, SQLite is a a commitment to actually try to produce all the tests something should have, right? And I think SQLite famously has what like a thousand times the lines of code in tests as regular code or whatever. It's more. Yeah, it's like way more. It's like orders of magnitude more testing code. So I mean one thing is that it's it's not super economical to try to somehow express every single test that encompasses every single emergent edge case. I mean that's one reason why you [clears throat] would just uh you your whole job would become just doing that all the time. Um yeah um let me give some examples of real failures we've seen before in production. One failure we've seen is you fync to a disk and the disk says the data is written there and the smart status smart is this um API where a disk tells you tells you if it's healthy or not. The smart status says it's fine and the data is just not there because the hardware lied to you. We've seen that happen many times. Um, we have seen CPUs branch incorrectly and um, where literally the CPU there's an if statement and the CPU goes the wrong way down the if branch. And you might say, "No, you just wrote the wrong code." No, we had the CPU guys fly out with a big old machine to test the to test the a very expensive machine to test the CPU. Yeah, the CPU branched incorrectly. Um, we've seen error correcting memory that corrupted memory. [laughter] That one happened once too. Hard is because your system runs in the real world, right? On unpredictable hardware that is not always correct. The more than that, we've seen situations where um an operator's done something weird, accidentally changed some data, right? Um or far more frequently like a misinterpretation of an API. We we tested we tested the spec and we thought it was correct, but we meant something different. One issue I had early early on at Dropbox is I accidentally deleted every every block of data in a storage system during development. So it wasn't user data because I thought we specified the names of of nodes via IP address but we used host name or something like that and I deleted everything. Um yeah didn't actually impact users right? So one thing we actually did so we did so this is not all doom and gloom right? So what we did at at Dropbox in particular, we have and and we have a convex, a big array of these verification systems running in production. So the whole point of this is not that life sucks. The point is that your job's not finished once you've written the code, right? So there is release process like you have to accept that bad things are going to happen. You cannot test your way to correctness. There's release process where you you roll out code, you see if it works, you roll it back if it's not working. You run tests in production. We run validation jobs to make sure the disk the data on the disk is the same as what we think it is. Right? You have this suite of validation services. Um, another trick we used which I thought was kind of cute is we had sometimes when an engineer writes a system, they bake in their own broken assumptions about the world into the tests. So if you misinterpreted how a system was meant to get used and you also wrote the test, the test is just going to validate your bad assumptions. So we sometimes have people who didn't write the system write the validation test for it without knowing how the system worked to ensure that um broken assumptions didn't didn't pass through from um from the implementation to the testing. Mhm. This is I mean this is part of why mocks are one of the most dangerous things ever, right? Can you say more about what you mean by that? Well, [clears throat] I I think there's you like let me put it this way. An engineer might argue and have a kind of point, right? That like you get less testing if you don't have mocks than if you do. And so therefore mocks are mocks meaning you take a component of your system and replace it with a simplified stub that has main functionality. Yeah. Like you take instead of S3 itself you have some library that implements an in-memory S3. Um so um so if you did not have the mock then there's a whole bunch of tests it might feel pretty difficult to write like they would have to become change from being like unit type test to like integration type tests and that could feel like a lot to set up and can be a pain in the butt. And so the temptation to put in mock in there so that you can like get more code coverage or whatever, right, is it's pretty high. But the the issue with mocks is that the real systems that they're mock they they almost never model the real system correctly. In fact, we just recently had some issues with the convex test because it doesn't actually model convex accurately. Yeah. And the tricky part is that mocks can sometimes inspire a false sense of confidence that in practice can almost be worse than not having the mock because you think you don't need to put in the work to do the whole integration test or to do the validation because you've sufficiently modeled the external system and and and then by the time you get surprised that oh actually the mock was not accurate and there was a a problem that emerged due to the divergence between the mock and the real system it might kind of be too late like you already lost the data or whatever. ever. So the false confidence that mocks provide is often a really big problem and and James and I are not huge fan of mocks for this reason to be honest. Yeah. Be like Convex has this. So we have Convex tests cuz people want it and I understand why it's needed in some situations. Jamie and I within Convex probably are the least fans of of convex tests. Certainly the engineers did great work on it, but um I don't trust it. I would much prefer people be testing their code on real systems. Um because then the real because ultimately really good engineers don't tend to make mistakes like off by one errors and things. They make mistakes on the on the boundaries or the understanding of the semantics of a system. And so mocks often really hide that kind of stuff. So um let's let's have some positives here, right? So um I writing tests is really really really really hard and no matter how much you test your code, you don't know it's correct. So what's what's the lesson here? Spend time thinking deeply about the state in your system. Use a platform that has very clear guarantees. Cough convex, right? Um try to do as much integration testing rather than unit testing. A a benefit of a system that has as many gu contracts and guarantees as possible, right? Is it just reduces that combinatorial space, right? So it just makes it easier to be closer to comprehensive in your tests, right? So when its design can reduce the kind of combinatorial state space that is one of the nicest things the system can do for you and that's what convex tries to do. All right Jamie we promise not every one of these is going to be a convex show. We honestly believe [laughter] well it's it's it's hard because you and I are the founders of convex. So if reason a lot of these beliefs well another one that certainly I mean I would say is built into convex but really is built into almost I won't say almost I would say built into every reliable system you use is a lot of engineers think when things start to go wrong the most important thing to do is do everything you can to make progress despite the fact that you're starting to have system problems and that is almost always the wrong instinct actually. So, um, the slow is not better. Slow is actually worse than broken. When things start going wrong, it's better to break than to slow the system down a lot. Yeah. What do you think about that one, James? I want to give people a thought experiment, right? Imagine you're running a service and the response time to your requests is 10 milliseconds, right? Um, and so you can run, you know, a 100 requests per second in a single thread, right? Um, and imagine if all of a sudden latency increases because a data database is slow or something's maybe is synchronously trying to talk to Claude, whatever, and all of a sudden your requests are taking 5 seconds now, right? You have 500xed your load in your system. You have increased the load in your the number of concurrent requests by 500 times, right? Because now if you want to service the same number of requests per second, you have to have 500 threads running instead of one thread, right? Which means your your system will fail like it it will grind to a halt because you probably haven't overprovisioned 500 times more memory in your service, right? Probably not. And so you have a choice at that moment, right? Do you just allow the system to keep running slow or do you just shed some load? Right? Because you have a choice. Do you have 0% uptime for everybody or do you have a 100% uptime for some requests so those people can get their job done and go away and at least there's some progress through your system? Yep. Yeah. I mean systems in general that are shooting to be run responsibly are usually running 50 60 70 maybe as high as 80% capacity. Right? So, and some a lot of times not even 80, right? So, you don't have uh 500x overhead for when requests get 500 times slower, right? So, to James' point, if you if you make everything slow down to the point it times out, you're failing 100% of requests, it's much better to sort of succeed some and even even if you have to fail most of them, right? Because it starts to at least have some requests go away instead of retrying. and and you can start to make progress on the problem. Um so slowing everything down just is is actually the the the most difficult state to recover from. You will destroy your system. Everything will everything will go to zero. Now this is also why you shouldn't and you honestly can't at scale run analytics on Postgress. Someone will always say Postgress is great because you can run a big analytics query. You can do a full table scan of your whole database and add up all the numbers and do something and do some mutation. Um whereas in ComX we actually force you to have fast query like we mandate that your queries are fast in convex which seems like a controversial choice. Um but what happens when you have one request that takes 30 seconds to run and locks the entire table because you're reading every table in the database and doing a a change the same time as a whole bunch of fast low latency requests like for a chat app. Well, guess what? your system locks up for 30 minutes and your database probably fails over like like things will probably fall apart very if you're lucky it just stalls for 30 seconds but if you're not lucky your service is down and it and it won't come back up without manual intervention of you going and like shedding load and like um changing your firewall rules to block everyone's IP address and then slowly allow new users back in. This is what you call congestion collapse. This is when the system has tipped over into repeated failure mode and just will not recover without manual intervention. This is what happens when you have very slow activities at the same time as very fast like low latency activities. Yep. This highly heterogeneous loads are really dangerous, right? So you let homogyny is is really really good for healthy systems. Uh to make it concrete, let's just say you've you've used Postgress to solve everything, right? So Postgress most of your queries are indexed and you're grabbing a few records and most of the time the database CPU is at 6%. You have tons of headroom, right? And let's just say there's one dashboard in your system and occasionally when a user loads that dashboard this the the database CPU temporarily spikes up to like 25%. Then it goes back down to like 6%. You're happy, right? Well, what if one day six people all try to run that at the same time? [laughter] That's all it takes. you just get unlucky and six humans try to do the same thing at the same time and now your database server is spending all of its time it's it's actually it needs more CPU it has in the machine to go through all those records in the table scan add up the dashboard query and in the meantime every single small index query is slowing down some of them start to time out they retry more people show up try to get their job done those requests retry and you congestion collapse your whole thing so from your perspective if you were just looking at that machine, you would be like, "Oh, my CPU 6% 6% 6 6% cool." And then all of a sudden in 20 seconds, the entire database server just completely got overrun and locked up and all it took was like six humans to do something to cause that to happen, right? So those are the kinds of characteristics of s systems that are really tricky to to to make stable when you allow that kind of thing to happen. I mean, this is not a hypothetical example. I've seen multiple times situations where there's like panicked like yelling in an office. Can everyone close the dashboard right now because like it's like the systems are failing literally because people have a tab open. This hasn't happened I don't think. Yeah. Yeah. That's the funny thing I think. Yeah. That's the funny thing is sometimes teams will be like uh oh all the userfacing queries are fast but there's some admin thing we can do right that uh that you know reads all the records or something and uses Postgress like a data warehouse and uh but all it takes is you hire a few more employees on the engineering team and pretty soon you don't realize everyone has that tab open in their browser and so the co your own company takes down your web app even though your users weren't doing anything. So, yep, those kinds of things are are pretty tricky. Yes. So, I'll just quickly address that. Um, you might say, well, if you're going to be broken anyway, what difference does it make? Um, generally, if you do a good job of shedding load, you can ensure that some users have their their data handled and their requests um handled and others have an opportunity to go away or try somewhere else. Right? So if you're building a system at the bottom of a stack and there's seven layers of services on top of you, if you just return, hey, I'm not I can't handle this request right now, at least the system above you has an opportunity to decide what to do. Try later, who knows, right? But if you just lock up, no one has an opportunity to react. Um, now what can we do? This is not all doom and gloom. What can you do? I've got two ideas, right? One idea is yes you should have short timeouts but what if you want to do slow long things right well you have to very clearly delineate in a system live site and non-live site services so any really well hardened system has a concept of this like userfacing low latency someone's clicking a button they need some they're purchasing something they need something to happen on the website and then background activities and background activities can stall. They can wait. Sometimes those background activities are just analytics queries and they should get run in an analytics database separately from your data altogether. Or maybe they should be run in a scheduled job. Maybe they should be run in the background when loads subsides. And so if you clearly architect your system separating critical live site stuff that in comics would be a mutation or a query and non-live site activities which in comics would be just um you know a schedule job through the workpool or maybe it' be some kind of action or maybe it's literally you running something on data bricks because you've mirrored it out by fiverr um then you can really insulate that live traffic. Yep. Yeah. The other thing too about that background stuff is that when it comes to databases, I mean the transactional actions are the ones that absolutely need the database primary because by definition they need to like read some records and change some records without having races with anyone. Um once once you do any traffic that does not have that requirement the beautiful thing about when you're operating on a replica whether the data is in a data warehouse or even just a postgress replica is that you can you can have 20 replicas if you want. So once as soon as you get out of that consistent barrier now you're into this like embarrassingly parallel like uh architecture and then it's really easy to have a whole bunch of traffic. If it's worth not delaying background jobs you can just have more replicas and then you're fine. So um yeah but slow is really bad and the slow is especially bad on the transactional core of your of your application. So there's a there's a very um there's a related concept um which is that steady state should be worst state. I this one is particularly unintuitive. Almost no one unless you're a very experienced systems engineer thinks like this. Um and so I'll give an example of a time I got it wrong. So this was designing the storage system at Dropbox and I'm going to simplify this heavily but um imagine if you um someone comes along with a read and you can service that read off a single disk. So the read comes in you read one disk and you send the data back to the user and that's a pretty fast um low overhead situation. But if that disc is not available for some reason, like that disc has failed, you then go and read from seven different discs um some partial state that's been erased and then you read all the data back and you combine it together and then you send it back to the user. And this seems on the surface like a a good design like most of the time the system's very efficient. You got single like you don't have a lot of load on your on the discs in your system. Um, and sure if something bad happens, well then just do the more expensive thing. And wouldn't it make sense to optimize for the common case and just let the bad case happen very very sporadically? Now, here's the counterintuitive wisdom. Think about what happens when low gets a little bit too high for that disc. Maybe um I think one example I think it was Dea Soul, some band, am I allowed to say that? I don't know. some some music group launched an album on Dropbox. And so everyone in the world who was into this band went to try to fetch this file at the same time and it was too much load for the disc. And so the disc just stopped responding because it it it was slow. It was worse than broken, right? The disc slowed down and the system said, "Well, guess what? I'm going to have to do a reconstruction read." So then all of a sudden you were in a situation where you had 1x load was too much. Then we translated it into 7x load, right? Which then obviously became too much because if one was too much, seven was too much, right? And then you can lead to cascading issues where under duress you're ending up doing more work than under steady state. And this is in any system where you may hit a load threshold, you never want to switch mode into a more expensive mode of operation. If it's like under normal situation, we read from this data center. Otherwise, we just go to the other data center and and latency is higher and load is higher, you are guaranteeing your system's going to go down, right? Yeah. I think um two changes I've seen over the last 20 30 years that have kind of embraced this I can think of is that I think 25 years ago there were more programs that were uh these are like Unix programs that were kind of precious about uh kill term versus kill kill. So signal 9 versus signal 15 or whatever term is and um and the idea would be well like give me a chance to catch the signal and like flush to disk or whatever, right? So um there were a lot of um there was a a kind of new generation of services once people were really depending on machines that started to have the philosophy that kill 9 is the only kill, right? So, and and what what that really means is that if you architect your system to be able to go down hard at any time, right? Then the day it happens and you weren't planning on it is no bigger of a deal than any other day. You just restart your system. The company right now that's pretty excellent. This I think is Planet Scale, right? So, Planet Scale has this philosophy. If you're a Planet Scale customer, you will learn quickly. Planet Scale fail fails over all the time. And the attitude 20 25 years ago and probably is still the attitude some places used to be that failovers are a really bad thing and you mostly just cross your fingers and hope you never have to fail the database over and then the day it happens guess what it probably doesn't work right or it's so slow when it happens or it takes so long to promote the new primary that your site is down for like four minutes you know and um or four hours right and Planet Scales attitude is we we make like every time you make a configuration change on a database server, they just fail over. Um, and if you don't do anything to the database server, they still fail over just to keep everything fresh and to keep themselves honest. So, it's another good uh example of it's not strictly performance related, but even for correctness, try to make your average operation the same thing as your disaster operation and then you know you're going to be okay. Yeah, if you have an emergency recovery case, you should just run that on every request. Now, that's interesting. You might say, "But wait a second, we don't want to run the expensive thing all the time because we're wasting money." And this is where like the cloud has kind of polluted people's minds a little bit. Now, if you are on a completely elastic platform, I guess you can get away with this. For companies like Convex, we're not on a completely elastic platform. We can't just um increase our systems 20x in scale. like we have to actually provision our services in terms of hardware, CPU, RAM, etc. network for the worst case scenario, right? So, you're already paying for that hardware anyway. So, most of the time like we'll be running our services at quite low utilization just in case something bad happens, we're ready to go uh with a higher load. So, you have to pay for it anyway. If you're already paying for the hardware, you might as well use it all the time. And this this reminds me of like I always find company origin stories funny because they're almost always a total lie. And that's true. I've said this before, but the AWS one is particularly hilarious. Like I've had multiple people tell me with a straight face that the origin of AWS was that Amazon has to buy a whole bunch of servers for Black Friday because everyone's like buying, you know, socks, right? Um, and but then when it's not Black Friday, they have all this spare hardware. So, they had to find a way to use that hardware. And so, they invented AWS, which just think about it for like 7 seconds, right? What happens next Black Friday, right? They can't turn everyone off, right? Like Amazon has to provision for the worst day of the year. Now, sure, they can maybe turn off some background activities. Maybe they you like some particularly large companies like um scale up and down a lot of their background work, their data processing, their model training, etc. to kind of smooth out their load throughout the day. But any major service has to provision their services for the worst day of the year. And so um yeah, and and I think that the the the mindset that can be useful here because people would say, well, isn't that wasteful, right? Well, it kind of is, but I think the right mindset is operationally plan on making every day your worst day and exercise that all the time. And then asynchronously, a second thread your company works on is from an efficiency perspective, try to just narrow the gap between your worst day and your average day. That's that can be a different architecture, different that's fine, but but the baseline is we make every day our worst day. And then if we also want to chip away at the gap between the average day and the worst day, we could do that. But but that's viewed as a second process in the invariant, the principle you won't break is we're going to make um you know our our basis based on the worst case scenario. Yeah. The last thing I'll say about that is just that if you want to build a good operational culture within your company, it is not about preventing bad things from happening because bad things just happen. It's like our first point. Bad things are just going to happen. machines going to fail, weird things are going to happen, you're going to get dodoed, etc. So, good operational culture is not about preventing bad things happening. It's about avoiding really really bad things happening, right? So if you're every day dealing with a like a automating um steadystate reacti reaction to to bad things, you get closer and closer and closer to the point where it's it's rare to have huge anomalous spikes of terrible things where everyone's all hands on deck. And the only way you get there is is not like just crossing your fingers and hoping it doesn't happen. It's just it's training your team and your systems to be always operating as close to as possible to worst case scenario at all times and optimizing that so it's not stressful. Yep. Agree. All right. Well, yeah. So we've talked a lot about things like even with like testing we talked about how you know you might have a unit test but then when you combine this with other parts of your system these combinatorial effects are are um can can become really difficult to reason about and then you you're still broken. So you know in general one of the principles that kind of keeps you thinking I think the right way about all these things we've talked about right is that um simple is much much better than sophisticated. Um, and I would go as far as also to say that complexity is um, inevitable to the point of all of the three things we've just been talking about. If you're, you know, if you build a toy app, you might high-five yourself and say, "How elegant is this?" And it's it's fully encapsulated. It's all test. It's in my head. If you're building something real and it has a lifetime, everything gets complicated. Everything gets complex. But um but if you don't focus on making individual things as simple as possible, then when the combinatorial complexity shows up, you're completely screwed, right? So like trying to be keep things as simple as possible is key. Yeah. Don't be so self-indulgent to design like complex systems because it feels cool. Everyone wants to write their own consensus protocol. And there's some some very very very few companies that need to do this but almost no one does right outside a very small set of companies. Um generally you should design your systems as simple and your code as simple everything about your company as simple as possible and if you succeed they will get complicated anyway right and for example one of the one of the big arguments we used to always get is like well why would you have a database like that maps a name of a file to the location of the file on disk you should have this like distributed hash table that uses cord and instead of you can just hash the name of the file and that will map to the to the data on disk and it's like more sophisticated. Well, guess what? That system one is hard to get right. Two, it's not verifiable. I can't walk over a distributed hash table and figure out if the data is all in the right place. If it's in a database, I can just look at the database and check where every block is where it's where it needs to be. And three, it's just doing the simple basic low overhead conceptually easy thing, right? Because when people when something fails and goes down, we can go and say, "Huh, is the data in the database anymore? Where is the data meant to be?" Not like, "Huh, how does that hash function work?" Yeah. If you can't reason easily about any one piece, you're really in trouble when things go wrong. And I I mean, maybe maybe on the like engineer satisfaction kind of um continuum I'll put out there for a second, right? is the scale that like I'm bored, right? I'm challenged slash excited is in the middle and then like I'm I'm [ __ ] is the extreme end, right? And the challenge is if you're already excited on the individual component level, your only answer is you're going to be [ __ ] at the at the composite level when you've when you've composed all these systems together. So you really you really want to be a little bored about how simple you've managed to solve individual things. Trust me, you will have an opportunity to be challenged because the composite system you'll end up with will inherently be sophisticated. So the challenges are coming. You you don't you don't want to take on more than you need to. You there's no need to do that. Now the the kind maybe the challenge to the audience, right, is that is to recognize that simple simple systems are the hardest to build, right? It's so easy to look at a simple system and say, "Oh, of course it looks that way." Well, guess what? If you say, "Of course, it was designed well, but that didn't mean it was easy. It took a lot of talent to get there." One of the I don't know what the opposite of a backhanded compliment is, let me like a a front-handed insult. Front-handed insult. Yeah. I've heard [laughter] I remember um someone describing the code of a senior principal engineer we're friends with and I heard someone say this guy writes code that looks like a toddler wrote it right and I was like yeah you don't understand how much of a compliment that is right this guy managed to solve really really hard problems at very impactful companies writing systems that just look so easy and simple and the code and in particular like here's one concrete example if you a Python developer and you have um decorators everywhere and operator overloading um and there's like um side effects happening whenever you're calling something you are not yet an excellent software engineer. I don't know. You haven't reached the top of the mountain, I guess. And there's a there's a time and a place for decorators, but like but the the very very best software engineers write code that just looks simple, right? So hard to do that. It's so there's a ton of parallels too to like the good good artists or good designers, right? Like someone is just starting and they can't really make anything yet. And then when they first start to sort of like master harnessing color or drawing or right like what you can do with a space to it like all the you know all the materials and the finishing might be overdone and there's just this exuberance to express your capability but it shows up as doing too much and then like but you can recognize excellence in someone when they create something simple but it's the perfect simple thing right and you go I would not have been able to come up with that. I would have been able to come up with something simple that was wrong and then someone more skilled than me would have been able to come up with a whole lot of stuff that wasn't quite right. But you were able to pick something that was simple and right at the same time and and coding absolutely has that as well. Engineering has that too. Yeah. I mean there's analoges in in music for sure. You know some of my favorite musicians are like some of my favorite drummers we got to be careful because we could talk for an hour about drummers. Jamie honestly Ringo from the Beatles. Like some of my favorite musicians are the people who just do simple sounding stuff and you're like, "Well, I could do that." I'm like, "Yeah, but you didn't you didn't make the Beatles like like like or the drummers or something." I like too. I like I'm not a massive Dream Theater fan, but like like I like complicated music too. loved Tulle and Dary Danny Kerry is probably my favorite drummer, but I have so much respect for like honestly L or from Metallica like Ringo because they made the most incredible music in a simple Yep. They pick just the right thing. Yep. If you think that's easy, you probably don't really understand music. And if you think writing simp complicated looking code means you're a smart engineer, you probably don't understand engineering yet. You'll get there. But um work so hard on simplicity. Yeah. By the way, it's also if you're starting a business, someone was asking me the other day about advice for starting an infra company selling to large enterprises. And I was like, that's complicated. Like selling to large companies is complicated because it's messy. They all have their own bespoke requirements. They have the legal teams. to have like you're adding complexity to the process of running a business while you're still early on as a startup. The the more streamlined and simple you can keep things early on in a in a company or in a team in a codebase um the better it'll be. Um by the way fully AI generated code bases are generally not very simple. Yeah, they are apparently. Yeah, I mean even just one good example of that, there's obviously a million of them, but I think almost everyone has run into this one without looking too hard is that a lot of times LLMs are not very good at eliminating dead code, right? So they because you know so even if they have a validation loop, it's mostly treated as did the thing I just do work going back to the testing thing by whatever limited definition of work you've been able to negotiate with the LM. And that's fine. But the problem is as a human though, when you do later on go and engage with this codebase, you'll often find there's just all kinds of stuff in there where there's like a conditional that is never true. And so that why is that in there? And so um yeah, but all of that stuff adds complexity and um getting LMS to make exactly the right thing. It's the the just enough thing is is really really difficult. They could generate a lot of stuff and that's impressive, but they don't always choose kind of the the perfect simple succinct bit of engineering. Now, you're off to a head start if you have good constraints, good guard rails, good architecture. Um, right. Is there is there a backend platform that has those things, Jamie? I don't know. [laughter] Is is this the requisite convex plug? We just hit another 20 minute mark at go to convex.dev and use convex. Yeah. [laughter] Uh, okay. On that topic, I just mentioned architecture. Um, architecture is way more important than raw performance. Um, yeah, tell us more, Jamie. Yeah, I mean I mean here here's just a question out of the gate, right? Like we might make fun of them, you know, we might roll our eyes at them. Um, but um, you know, is Oracle the fastest database in the world? Like I have no idea, right? Like I I I don't know. Does anyone really know how fast it is? I'm sure it's fast enough, right? Um if we look at other winners in their category, right? Is data bricks the fastest, right? Or or whatever. Um and uh yeah, I mean at the end of the day, what matters is that you're fast enough. There's a lot of things that matter a heck of a lot more than uh being fast in one particular benchmark, for example. like benchmarks end up being pretty arbitrary and it's it's actually solving the problem that your users or your customers need. Um performance is one element of those but um but to be honest it's often times even a small one. It's more of a binary, right? It's not like no very few customers actually say I'm going to what I'm going to do is I'm going to make my purchasing decision based on whatever has the most requests per second, right? um you have a a need you're trying to solve and there's a million different dimensions and the performance side is often very binary, right? It's just it will this thing be fast enough for what I need. Um and so because of that, yeah, the architecture which is normally much more connected to what capabilities you're going to be able to deliver for this customer is much much more important than the quote unquote performance as some kind of you know universal measuring stick. Yeah. Now I mean one one way to think about this is is you can have two systems two um not two databases but two architectures two two products right and one is caching requests effectively and you serve a million requests out of the cache for free on the database basically and the other is not caching uh and those a million requests each trigger a request waterfall that issues multiple requests right and you might find yourself very quickly with eight orders of magnitude in different workload. That sounds like eight orders of magnitude is an obscene number, right? But it's it really can happen, right? Um or let's just say six to to play it safe, you know? Um now the difference that's that's as soon as you're dealing with six orders of magnitude, no benchmark comparison matters, right? Now it is important that the system is capable of running large scale, right? But ultimately the the the factors that matter in system scale are how it's designed. Are requests happening fast and small? Do you have parallelism? Um do you avoid locking that locks up your database? Are you getting you know aggregate throughput through a system? Are you doing good caching? Um, and also does the API do the building blocks encourage you to design systems simply because for example again this turning into a convex plug honestly these are these are actually our real opinions but we kind of took all these and bundled them up into convex, right? You can build a chat app that does polling, right? That you ask the server like is there a new message for me? Is there a new message for me? That's the naive way to build a a chat app, right? Or you can have a subscription that wakes you up when that message um a new message shows up. The difference between these two models could be easily thousands of of of times more load, right? If you're receiving a, you know, an update every now and then. And so architectural changes, particularly architectural changes around endto-end data sync and state management are really really core to the performance of your system. Um as you know it's why we why we wrap them up in a in a in a model and turn it into a company basically. Yep. I mean another even to do a example taking convex out of it for a second. So um you know if you imagine something like um uh a content management system, right? So a content management system if they just tried to beat benchmarks they could you know a new startup content management system could you know spend years making a system which had the highest post update rate in the world right [laughter] you know but then but it ends up that people only update posts you know 20 times a day even in a very big company or something right it's like it would been completely wasted And in fact there's so few updates that the read side they they could should just be static site generation and they just read generation static files and push that to CDNs. Right? So if you there's so many ways you could become faster. Almost all of them are the wrong way right now. Right? The only right ways to become faster are ones that are derived back from what your users actually need. There is no such thing as an absolute fast. Right? There's fast for your customers and for your use case. Otherwise, you'll you'll you'll build the wrong thing. You know, this applies to us, too. Like, we're constantly working on making convex faster. We're not making convex faster by buying faster discs or something, right? We're not buying them faster by putting our servers closer to the database. So, the speed of light takes less time. We're making them faster by changing our architecture, right? There's architectural reasons why convex could be faster. It wouldn't constantly. So, that's how we're designing things. re rearchitect things to make them faster. And the same applies to you, right? Architecting things, right, is way more important than just just turning up the knob on a on a kind of performance number, right? Or even just getting lost in the weeds making the ultimate hasht implementation or something like, you know, if you don't yet know what matters, uh, then find out. And then once you find out like you're probably going to discover the architecture you should make, not the not how fast I need to go. You'll be like, "Oh, this is the architecture my system should have." And then the performance will be good enough for the use cases. You know, Jamie, I'm not sure if I'm remembering this correctly, but I think this might be Mal's law. Mdall's law says that you can't speed up you can't increase throughput of a system more than the critical path that can't be parallelized. So this came from an era of of of supercomputing and parallel computing. Right? So it says basically if you have a system that has some thing that's in the critical path like committing a transaction to a database for example um that can't be parallelized you can make everything else as fast as you want. It won't or as parallel as you want and it won't ever get faster than than than the critical section. So it doesn't make sense to optimize stuff if it's not the things contributing to why your system is slow. If your hasht is not actually making your system slow, speeding it up is not going to make any difference to your users, right? Even though it might be fun. Yep. I mean, this is probably a lot of the stuff we're did or a lot of the a lot of our thinking around this particular point really does cue back to this general premature optimization thing, right? Like if you're not measuring and it's not anchored in a real problem that people are running into, it almost certainly is not yet worth making something faster. You need to understand first what matters. Now um one of the most useful classes I think I studied um in university in my unfortunate many years in university was uh being a network researcher. For a while I was I studied computer networks. Um uh and and did queuing theory and learned a lot about how people design routers on the internet. Um and there's so much to learn about systems design from networking people because networks computer networks are distributed systems, right? And they're very large scale distributed systems like super duper large scale. And one of the biggest issues in uh in queuing theory in in routing is called buffer bloat. Mhm. And so buffer bloat happens when you add you make the cues in your routers too large. This is I'm getting to the queueing is bad, right? You make cues in your routers too large. So you see it it seemed like if Cisco's selling a router, putting one gigabyte of RAM in there is better than putting one megabyte of RAM in there because you can buffer up all the requests, right? And if more requests show up than you have the capacity to process right now, well, you just buffer them up in a queue and then you process them once that queue drains once once load depletes. And buffer bloat has been a massive problem in um in the routing kind of field. And so you know network researchers have invented so many algorithms to proactively drop um requests in a network like random early detection and all these algorithms to proactively just drop requests if anything slows down a little bit because of all this buffer bloat and buffer bloat is queuing and queuing contributes to slow is worse than broken initially. Now, if you have a system that is receiving more work than it can adequately process, there are just two things you can do. Well, there's only one thing you can do. You can either like refuse. You can you can issue back pressure. You can drop the request. You can say, "Hey, I'm too busy right now. Come back later." And give the system higher than you in the stack the opportunity to go away, do something else. Or you can cue it and turn that overload into latency. And that latency just causes systems to grind to a halt. And so often we see systems fail because someone a well-intentioned developer has slapped a queue in front of something thinking that the queue will make the system work better because you can do some retries and and ultimately a little tiny little bit of queuing is okay. sometimes a couple of requests, but once Q's exceed a certain threshold, they almost always lead to systems failing. Mhm. Yeah. Q's are Q's are bad in general like [laughter] you know there can be you know judicious use of and I would say in particular like fixed size Q's like Q small cues of uh you know that do not grow and reallocate can help with variance small amounts of sec like second to second variance but that's about it and so I do think that that and um it's a thankfully it's a little less popular than it used to be But the single most overapplied system I think in in uh infrastructure over the last 20 30 years has been Q's. So people just for a while were just putting Q's in between everything. You know you have this service then you have a Q and then you have that service and then you have a Q and they use the Q as a gobetween for every single thing. And there is this exercise you can ask where you're just like what is that Q what what use is that Q doing other than being another another cause of back pressure and and bloat and you know so another thing to manage. So slapping a queue in front of something is such a noob move because it's like it's it's it's almost like a let me say something real strong. It's a sign of weakness [laughter] sometimes because you that is that is a strong statement. Yeah. you're not willing to accept the reality of the situation is that you are getting more load than you can handle right now. Right? Sometimes you just got to say sorry I can't handle it. Now there is a use. So firstly there's a few uses for cues. One background jobs an incredible use for a queue right if you have a bunch of background processing you want to reindex all the emails in your system put them in a queue process them slowly over the next week whatever that's is out of the critical. So we're talking about Q's in the critical path, right? Secondly, sometimes a very small Q is valuable to deal with kind of load spikes to get get absorbed very fast. Now if you have to have Q's in your system and you want to run it at large scale, there are some weird tricks people play and I can give a few weird tricks we play. Um I've talked about this before, but so in there's a few times we just kind of have to have a queue and we use um um an algorithm called Kodell basically but we use basically two techniques. One is is called adaptive LIFO and one is called what's it called? Proactive dropping or something I don't know it's even called. So, two things. Once your system is overloaded, the system is receiving more work than it can handle, right? Um, is possible for your queue to grow longer and longer and longer, right? And all of a sudden, it and if your system's extra slow, it takes a long time for messages to get to the front of the queue. And if you do this particularly poorly, you will find that by the time the message gets to the the front of the queue, it's too late. Like, the user's already gone away or the system's already timed out. But they're seven layers above you, right? So you're spending all your time processing requests that are now useless. No one cares about them anymore, right? So sometimes uh people adopt techniques like adaptive LIFO. So Q's are normally FIFO, first in first out. That's the obvious way to process a queue, right? Under overload situations, i.e. Once the queue gets beyond a certain length or the service time gets beyond a certain length, you can switch to processing the queue in reverse order. Process it like a stack instead. Seems really strange, but at least you're guaranteed that you're processing requests that haven't timed out yet. And the other is just I forget the the terminology. Someone can comment in the in the in the comments and and tell me what the terminology is. Proactive discard, adaptive, whatever. you you proactively start dropping requests. You just look at the queue and just start dropping things out of it, right? That starts draining your queue and it gives a signal to folks higher up the stack that hey, they should probably back off and do something else. This is very very common in um in computer networks where proactive discarding of packets is is very very common. So there are techniques people use, but if your eyes just glossed over and you're like, "What the hell, James? I don't want to think about these things." Probably shouldn't slap a Q in front of things. Or be very, very judicious about about doing so. Yeah, it's probably a safer model to say putting a Q behind things is a better idea than putting Q in front of things, right? [laughter] That's a huge simplification. But if your thinking is more a queue is a place where things rest so they could be visited entirely asynchronously in pipelines which can stall cool that's fine. Um but if a que is in front of something and you think it's adding to maybe the operational robustness um yeah I mean go back to slow is worse than broken and all these other things. I wanna if if this point hasn't been driven home yet that like cues are complicated and messy and you should avoid them. Um I want to give you an example of a very hard bug we have to debug once. So we were moving a whole bunch of data off of Amazon S3 to Dropbox and I think we're running uh 768 gigabits per second. we were running almost a terabit per second uh between AWS and and Dropbox. that's requires some big expensive cables and and how these cables work from the engineering team's perspective is like hey James and Jamie here are some 40 gigabit connections that you can use right so you can copy this data over 40 Gbit connections and so we start using these and we're finding out that you start off pretty close to 40 Gbits but then the the data throughput would drop off and off and off and off and off and then we'll stabilize around 10 Gbits per second and you're like why are these 40 Gbit per second links only giving us 10 Gbits per second? So the answer is that they weren't 40 Gbit per second wires, right? There were 10 GB 10 Gbit per second cables, right? In a bundle of four, right? And there was a connection pool in front of these. And so how does a connection pool work? So a connection pool maintains like a an array a a stack of connections to another service. And so when you get a request to fetch something, it takes a connection off that stack, uses it, and once the connection returns, it puts it back on the queue. Right? Now, what happens if one link slows down a little bit for some for reasons, right? It's more loaded than the others, the the requests come back a little bit slower than the others, right? So, you have a load spike, the connection pool increases in size, you issue a bunch of concurrent requests, and they all finish at different points in time. And the ones that finish first get put back on the stack. And the ones that take a long time to finish, they just eventually get get dropped because we're trying to maintain a certain size connection pool. And over time, if you keep iterating on this process, you'll find that the slow links end up getting evicted from the connection pool and the connection actually the other way around. The fast links end up getting connection pool and you end up with a connection pool basically full of one circuit, right? And so we were eventually ending up with like all our traffic going over one circuit. And I explained that really poorly to be honest, Jamie. But [laughter] yeah, to the point where I almost want to pause it and re-record that. [laughter] Maybe you can say better than me. The lesson I was going to say here is yeah, it's very complex to manage when you have connection pools and cues. Very strange emerging properties occur. Yeah. Yeah. Yeah, if you imagine I mean one way to think about it, right, is if you imagine there's a population of 40 connections and periodically a kind of purge is run that removes connections out of like the free pool, right? Um that you don't need anymore. You don't need 40. You're only going to have 10 spare or whatever. The truth is the fastest links uh that got the requests done quickly returned to the free pool. So they're the ones that are a they're the ones that are available to to remove. So thank you Jamie for clarifying that my poor explanation. Yeah. [laughter] So by by virtue of being fast they're sitting around idle and so therefore they get terminated and pretty soon the only links that are left are the ones that are taking a long time and therefore are never happen to be free. Um they're always in a request because requests are taking so long. So yeah connection pooling is tricky. What's the point of the story? Well one, it's an interesting bug. It's a very unintuitive bug, right? And I think the answer here is that Q's and connection pools which are kind of bundles of of cues in many respects are really hard to manage. In fact, my SQL and Postgress are terrible at connection pooling, right? Yeah. Um and because it's hard, right? And so the answer is just, you know, you probably want to avoid these problems all together. Yep. Yep. Um Oh, definitely. I would say uh cues are are a heavily used thing. And I think connection pools are also connection pool is everywhere and connection pools are hard to get right. All right. So I think we have some wisdom. People can decide if they're wise or not. But I will say these are things that like we I I'll say I did not know these things, right? I did a I did a PhD in distributed systems. I didn't learn these things. There was no class that told me this stuff. This is comes from like mostly from pain. This is kind of like this comes from building stuff and it not working and then be like, "Oh, that's not good." Yeah. Um, and here's my plug for like aic development. Make sure you're experiencing a bit of pain. Make sure you've you're struggling a little bit. Make sure every now and then you're stuck trying to solve a problem. That'll help you develop um wisdom of your own. Um, so to summarize, you can't test your way to correctness. Testing is really, really, really hard if you're doing it well. If it's not hard, you're not doing it well. Um, and two, even if you test a system, it's not going to work for sure. Uh, slow is work worse than broken. Steady state should be worse state. Simple is better than complex than sophisticated. Architecture is more important than performance. And cues are hard and usually bad. That sounds like a great summary of the six things, James. There [laughter] we go. There we go. So, if you just memorize six things, that's it. you've known you know everything about systems and uh no there's lots to talk about in systems and we'll hopefully do some more episodes talking about systems wisdom uh but this is just some stuff we wanted to share with you today I think this is a really good set of a lot of the most common things that there's a long list but I will say that these are probably the most frequently encountered kind of counterintuitive things in my experience so if you really try to put these to work um yeah you'll probably find that You just run into things going when things go wrong, they don't go as wrong. That's the goal, right? Things will go wrong, but we we want them to to go wrong in a way that is not too bad. Systems tend to complexity. Systems want to fail and systems want to collapse. They don't want to fail a little bit. They want to fail entirely. Right. And if you kind of follow some of this wisdom, it's really mostly about simplicity and eliminating complexity architecturally. Yep. All right. Cool. I think that was good. Episode two. Thanks everyone. We got it done. Thanks everyone. Until next time. All right. See you.