Stop Talking About The CAP Theorem
On Avoiding "Buzzword Driven Development"
Focusing on more useful interview topics, I’ve been genuinely concerned about this trend in System Design interviews. SD are my bread and butter, and they’re a large part of the draw for candidates to working with NextPath. I’ve nearly worn out my soapbox on this topic, so rather than beating this dead horse, I figured I’d write up my thoughts to share.
In addition to my rant, I highly recommend you read the Wikipedia article on CAP Theorem, Vivek Bansal’s excellent primer on the subject, and Martin Kleppmann’s post about the technical deficiencies of the theorem as it relates to databases.
Also - if you have questions, drop them in the AMA box, or even better, come to the NextPath “Ask A Hiring Manager” event July 16th! It’s free, and will be a great opportunity to ask anything you’ve ever wondered about the industry, the hiring process, and how to get good at engineering.
Okay, lock in. You’re interviewing a candidate for a Senior Fullstack role. It’s the third round for this candidate, and they’re the seventh to get this far. Their resume is promising, but it’s the System Design interview - this is the one that has tripped up most of the prior candidates.
The candidate has started off pretty solidly. They asked good questions to define the scope, didn’t introduce unnecessary complications when discussing requirements, and while they didn’t blow you away with their instincts on acceptable latency, they were directionally... passable.
Ok, now here’s the moment. Brace for it. Maybe it won’t happen?
No such luck.
“Now let’s discuss the the CAP theorem and whether we should prioritize consistency or availability.”
A week later I was meeting up with a hiring manager at a local AI startup. We were discussing trends in interviewing, and I finally realized I wasn’t alone.
Them: I’ve had candidate over candidate who do so well in interviews but then we get to System Design and it just... gets weird
Me: How so?
Them: They just seem like robots. They ask the same questions, use the same approaches, and just aren’t thinking about the problem at all.
Me: What kind of questions are they asking, by chance?
Them: Well.. this is weird but, they keep asking-
Me: About the CAP theorem?
Them: YES WTF
If you’re new here, I want to give you some context on myself. I’ve been building teams at tech companies for well over a decade now. I’d heard of the CAP theorem in college, but it never really came up afterwards. Suddenly, starting in the past year, it’s been brought up in roughly eight out of ten system design interviews I’ve conducted (I did the math). I won’t go into why this is suddenly happening - despite never coming up in any interviews I have taken, as candidate or part of the hiring team, or in any job I’ve ever worked in. Let’s just say there’s a certain interview prep website that has gotten fairly popular during that timeframe.
But isn’t the CAP theorem useful?
Absolutely. Let’s first cover what the CAP theorem is:
The CAP theorem states that distributed data systems can simultaneously guarantee at most two of three properties:
Consistency (all nodes see the same data at the same time)
Availability (every request receives a response)
Partition Tolerance (the system continues to operate despite network failures)
In a distributed system, you have Network Partitions. These are parts of your system (databases, typically) that cannot communicate with each other. Because of that condition, it is impossible to have both High Consistency (HC) AND High Availability (HA) as priorities in your system.
Put plainly, if two servers cannot talk to each other, they either have to wait until they have the most up-to-date information to respond to requests (HC) or send along whatever info they have regardless of accuracy (HA). So therefore, the decision is whether or not to favor HC or HA.
I Think I Get It… Wait No, I Don’t.
Here’s a real life example from my time at Wayfair. I was leading development of their personalization service, an API that served hundreds of millions of recommendations per day. The API I was building needed to serve relevant recommendations based on pre-crunched data from hundreds of machine learning models, which were retrained daily.
If the data was stale, that wouldn’t be the end of the world - yesterday’s recommendations were good enough. And if the service timed out or went down, the frontend just hid the recommendations section or provided hardcoded products so there was something there. This is an example of favoring availability.
The actual recommendations, however, had to be in stock and available. If we stopped selling, or sold out, or a given product, we never wanted to recommend it no matter what - so we needed to make sure that we could get the ok from our product availability service. This is favoring consistency.
Before I go on, I’ll remind you that this is a pretty high level explanation meant to bolster my argument that we need to stop discussing the CAP theorem in the vast majority of software engineering interviews. As for the CAP theorem itself, Martin Kleppman said it best in his blog post Please Stop Calling Databases CP Or AP.
Now let’s cover what the CAP Theorem is not.
It is not black and white: it is a sliding scale problem, where you need to make micro decisions which will cause your system to lean one direction or the other, and those decisions will need to consider both local optimizations as well as global requirements.
It is not unproved: it was formally proven in 2002 by researchers Seth Gilbert and Nancy Lynch of MIT in their paper “Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services.”
It is not relevant to non-distributed systems: You can absolutely have both HA and HC in any system that has a centralized data store, as long as that database is alive and well (otherwise, you have no availability OR consistency, but this isn’t a CAP issue- it’s a liveness failure). If you don't distribute your data, you don't need to worry about different nodes having out-of-sync information, because there's only one node. With nothing to keep in sync, there's nothing to partition, and therefore no forced choice between consistency and availability.
Once again, it is Not. Relevant. To. Non. Distributed. Systems.
So why are we talking about it?
Certain interview prep materials, that may or may not be popular because of the pedigree of the materials’ creator, can in fact be limited in efficacy because of that same pedigree. Say, for example, the creator of the materials comes from a FAANG background, they would be particularly specialized in helping candidates interview for similar companies.
Meaning:
Companies that have large, interconnected systems that serve hundreds of millions of customers
Companies that have massive engineering teams of tens of thousands of developers
Companies that process petabytes of data on the daily
So I’m not saying never discuss the CAP theorem, but definitely don’t always discuss it. The CAP theorem is only relevant to distributed systems, and if you’re discussing it before you’ve indicated that you’re going to use a distributed database (and why), the cart is way before the horse.
Here’s the secret: interviewers just want you to discuss concepts like a human being. They want to hear you explain the tradeoffs you would consider, and why. For example:
“I guess it’s ok if the recommendations are old, so we can introduce a persistent cache that always has the latest data, even if that data has expired. That way if the update process fails, we aren’t left without product to recommend.”
“Question: Do we ever want to recommend an out of stock product? No? Ok, then, before we return a product as a recommendation, we should hit this product availability service. If we don’t get a response for some reason, it’s better that we don’t return any data, rather than accidentally return an out of stock product.”
Why It’s Dangerous
When someone brings up the CAP theorem, unprovoked, and in a situation where it may not be appropriate, it sends really negative signals.
For one, it signals that you aren’t thinking about the problem presented to you, and are instead using a one-size-fits-all approach. This is very bad! As software engineers, we are expected to be able to solve a very diverse array of problems, efficiently, in the way that makes most sense for the given scenario. Bringing up the CAP theorem (or any topic) when it isn’t relevant is like asking how many bottles of tequila you should order for a teenagers’ birthday party - your judgment will immediately be questioned.
Another issue with this type of approach is a bit more subtle. For all the times I’ve had a candidate discuss HA vs. HC during the initial stage of a System Design interview, I almost never have seen it come back. Like a failed use of Chekhov’s gun, if you make the assertion that a system should prioritize consistency over availability, but then never make decisions that support this claim, the interviewer is left unsatisfied.
And finally, it’s just plain wrong. If you’re discussing whether a system should prioritize availability over consistency, when you can have both, you’re bound to fail. Additionally, the CAP theorem is not applied to an entire distributed system - merely each network partition, of which there can be numerous. So if you’re going to discuss a tradeoff such as HC vs. HA, don’t discuss it globally - it should be on a per-feature, or per-subsystem basis.
A Warning
There’s a lot of resources out there for system design, and each one is on a spectrum of “pretty helpful” to “complete shit”. Regardless of how useful any one resource is, however, remember that designing systems is not something you can just learn via a subscription. It’s something you need to study, experiment with, and fuck up before you get even close to being good at it.
So if you’re reading this in between sessions with the same old prep service that approaches every problem as if it’s just designing twitter/facebook/instagram/bluesky/threads/whatever, maybe try something different. Go read tutorials of existing technologies, find some blog posts from engineers with different experiences, or watch one of the two enthralling videos I managed to put up on NextPath TV over the past year. Whatever you do, just remember to use your own brain in an interview and don’t be a clanker about it.