Depending on what you try to achieve in life, you might want to advance in your career or simply in the profession of a software engineer. Perhaps you want to become really good before you start your own business, or you want to climb the corporate ladder. But no matter what your motivation is, growing as an engineer does not only have a lot to do with your hard skills. It's also about soft skills and your own mental model.

Here are the five lessons, as far as I can recall, that helped me immensely to grow as a software engineer. Let me share them with you in the hope of them helping you as much as they helped me.

Learn to listen

There is a German saying: “Reden ist Silber, Schweigen ist Gold”, which roughly translates to: “Talking is silver, being silent is gold.”

What it basically means is: Sometimes, it’s better to not say anything unnecessary. In an even broader scope it can be understood as: Listening to others can often be more valuable than talking yourself.

Especially in software engineering, just listening to your peers can be a very valuable action to perform. You don’t always need to comment on everything, especially if a comment wouldn’t add a lot of value. The amount of knowledge you can gain by simply letting your colleagues talk about their experience with certain problems, or approaches they have taken in the past, is invaluable.

You also don’t need to interrupt your peers if you have questions. Let them finish first. If you are afraid to forget the question, write it down. Ask the question later.

Accept you don't know everything

Software engineering is too broad and too deep as if we could ever really know everything. You need to accept and understand that. The more senior you become, the more you realize what you don’t know. But that’s not a bad thing. It is just what it is.

There are many different fields in software engineering, and even many fields are so deep that you can never learn everything about them. Even if you try, your older knowledge will be outdated the moment you reach a quarter of what there is to learn in your field of interest.

Instead, focus on what you have most interest in and let projects drive your curiosity. On-demand learning can often be way more powerful than learning things only for the sake of learning them. Especially if you have a solid foundation of knowledge built up, it becomes way easier to absorb new concepts and technologies on the fly.

You will never catch all edge cases

Bugs are everywhere, and they will always be. It’s not optimal, but once again, it is what it is. That is what quality assurance measures like tests are for, to catch as many of them as possible. But at some point, it’s not worth putting more thoughts into catching all possible cases. No matter how much energy you put into it, there will always be cases you can’t even think of. It’s better to preserve this energy for more important tasks.

A better way to deal with this circumstance is to try to catch the most common ones you can imagine and then guard against the unforeseen cases as good as possible. In the end you need to make sure that bugs you couldn’t catch don’t bring everything down. Sometimes, a circuit breaker can lessen the blow of even the craziest cases of upstream errors, and sometimes a generic exception handler makes your day. And sometimes, you can even implement software in a way that leaves no room for unforeseen cases. That is also something you learn with more experience.

Software engineering is a team game

There is not much space for lone wolves in software engineering. We work in teams for a reason. Those teams are usually assembled to make working on difficult problems easier by having an available pool of talent who can learn from and help each other. Whether it’s just talking about an issue with your peers, a system design session with the whole team, or just the reviews your colleagues do for the code you produced, team members are a part of the process.

Not leveraging your peers as what they are (next to hopefully being awesome humans), valuable resources to make use of, is a waste of said resources. It’s even counter-productive.

You should also always take care of preserving a good atmosphere within your team. It makes no sense to ruin that atmosphere only because of a small dispute between you and one of your colleagues. Make sure to keep everything as inclusive as possible, including reviews. Healthy and happy teams are known to produce better results, and a good atmosphere, paired with great results, helps to keep your own morale up. You could call it a win-win situation.

Automated testing is not optional

Look anywhere online, and you will basically identify two camps of developers and engineers. The first one advocates for writing tests because they’re helpful. The other one advocates against tests and says that they are nice to have, but should never block you from launching something new.

The reality is, however: Automated tests are nothing else than an (drumroll, please) automation of all the manual tests you do over and over again. Especially if you launch a product on your own, your users won’t have infinite patience with you. If you repeatedly break something, users will drop your product in favor of another, more stable one.

Look at it from the perspective of a lazy human being: Why bother to manually click through a frontend over and over again if you can just let your computer do the work? The only thing you need to learn is a few more APIs, and you will probably need a little creativity to really test what you want to test.

Testing can also have a pretty positive impact on your career. Always having to ship hotfixes after each release will sooner or later catch your manager’s attention, but not in a positive way.

Summary

Before we end this article, let's do a quick summary of the five tips previously presented:

- Learn to listen

- Accept you don't know everything

- You'll never catch all corner cases

- Software engineering is a team game

- Automated testing is not optional

Those are the most important lessons that helped me grow. They are really easy-to-implement and won't cost you a lot. So, what are you waiting for? Go and try them out, they might help you to grow, too!

You have (finally?) come to the end of this issue, so let me tell you something:

Thank you for reading this issue!

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver

Developing software is easy. Just implement a few features, add a few tests if you feel like it, and that’s it. Or is it? Well…it depends.

Let me tell you a story:

There once was a team that built software like many other teams. They implemented features, they tested thoroughly, they even had regular load tests, they added all kinds of alerts, and they monitored their software closely. Their software ran on Kubernetes and was thus deployed as a container on a pretty large cluster. Their deployment strategy was simple: Just a rollover of the deployment. Release a new version, use Helm to trigger the rollover, and let all the magic within Kubernetes do its thing.

The software did not have many dependencies. It just needed a Redis to store some data inside, and that was it. Get a request, make a Redis lookup, do some more processing, and then return a response. Not much that could actually fail.

Everything ran fine for one and a half years, until one Saturday evening, they got a call from a Vice President. Everything was down because their service didn’t respond anymore. Something was odd. All requests took thirty or more seconds. All clients only showed the loading placeholders of doom. Customers were unhappy. Management was unhappy. Nothing worked.

What had happened?

As it turns out, a massive spike in traffic had actually overwhelmed the network interface of the ElastiCache cluster (in the end, just a Redis with an interface in front of it) the team used to store their data in. Additionally, the cluster had so much data that the machine began to swap memory. This meant that all Redis commands took forever to finish. And this, in return, meant that everything took forever.

But how could this happen?

It turns out that the Redis connection was just a plain Redis client, and the logic worked as follows:

1. A request comes in

2. Take a connection from the pool

3. Send the Redis commands

4. Get the responses

5. Return the connection to the pool

6. Process the responses

7. Send a response to the client

Just plain logic. No fallbacks. No safety measures.

Spot the issue? Yes. A feature that had worked for one and a half years didn’t work in one particular situation. A feature that had been load tested thoroughly and never broke finally gave in and broke. It broke in a situation with so much entropy that no one could ever have come up with a suiting load test to reproduce such a scenario reliably. Nevertheless, it happened.

A very important lesson about resiliency

The short story above is just an example, but scenarios like this one happen probably every day around the world.

They teach us one crucial thing, however:

We actually don’t build software for 99.99% of cases. We build it exactly for the 0.01% of cases where something really breaks. Or better: We should.

But, resiliency is difficult. Our job as software engineers is not done when we think it’s done. It’s done when we’ve put in our best efforts to protect our software against nearly unimaginable failure scenarios. And we can be sure of one thing: We will never get it right. This is why I said “best efforts,” because failure is inevitable.

Even with a lot of experience, you will still overlook a possible edge case from time to time. What really counts in these circumstances is all the measures you have taken at least soften the blow as much as possible. And even these take a lot of experience and imagination to come up with.

Next to that, resiliency ≠ resiliency. It’s a situational thing. Some applications need more resiliency than others. A static website that you deploy to Cloudflare Pages, Netlify, or Vercel needs way less resiliency than a huge streaming portal with many moving parts and hundreds of micro-services, deployed to Kubernetes on AWS. The static website already profits from what the serverless hosters offer, the streaming portal is so complex that it needs a lot of custom-tailored solutions to increase its resiliency.

Lastly, some measures to increase your resiliency cost way more than others. At some point, the decision about whether to increase your resiliency becomes a matter of your available budget. And more often than not, you will probably have to decide to not make your application more resilient because it’s not economical anymore.

Actually, there is a pretty simple rule about resiliency and uptime, which goes as follows:

Achieving 99% uptime is easy. 99.9% uptime is already exponentially harder and more expensive. Every additional 9 you want to add to the decimal places of your uptime will cost you exponentially more than the previous one.

And finally, you will never be able to make any system so resilient that it will forever stay at a 100% uptime. It’s impossible. It will never happen. There is always a single point of failure that you can’t make more resilient. If you try to, you create another single point of failure. If you try the same procedure again, you end up where you started. You’ve basically reached the end of it. At some point, everything else is out of your control.

There exist a few memes for a reason. One example is one about AWS’ us-east-1. Even a company like Amazon can’t eliminate all its single points of failure. If us-east-1 is down, many other regions and services go down.

This proves one point:

Resiliency is difficult, and sometimes, you just can’t get it right.

A few examples of what you can do to improve resiliency

Although you can never get to 100% uptime and completely resilient systems, it’s still worth investing in the parts of your system where it makes sense. Sometimes, it doesn’t take much to improve the resiliency of a single service so much that any further investments make no more sense. The rest is just the risk you need to live with.

But let’s take a look at a few examples of how you can improve the resiliency of your software, even on a smaller scale.

Circuit Breaking

Circuit Breakers are one of the most important resiliency patterns out there, and yet they are not used often enough. They would also have been a pretty great way to prevent the issue of our small example from the beginning.

In electronics, a circuit breaker is an intended breaking point. In case of an overloaded electric circuit, a ground fault, or a short circuit, the circuit breaker trips intentionally and prevents further damage to devices and the system itself. The circuit breaker itself is designed in a way that it can break without becoming broken. It can trip over and over again (at some point you have to replace it, but that usually takes forever).

Circuit Breakers in Software Engineering

In software engineering, a circuit breaker works similarly. A method or remote call is protected by a circuit breaker that constantly measures the behavior of what it protects. If one of the metrics the circuit breaker observes exceeds a certain threshold, the circuit breaker trips temporarily to prevent further damage to the system. After a while, the circuit breaker slowly opens the flow again for a percentage of all calls, while observing its metrics. If the metrics stay below the threshold, the circuit breaker opens up the whole flow again. Otherwise, it interrupts the flow again.

Conceptually, a circuit breaker in software engineering has three states:

1. Closed

2. Half-Open

3. Open

Closed State

In this state, everything works normally. Closed in this sense is more to be seen in the sense of a normal circuit breaker in electronics. As long as the circuit is closed, electricity (or requests in software engineering) flows.

Half-Open State

In this state, the circuit breaker has already tripped. It was previously open and now closes the circuit only for a certain percentage of requests to see whether everything has gone back to normal. If so, it goes into the closed state again to allow calls to flow through. If not, it opens and prevents all requests again.

Open State

In this state, the circuit breaker has tripped. One of the metrics the circuit breaker observes has gone over a certain threshold and all further requests are rejected temporarily.

Additional traits

Unlike a circuit breaker in electronics, the one we use in software engineering has a few additional traits. Instead of only the electric current, it can observe multiple metrics. Usually, a circuit breaker has at least support for measuring and handling timeouts, exceptions, and other errors. Additionally, most circuit breakers allow for a fallback. This fallback is used whenever the circuit breaker trips.

What a circuit breaker can be used for

Circuit breakers are usually used to protect remote calls. In distributed systems, other services and components are usually way more likely to break than internal components within a service.

A circuit breaker can protect calls to upstream services, but also to databases. Sometimes, it also makes sense to protect calls that are otherwise internal but make use of system resources, especially in containerized systems. If a service uses an SQLite database that is mounted through a volume inside a Kubernetes cluster, it can indeed also be worthy of being protected by a circuit breaker.

Usually, a circuit breaker wraps a function call, which itself performs a remote call (or accesses the file system, like an SQLite database).

Common misconceptions about circuit breakers

A circuit breaker doesn’t magically make everything better. It also doesn’t prevent issues from occurring. In fact, it actually makes things fail faster and it also creates another category of errors. But in the end, these are intended measures to soften the blows of failure.

Many engineers are afraid of intentionally letting things fail. They work under the assumption that sending a response somehow (even if it means blocking resources for tens of seconds) is better than showing users an error. But this assumption is wrong.

The reality is that protecting the system for at least a certain percentage of users is better than letting it fail for everyone. If a circuit breaker in your service gives another service time to breathe and recover, that’s better than continuing to put that other service under pressure until it completely fails.

More often than not, you can implement a reasonable fallback. Sometimes, serving static fallback content can save the day. It might not be the most accurate content and may not be personalized, but once again: Something is better than nothing. And even if you cannot reliably serve any fallback content, you can still ensure that your application only fails for a certain percentage of users.

Circuit Breakers as Infrastructure

If you work with Kubernetes and deploy containers to clusters, you have even more choices. A good Service Mesh can provide circuit breaking capabilities out-of-the-box, for example.

Services Meshes are designed and implemented to deal with the chaotic nature of distributed systems. They control how data flows and help with exactly that. Most of them provide sidecar proxies that additionally come with built-in circuit breakers. And usually, you can fine-tune these with resource definitions and tailor them to each individual upstream service. The sidecar proxy intercepts your outgoing traffic, proxies it, and handles things like circuit breaking for you, while returning the response as if nothing had ever happened.

Deploying a service mesh and actively putting your services inside that mesh often frees you from having to manually implement circuit breaking for upstream APIs. It may not free you from implementing them for your database calls, though. But still, if you configure them correctly, you don’t have to write code yourself, which alone is already worth a lot.

Istio, for example, comes with an excessively configurable circuit breaker. You can add a so-called DestinationRule for each upstream, which additionally frees you from having to rebuild your software only because you want to reconfigure one of your circuit breakers. Changing the configuration only involves updating a resource definition within the cluster, which is usually way faster then rebuilding your whole software and deploying a new container.

Dealing with Backpressure

Just try to imagine that someone pushes you a few times and at some point you fall. But when you try to stand up again, that someone continues to push you. Standing up becomes nearly impossible at this point. This is too much backpressure.

Every service or component in a system has a certain load it can take before it breaks. Even if scaled horizontally, there are situations in which single instances of a component cannot reliably take any more. In this situation, they are just overwhelmed by the requests they get and give in at some point. When this happens, recovery is often difficult, especially when the requests don’t stop.

Interestingly, many software engineers still ignore handling backpressure and instead focus on other things. They do nothing to protect their services and components. If their services get overwhelmed, they have a hard time trying to recover the system.

Handling Backpressure

Although circuit breakers exist, you should never rely on everyone playing fair. Maybe some of your colleagues didn’t have the time (or sometimes will) to implement a circuit breaker with reasonable timeouts and a back-off strategy for calls to your component. Sometimes, clients are not under your control. On the internet, everyone can call a public API, and even if it’s authenticated, it doesn’t protect you against getting requests at all.

It’s your job to find out at which point failure is imminent, by, for example, load testing thoroughly. You need to find important metrics your service depends on. Perhaps a component is CPU- or Memory-bound, or it’s actually depending on the amount of file descriptors available. These are metrics you can actively observe, even within a component itself.

You need to discover thresholds at which a component needs to say: “No more.” Depending on where you deploy your software, you might even need to find multiple. On Kubernetes, for example, you can gather one set of thresholds at which Kubernetes should direct no more traffic toward a pod (that’s the readiness probe), and one set of thresholds that puts a service into emergency mode, stopping most of the work and actively rejecting requests. While the first set’s values should be below the values of the latter, both should still be a little away from the point of complete failure.

Handling Backpressure in HTTP Services

HTTP has a strategy to tell clients to “back off,” HTTP status codes and HTTP headers. Although it’s no requirement for a client to implement, it’s still a good first start for you because at least a few HTTP client libraries respect it.

There are two status codes that you can use for this purpose:

• 503 Service Unavailable

  • This indicates that the service is not ready to accept any traffic at the moment (which means that no one can currently be served)

• 429 Too Many Requests

  • This usually indicates that a specific user has sent too many requests and is probably being rate limited (which means that other users may still be served)

Additionally, there is the Retry-After header that you can use to give your service some room to breathe (at least from clients that respect it).

If an instance of your service detects that it’s close to collapsing, it should actively start to reject requests. In this case, you can decide between one of the two status codes, mentioned above, to return. In most cases, if you don’t selectively rate limit, a 503 is the better choice, though (at least semantically).

Additionally, you can set the Retry-After header to one of two possible values:

• A date

• An integer value in seconds

If you set a date, some clients will respect this value and only try to issue a new request after the date specified. In case of an integer value, clients are asked to retry the request after the amount of seconds specified within the header has passed. If your service only receives requests from components inside a closed system, you can require everyone else to honor this status code and header combination (or even make it a standard for everyone in your company).

The important task for you, however, is to implement a backpressure strategy at all and as early as possible. Rejecting a request if your service can’t take any more should happen before it spends significant resources trying to process a request. This usually means implementing this kind of logic in a middleware layer.

If you place these security measures as early as possible in the call chain of any service, it becomes less important whether clients honor an HTTP status code and a Retry-After header. Clients respecting it are just a nice added bonus on top. Rejecting most of the work as early as possible still gives your service more room to slowly recover. If your service is deployed on Kubernetes, it can additionally intentionally signal that it’s not ready to receive traffic by returning a non-successful status code at its endpoint for the readiness-probe. Kubernetes will then shift traffic to other pods until the readiness-probe signals a recovery by returning successful status codes again.

Handling Backpressure in Other Components

For all other components that are no HTTP services or similar, there are usually other strategies. Sometimes, they are directly baked into the component itself, sometimes S/P/IaaS providers provide them for you, and sometimes, you need to handle it yourself from the outside.

For these components, it’s important to do some research. You need to find out whether the components you want to use have a backpressure strategy in place or whether there are additional components (like proxies) that you can deploy to do it for you.

If there is nothing available, you probably need to implement a strategy in components that are under your control. For a database, that means circuit breaking all requests actively and trying to allow that database to recover if its overwhelmed. Often, this includes especially configuring timeouts because long-taking queries can be a good first indicator of overwhelm.

In the end, you need to get creative when dealing with such components because there are so many different of them. It can also take quite some time to come up with a suitable strategy. Nevertheless, you should invest time and resources into it because when things really get bad, you’ll profit off that investment massively.

Caching

Caching usually has different applications, but it’s also a good way to increase the resiliency of software components. Caches (whether in-memory or remote) are usually faster than calling the sources of their data. Calling a Redis to get some data is, for example, often faster than performing an SQL query, which provides the source of that Redis’ cached data. Having an in-memory cache in front of your Redis in return is faster than issuing a call to your Redis at all. Having an application cache in front of all your logic can save your service from performing too much work overall.

Which types of caching you should implement does, of course, depend on your specific use case. Sometimes, caching your database data is enough, sometimes, an in-memory cache is enough, and sometimes you need it all. Nevertheless, all strategies have their own use cases.

Caching Upstream Responses

Caching upstream requests is pretty common. In fact, the HTTP spec even encourages it by providing a Cache-Control header. Any upstream service can send such a header with its response and tell you how you are allowed to cache the response. Browsers also honor the header and cache responses accordingly.

The spec is quite extensive and the header itself has many fields, but all of them have a right to exist. There are also quite a few HTTP clients that already have their own caching mechanisms that work exactly by processing this header. But even if there is not, you can at least partially implement a caching mechanism yourself that processes the Cache-Control header.

Caching directly impacts your system’s resiliency by decreasing the number of requests your upstreams have to handle. If they correctly set their Cache-Control header, they allow you fire less requests against their API because you can reuse cached responses for as long as they are valid.

Another mechanism that even improves your whole system’s resiliency is the stale-if-error field of the Cache-Control header. By setting this field to a positive integer, an upstream service allows you to reuse a cached response, even if its stale (which means it’s already older than its max-age field allowed), in case a call to it returns a 5XX HTTP status code (in other words, an error). This means that even if an upstream service you need to function properly is down or experiences other issues, you can still continue to serve at least some requests.

Caching Database Results

Depending on which database you use and how you store your data, you might deal with more or less scalable components. But even a NoSQL database can become pretty slow if you issue many complex queries (given that you can’t optimize them further).

Caching database results is usually a little more complex than caching upstream responses. Databases don’t send a Cache-Control header that tells you how long you can safely cache their responses. Even worse, you either have to make an educated guess and decide for yourself how long you think you can safely cache a query result (But keep in mind, data tends to change. Sometimes even rapidly), or you need to implement some form of eventing and leverage database triggers to notify all clients to invalidate specific cached entries (which is obviously not trivial and adds a lot of complexity).

Caching query results can greatly reduce the load on a database and give it some room to breathe. It naturally reduces the load a database has to handle because queries are not always issued but only when really necessary. Additionally, if implemented carefully, cached results can also be used to still serve requests even if the database is down. This usually won’t make all requests succeed, but a few are still more than none at all.

Nevertheless, database caching isn’t simple. That’s sure, and there are also other alternatives you should try first (you will learn more about them in the next strategy, further down below), but you should not forget that you can and should (if necessary) also consider caching database query results.

Distributed Caching

Most services you deploy aren’t isolated, lone components. Nowadays, we usually scale our software horizontally. Either by leveraging serverless offerings, or by deploying to Kubernetes. Horizontal scaling is just easier than thoroughly optimizing our code or scaling vertically by adding more and more hardware (although I have heard that these multi-million Oracle Database servers with Exabytes of RAM and hundreds of CPUs still exist).

Scaling horizontally comes at a cost, though. Instances don’t share their memory. An in-memory cache thus doesn’t work reliably. In this case, you can either accept that you have to do certain work n times (n being the number of instances of your service), or you can reduce the load on upstreams and databases even more by providing cached results to all instances of your service.

A common pattern is to deploy a Redis or a Memcached, which acts as a distributed cache. When one instance of a service makes an upstream request or sends a database query, it puts the result into one of these distributed caches and thus makes it available to all other instances. This can greatly reduce the number of requests all instances make overall, which in return improves resiliency once again. And in case of any errors of upstream components, all service instances can reuse the cached data to respond to requests for as long as they are allowed to.

In-Memory Caching

In-memory caching is the easiest (but sometimes also hardest) form of caching. Cached responses can simply be stored in-memory and reused from there whenever necessary (or possible). Memory-based caches are also the fastest caches available because there is nearly no overhead involved. Accessing memory is just faster than anything that involves network calls (which also includes leveraging distributed caching mechanisms), and it’s still faster than accessing the file system (which you can theoretically share among pods in a Kubernetes cluster, but I’d really advise against it unless you really know what you are doing).

A common pattern is to put an in-memory cache in front of your distributed cache, which is put in front of any cacheable remote call. The call chain then usually works as follows:

1. Try to find a cached result within the in-memory cache

2. If found, return that result

3. If not found, try to fetch a cached result from the distributed cache

4. If found, return that result

5. If not found, try to fetch a fresh result from the upstream component

6. If found, return that result and put the result into all cache layers

7. If not found, use a fallback strategy or else

Next to improving the overall response times of your service, in-memory caching, once again, adds to the resiliency of your component. Even distributed caches can experience issues and outages. Having at least a few cached responses available in-memory allows you to serve a few requests instead of none at all.

Application-Level Caching

Application-level caches sit in front of all business logic inside your service. They can be leveraged before you even have to put in any compute resources to calculate or fetch a response for your caller. Right at the beginning of processing a request, you can make a lookup in all your different tiers of caches and see whether you already have a pre-calculated result. If so, you can return that. If not, you can still put in effort to compute a new result (which can then, in return, be stored inside your application-level cache, so that you save computing power when the next request asking for the same result comes in).

As you can probably imagine, this reduces the load on all upstream components even further. You won’t usually end up with a 100% hit rate, but every percent counts. And once again, if some components are down, your application-level cache might still make someone’s day because they get a result instead of an error.

Deduplication

Deduplication is a strategy that tries to prevent the same work occurring concurrently. To better understand the problem deduplication solves, imagine the following scenario:

Three clients make a request to a service at the same time. All three requests hit the service at roughly the same time. They all request the same data. All of them are processed at the same time. What now usually happens is that the service still does the same work thrice (except there are some locking mechanisms preventing them from doing so, but that’s usually not the case). If instead of three clients, you have to deal with a few thousands, the problem quickly becomes clear: Although you are probably already caching results and doing everything else, you still have to deal with a few thousand, non-synchronized requests.

In-Memory Deduplication

Deduplication tries to solve the issue of dealing with non-synchronized requests. In its simplest form, deduplication creates an in-memory queue handling requests to your service. All requests that ask for the same data get queued up. The first request that hits your service in that chain creates a new task within that queue, while synchronization mechanisms prevent other requests from advancing further in the processing chain. After the task is created, the first request gets subscribed to the result of that task. Then all other requests that ask for the same data can advance further, but instead of starting a new task, they also subscribe to the task that the first request created. Then, all work to complete the task is done exactly once (including looking up a potentially cached result in all caching layers), eventually cached (if computed again), and its result then returned to all requests subscribed, which means that all clients get the same result, which only had to be computed exactly once.

Deduplication requests does indirectly increase your service’s (and thus systems’) resiliency. It directly reduces the computing power you need to serve requests and it also saves upstream components from having to put in unnecessary duplicate work. The less work a system needs to do, the less likely it becomes for it to break at all.

Distributed Deduplication

In a more complex form, deduplication can even be performed in a distributed manner. Discord, for example, decouples requests to its ScyllaDB cluster, which stores all messages for all Discord channels, by leveraging proxies and so-called data services.

First, a proxy distributes requests for channel messages by a routing key and then routes all requests with the same routing key to a specific instance of a so-called data service. These data services implement an in-memory queue (as described above), which ensures that requests for the same data only get processed exactly once at the same time. Within these data services, the corresponding queries are implemented to fetch certain types of data, which, in this case, only have to be sent to the database cluster once instead of multiple times.

This is only a small addition to the in-memory variant of deduplication (as described above), but the additional routing proxy ensures that the same work is really only done exactly once, and not by multiple instances of a service at the same time.

Admittedly, distributed deduplication is probably something you only need to do at a very large scale. But it’s still nice to know that there is a strategy available, should you ever find yourself working in a scale that justifies even measures like this one. You can view it as the royal class of deduplication with most benefits for resiliency.

Failover

Even the most sophisticated methods of increasing resiliency we have already taken a look at can’t save you from all kinds of failure. Sometimes, components just break, no matter how much work you save them from. Infrastructure components or databases can just die out of the blue, and sometimes a whole data center loses power (like Cloudflare just experienced, which in return brought all of npm down).

Failover is a strategy that employs a backup in case of an outage or emergency. A failing database, for example, can, in such a case, be replaced by a failover cluster that has been synced in the background all the time. If the main database fails, the failover automatically (hopefully) takes over.

There are multiple strategies for failover, but the most prominent one nowadays is probably Multi-AZ (short for Availability Zone) deployments.

Multi-AZ Deployments

All major cloud providers provide so-called availability zones (sometimes just called different because … marketing) within their regions. These can be separate data centers (located geographically close to the main data center), or just other rooms within the same data center. No matter how a cloud provider implements them, they all usually have their own networks, power supplies, etc. This improves the chances that even if something fails in one availability zone, others can take over because they are all separated from each other.

On AWS, for example, you can deploy everything in multiple availability zones. Even the nodes of the same Kubernetes cluster (usually just EC2 instances) can be hosted in different availability zones. Pods can then be spread among the nodes of different availability zones evenly (or oddly, or however you like. It’s fully configurable) by making use of topology spread constraints and else. All databases you can deploy on AWS, of course, also have an option for a Multi-AZ setup. In case of failure, they offer an option for an automatic failover strategy (which you still have to explicitly activate, though).

Multi-AZ deployments throw money at a problem that doesn’t often occur. But if it occurs, you’ll be happy to realize that your investment has paid for itself. The issues covered by this strategy are usually something completely out of your control.

Canary Deployments

Even if you implement all of the above strategies, there is still a pretty high chance of you just introducing good old bugs. Mistakes just happen. But gladly, you can protect yourself to some extent.

While there are also other deployment methods that have the same goal in mind (reducing the impact of deploying broken software), canary deployments bring the most automation to the table, so we will take a look at them representatively.

Conceptually, a canary deployment is a form of an automated rollover strategy with progressive traffic shifting. Instead of just rolling over a deployment (replacing the old software with the new one step by step), a canary deployment rolls over a deployment progressively and slowly and automatically shifts and divides traffic between a stable and a canary deployment.

How a Canary Deployment Works

A Canary deployment works as follows:

• A new version of a software is deployed

• The old, stable deployment still remains active

• Automatically, a small percentage of traffic is shifted over to the new deployment

  • The shift can either be sticky (session-based, for example) or happen randomly. That’s usually configurable.

• While traffic is shifted over to the new deployment, a controller closely monitors key metrics like error rates

• If all observed metrics stay below a configured threshold, more and more traffic is gradually shifted over to the new deployment

• If during this process, certain thresholds are broken, the traffic is automatically and completely shifted back to the old deployment

• If no thresholds are broken, traffic is slowly shifted over until it hits 100%, which makes the canary deployment the new stable deployment

How a Canary Deployment Improves Resiliency

Even if a new deployment has bugs, it can never bring the whole system down. Due to the (usually) automated nature of the process (given that everything is configured correctly), issues are spotted early before they affect too many users. The whole system can usually never collapse due to one broken deployment.

Additionally, a canary deployment allows for rapid development. New features can quickly be tested in a production environment, which usually yields more useful data than synthetic tests. This indirectly improves the stability of the software because it’s tested against real traffic more often. Importantly, it also allows smaller chunks to be deployed more frequently. The smaller a release, the less likely the chance of major issues becomes. More features that have never seen the reality of production just increase the chance of bigger outages due to multiple bugs affecting each other.

You have (finally?) come to the end of this issue, so let me tell you something:

Thank you for reading this issue!

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver

We are funny beings. I mean us as software engineers, because we love to build stuff. Give us any problem and we will probably get stuck for a few hours and wish we were never born a few times, but we will try to get through it, and we will come up with a solution. More often than not, we will build something ourselves, however, and we will be hellish proud of our accomplishments.

Interestingly, most companies love exactly this. We solve their problems for them, and we even discover a few more along the way and use our precious time to solve them, too. What more could you ask for?

Well, did you know that this is one of the biggest problems in tech? We simply don’t know when to stop. We get so lost in what we love to do that we stop asking ourselves whether we spend our time wisely at all. Only a few companies and managers have realized this, as well. And if they have, they will try to stop us.

If you are a little lost now and ask yourself what I am talking about, wait a second. Let me explain.

Software Engineering in the industry

There is a huge difference between software engineering and industrial software engineering. The first is what we do (or better, want to do), the second is software engineering within a business context, or stated differently: economical software engineering.

The reality is that not all companies have unlimited resources. To be even more precise: No company has unlimited resources. They all have a certain budget they are willing to allocate and another budget they can spend at most.

Money is finite. Every bank account of every company has a specific number inside it. If you go below that number, the company either goes bankrupt, or needs to lend money to continue to operate. If a company lends too much money, it definitely risks going bankrupt, and that is something most companies want to avoid, of course.

What has that to do with us? Well, we somehow like to get paid for what we do. Additionally, we love to spend money. We spin up a few EC2 instances here, create a few ElastiCache clusters there, add a CloudFront on top of it, and now we do the same on Azure or GCP because multi-cloud definitely increases our resiliency. That usually leaves your employer with a high bill, but at least it’s not you who has to pay it, right?

The scenario above is not necessarily bad. If going through your requirements results in a large cloud infrastructure on multiple cloud platforms, then be it. At least Jake from the business unit can grab your cloud engineer Sarah and go through your cloud dashboards and buy reserved instances for the next three years to save a lot of money and even get a few tax exemptions in some circumstances.

So, if that is not bad at all, do we even have a problem? Does this issue even make sense? Could you have spent your precious time more wisely than to actually read all the above, only to realize that I have fooled you?

Nope. There is still a problem. Believe me.

What we haven’t talked about yet is the countless hours we pour into building our CI/CD pipelines to deploy all of our software to our two cloud systems (by “we”, I mean us and the twenty other teams who do the same work). What we also haven’t talked about yet is all the work we put into setting up our own Kubernetes cluster and the dozens of operators we need. We add a service mesh, and definitely need to host a few databases ourselves, right within the cluster because neither AWS, nor Azure, or GCP have a native managed MongoDB (No, DocumentDB is a relational database with a MongoDB interface. Sorry.), or a Redis with more than its default functionality.

Oh, and after all that, we have decided to build our own authentication system, and implemented twelve Lambda functions and five micro-services, which all have nothing to do with our core business, but are still needed for our system to work.

And here comes the crux: There is no “reserved engineering capacity” at a discounted price. We get and deserve our salaries. No matter what happens. And this leads to the most difficult thing in tech.

Finding and sticking to our core competencies

Core competencies are exactly what differentiates businesses from each other. They are what a business makes money with. And they are what you should usually focus your engineering efforts on. Engineers cost money, and you don’t usually get discounts. Everything you build and need to maintain thus costs usually more than your cloud resources or SaaS' cost you because you always need to pay someone to take care of it for you.

If you build a fancy new AI app, your core competency probably lies somewhere in the problem space you are working in. If that is creating headshots of people with AI, it makes most sense to focus your efforts on building a great user experience and creating ML models that produce awesome headshots. It’s neither your core competency to create your own authentication system, nor is it to implement your own vector database.

In such a scenario, it’s essential to make a few make or buy decisions. The leaders of your company need to ask themselves the following questions for everything you need for your product to work:

1. Is this one of our core competencies?

2. How much does it cost us to build competence in a specific field of expertise and build and maintain the solution for the foreseeable future

3. How much does it cost to simply buy an existing solution?

This is where many companies and engineers fail.

Business leaders and budget owners see the cost of an SaaS solution priced at a few hundred thousand dollars a year and immediately begin to worry about their budget. What they don’t understand is that there is usually more cost associated with building something new than only paying for the time some engineers need to integrate an existing solution.

Engineers, on the other hand, would love to create a solution themselves because it’s an interesting problem space they want to work in or it’s something that gives them the most opportunity to learn. They don’t care about the business aspect (which is fine because it is not one of their core competencies), they care about the engineering (which is exactly their core competency).

The harsh truth, however, is that the most efficient way to deal with these issues is to try and make a perfectly neutral and objective decision. It’s exactly as stated above. If it’s not your core competency, try to buy it. Only if all solutions that fulfill your requirements are more expensive than building something yourself (with all true costs associated with it, like long-term maintenance, etc.), you should ask yourself whether it’s worth building a new competency from the ground up and pay all costs associated with it (including frequent failure and outages until your solution is production-ready and solid enough) or you should simply pay someone else to give you access to a solution they take care of.

If something is just too expensive and you also cannot build it yourself because that, too, would blow your budget, it’s probably time to try to come up with an alternative solution. Going back to the Headshot AI app scenario: Perhaps, you cannot build a full model yourself because you can’t afford the GPU time associated with training a model from scratch. In this case, you could come up with the idea to retrain an existing open source model. Maybe, you need some time to get good results, maybe you even never reach the perfect quality, but perhaps it’s just enough to get your first few thousand customers and bring in cash you can reinvest into developing your own model from scratch.

We, as engineers, still have to learn

Even if some of our business leaders are bad at finding and sticking to a company’s core competencies, we as engineers can do better. We are the ones whose job it is to come up with solutions, but it’s also our job to offer expert advice. This can and should also include advising against building certain things ourselves only because we want to. Sometimes, we need to argue to spend a few hundred thousand dollars a year on a (let’s just say) ready-made monitoring solution instead of setting up Prometheus, Grafana, OpenTelemetry and what else ourselves. Sometimes we need to argue that Auth0 (or another auth solution) is better than to take an existing open source solution and customize it ourselves if auth is just not our core competency (hint: it usually isn’t).

It’s really difficult. I know it myself. More than once I have designed full solutions in my head before realizing that I had already gone too far because the component in question was a clear buy candidate. I’m an engineer. I love to build difficult and fun stuff. I love to learn a lot along the way, and I love to later talk or write about the awesome things I have created. But it’s also my job to try to stay calm and assess a situation professionally. This includes sometimes having to speak against adding something to our stack, which clearly doesn’t belong to our core competencies.

At some point, however, you work at a scale where ready-made solutions simply don’t work anymore. I have, for example, made some “fun experiences” with MongoDB clusters on Atlas costing nearly a million dollars a year because our scale had just become too big. At this point, it was clearly time to expand our competencies, even if that meant having to hire people, train them, and deal with running a MongoDB cluster ourselves. It just became cheaper to do it ourselves than spending so much on a cloud service.

I also currently gather experience building a fully-fledged GraphQL Edge CDN. We don’t do this because we love to deal with global availability, low-latency responses, the complexity of dealing with cache entity tagging, or high-resiliency. We do it because other existing solutions would cost us thrice as much a month as we spend on our whole AWS bill a year (it’s definitely more than seven figures). So, we simply have no other choice.

The examples above also show one of the most important things we, as engineers, have to learn: We need to learn how to correctly make make-or-buy decisions. And we need to learn how to take every nuance into account. Well, technically, not only we, but also our managers because we usually don’t get many insights into business KPIs like cost of development, and more.

How to compare apples to apples

If you want to make really informed decisions, you need to take into account as much as possible. You can’t just compare apples to oranges. You need to get it right.

As already stated, you always need to think about whether you deal with something that is a part of your core competencies first. If not, you should think thrice about whether you really want to build a solution yourself. If you still come to the conclusion that you should at least find out whether making or buying comes cheaper, you will have to do some math. This, however, needs a clear process and is not easy, either.

To give you a general idea of how to come up with numbers you can compare, here is the process I usually use (but keep in mind that this might vary depending on how or where you work):

1. Estimate how much time it takes you to build your own solution (I usually estimate hours)

2. Add 20% on top because engineers tend to underestimate complexity and usually can’t reliably foresee complexities arising during implementation

3. Multiply your final estimation with your calculated cost per developer (per hour)

4. Estimate how much time it takes to maintain the solution for a longer time (I tend to let my team estimate how many hours on average they think they will spend per week)

5. Once again, add 20% on top

6. Multiply this estimation with your calculated cost per developer, again

7. Don’t forget the cost it takes you to potentially hire new people or train existing staff

8. Calculate how much any outage due to your solution failing will cost you (This one is the most difficult)

   1. Expect self-built solutions to fail at least once

   2. Expect to lose users due to this and calculate how much that churn will cost you

   3. Take into account that it also costs you money when you or other engineers need to solve the problem while not working on other features (which in return costs you users and money once again)

9. Sum everything up

10. Take what a ready-made solution will cost you in the same timeframe that you need to build your own solution (if it takes you two years, you need that ready-made solution for at least two years)

11. Add 1-5% on top because even a bought solution will fail at some point and probably cost you a few users (it’s just not under your control to fix it but at least you pay someone to interrupt their well-deserved night and work until the issue gets fixed)

12. Sum the previous two points up

13. Compare the two sums

Don’t worry. If you are just an individual contributor, someone from the business side will have to help you. In this case, you can’t know how much money one day of an engineer’s time costs the company. You also can’t know how much churning users cost the business. If you are an entrepreneur, though, you will need these numbers at hand. If not, try to get them as soon as possible.

In the end, you end up with a pretty neutral number. It either tells you that building a solution yourself is cheaper, or the opposite, buying something, has less cost associated with it. The only thing left is actually being strong enough to make a decision and stick to it for the foreseeable future.

But that’s it for you. If you have come this far in the process, you have made a very professional decision (or at least contributed to it). It might not bring you joy because you are missing out on an incredible opportunity to do something new and/or crazy, but it will help you keep your sanity because everything you build fails at some point. The more complex the solution that fails, the more “fun” for engineers to fix it ASAP. Stated differently: The less you do yourself, the less code you need to maintain. The less code you need to maintain, the less debt you have. The less debt you have, the more peace of mind you gain.

See? It’s not that bad!

You have (finally?) come to the end of this issue, so let me tell you something:

Thank you for reading this issue!

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver

Still, these interviews exist, and being somehow prepared for them doesn’t hurt. Sometimes, it’s the company of your dreams that still uses such processes. In these cases, just applying somewhere else is no choice.

To make your life as easy as possible, and also give you a process that you can use outside of interviews (to design systems yourself), I want to share my personal 7-step process to design any system with you. It’s the process I still use when confronted with such interviews, and it’s the same process I use when I have to interview applicants at the senior staff level and upwards (at some levels, you want to make sure an applicant knows their craft), and design systems myself (of course).

Ready? Let’s get into it.

Step 1 - Clarify The Requirements

Ask questions and a lot of them.

• Try to find out what exactly the interviewer expects from you.

• Try to narrow it down as far as possible to the exact scope of the problem. Even if the question is to (re)design an existing system, ask.

The amount of questions you could ask is nearly endless, but here are some ideas for you:

• Are there multiple frontends (App + Website + more)?

• Are there regulatory requirements?

• Do we need our own auth, or can we leverage an existing one (like login with Facebook/Google, etc.)?

• Are there other consumers than the known clients planned for our APIs?

• Do we need third-party integrations?

• Do we want to incorporate a data lake for analysis?

• Any requirements on data consistency?

Step 2 - Define System Interfaces

Define all APIs that the system will (probably) need. Explain what each API is for as detailed as possible so the interviewer can jump in and tell you if you got a requirement wrong.

If you got something wrong, no problem, adjust accordingly!

To give you a general idea, take a platform like Instagram for example which needs APIs to:

• Upload and view images,

• Retrieve the feed

• Store and retrieve comments for an image

• Store and retrieve likes

• Manage your profile

• Manage people's followers

• Store and retrieve stories

• etc.

As you see, existing systems are already pretty large, so it makes sense to ask your interviewer if you should really lay out everything you could possibly think of, or if you should only focus on a part of the existing system.

Illustrating System Interfaces

You can, for example, start with a use case diagram, which is a great way to identify and showcase possibly needed APIs.

Step 3 - Estimate The Scale Of The System

Give estimates to the scale of the system and communicate them clearly. Once again, if you got a requirement wrong, the interviewer can tell you, and you can adjust. If not, the interviewer can better understand your design decisions.

You don't need to give an exact estimate here, simply throw a number in the room which you find reasonable. In the case of well-known platforms, inform yourself prior to the interview. The numbers for Instagram, Facebook, etc, are all out there on the internet.

And regarding numbers, we talk about daily / monthly active users being transformed to an estimate of how many API requests/s (or minute) your system should be able to handle

If you interview at an organization that has such a well-known platform, you can impress your interviewer if you have the numbers at hand. This shows that you are really interested in the company and have made your homework.

You don't have to memorize those numbers exactly, it's not even about 1 or 2 million more or less, but you shouldn't be off by hundreds of millions. Being off by such a large amount could lead to design decisions you would not have made otherwise.

Step 4 - Define The Data Model

Define the data model of the system, and define how data flows. Ask how detailed this step should be, and if your interviewer requires it at all.

If it's required, also explain and maybe visualize how data flows between the components of the system.

Leverage relational models where applicable, but don't forget that there are other types out there, as well, like documents, graphs, and many more. Try to use the most appropriate ones for the entity at hand. It might, for example, make sense to leverage graphs for friendship-like relationships or for follower relations. If you find such a graph model fitting, use and model the entity accordingly.

Don't forget to lay out how the entities move through the system, like for example: The feed service fetches a user from the user service. The user entity thus travels from the user service to the feed service, and maybe even further.

You don't have to come up with the exact solution that the engineers of an existing platform came up with. That's not what an interview is about. Simply add the properties you may find fitting, and which are just enough to fulfill the requirements.

Your users, for example, will most likely need a forename, a name, an ID, an email address, and so on.

Illustrating Entities And Data Flow

You can use a UML class diagram, draw a box with properties in it, or simply list the properties of an entity to describe the data.

To illustrate the data flow, you can use simple illustrations to showcase how the data flows for certain operations.

Step 5 - Draw A High-Level Design

Most systems only have a few very important components that make up the core of the system. Try to identify them and draw boxes for the five most important ones, and visualize their interactions. Especially in modern micro-service environments, services communicate with each other (a lot!).

If service A needs some data that is managed by service B, you already have two boxes and one arrow to draw to illustrate that communication.

Remember: If you have already modeled the data flow of certain entities, reuse this information here.

Illustrating The High-Level Design

Once again, a simple illustration with some boxes and some arrows, including descriptions, is enough to showcase the design you have in mind.

Step 6 - Design Components In Detail

Explicitly ask if you should focus on specific components for this step. If not, choose the three most important ones and visualize/explain how they should work and what they should do in detail. A good interviewer will guide you.

Take an image service, for example, that is one of the core components of an Instagram-like system. It offers an API to retrieve images, for example, which is handled by an internal implementation. To store images, it needs access to some kind of storage, which is handled by another internal implementation. Lay all that out, draw diagrams to illustrate your ideas, and help the interviewer understand what you are designing.

Illustrating Detailed Components

You could try to resemble a UML component diagram to illustrate a detailed view of a service, for example.

Step 7 - Explicitly State Bottlenecks

Every system and every design has its limits. If there are bottlenecks in your system (and there will always be), state them explicitly. Give detailed explanations about what affects them, and how you could mitigate them.

This step requires the most knowledge about the components you used to design your system. You'll have to know, for example, when relational databases fall behind NoSQL storages noticeably, or in which situations certain message brokers are not the best choice anymore.

Maybe a relational model was good for the anticipated scale of the system, but migration becomes a pain at some point? You can be pretty creative in this step, but once again, you need to know the advantages and disadvantages of components and also the design principles you built your system upon. You don't have to be able to recall each and every disadvantage, but be able to state the most important ones.

Sometimes interviewers will also jump in and give you hints or state a bottleneck they found explicitly. That's your chance to discuss it with the interviewer and find a solution together.

Those 7 steps listed above can be a guide for you to get through a system design interview. They will support you to design a system in a structured way, but they won't obviously help you to ace the interview automatically. That's up to your skills, creativity, and imagination.

From my very own experience, I never expected candidates to come up with a solution that you could implement right away and that would work flawlessly afterward. For me, the system design interview has always been more of a creative and structured brainstorming session, in which I get to know the candidate's ability to design systems and recall important traits of certain components better.

What I usually found pretty helpful for me and all candidates, though, was exactly that structured approach. Starting from the very beginning, slowly going deeper and more in-detail. This way candidates can structure their thoughts better and I am able to follow along better, which is a win-win situation.

You have (finally?) come to the end of this issue, so let me tell you something:

Thank you for reading this issue!

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver

You either need your frontend to talk to your backend (that’s basically most products out there), or you want other people’s frontends or systems to talk to your product (Stripe is a prime example of such a product).

Creating APIs, however, can turn into both an art form and an organizational nightmare.

Here is why:

1. Good APIs need to be well-crafted. You can’t just create some REST endpoints without much design and call it a day. You need to put some real effort into them to make them usable and scalable.

2. Dedicated API teams usually create a middleware-layer between frontends and backends, which basically adds a whole additional iteration to your feature cycle. Instead of two iterations (first the backend, then the frontend), you now need three iterations until a feature of your product can make an impact for customers.

Companies like Netflix, PayPal, and many more, however, seem to have finally found a way to decrease the organizational burden of creating their APIs.

It’s the same technology I work with daily, nearly at the same scale as these companies, and in this issue, we are going to take a look at what exactly these companies do, and what we can learn from them. But first, we will take a look at common challenges when creating APIs before we dive into how you can solve most of them.

Setting the foundation

Before diving deeper, we should first set a foundational layer of understanding. In this case, you need to understand what type of APIs we talk about here, and which ones we don’t.

In this particular case, we talk about “THE” API between frontends and backends, so basically the entry point to any system.

The backend itself usually contains many more APIs on a micro-service-level, with many different protocols used, and a lot of inter-service-communication.

To get a better idea of what we talk about, take a look at the following image:

It’s exactly as stated. A more or less public API at the front of the system that connects backend and clients, and in this issue, we will focus on exactly this public-facing API.

Stated as simply as possible: We talk about the public-facing remote API of a system.

The issues with APIs

Creating APIs comes at a cost. To be even more precise, there are several issues you face and hurdles you have to take to get a good API up and running.

All companies building large-scale systems face these issues at some point, and we will now take a look at what these are.

1. Designing APIs is difficult

There are so many things you need to think about when you want to create a really good API, it’s mind-blowing. You need to think about

• Usability

• Scalability

• Security

• Maintainability

to only name a few. And even if you are done thinking about these, you still need to implement your API, which comes with its own problems once again.

Stripe is a great example of a company that puts a lot of work into its API (but that is no wonder because this API is Stripe’s product). Their API just “feels good”, and it’s easy enough to use, but that is also no wonder because Stripe’s engineers put a lot of work into creating it.

Stripe’s engineers go through many feedback loops until a new API endpoint finally gets released into the wild. This means that changes aren’t that easy or fast to make. Instead, doing it slowly and carefully is king in this particular scenario. That’s nothing for every company out there. Some of them want to iterate fast (or even faster), most probably also without compromising on API quality.

2. Creating APIs is even more difficult

When creating an API, you usually also face organizational issues. Who designs the API? Who implements it? Who runs it? Who maintains it? (You get the point)

In many cases and for many companies, the public-facing API is created by a dedicated team, which creates a new layer of responsibility, and a few more issues.

The API team gets its requirements either from the business-side (product managers, e.g.) or client teams (who want to implement features in their clients, which they need new or additional data for). It basically looks like shown in the following image:

Every new feature you want to introduce needs to run through these three layers, usually in the following order:

1. The backend needs to add new data or new endpoints

2. The API team needs to incorporate the new data or endpoints into the public-facing API

3. The client teams need to use that data to create their features

You could now argue that all teams can still work on a feature concurrently, but there is still some delay between getting a basic idea of how to implement something and providing mock data and/or mock endpoints for other teams to use. Nevertheless, the feature is only finished when all teams are finished.

On the other hand (and to be fair), having a dedicated API team also has one crucial advantage: You have exactly one team that needs to know how to create great public-facing APIs. That’s a fact you also should not underestimate.

3. Increased mobile usage creates even more requirements for APIs

Nowadays, the world is mobile, and mobile-usage is only going to increase in the future. Everyone I know has at least some form of smartphone, and they use it regularly. Chances are thus high that most users of any product are going to use exactly that product from a mobile device sooner or later.

We don’t have mobile-first web design for no reason. Love it or hate it, but you will most probably have to deal with mobile users if you don’t want to exclude quite a few potential customers from your product. Most products even have a mobile app for exactly that reason (because people somehow like “native” apps more than mobile websites).

Mobile websites and apps have an issue, though: They often need data from your backend, and mobile coverage isn’t always the best. Additionally, not all countries have massive amounts of data for a reasonable price, which puts additional constraints on your design. You can’t let your users fetch megabytes of data only to display a small portion of it. This over-fetching is a common issue of REST APIs.

Lastly, the times are over where you could get away with only a web client and one mobile client for iOS and Android each. Depending on what you build, you could even need somewhere around twenty or more different clients.

Believe it or not, but Netflix, for example, really has to deal with twenty or more different clients. As a streaming service, they not only live on the web and on mobile devices, but they also need to provide content on Smart TVs (Especially Smart TVs are very fragmented. Every manufacturer has their own OS with different constraints), Google Chromecast, Amazon FireTV Stick, Android TV, and more (I know what I’m talking about because I also work for a pretty large streaming service provider).

It’s difficult to build a single API that serves so many clients. Clients that all potentially have slightly different requirements of data or ways to get that data. A general-purpose API quickly becomes too inflexible.

If general-purpose APIs don’t do the job, what then?

To serve so many different clients with slightly differing requirements you would usually have to build the ultimate general purpose API. But that’s nearly impossible.

Your API would probably consist of many different routes/endpoints and often duplicate logic, only changed slightly, so all clients get the data and functionalities they need. This quickly gets out of hand.

That problem isn’t new, though, because Sam Newman already wrote about the so-called Backend-for-Frontend (BFF) pattern in 2015. (To be honest, this pattern had already been applied prior to Newman’s article, but he was the first one to talk about it publicly.)

The general idea goes as follows:

Every client with its special needs gets its own API, called Backend-for-Frontend, which basically looks like this:

If your iOS app wants to fetch data differently than your web app, you only need to add a special endpoint in the iOS BFF. The requirements of your iOS app don’t affect your web app because it’s a totally different API that needs to be adjusted.

This solves one of the many issues: Delivering exactly the data a client needs because all APIs are tailored to the needs of that specific client.

Instead of creating different endpoints for specific clients in a general-purpose API, you assign dedicated teams to each BFF (or even let full-stack engineers build both the client and their own API). This increases the flexibility of your API(s) and additionally eliminates (or at least lessens the impact of) the bottleneck a single API potentially creates.

(The pattern works well, as Soundcloud, for example, has successfully proven in the past.)

At that time, other companies also had issues with different clients and varying requirements for their APIs. To tackle the problem with traditional REST APIs and mobile apps, Lee Bryon, Dan Schafer and Nick Schrock invented GraphQL internally at Facebook in 2012 (they got the idea when working on a redesign of their iOS app). In 2015 Facebook open-sourced GraphQL, and since then, GraphQL has seen massive adoption throughout the industry.

GraphQL solves an issue Backends-for-Frontends create: Instead of having to provide multiple APIs for different clients, you can shrink your API layer down to one again. At the same time, each client can fetch exactly the data it needs. So you can basically hit two birds with one stone. Not bad, isn’t it? It doesn’t matter whether your GraphQL entities have 50 or more fields. Clients only fetch what they really need.

But GraphQL comes at some of the costs of a general-purpose API (or let’s better say it reintroduces it): You are once again down to one (or less common two or more) teams that work on that single API. Chances are high that the resources you allocate to your GraphQL API are still not enough to serve the needs of all stakeholders at the same time. You have reintroduced a bottleneck.

The architectural pattern allowing Netflix, PayPal and other big tech companies to evolve their APIs into something far better

If a general-purpose API is too inflexible and creates a feature bottleneck, BFFs take too much work, and GraphQL itself reintroduces the feature bottleneck, what then?

Time to finally take a look at how big tech companies evolve their APIs and solve 95% of problems that all other solutions we have previously taken a look at create or cannot eliminate.

The answer is an architectural model and protocol addition to GraphQL itself, called GraphQL Federation.

GraphQL Federation is also known by the name Apollo Federation, which includes the name of the company that develops the architectural model (partially together with Netflix) and offers commercial and open-source software based on it, Apollo GraphQL.

GraphQL Federation brings something to the table that completely eliminates the need for a middleman: It allows engineers to directly attach backend services to the API, automatically, without the need for a team that does the grunt work of connecting clients and backends.

Instead of creating micro-services that provide RESTful APIs, or leverage gRPC, Thrift, or whatever else seems fitting, teams can directly implement their APIs using GraphQL. All changes to their own APIs are reflected on the public API as soon as that team updates their public schema. This completely eliminates the middleman.

Conceptually, Federation consists of three parts:

1. Well-designed additions to the GraphQL spec in the form of specific directives (directives are like annotations in Java or decorators in TypeScript).

2. A schema registry where teams of individual services upload their GraphQL schema to and clients, as well as a central component, can fetch an aggregated single schema from.

3. A central component, called Router, that dismantles queries, creates a query plan, and calls multiple backend services (called subgraphs) under the hood. All individual responses by subgraphs are then reassembled into a single response.

But let’s take a closer look at what these are and how they work.

1. Federation-Specific Directives

Conceptually, nothing changes when working with Apollo Federation. A graph always consists of a well-defined set of types, queries, mutations and (recently added) subscriptions.

A type is like an entity, some form of data your graph offers. It consists of simple fields or sub queries, which are like remote methods that any client can call.

This is an example of a simple entity that models a blog post:

type BlogPost {
  id: ID!
  title: String!
  description: String
  author: String!
  publishedAt: DateTime!
  lastUpdated: DateTime
}

And this is an example of a corresponding query:

type Query {
  # fetches a single blog post by its id
  blogPost(id: ID): BlogPost
}

In a federated graph, entities can be accessed between graphs. In the above case, a blog post could be provided by a blog post service. Blog posts usually have comments, and these might be handled by a completely different team, with their own micro-service and their own data storage.

In this case, the schema for a comment could look like this:

type Comment {
  id: ID!
  author: String!
  text: String!
}

In a “classical” GraphQL API, the API service would implement the schema for all types and fetch the corresponding data from different services if certain fields are requested. The comment schema above would move into the original schema and be provided by the one single API service. In a federated API, however, things are a little different, and this is where federation-specific directives come into play.

Each subgraph (every service attached to the federated graph with its own schema) can mark certain entities as “accessible by other graphs”. It does so by adding the key directive to every type it wants to make accessible by others.

In the scenario from above, the comment schema would look as follows:

type Comment @key(fields: "id") {
  id: ID!
  author: String!
  text: String!
}

The only addition is the key directive next to the name of the type. It just makes one statement: “You can uniquely identify this type by one field, namely id.” This makes the Comment type a federated entity.

If the blog post service now wants to include a Comment in its schema, it can do so by simply referencing it as follows:

type BlogPost {
  id: ID!
  title: String!
  description: String
  author: String!
  publishedAt: DateTime!
  lastUpdated: DateTime
  comments: [Comment!]
} 

This is everything. The BlogPost itself needs no key directive if no one else must be able to include it in their types, and it can simply include the Comment type within its own schema. The only thing the providing service of a federated entity needs to do, is to implement a special method that allows the federation infrastructure to fetch one or multiple entities by the fields specified within their key directive. Everything else is handled by the rest of the federated GraphQL infrastructure.

This also eliminates a primary need of inter-service communication in a system. The communication is implicitly done by the Router in the front. Services themselves don’t necessarily need to talk to each other if they are connected through a federated graph. If a client fetches multiple entities that are connected through the schema, the Router in front of the system will issue the corresponding requests to fetch all necessary data from multiple services.

Theoretically, other subgraphs can also extend existing entities with single or multiple fields, without providing new types themselves, and there are also ways to make certain fields resolvable by multiple subgraphs, or override them. These are lesser-used directives, however, so looking at the key directive specifically shall suffice to give you a general idea.

2. Schema Registry

The schema registry is a central part of a system that leverages a federated graph.

Every subgraph (remember, a micro-service that offers its own partial GraphQL endpoint) uploads its schema to the registry. The registry then takes all available schemas and wires together two new schemas:

1. An API schema

2. A supergraph schema

The API schema is the public schema that all clients use to communicate with the GraphQL API itself. This schema doesn’t contain federated directives and only provides what is really necessary for clients to generate types and clients from. It contains all types, interfaces, queries, mutations, and so on from all subgraphs.

The supergraph schema is a special schema that has the same contents as the API schema, but it additionally contains federated directives and special control directives, which are used by the Router to create query plans with. This schema is usually not public.

3. Router

The Router is the entry point to a federated graph. All clients send their GraphQL requests to this component, which then uses the supergraph schema to create a dedicated query plan.

First, a large query is split into multiple different queries (if necessary). Then the resulting subqueries are sent to the respective subgraphs. After that, all individual responses are stitched back together into a single, unified response, which is then sent back to the client.

The Router also takes care of handling errors that inevitably happen in distributed systems. Sometimes, errors from subqueries can be aggregated into the final response, and sometimes if the schema doesn’t allow for it (non-null values missing due to errors, for example), the whole request needs to result in an error.

Architecturally, you can imagine the Router being placed in a system as follows:

How Federated GraphQL helps Netflix, PayPal, and many more with creating APIs at scale

Now that you have a general idea of how GraphQL/Apollo Federation works, it’s time to take a look at what exactly it offers that makes it such a valid alternative to other solution that big tech companies go all-in on it.

First of all, it’s still GraphQL. GraphQL’s huge advantage over REST is that any client can exactly specify and fetch which data it wants and really needs. Backend or API teams don’t need to deal with any of this. They can simply provide data they have available and even extend their types on-the-fly without breaking anything, decreasing the performance, or even increasing the network overhead. No more over-fetching.

Second, it eliminates the bottleneck a centrally maintained API creates. Features don’t need to be worked on by additional API teams. As soon as a backend service or multiple services have implemented a new feature, clients can begin fetching the new data or using the new remote procedures. Changes and fixes can be reflected way faster. This is a massive speed increase.

Third, it’s still GraphQL (yes, again), and this comes with human-readable schemas that are way faster to scan and understand than OpenAPI specs for RESTful APIs. That’s a huge advantage of GraphQL schemas. They were designed with humans in mind.

Fourth, by spreading the API all over the backend, every engineer working on a backend service integrated into the federated API also becomes an API engineer. It may not sound like much, but frontend and backend engineers have always had very differing views about usable APIs. More often than not, backend engineers simply expose their database schema as their API schema and call it a day. In 99% of all cases, this results in a very poorly usable API. Frontend teams often imagine different, more usable APIs, with better user/developer experience. In a federated GraphQL API, backend engineers can be held responsible and can’t hide behind the fact that someone still has to build a facade (the public-facing API) and call their API under the hood.

Now that you know what GraphQL Federation offers, it’s time to take a look at some case studies to understand how two selected companies use GraphQL at scale; with hundreds or thousands of services and billions of requests a day.

The story of Netflix and GraphQL Federation

As a GraphQL Federation user (and contributor) of the first hour, Netflix had already used GraphQL very early. But interestingly, Netflix began using GraphQL Federation in a not-so-public system first: Its Studio API.

Netflix has been producing original content for quite a while, and a large portion of Netflix’ staff works on these original productions. A large chunk of Netflix engineers also work on these systems that help Netflix staff manage these productions, from the very first creative pitch to post-production and the final release of a movie or series.

All these workflows are handled by the Studio API, which allows custom clients to be built on top of it. Thanks to the flexibility of GraphQL, these clients range from mobile clients for creative scouts, to web or desktop apps for Netflix Studio employees doing their work in the office.

The Studio Graph consists of at least three services or “domain graph services” (as Netflix calls them), which all model a partial but important part of the overall domain.

Next to its Studio API, Netflix also started working on migrating its mobile clients from its API framework Falcor over to Federated GraphQL in 2022.

Prior to this, the API was maintained by a single team that used the Java Version of Falcor to implement the mobile API. But this created exactly the issues you have already read about. The API team always had to manage communication between both involved parties (frontend and backend), and additionally had to catch up on domain knowledge to understand the needs and desires of its stakeholders.

After realizing the potential gains of GraphQL over their custom implementation Falcor and the added productivity gained by directly connecting backend to frontend engineers, Netflix undertook a very interesting migration (which is a story too long to cover here, but feel free to read about it, it’s worth it) with zero downtime and a lot of A/B Testing, Replay Testing and Sticky Canaries.

Since June, 2023, all of Netflix’ mobile traffic is served through a single Federated GraphQL API, with globally distributed instances of Netflix’ custom implementation of a Router. We don’t know exactly how many subgraphs the whole graph contains, but we can safely assume that it’s a few hundreds at least.

Next to this, Netflix invests heavily into GraphQL by building its own GraphQL infrastructure. Engineers are working on at least a custom Router, a custom Registry, and have also open-sourced their personal Spring-based DGS framework.

The story of PayPal and GraphQL Federation

PayPal’s journey into GraphQL started in 2017, when they faced a growing problem: Their Checkout was powered by a RESTful API, or better, multiple RESTful APIs: Backends-for-Frontends. Core teams built the system in the backend, creating APIs with different protocols, not caring much about how that data would be used. Multiple other teams would then build BFFs for specific clients or products.

A very crucial issue PayPal’s engineers noticed at that time was a lot of repeated low-value boilerplate code in all BFFs. Data had to be fetched, filtered, mapped, and sorted over and over again. This used up valuable time that engineers could have used to do what they were actually meant to do: Building (UI-) features.

Initially, PayPal’s Web Platform team built a monolith GraphQL API, as a proof of concept. With a lot of success and many issues solved, the team soon realized that building an API themselves would not suffice forever. This is when they stumbled upon GraphQL Federation and started to champion the architectural pattern and technology internally.

Soon after, PayPal started migrating over to GraphQL Federation and has added more and more services to its Graph throughout the years. We have no explicit data on how many subgraphs the PayPal graph contains, but we can approximately assume that it must be hundreds right now.

PayPal’s engineers have since been quoted praising federation and its added benefits, as it increases development speed of new features and frees capacity to work on more important things than dedicated APIs.

As far as we know, PayPal has made GraphQL its default choice for all new UI apps. With over 50 apps and products already connected to the graph, this number will only grow. Additionally, some of its core services, Identity, Payments, and Compliance have fully migrated to GraphQL. Additionally, PayPal even offers a public GraphQL API to interact with its services.

From some of PayPal’s blog post, we know that PayPal has no dedicated GraphQL infrastructure team. They leverage Apollo’s platform and use the components offered by its product GraphOS.

You have (finally?) come to the end of this issue, so let me tell you something:

Thank you for reading this issue!

Although this issue was incredibly large, I still hope you found it useful.

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver

Subscribe now

I can still remember myself, a few years younger, searching the internet every day, just to find out what cool new tech there was out in the world.

My reasoning behind this was actually pretty simple:

The fresher and crazier the new tech I use, the better my chances must be to get a high-paying job. The better I keep up with the latest tech, the better my overall chances.

- A younger naive me

But guess what? It didn’t work. It never worked this way.

I know for sure that many aspiring and experienced developers out there still try to chase the latest tech, worrying about their careers or trying to take the next step.

If you’re one of them, stay with me. Let me try to show you what I learned over the course of my (already very long) career, and what helped me calm down and focus on the only things that really matter.

The problem with FOMO

The biological problems

FOMO, the fear of missing out, puts pressure on you (when experienced for a longer period of time).

Sadly, most of us humans are pretty bad at handling pressure. We are not made to sustain a certain amount of pressure over a longer period of time.

Biologically, we were made to be relatively calm beings. Our genetics are still made for us to be hunters and gatherers. We are made to walk for miles and search for something edible. We are made for short intense physical activities like hunting or running away from danger.

This is also why adrenaline exists, the hormone that (in theory) should help us. If you’re put under pressure, your body releases adrenaline as a way to improve your senses and help you over a short period of time. That adrenaline rush then allows you to hunt down your next meal, or run away from a danger that could otherwise kill you.

Adrenaline rushes are usually triggered by one of the following:

1. Fear

2. Excitement

3. Anxiety

4. Stress

Three of the four triggers of an adrenaline rush already have to do with FOMO. The fear of missing out. Your fear can lead to stress, that stress can lead to anxiety.

The result?

• Jitters and irritability

• Trouble falling or staying asleep

• Frequent headaches

• Difficulties with concentration and memory

• Your muscles constantly feel tense or sore

• Unexplained weight loss

I think we don’t need to argue about the biological implications on your health. They are bad, and you probably want to avoid them like hell. You even should avoid them like hell.

The professional problems

Professionally, there are also quite a few issues with FOMO.

Always running after the latest tech robs you of crucial experience. You usually don’t need more experience with a specific tech stack if you suffer from FOMO. What you actually need is more general experience solving problems.

Changing “your” tech stack regularly, in the beginning, resets your whole learning experience each time. Instead of continuing to learn new concepts and encounter new problems, you confront yourself with a new syntax, new APIs, and new technology-specific problems.

It’s basically like trying to take the steps up somewhere but deciding to take another stairway mid-way by basically jumping down to the bottom again. You never reach the top this way.

In the end, you end up with mediocre experience in a few technologies, without the traits that are usually expected from a more experienced developer.

Next to that, I have already seen quite a few CVs of developers who were definitely suffering from FOMO. Their skills section contained more bullet points and was longer than the section with their previous projects and prior experience.

More often than not, engineers who suffer from FOMO, are only mediocre developers because they have never dived deep enough. They just stopped at a certain level and decided to reset everything because a specific technology was more important to them than all the fundamentals you can learn in (really) any programming language and with any library.

In the end, FOMO kills an engineer’s career because the reality is:

You only advance in your career if you become a better engineer, not a better “user of technology xyz”.

But, what then?

By now, you have probably realized that running after the latest technologies is not the way to get further in your career. But what should you do then? What does really matter?

First of all, calm down. There is no reason for FOMO.

Yes, there are companies out there that solely hire based on experience in very specific programming languages, technologies, and frameworks, but the ones that really matter hire you for your experience as a software engineer.

Google, for example, only hires Software Engineers. You are hired based on your experience and then assigned a team that uses specific languages and tools.

The reasoning behind it?

Software engineers are not defined by the technologies they use. These tools are just that: tools to get a specific job done, with engineering principles.

This is exactly the way it should be. A company usually has one or a few problems they want to solve with software. These problems somehow need to be solved. The solutions need to be usable for the users they are meant for.

All that needs to be done in an economically sustainable way, and the cost must not exceed the budget. Otherwise, the project is either canceled because there is no money left, or an automated solution to a problem becomes more expensive than the prior manual solution.

And how do you do that?

By putting a lot of work into designing a solution and deciding on technologies to use.

The most optimal way, in this case, is deciding what to use based on the key data you have available. What does your budget allow for? What problems need to be solved? Which technologies are capable of solving these issues? And so on.

And, what talent do you need if you really want to do it this way?

Well, you need engineers who have already seen a lot and who are capable of getting into some new technology as fast as possible while still being able to leverage their existing experience.

That experience usually has more to do with the problems themselves, like:

1. How do you efficiently store data?

2. How can you decrease the latency as far as possible?

3. How do you get as much UX as possible into your solution (on the tech side of things)?

4. How can you work with petabytes of data efficiently?

5. etc.

This experience is not necessarily gained with different technologies.

Second, get your priorities straight

You need to ask yourself a few questions first:

1. Where do I want to end up?

2. What do I really enjoy?

3. What am I ready to do for it?

Number 1 sets your goals straight. What is your end goal? How far do you actually have to go?

Number 2 ensures you keep your sanity. It makes no sense to learn or work with something you do not enjoy using at all.

And number 3 ensures that you are aware of what you are ready to sacrifice or put into it.

If you put all these answers into a single sentence, it will usually look something like this:

“I want to become a skilled software engineer who enjoys solving problems in [the (frontend|backend|fullstack|mobile|gaming) space | xyz space) (regex is everywhere…), and I am ready to allocate n hours a day to get there.”

Great, isn’t it? A single sentence that defines all of your professional life for the foreseeable future.

But don’t worry, it’s only a mantra. What matters even more is what follows now.

Wherever you want to end up, whatever you enjoy, and whatever you are ready to sacrifice, you won’t get there the way you imagined.

There is one, simple, way to gain all the experience you need, and that is the following:

1. Pick technologies you like and enjoy using

2. Build as many projects as you like (although the more the better)

3. Ensure you face many different problems

4. Repeat for as long as you enjoy doing it

That. Is. All.

There is usually a way to do everything with as few technologies as possible, without you having to always switch around because you could potentially miss something.

You can do everything with JavaScript and Mongo, you can also build fullstack apps with Python (Hey, Django) and Java (GWT anyone?), or build frontends with Go, and you can even potentially do data science and machine learning in any language. You can use vectors with Postgres, or perform ML with it, and even Redis is a suitable main database nowadays.

This is the crucial thing. Your priority is becoming a competent software engineer, not a competent JavaScript/Python/Whatever developer. A loop is still a loop, a network still works the same way no matter which language you use, and the fundamentals of databases still remain the same, no matter which one you use (a document DB is a document DB, a relational database a relational one, etc.).

You won’t lose much by going down this route.

Yes, there will be companies that won’t hire you as a frontend developer because they expect two years of experience with React instead of Vue. There will be companies that will reject you because you know Java instead of Go.

This isn’t even bad, because a company that is this short-sighted usually also isn’t a very good company to work for.

More than enough companies have a hiring process that does not look at specific technologies but instead puts an emphasis on your experience in a specific area. For these companies, experience in Vue is as good as in React, because they both follow very similar principles, and experience in Go is as good as in Java.

Third and lastly, do everything to enjoy what you are doing

Enjoying the journey that learning software engineering is is way more important than grinding through it and bringing yourself closer to burnout, only because you fear not being accepted the way you are (the reality is…most seniors still suffer from this, so you’re not alone).

If you’re at the beginning, you still have a few years to learn until you can call yourself a senior. If you’re already further down the line, congratulations, you still have years to come. Learning never stops for a software engineer. Really. Never.

Just view this journey as a marathon instead of a sprint, and enjoy some peace of mind. You deserve it.

If you have come this far, let me tell you something:

Thank you for reading this issue!

And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.

Do whatever makes you happy. In the end, that’s everything that counts.

See you next week!

- Oliver