The most difficult thing in tech is sticking to your core competencies
What you as an engineer can do to improve the situation
We are funny beings. I mean us as software engineers, because we love to build stuff. Give us any problem and we will probably get stuck for a few hours and wish we were never born a few times, but we will try to get through it, and we will come up with a solution. More often than not, we will build something ourselves, however, and we will be hellish proud of our accomplishments.
Interestingly, most companies love exactly this. We solve their problems for them, and we even discover a few more along the way and use our precious time to solve them, too. What more could you ask for?
Well, did you know that this is one of the biggest problems in tech? We simply don’t know when to stop. We get so lost in what we love to do that we stop asking ourselves whether we spend our time wisely at all. Only a few companies and managers have realized this, as well. And if they have, they will try to stop us.
If you are a little lost now and ask yourself what I am talking about, wait a second. Let me explain.
Software Engineering in the industry
There is a huge difference between software engineering and industrial software engineering. The first is what we do (or better, want to do), the second is software engineering within a business context, or stated differently: economical software engineering.
The reality is that not all companies have unlimited resources. To be even more precise: No company has unlimited resources. They all have a certain budget they are willing to allocate and another budget they can spend at most.
Money is finite. Every bank account of every company has a specific number inside it. If you go below that number, the company either goes bankrupt, or needs to lend money to continue to operate. If a company lends too much money, it definitely risks going bankrupt, and that is something most companies want to avoid, of course.
What has that to do with us? Well, we somehow like to get paid for what we do. Additionally, we love to spend money. We spin up a few EC2 instances here, create a few ElastiCache clusters there, add a CloudFront on top of it, and now we do the same on Azure or GCP because multi-cloud definitely increases our resiliency. That usually leaves your employer with a high bill, but at least it’s not you who has to pay it, right?
The scenario above is not necessarily bad. If going through your requirements results in a large cloud infrastructure on multiple cloud platforms, then be it. At least Jake from the business unit can grab your cloud engineer Sarah and go through your cloud dashboards and buy reserved instances for the next three years to save a lot of money and even get a few tax exemptions in some circumstances.
So, if that is not bad at all, do we even have a problem? Does this issue even make sense? Could you have spent your precious time more wisely than to actually read all the above, only to realize that I have fooled you?
Nope. There is still a problem. Believe me.
What we haven’t talked about yet is the countless hours we pour into building our CI/CD pipelines to deploy all of our software to our two cloud systems (by “we”, I mean us and the twenty other teams who do the same work). What we also haven’t talked about yet is all the work we put into setting up our own Kubernetes cluster and the dozens of operators we need. We add a service mesh, and definitely need to host a few databases ourselves, right within the cluster because neither AWS, nor Azure, or GCP have a native managed MongoDB (No, DocumentDB is a relational database with a MongoDB interface. Sorry.), or a Redis with more than its default functionality.
Oh, and after all that, we have decided to build our own authentication system, and implemented twelve Lambda functions and five micro-services, which all have nothing to do with our core business, but are still needed for our system to work.
And here comes the crux: There is no “reserved engineering capacity” at a discounted price. We get and deserve our salaries. No matter what happens. And this leads to the most difficult thing in tech.
Finding and sticking to our core competencies
Core competencies are exactly what differentiates businesses from each other. They are what a business makes money with. And they are what you should usually focus your engineering efforts on. Engineers cost money, and you don’t usually get discounts. Everything you build and need to maintain thus costs usually more than your cloud resources or SaaS' cost you because you always need to pay someone to take care of it for you.
If you build a fancy new AI app, your core competency probably lies somewhere in the problem space you are working in. If that is creating headshots of people with AI, it makes most sense to focus your efforts on building a great user experience and creating ML models that produce awesome headshots. It’s neither your core competency to create your own authentication system, nor is it to implement your own vector database.
In such a scenario, it’s essential to make a few make or buy decisions. The leaders of your company need to ask themselves the following questions for everything you need for your product to work:
Is this one of our core competencies?
How much does it cost us to build competence in a specific field of expertise and build and maintain the solution for the foreseeable future
How much does it cost to simply buy an existing solution?
This is where many companies and engineers fail.
Business leaders and budget owners see the cost of an SaaS solution priced at a few hundred thousand dollars a year and immediately begin to worry about their budget. What they don’t understand is that there is usually more cost associated with building something new than only paying for the time some engineers need to integrate an existing solution.
Engineers, on the other hand, would love to create a solution themselves because it’s an interesting problem space they want to work in or it’s something that gives them the most opportunity to learn. They don’t care about the business aspect (which is fine because it is not one of their core competencies), they care about the engineering (which is exactly their core competency).
The harsh truth, however, is that the most efficient way to deal with these issues is to try and make a perfectly neutral and objective decision. It’s exactly as stated above. If it’s not your core competency, try to buy it. Only if all solutions that fulfill your requirements are more expensive than building something yourself (with all true costs associated with it, like long-term maintenance, etc.), you should ask yourself whether it’s worth building a new competency from the ground up and pay all costs associated with it (including frequent failure and outages until your solution is production-ready and solid enough) or you should simply pay someone else to give you access to a solution they take care of.
If something is just too expensive and you also cannot build it yourself because that, too, would blow your budget, it’s probably time to try to come up with an alternative solution. Going back to the Headshot AI app scenario: Perhaps, you cannot build a full model yourself because you can’t afford the GPU time associated with training a model from scratch. In this case, you could come up with the idea to retrain an existing open source model. Maybe, you need some time to get good results, maybe you even never reach the perfect quality, but perhaps it’s just enough to get your first few thousand customers and bring in cash you can reinvest into developing your own model from scratch.
We, as engineers, still have to learn
Even if some of our business leaders are bad at finding and sticking to a company’s core competencies, we as engineers can do better. We are the ones whose job it is to come up with solutions, but it’s also our job to offer expert advice. This can and should also include advising against building certain things ourselves only because we want to. Sometimes, we need to argue to spend a few hundred thousand dollars a year on a (let’s just say) ready-made monitoring solution instead of setting up Prometheus, Grafana, OpenTelemetry and what else ourselves. Sometimes we need to argue that Auth0 (or another auth solution) is better than to take an existing open source solution and customize it ourselves if auth is just not our core competency (hint: it usually isn’t).
It’s really difficult. I know it myself. More than once I have designed full solutions in my head before realizing that I had already gone too far because the component in question was a clear buy candidate. I’m an engineer. I love to build difficult and fun stuff. I love to learn a lot along the way, and I love to later talk or write about the awesome things I have created. But it’s also my job to try to stay calm and assess a situation professionally. This includes sometimes having to speak against adding something to our stack, which clearly doesn’t belong to our core competencies.
At some point, however, you work at a scale where ready-made solutions simply don’t work anymore. I have, for example, made some “fun experiences” with MongoDB clusters on Atlas costing nearly a million dollars a year because our scale had just become too big. At this point, it was clearly time to expand our competencies, even if that meant having to hire people, train them, and deal with running a MongoDB cluster ourselves. It just became cheaper to do it ourselves than spending so much on a cloud service.
I also currently gather experience building a fully-fledged GraphQL Edge CDN. We don’t do this because we love to deal with global availability, low-latency responses, the complexity of dealing with cache entity tagging, or high-resiliency. We do it because other existing solutions would cost us thrice as much a month as we spend on our whole AWS bill a year (it’s definitely more than seven figures). So, we simply have no other choice.
The examples above also show one of the most important things we, as engineers, have to learn: We need to learn how to correctly make make-or-buy decisions. And we need to learn how to take every nuance into account. Well, technically, not only we, but also our managers because we usually don’t get many insights into business KPIs like cost of development, and more.
How to compare apples to apples
If you want to make really informed decisions, you need to take into account as much as possible. You can’t just compare apples to oranges. You need to get it right.
As already stated, you always need to think about whether you deal with something that is a part of your core competencies first. If not, you should think thrice about whether you really want to build a solution yourself. If you still come to the conclusion that you should at least find out whether making or buying comes cheaper, you will have to do some math. This, however, needs a clear process and is not easy, either.
To give you a general idea of how to come up with numbers you can compare, here is the process I usually use (but keep in mind that this might vary depending on how or where you work):
Estimate how much time it takes you to build your own solution (I usually estimate hours)
Add 20% on top because engineers tend to underestimate complexity and usually can’t reliably foresee complexities arising during implementation
Multiply your final estimation with your calculated cost per developer (per hour)
Estimate how much time it takes to maintain the solution for a longer time (I tend to let my team estimate how many hours on average they think they will spend per week)
Once again, add 20% on top
Multiply this estimation with your calculated cost per developer, again
Don’t forget the cost it takes you to potentially hire new people or train existing staff
Calculate how much any outage due to your solution failing will cost you (This one is the most difficult)
Expect self-built solutions to fail at least once
Expect to lose users due to this and calculate how much that churn will cost you
Take into account that it also costs you money when you or other engineers need to solve the problem while not working on other features (which in return costs you users and money once again)
Sum everything up
Take what a ready-made solution will cost you in the same timeframe that you need to build your own solution (if it takes you two years, you need that ready-made solution for at least two years)
Add 1-5% on top because even a bought solution will fail at some point and probably cost you a few users (it’s just not under your control to fix it but at least you pay someone to interrupt their well-deserved night and work until the issue gets fixed)
Sum the previous two points up
Compare the two sums
Don’t worry. If you are just an individual contributor, someone from the business side will have to help you. In this case, you can’t know how much money one day of an engineer’s time costs the company. You also can’t know how much churning users cost the business. If you are an entrepreneur, though, you will need these numbers at hand. If not, try to get them as soon as possible.
In the end, you end up with a pretty neutral number. It either tells you that building a solution yourself is cheaper, or the opposite, buying something, has less cost associated with it. The only thing left is actually being strong enough to make a decision and stick to it for the foreseeable future.
But that’s it for you. If you have come this far in the process, you have made a very professional decision (or at least contributed to it). It might not bring you joy because you are missing out on an incredible opportunity to do something new and/or crazy, but it will help you keep your sanity because everything you build fails at some point. The more complex the solution that fails, the more “fun” for engineers to fix it ASAP. Stated differently: The less you do yourself, the less code you need to maintain. The less code you need to maintain, the less debt you have. The less debt you have, the more peace of mind you gain.
See? It’s not that bad!
You have (finally?) come to the end of this issue, so let me tell you something:
Thank you for reading this issue!
And now? Enjoy your peace of mind. Take a break. Go on a walk. And if you feel like it, work on a few projects.
Do whatever makes you happy. In the end, that’s everything that counts.
See you next week!
- Oliver