You might be thinking that you want to build an internal developer platform on Kubernetes. If you are not already on this journey you will be in a few years. There might be 100 product managers and engineering managers in the world that find themselves reading this article. You are wondering how to navigate this journey for your organization. Your organization is asking you to help it not fall behind its competition. Software developers are leaving to work on newer technologies of your competitors. Your ossifying virtual machine platform and decrepit ticketing systems delay important work tasks. All your systems are fighting against you. You read wonderful stories of hard-working teams creating great products under tight deadlines.
Maybe you’ve already started building a new platform with a team and you’re looking for validation. It’s worth it, it’s a hard road, hang in there the light is at the end of the tunnel. Think again. Internal developer platforms are unlikely to be your market differentiator. Tread lightly here unless you happen to have constraints like mine. Are you in a regulated industry? Allergic to the public cloud? Own and enjoy operating your own data centers? Think long and hard if this is a journey you want to go on. There are options out there! If you’re thinking about a Kubernetes platform maybe the big public cloud providers are already not a fit for you? Or maybe they are. Software-as-a-Service is lovely compared to a long hard road for this journey.
Me? I have fortune on my side. Top software engineering and operational/administration talent? (did you think I was going to say DevOps?) check. Unparalleled company culture? check. Management support? check. Security buy-in? check. If this is your potential future I want you to walk into it with open eyes. I love hard challenges. If you at least consider avoiding unnecessary problems due to this article I will be satisfied.
I know what you’re thinking. The guy that is telling me that my internal developer platform sucks is telling me not to build it myself? Is he serious? Whoa, slow down don’t close this article yet let me walk you through what you will need to do if you still want to build your platform. First, check out these two books.
You may think you are investing in a platform. What you are truly doing is starting many journeys at once. You are asking people to change how they do their work and organize. Are you struggling with agile? Are you still operating with a waterfall process or general chaos? Kubernetes requires teams to use different tools that demand new habits and mindsets.
What you could expect when you build your internal developer platform
Are you prepared to:
- train your staff on these new technologies and how they’re different than what you offer today?
- negotiate with internal centralized infrastructure teams that might be opposed to your needs?
- develop many new vendor relationships with companies that may go out of business or be acquired in a few short years?
- wrestle with internal stakeholders that are not interested in quick wins and prefer to take big risks?
- screw up the first version of your offering and mitigate its shortcomings over a couple of years?
- risk trying to be everything to everyone and fail at creating a sustainable offering you can maintain?
- be surprised by some component that is incredibly expensive compared to everything else?
- choose some technology that is completely the wrong fit you are trying to solve?
- augment or improve your incident management practices now that software is more resilient and now breaks in fun new strange ways?
- discuss the merits of microservices versus service-oriented-architectures?
- make anemic nano services, make microservices that need other services up in a certain order first, make distributed monoliths?
- regret making all your services call each other using synchronous calls?
- design health checks that need all dependencies to be up constantly and when they are not severely ruining your day for no good reason?
- realize that eventing, as well as reusability, is hard to get right and you’re probably going to get this wrong?
- be told that some library is the solution to all your Kubernetes YAML problems and everything will still suck in different ways?
- involve change management, incident management, and quality assurance engineers?
- report on your progress weekly until you have a product you can show and illustrate the value of the platform?
- entertain wonderful ideas of projects for your software developers to do since a platform has not been delivered yet?
- demo the product constantly?
- write blogs explaining what cloud-native is and means to various stakeholder personas?
- record videos of demoing the product or discussing how the internals work or what the debugging experience is like?
- rectify diverging alignment of priorities of stakeholders that have contradicting missions?
- balance a wide spectrum of product ideas of how the platform should be consumed and operated?
- encounter unclear boundaries of systems that when glued together create a production product?
- delete documentation that once was helpful but now is wasteful or misleading?
- struggle to get that first product or team on your platform?
- tell the story of your product?
- share the feedback from users to your team no matter how little or minute it sounds?
- misunderstand some Kubernetes primitives like not setting self pod anti-affinities until you learn the hard way?
- be tempted to overpromise the benefits of eventing, microservices, and other architectural paradigms?
- underestimate the amount of training and effort required to enable the success of teams?
- plunge into using service mesh that requires a bit too much expertise and knowledge than you realized?
- irritate developers with a time-consuming CICD process when you could be offering them rapid inner loop development?
- groan when you realize it is time to automate cluster creation after you missed essential backups or inserting required secret values?
- wring your hands when DockerHub rate limits you again but you didn’t bother to set up harbor or sign up for a DockerHub account?
- shriek when you lose a quarter of progress because security told you to stop running pods as root and not to run everything in the default namespace?
- hold your head in your hands after realizing running databases on Kubernetes is for someone more adventurous than you?
This is not a comprehensive list of all possible things you might encounter. I hope it provides some context of what you are getting yourself into. If you are interested in learning about other ways Kubernetes could be difficult technologically check out Kubernetes failure stories.