Cloud Native Landscape for DevOps Engineers (mid 2018)
Saturday, Jun 9, 2018
^ Disclaimer: I live and work in Melbourne, Australia, so things may differ in your area.
Look at all the logos!
It’s an exciting mess, especially as a DevOps engineer.
Are you going to learn all of them? Nope, it’s impossible. There are changes daily on the landscape, and there are millions of people behind these logos and a huge eco-system and community.
Big companies care about every single aspect of this landscape, and that’s why these things are there and a lot of them became bigger and bigger. Now let’s look at each aspect of it!
Public and Private cloud, both of them have a reason to exist. The cloud is now making businesses’ lives easier than ever. No more hardware provisioning and months long “design to production” cycles. For DevOps it’s the time to write your infrastructure provisioning scripts that quickly spin up, scale and shut down different services across different clouds. Sometimes you do, however, need to know some details about how each cloud works and what are the tricks and workarounds to each of them.
As real clouds, they do rain from time to time. This is were you may struggle to explain why your application is unstable or even unavailable, so knowing how to reliably deploy and design the applications are very important too.
And finally (or firstly I should say) - security. Sometimes because of the ease of deployment and management, we forget about stuff, or we get lazy. You may think leaving a machine there with SSH or RDP wide open is not a big deal because this little VM you just deployed seems naive and simple, until one day you found that suddenly you have malware mining coins on your server… So, know security and make sure everyone in your team has the easiest way to do their work in a secure way by default.
The story of Ansible v.s. Puppet v.s. Chef v.s. SaltStack never gets bored right? What about Containers? What about AWS CloudFormation and Azure Resource Management templates?
You may find yourself stuck with 20 tools and any 5 of them combined can get your task done. And you may also find that make sure IAAS resources are under control is easy (secret: just don’t give people access and only use your Ansible Puppet Chef and SaltStack). However the PAAS and SAAS type of things and the “Container” can kill you quite quickly. You will find apart from you 200 other people have access to the “cloud” and can make changes and the 20 environments you are provisioning are never the same look you left them yesterday. So what can you do?
Sorry you may need to live with it, at least for now. PAAS means you are locked into a “Platform”, as the “P” suggests, and it also means that only the platform will have the incentive to provide better tooling for you to manage itself. As of now, making PAAS environment “secure and consistent” is quite hard. Not saying it’s impossible but requires very good process (and self control). And you will be lucky if you are the only person in a team making calls of how to provision PAAS, because a lot of times there are just too many ways to do something, and each member will have his/her own way of doing it.
I would say the old way still works, but it’s all about containers now, so obviously the eco-system is now shaped around how to run containers better and get them to communicate with each other.
The difficulty here for things like Storage is that you have a lot of options, and you don’t really know which one is better and you cannot simply just try some of them because they are either commercial software or it requires a lot of effort to make it working. Never to mention that you want to run it efficiently and with confidence.
Similarly with network you need to know quite a bit of how each of them works and why you pick one or the other. My suggestion here is that just learn maybe to top 2 of each category and leave the decision making to the architects.
Orchestration & Management
This is where it’s getting interesting. Look at the cool kid - Kubernetes. It seems that a lot of things are now built around Kubernetes and it seems that it is going to win for at least the next few years. End of the day it’s a lot easier to sell to management saying: Google is using this so we should be fine!
My suggestion is to build a Kubernetes cluster yourself from scratch with a couple of virtual machines (at least 4 and try to make it highly available). After all the pain and suffering you will know a lot more about how exactly Kubernetes works and why things are they way they are. And I promise you after this finding a job should be easy for you for at least the next 5 years. It took me 3 months on and off to fully built my Kubernetes cluster and understand how it works. I did it using 4 old Dell Optimum desktops from eBay.
App Definition and Development
You are the DevOps person? Cool can you configure some Cassandra and at the same time set up geo-replication on our Microsoft SQL database and at the same time figure out why our Redis instance is not performing well? Oh and also can you come up with a plan to backup our MongoDB?
Here is where people are expecting the most from DevOps, not only the databases, and of course hopefully your team has a dedicated DBA that is specialised on SQL Server, MongoDB, Cassandra and Redis…? Oh sorry I forgot better to know Azure CosmosDB and AWS DocumentDB, thanks!
This was just the start…
Streaming is gaining popularity so know a little bit of things like Spark and Kafka, just so that when people are talking about it you know what they are talking about.
In the mean time, now the fun part, you need to know at least 5 of the CI/CD tools to make you hireable. As a starting point I would say learn Jenkins, Octopus Deploy, TeamCity, Bamboo and Microsoft Visual Studio Team Services. Of course you need to also be an expert in Source Code management tools like git, and pray that your team doesn’t use Microsoft TFVC.
… Oh BTW Microsoft just bought Github, so expecting things to get a lot more connected.
Don’t stress too much about platforms, if you work in AWS and people want Azure, just say you know it and you can learn that really quickly. It’s good, however, to learn some basics of Azure, AWS and Google Cloud so that you can cover at least 80% of the cases.
Again Kubernetes is getting more and more popular which means that there will be a lot more platforms offer managed Kubernetes service. As of Jun 2018, just think twice before you decide to move everything to that platform if it is not Google Cloud.
Observability & Analysis
Don’t overlook this section, it is very important! I repeat, this is very important!
Be able to use the right tool to do monitoring and analysis is very important for companies, especially if you work with big enterprises and corporations. Your task as DevOps is mostly to know what tools should be used in which situations for both application and infrastructure monitoring.
Do you remember I mentioned Kubernetes? Try to understand how to monitor and analyse applications running inside Kubernetes can be quite useful. So after you learn 2 to 3 different tools try to see whether you can use it inside or alongside Kubernetes. In the mean time don’t forget to learn some log visualisation tools as people still have love with dashboards.
Remember, running microservices and be good and monitoring and debugging them can be really tricky, so if you can do well in this it’s a really good selling point.
It’s becoming more and more popular these days but in my opinion it is kind of platform locking mostly, so I would learn more about how to auto-provision and deploy serverless applications and how to worry about monitoring and logging.
Currently not many people are doing Serverless in Kubernetes now, but I guess it will be more and more popular…?
Too little time for too many things to learn… but isn’t that always the case for life?
Also whatever you are doing remember things can get messy very quickly with all the things you added to the system, so keep things simple is essential, and for whatever you are doing and however you are doing it, make sure you think about security.