How to Become an MLOps Engineer as a DevOps Engineer

Getting into Machine-Learning-Ops

This article is for people who are already DevOps Engineers – who work day and night with infrastructure automation, continuous integration and continuous delivery pipeline setup and maintenance, and set up monitoring and alerting for applications and systems.

I assume you know the concepts and implementations of the "DevOps Roadmap", and now you are looking at a team with 5 PhDs doing something called "Machine Learning". You may be wondering: Are they using automation? The answer is most likely no. Looking at Google Trends MLOps vs. DevOps as of early October 2020, the search term MLOps is almost non-existent, but if you search for machine learning you can see how much growth there has been.

What does this tell you? There may be a gap between the development of machine learning and the level of maturity needed to productionise machine learning projects' artefacts.

What is the difference?

The first thing you need to understand is the key difference between software engineering projects and machine learning projects.

Software engineering projects are built to produce a deterministic result given an input based on the business requirements, while machine learning projects give you a probability or prediction of a result given the input data. If you still do not get it, let us make it simple: software engineering projects are like multiple-choice questions, but machine learning projects are like writing essays – there is mostly a known answer for a software engineering problem, and mostly no known answer for a machine learning problem.

Practically, what does this mean?

It means software engineering projects have black-and-white answers, and they most likely have an "end state" where the software is behaving as expected. You can write tests with great confidence that, as long as your infrastructure is up and running, the software should behave correctly as expected.

However, that is not the case with machine learning projects. They do not really have an "end" because the result is a probability, and if a problem can be solved with 100% probability, then it is likely that you do not need machine learning. This means, if you want, you can always improve something in a machine learning project – algorithms, parameters, training data …

So, what should I learn?

Personally, I think here are the things you need to learn:

1. At the very beginning you need to know the machine learning workflow.
2. Along the way, you need to learn the terminology.
3. You then explore what tools people use to do machine learning.
4. Understand how to do CI/CD for machine learning projects.
5. Understand how to do monitoring & alerting for projects with machine learning components.

Now let us dig a bit deeper into each section!

Machine Learning Workflow

Typically, in a software engineering project, you follow the process below:

1. Define the business requirement.
2. Translate the business requirement into technical requirements.
3. Design the technical solution.
4. Implement the technical solution.
5. Test the technical solution.
6. Deliver the technical solution.
7. Then you repeat the cycle, from step 1.

However, this is not what a typical machine learning project looks like. It is more like this:

1. Define the business requirement.
2. Gather the data that could be relevant to solving the business problem. This may include data cleaning, transformation, etc.
3. Perform experiments with the data. This may include performing feature engineering, picking and choosing the correct algorithms and parameters, etc.
4. Build machine learning models. This may include selecting the features, algorithms and parameters, and building models.
5. Deliver the machine learning model and look at the performance.
6. Improve the model by repeating the cycle, from step 3.
7. Then when you are happy with the model, you repeat the outer cycle, from step 1.

As you may have noticed, the way machine learning projects work is not a simple build-and-deliver model. It has more experimental cycles before moving on to the next business requirement. This means MLOps engineers will work more closely with data engineers and data scientists, providing solutions to facilitate the need for continuous experimentation and engineering.

Terminology

There can be a lot of new terminology for you, which I consider the elephant in the room. At the end of the day we are talking about data science, not software engineering. As scientists always make things sound more complicated, it is necessary that as MLOps engineers we understand the terminology they are using.

There are a lot of places where you can find great references about these terms:

• https://developers.google.com/machine-learning/glossary
• https://ml-cheatsheet.readthedocs.io/en/latest/index.html
• https://machinelearning.wtf/
• https://yanndubs.github.io/machine-learning-glossary/

Do not stress too much if you do not understand what something is. Personally, my suggestion is that a lot of times you want to pay more attention to the verbs than the nouns. For example, you may want to know what "Dimension Reduction" is, but probably do not need to know "Markov Decision Process".

Tooling & CI/CD

This will be similar to what you need to learn when you start your DevOps journey, but focused on the set of tools you need for machine learning. Putting aside your usual tools like scripting languages and CI/CD platforms, you may want to learn some new tools that are designed to solve machine learning problems.

Some of the things you are familiar with may need to be changed, and you may need to find new ways of doing the things you know. For example, when you think about testing, what type of testing would you like to do, with what tools, and at what stages? You may find yourself unable to do unit testing on the order of seconds or minutes, or you may not even be able to do unit testing at all in some stages like model training. Another example could be the artefacts you will be storing and serving. What are the artefacts at each stage, how do you store them, how do you version them, and how do you distribute them? There are some great resources out there, like https://neptune.ai/blog/best-mlops-tools.

The main complication here is that as of today the machine learning ecosystem is not as mature as the software engineering ecosystem. Therefore, you may find a lot of new tools that have not been fully tested and/or are still in beta. You may also need to talk to your data engineers and data scientists to convince them to use some tools over using no tools at all, or even more excitingly (or worse, depending on whether you like it or not), you would need to skill up your team and help them to follow the so-called "engineering fundamentals" – because a considerable number of data scientists, despite being super talented and smart, have not been exposed to writing production-quality software.

Monitoring & Alerting

Monitoring and alerting in machine learning projects could mean the same or totally different things compared to software engineering projects. Of course, you want your server clusters to stay up, you do not want your data store to reach its storage or throughput limit, and you set up alerts on your budget in the cloud so you do not get a heart attack when seeing the bill. But on top of that, there are things specifically needed for machine learning projects.

For example, you may want to monitor the machine learning models to make sure that the model does not change unexpectedly. If your model is providing recommendations to an e-commerce website, you may want to see if it is still providing relevant recommendations or is providing any recommendations at all. Thus, your approach may need to be changed compared to your traditional software engineering monitoring – it is no longer simply "if the request returns 200, then the dashboard is green".

There could be more subtle things to monitor that are more difficult to design monitoring for. If you have a machine learning model that self-improves based on user behaviours/interactions with your application, you may want to monitor the model to make sure that it does not do unwanted things. When you deploy your next Tay Bot, you may want to make sure that it's not going down the old route again.

What is next?

Congratulations! You have now opened a new area of possibilities for your career to help more people! With the knowledge you gained while learning MLOps, you can then think about picking up DataOps – a similar but slightly different area of Ops, which will help you to be better at providing your team with a framework for data administration, collaboration and automation!