This is the second part of an article about the work organization of my DevOps team. You can find the first part here
- Small batches of work
Without entering the rabbit hole of the Toyota production system and the theory of the value stream, I remember how most of the IT professionals I’ve worked with have had an “in-house big side project”: a large refactoring of that part of the infrastructure or an update of all the operating systems in that part of the company network. It’s something they’re doing every day for a small percentage of their time. It’s hidden work as well, as usual, their managers don’t know about it. It cannot be measured, no one knows if/when it will be finished and ends up providing little value to both the company and the employee that gets no reward for the job done. It needs to be approached like this because it is too big and would not fit any estimation. At Curve, we try to go around this problem by organizing the DevOps team as any other development team. We use Agile, estimations, tasks, epics, bugs, and reports.
We strive towards the goal of small batches of work, for a large refactoring an Epic is created and smaller tasks are attached to it. As a rule of thumb, the tasks must be small enough to be completed in 1 or 2 days. No more than that. And in the same way, we apply the principles of continuous integrations to our environments: all the code describing the infrastructure is versioned in git and we don’t keep feature branches out of develop, our mainline, for longer than the duration of a task.
Creating smaller chunks of works, allows us to recognize repeatability and use it in our favor. Not to mention, a useful psychological effect of “visualizing progress” on a taming project.
- Separate IT/Administration tasks from DevOps
In a startup, especially at the beginning, everyone does a bit of everything 🙂 And this really is a great learning experience! But at some point, especially if the company is doing well and you have new people come in daily, you will find an (expensive) DevOps team struggling with a lot of IT/Administration tasks (new laptops, new office, new wifi).
When you reach this point, I think it is of utmost importance breaking out of the startup routine and separating IT/Administration tasks from the actual “DevOps” job. Assigning dedicated resources for these tasks will help you free the hands and the minds of the core engineers. Outsourcing can go a long way, in this case: if you’re not in the business of IT, getting a company doing the work on the hardware for you is usually cheaper than doing it in-house.
Nonetheless, certain “system administration” tasks will stay and need to be performed in the company: creating users, fixing the occasional printer, etc…
For this reason at Curve, I’ve created the role of: “IT/Operations Engineer”: a (very) junior DevOps that are trained by the company. They are doing part-time IT support and part-time DevOps tasks. It’s a win-win situation, the company gets a motivated employee doing needed work and the engineer gets the opportunity to grow his skillset set by working on more complicated stuff that they would usually not be allowed to touch.
In other words, if you have to do the repetitive work of creating users in AWS, you might as well assign the tasks to do it in Terraform to a younger engineer allowing them to learn a few things on the way.
This ends up creating a virtuous cycle, as over time, this engineer can be promoted to a full-time DevOps role (at this point, already trained on the system used in the company) and a new younger engineer can be brought in.
- We’re doing Continuous Integration and Deployment…why, not Continuous Learning?
This is one of our basic principles. Personal development should be top priority _for real_. Each member should have a clear growth plan, and ideally every day we should work on it. It should always be in the back of our minds. This is as important as work being done. Conferences, meetups, networking are always encouraged and, at Curve, sponsored by the company.
- Accept Requirements but not work from the dev teams
In the “average” startup the operations team will quickly find itself outnumbered by the developers with the concrete risk of becoming a bottleneck for the global velocity. To solve this issue we do our best to empower the devs, offering them a platform and the strongest automation we can build to allow work to proceed without our direct intervention. Self-services, deployment pipelines, and automated testing tools have been configured, production-grade monitoring have been (or “are about to be”) deployed. It’s a form of guided ownership that is the foundation of our work. At the same time, we should not do their work (e.g.: “fixing the environment after a bad deploy or a bug in code“) but train them to solve their problems, ensuring at the same time that the operational concerns of running their code are taken into account when estimating the work.
- The job is not (only) coding
I strongly believe that DevOps engineers are only as good as the culture they bring with them. If you’re just coding/configuring stuff all day, you’re probably only doing 50 % of your job. By focusing on the platform, we’re offloading some of the traditional operational concerns to the devs running the code; that at this point, counterintuitively, must become much better at Operations. For this reason, the DevOps team, should always foster the continuous learning culture in the other teams and offer operational training when needed.At Curve, every month we have a “Production meeting“, with most of Engineering, in which we talk about the state of the production environment, the issues that we faced during the month or the recommendations that we have. This meeting is also an opportunity to do operational training on technologies or portions of the system that are less known or that are important for performance reasons. To ensure engagement the session is organized like a group game rather than a lecture.
The inspiration for this article came from a couple of books that I’d suggest everyone to read:Kim, G., Humble, J., Debois, P. and Willis, J. (n.d.). The DevOps Handbook.
Kim, G., Behr, K. and Spafford, G. (n.d.). The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.
Beyer, B., Jones, C., Petoff, J. and Murphy, N. (n.d.). Site reliability engineering.