Written DevOps Training
Why Do You Establish Your Continuous Review Process To Ensure Quality?
Your organizations heavily rely on reviews, audits, inspections and approvals just before production deployments. These inspections are usually conducted by people who are only remotely involved in your work, if any. Worst of all these inspection and approval authorities sometimes have incorrect understanding about your work, so you need to educate them just before you want to deploy your code in production.
Problems With External Control Mechanisms
In most organizations building control mechanisms is easier than building mutual trust. Therefore, adding more questions to change control forms, adding additional approval layers in the hierarchy and requiring extra lead times to understand and approve changes are typical problems with external control mechanisms.
Low trust environments of command and control cultures are doomed to live with a lot incidents which continuously repeat themselves. A vicious negative cycle emerges in these organizations because there is no clear transparency about real work performed, because there is no transparency about recurring problems, and because work is performed in large batches to keep number of painful deployments as low as it is possible. Longer lead times to deliver hurts the time-to-market competitiveness. Slow, late and non-actionable feedback to your teams makes learning from mistakes impossible.
A competent DevOps organization like yours know that: People who are closest to real work performed and its associated problems know the most about them. Therefore, having additional external control and approval mechanisms do not bring added value to your value stream. Which brings us to the largest management and leadership challenge of our time. Building high trust cultures where the work performed is transparent, where genuine respect and mutual trust are essence of daily work.
You need to shift reliance away from periodic inspections and start relying on peer reviews part of your daily work. Make your developers, operations engineers, quality assurance and information security specialists in your DevOps team constantly collaborate and review work of each other to make safe deployments possible.
Coordination Of Complex Work Without External Control Mechanisms
It is already complex when one single team works on a project, but work can become even more complex when multiple teams work on different components of the same system. Some measures to keep you and your DevOps team on the top of your game are:
- Change boards who are subject matter expertises themselves, not only managers, can help you identify dependencies of various teams before changes happen.
- Loosely-coupled architectures based on SOA and micro services reduce dependencies and communication effort.
- Technical representatives of DevOps teams should identify dependencies, development and deployment sequences.
- Continuous communication with common chat platforms to synchronize changes and comprehend and review the understanding of others.
- More impactful and risky changes such as migrations or infrastructure changes require rehearsals on test environments and technical countermeasures such as failover and full-rollback mechanisms.
High Performer DevOps Organizations Rely On Peer Reviews And Less On External Approvals
Many prominent DevOps organizations including Amazon, Netflix, Etsy and GitHub use “Pull Request” Review & Deployment Process. Here is how it works:
- The developer creates a separate development branch from trunk. This branch should not live more than a few business days and it should have a clear, descriptive name such as “login-screen-v2”. The developer works on this branch locally on her workstation and regularly checks-in this branch to version controlling system.
- When the developer thinks that the development branch is ready for merging trunk, she opens a “pull request” to ask review feedback.
- After the developer gets review feedback and approval from other members of her DevOps team, she merges her code to the trunk. Then she deploys her code into production systems.
For high profile changes such as database schemas or changes that may impact information security of applications, the developer can also send additional “pull requests” to get feedback from other subject matter experts in your organization such as database administrators or information security specialists.
Small batch size principle ought to be used again while codes are being viewed. Otherwise reviewing code takes longer and this puts a lot of burden on reviewing engineers. When the change gets bigger, the risk of this change gets exponentially larger and the review becomes less reliable. If a change is too big and difficult to understand, the developer can be asked to split her change into multiple smaller and understandable chunks.
Everyone in your DevOps team regardless seniority and experience level must have someone review his or her own changes. Everyone should monitor commit streams, so potential conflicts can be quickly identified and solved before they cause larger issues in deployment pipeline, even worse during production deployments.
Pair programming by itself and pair programming in combination with test-driven development (TDD) (one developer writes code of application and another developers writes automated test) can enable your DevOps team to conduct peer reviews while the code is being written. Given the same team size and project requirements, it is measured and proven that pair programming takes 15% more time, but it reduces 85% of coding errors. Therefore, pair programming is an important practice you and your DevOps team should evaluate to use.
Furthermore, over-the-shoulder reviews, automated notifications of check-in steams and tools to assist code reviews and increased code quality are other alternatives you should be considering to implement.
CONCLUSION
Make every DevOps team in your organization be responsible for own quality of its deliverables. Build a culture that values continuous and high quality code reviews as much as writing high quality code.
