International DevOps Certification Academy™
How Should You Build Your DevOps Organization And Design Your Software Architecture?
According to Conway’s law, organizations which design systems are constrained to produce systems which are copies of their own communication structures. In other words, your software cannot do any better than how efficiently your teams communicate and interact. Therefore, how you structure your teams will surely impact your software architecture, IT and finally business performance as well.
How You Configure Your Organization Defines What Kind of Software You Get
Most companies, probably including your company too, compartmentalize their software delivery organizations in a number of teams, and they end up producing their software architected with the very same number of layers. The controlled experiments have also proven that when an organization of 6 teams was asked to build a software, their teams came up with an architecture of 6 layers. When another organization with 3 teams was asked to build the very same software, they came up with an architecture of 3 layers. In fact, there are many similar examples.
A renowned insurance company with 86,000 employees worldwide was coincidentally structured in 2 major functional teams for one of their IT organizations which delivers the backend to manage their customers, contracts, invoices and services. One of these functional teams was oriented in Java programming language and the other functional team was oriented in PL/SQL stored procedures. Furthermore, they were dependent on organization-wide and centralized IT operations, database admins, IT infrastructure, IT networking, IT security and IT quality assurance service desks which made their lives even more challenging to be able to safely deliver their software.
The other challenge of this organization was that: Although the teams were separate, in their architecture they couldn’t competently separate business orchestration logic in Java from data access logic in PL/SQL. They delivered hundreds of features, each feature being a spaghetti of Java and PL/SQL codes. Therefore, every change and new feature request required tens of handovers from one team to another, and they resulted in long waiting queues and lead times to get even minor changes done. Moreover, in this complex environment, as overall impact of even minor changes was very difficult to foresee, they continuously broke existing features in production and caused downtimes which also required long lead times, rework, escalations and stress to resolve.
To recover from this modus operandi, these two functional teams merged into one single product team. They gradually redesigned their software by converting their data access layer into a set of API functions. In addition, they built a new business system completely decoupled from the internal dynamics of their data access API. Even in its early stage, this initiative improved the team morale because both Java and PL/SQL experts started working for the success of their joint product team instead of motives of their past functional silos. As they built a loosely-coupled architecture, now the impact of changes are easier to identify, changes are easier and quicker to implement and defects are more straightforward to locate and fix. As a result, average lead time of new features reduced from 4 months to 3 weeks, and incident queue of the team is now almost empty, so they profit from this free capacity by further reviewing, refactoring and improving their codebase.
Matrix Organizations, Functional Teams and Illusion of Cost Optimization
Your companies are organized with a number of functional teams around various different subject matter expertises such as databases, networking, operations, information security, quality assurance, release management, project management, product management and so on. Project teams are built by people from different functional silos. In these matrix project organizations which attempt to deliver business products and services, project managers are usually responsible for getting something delivered, and product managers are responsible for ensuring that correct thing is being delivered.
Your problem in this organizational configuration is that functional teams have no to little understanding about the the extent of the work they contribute. In extreme but often typical cases, your functional teams neither care the big picture nor the overall IT and business throughput of the product and service they contribute. What they care is to make sure that none of their doors are left open after projects will go nasty and everyone starts to finger-point.
As your functional teams usually have to manage long queues of tickets, they usually require long lead times to support your project. Because projects fight for functional resources, escalations are the only way to get quick attention for your project. Escalations over escalations obviously pollute the working climate and trust between your teams.
Multiple handovers from one team to another, delays, quality issues, reworks, bottlenecks and stress are now part of your daily job. This is because your matrix organizations are not meant to do any better than that, as long they continue focusing on a opaque and fake illusion of cost optimization. In fact, due to quality issues, reworks and delays, functional organizations are probably even more expensive than any other random reorganization you can ever imagine. In addition to this, if your organization has already managed to outsource some of its vital functional skills such as quality assurance and IT operations to geographically remote locations, then your IT organization must be now barely surviving to safely deliver and fulfill demands which are critical for your business continuity.
This is indeed a Big Problem. How Does DevOps Solve This? Answer: With Product and Service Oriented Teams
In order to solve this problem, DevOps suggests you to switch gears from cost optimization illusion of functional teams to DevOps’ valid and proven speed optimization. In fact, done correctly, DevOps will anyway enable you to save costs while you and your team quickly and continuously deliver.
For you DevOps’ suggestion is product and service oriented teams. Product and service oriented teams are independent, cross-functional, autonomous, self-sufficient and usually small teams which are able to cover all phases of software engineering life cycle of a given product and service (but not a temporary project) including architecting, designing, coding, testing, experimenting, deploying, operating and maintaining software. Yes. You read it right. Who else can operate and maintain software better than their own creators? The answer is of course: No one. Therefore, it is no surprise that the most successful DevOps teams at Amazon and Netflix do not abandon their products and services once they release them. They are also in charge of operating and maintaining their software in their own production systems.
Functional Teams vs Product and Service Teams
(Source: Lean Enterprise, DevOps Handbook)
Your Organization Does No Longer Finance Temporary Projects, But It Finances Products and Services
To make its business performance flourish, your organization does no longer finance temporary projects whose tangible business outcomes are not trivial to evaluate in long run. Now with DevOps, your organization finances its own mission, its own purpose and its own products and services associated with this mission and purpose. The success of your teams are now assessed and evaluated based on their IT and business performance. Based on return of investment within their particular business domains where they serve their products, services and micro-services to their internal and/or external clients. Your teams now act like owners of products and services they create and provide, instead of merely being members of functional silos who don’t pay much attention to business outcomes.
How Can You Build Your DevOps Teams and Software Architecture?
DevOps does not of course suggest you to break and reorganize all ongoing projects at your organization in one go. A non-disruptive, but still impactful way of adapting your teams for DevOps methodology is to inject functional experts into projects teams. Once your teams get functional experts in their desired domains such as quality assurance, information security, databases, networking, servers, operations and so on, then you can expect these teams to independently and self-sufficiently design, deploy and maintain their software products and services.
In these new product and service oriented DevOps teams, availability, quality, performance, information security and compliance are everyone’s daily job. How can your teams build highly available, secure and high quality software applications which are able to cope with stress and load demanded by your clients if the teams never pay attention to these non-functional requirements until they finish their design and development? How good can external experts judge and validate the security and quality of your software applications without being involved at any software engineering stage of your products and services? This is why high performer DevOps teams rely on external subject matter experts only to get consultancy, but they still fully own all non-functional requirements at every stage of their software engineering lifecycle.
There is an Alternative You Can Consider
One exceptionally successful DevOps organization which still heavily uses functional teams in combination with product and service oriented DevOps teams is Google. At Google DevOps teams and functional teams see their organizational mission with their products and services as a shared goal. Functional teams build blueprints (such as Templates, Checklists, Application Skeletons, etc.) and self-service systems (one example: to independently provision and configure servers and networking) to make sure that they never become bottlenecks which slow down DevOps product and service teams. Functional teams at Google collaborate and support product and service teams at early stages of software engineering life cycle, they constructively perform peer-reviews of designs, codes and tests, and they continuously provide consultancy to DevOps product and service teams.
How Big Should Be Your DevOps Team?
The ideal size for a DevOps team is 5 to 10 people. Such a limited team size reduces complexity of communication and alignment within your team. Furthermore, your team lead and team members do not spend and waste much time with errands and overhead. This also keeps the size of product and service your team is responsible for up to a certain limit which further reduces the complexity, maintenance and operations difficulty of software applications. Every team member in such small teams sees the big picture, and everyone collects little bit leadership experience by becoming part of a crucial mission for their organization. Your team lead works with upper management to understand goals and translate them to your team members.
What If Your Product and Service Can No Longer Be Handled by a Team of up to 10 People?
Then your solution is to spin out a new product and service, and to build another DevOps team which takes it over. Here you shouldn’t conceive product and service concepts only as entities served and provided to external clients who pay for them. But also you can freely build internal products, services or so called “micro-service APIs” and their respective DevOps teams for your internal clients. For instance if your billing system becomes too big for a team up to 10 people, then you should spin out another DevOps team which takes over database access API. May be another one to take over online payment API. And then may be another one to take over batch processes. Of course, all these teams should be using a common code repository and a joint deployment pipeline to ensure continuous integration, quick delivery and success of their organizations.
As you already know in a tightly-coupled architecture, small changes in one application can eventually cause many adverse effects for numerous workflows. Therefore, products, services and micro-service APIs in your architecture must be loosely-coupled. Each DevOps team must be only responsible for one piece of an loosely-coupled architecture. Each DevOps team can independently design, develop and deploy their software. Early alert mechanism built in the deployment pipeline should automatically and rapidly inform DevOps teams about potential adverse effects any code check-in causes.
How Can Such Small Teams up to 10 People Can Self-sufficiently Handle All Phases in Software Engineering Life Cycle from Architecture to Operations by themselves?
To manage this, you should encourage everyone in your team to become a generalist. You should encourage and enable them to continuously build new skills.
You should only hire team members who are eager to learn and grow regardless their effective level of knowhow and experience. You should strictly avoid people who expect to be evaluated in a fixed set of roles and responsibilities. You already know that neither your organization, nor your products and services remain fixed.
The demand will change tomorrow, if not today.
CONCLUSION
In this chapter, we explained suboptimal outcomes of matrix organizations, functional silos associated with them and DevOps’ solution to address this.
DevOps’ suggestion for you is to build product, service or micro-service API oriented small teams up to 10 people.
These DevOps teams should constitute generalist full-stack software engineers which are able to self-sufficiently cover all phases of software engineering life cycle from design to maintenance.
DevOps relies on loosely-coupled service oriented architecture (SOA) in which every DevOps team owns and operates one piece of your loosely-coupled architecture.
DEVOPS-CERTIFICATION.ORG
