One critical focus of DevOps is eliminating as many “handoff” events as possible. Handoffs between teams cause considerable delays in a smooth process. Handoffs also leave teams open to quality issues: errors in manual processes, mis-communication, false expectations, etc.
Transitioning to DevOps requires significant planning, support, and patience. First and foremost organizations have to foster a supportive culture across all layers of people. Significant investments must be made in tools and people to work with those tools. Integration across an organization’s collaborative units will require serious attention to ensure effective communication. Finally, the entire organization must be open to continual learning, optimization, and improvement.
An organization’s culture is the foundation of any change or improvement initiative, regardless of what that initiative is. Organizations embarking on a transformation must take the steps to ensure their people understand expectations, see clear support for time and effort, and understand why the transformation’s taking place.
Everything in your organization starts with your people: the good, the bad, the mediocre. Your organization needs to be staffed with motivated, empowered people. Moreover, roles and skills will need to adapt as you move to a DevOps delivery style.
Every individual in your organization needs to clearly understand their role and expectations in delivering successful value for the business. That means your leadership and management need to be constantly communicating downstream to those individuals. Upstream communication needs to be effective as well, so your people can clearly express and resolve challenges they’re facing.
High-performing organizations understand their people, regardless of role, need time on the clock for learning and self-improvement. Obviously this is a critical factor of Culture (more later); however, it can’t be overstated: Individuals need time for learning, and a clear expectation they’re to actively work on that improvement.
Closely tied to an expectation for continual learning is a clear, emphatic expectation of excellence. “Do your best, always” has to be a closely held belief for everyone in your organization. This helps guide individuals to avoid “I’ll fix it later” habits, and it also helps individuals in the organization hold each other accountable. “Hey, wait. That’s not our Way. Let’s take an extra hour and get this done right.”
Many organizations have technical teams that specialize in one domain or area such as front-end, web services, or data access. Such a restricted skill set creates significant bottlenecks, dependencies, and delays due to handoffs. Teams and individuals must move away from this specialization and gather domain and technical skills to take features from envisioning to deployment. Obviously this is a significant undertaking; however, it’s well-worth the investment as organizations find themselves much more flexible and able to faster execute in such a model.
Because DevOps pushes new responsibilities down to teams, individual skills will need to move outside the “normal” development domain and in to areas like scripting, environment management, database management, etc. This doesn’t mean people who previously specialized in these areas find themselves out of jobs. Some headcount reduction is possible, but many organizations move those specialists directly on board delivery teams where they share responsibility and knowledge.
After ensuring solid people policies are in place, leadership must immediately take on a significant challenge: Building an organizational culture that’s not afraid of change, indeed a culture that cherishes and embraces change. This takes a huge amount of effort over a significant period. Moving through a transformation as significant as Agile or DevOps will be much more effective if the organization’s people have been living in a culture that’s been supportive of transformations—and that support doesn’t come overnight.
One of the most important factors in a positive culture is ensuring there’s a clear expression of “safe to fail.” Great organizations make it clear it’s safe to fail instead of punishing failure. That’s not to be confused with making failure a desired outcome! Instead, the emphasis should be on experimentation and learning.
Experimentation means teams and individuals approach problems with the mindset of carefully thinking the problem through, coming up with a hypothesis for a solution, then work the solution as a test. Outcomes are examined and lessons drawn both from successes and failures. Great outcomes are celebrated, and the lessons learned from less-desirable or outright failures are fed back in to the next round of experimentation.
Many great examples of high-performing, positive cultures in high-tech software delivery and service companies exist. Look to Netflix led by Reed Hastings, Dan Gilbert at Quicken Loans, or Microsoft with Satya Nadella at its helm. Naturally great culture and leadership examples in business exist outside the technical domain as well.
Needless to say, all this requires a highly collaborative, adaptable environment. It also requires an environment where processes support, not constrain, the evolution of the organization.
“Process” is a bad word in many organizations. Why? Because too often processes, and those who regulate them, have lost sight of the why of the implementation. Ceremony and paperwork takes precedence over logic in too many situations.
Most processes spawned from a valid reason. Many processes exist to support a specific business value, or mitigate a specific business risk. Compliance and auditing in financial or health care industries exist to reduce the business’s risk of being sued or prosecuted—not trivial things!
Organizations moving through Agile to DevOps absolutely need to work with existing processes to understand the “why” those processes exist. Collaboration with process control staff needs to ensure those staff understand the need for flexibility and change. Organizations need to ensure process control workers are supported and see safety in changes—often resistance is due to concerns at the base of Maslow’s pyramid: basic security of position.
One highly effective approach when working with process control is to find “champions” who are experts in the business reasons behind the processes—the “whys.” Work closely with these champions to bring them close to transformation initiatives. They become the go-to people who can help projects and teams quickly navigate process requirements for fast resolution. Those same champions can help adapt existing processes or create new ones where needed.
DevOps focuses on moving to a continual delivery model by reducing handoffs and eliminating bottlenecks. This requires building out a DevOps or delivery pipeline of tools and processes that automate processes previously done manually. As the toolchain grows and matures it encompasses all aspects of pushing a feature to production.
This toolchain, also called a pipeline or delivery pipeline, isn’t just about the mechanics of pushing to different environments. It’s also crucial to bring your various quality gates into this automated chain. Moreover, you’ll need to ensure your testers understand they still have crucial quality-related work to do—they’re not automating themselves out of a job, they’ll be using their skills far more effectively and in more high-value ways!
Delivering value to production involves managing a huge number of discrete tasks and responsibilities. Less-mature teams often accomplish these tasks manually using steps fraught with fragility. (“Hey, I’m starting a build! Don’t check anything in to source control!”) Teams adopting DevOps need to either have moved past this stage, or make it their first priority.
Orchestrating these tasks is typically handled via a continuous integration (CI) server that automates jobs to manage the overall process. CI servers have exploded over the last ten years, giving organizations the ability to choose from many options, both commercial and freeware/open source. A few of the currently popular approaches include GitHub Actions, GitLab CI/CD, AWS and Microsofts’ pipeline automation tools as well as old favorites such as Jenkins or Circle CI..
You’ll see the tasks above include a wide range of jobs including compiling, interacting with external systems, driving infrastructure, etc. Popular CI servers include many connectors to external systems. At the end of the day, however, teams will need extensive scripting skills to handle the numerous, inevitable tasks that can’t be accomplished by default plugins/connectors.
Organizations likely have the required skills already on-hand with existing staff: database administrators, operations teams managing servers, network security teams, etc. It’s a matter of bringing those staff in closer to the teams to help boost on-team skills and speed execution of these tasks.
A transition to effective DevOps absolutely requires moving a great deal of quality-related activities into the automation domain. This requires changes to how traditional-model development teams may be working. It will also require changes to monolithic architectures and code designs that are webs of complex interwoven dependencies.
One of the biggest challenges to automating more quality checks are teams with “traditional minded” architects and developers who aren’t well-versed in disciplined, skilled software craftsmanship principles. Teams in a mature DevOps culture need solid skills in coding practices that enable the various forms of test-first development. These skills and practices are absolutely necessary to ensure automated unit and integration tests are accurate, effective, and maintainable.
Automated checks via unit, integration, and other tests aren’t enough. Projects also need automated code health checks via static analysis tools such as SonarQube. These tools ensure the teams are meeting industry standard metrics for complexity, dependencies, and other critical factors. All major CI servers have easy integration to these tools via connectors, plugins, or other easily available modifications.
Other critical quality checks must also be automated for an effective DevOps pipeline, including:
- Acceptance testing
- Accessibility
- Performance
- Security
You’ll see many of these automation targets address tasks traditionally requiring formal handoffs. Actions like acceptance testing, security validation, and performance/load testing have traditionally been handled by separate teams. Again, with DevOps’s emphasis on eliminating handoffs, these specializations need to be decomposed and brought on to the individual projects and teams. Specialists will always be needed to help guide efforts and troubleshoot; however, the vast majority of these domains should be moved to the teams.
Occasionally organizations look at the level of quality related automation and think they can dramatically reduce or eliminate specialized testers. In reality those testers aren’t eliminated, they’re working much more efficiently far earlier in the delivery cycle.
Many traditionally-minded organizations have relied on testers writing numerous, long, detailed test cases that are manually executed. Industry evolution has found such work to be far less effective than the close collaboration demanded in fast-paced Agile and DevOps cultures. Indeed, these environments offer testers to bring their skills and expertise to another, far more effective level.
Testers, even non-technical ones who might not know how to write test automation scripts, will find themselves working in dramatically new ways. Far more testing activities will be at the start of project work, even as early as the envisioning phase. Instead of drafting and maintaining tedious, low-value test cases, testers will be working harder on effective requirements and specifications elaborations—directly with product owners and users. Instead of waiting for “The Testing Phase” testers will find themselves working in parallel with developers, sometimes even pairing with them as they code, to test far smaller chunks of functionality then they’ve been used to in the past.
This close collaboration can be a huge challenge, especially in traditional-model organizations where testers are devalued and culture favors “throw it over the fence” handoffs. Leadership and management must work hard to foster respect and facilitate the close communication necessary. Moving to true cross-functional feature teams where every role is a member of the same delivery team (versus separate development, testing, ops teams) is critical. Implementing simple process changes, such as adding in Three Amigos conversations, quickly works to further break down those walls.
Once the basics of the DevOps pipeline is in place, it’s time to look past the mechanics of pushing to production and focus on integrating the efforts with other aspect of the business. The best DevOps environments give leadership constant visibility into the state of their systems for better decision making.
In the past leadership has often focused on simplistic metrics that didn’t give an accurate representation of their projects. It’s been hard to gather the right sorts of data and transform that in to actionable information to support good business decisions. Good DevOps practices, coupled with advances in data gathering and reporting, can help close this gap.
Previously organizations focused on easily gathered, but often misunderstood metrics like defect counts, velocity, or number of test cases. These are all simplistic metrics which don’t provide true exposure into how the system is delivering business value. Constant delivery of high-value features gives organizations the ability to truly see how the system is functioning in live environments, and how users are working with the system to solve business problems.
Additionally, DevOps provides leadership and management access to much richer information at all stages of the process. What’s the state of quality in the user acceptance environment? Where could we apply more people and resources to mitigate bottlenecks and speed delivery? If we’re properly sizing our work, how long will it take us to complete this next major block? What does the user experience for performance and accessibility look like in the live environment?
Such information gives leadership the opportunity to make better strategic business value decisions based on richer data.
One of the best payoffs of adopting solid DevOps practices is the ease of continually improving an organization’s delivery processes. By building out a solid foundation of automation, integration, and reporting organizations make it easy to constantly try small experiments.
Looking to try out a new micro-service architecture? Your DevOps infrastructure tooling will make provisioning new environments, complete with dependencies, a snap. You’ll have those environments ready in minutes, versus days of building, configuring, and testing.
Are you looking to experiment with new team structures in the hopes of improving quality and productivity? That team can get to work immediately, thanks to tooling which can quickly stand up environments, build pipelines, source control, and apply default process templates for a new project. That experimental team will also be able to make use of rich data analysis reports to help them understand how they’re performing against expectations set out at the start of the project.
Do you want to do A/B testing on a new feature in a segment of your live environments? Again, your DevOps pipeline and tools will enable you to quickly roll that out, monitor and measure it in a live environment, and either roll it back or roll it out further depending on your decisions.
Solid DevOps environments also enable an organization to constantly move though the cycle of identifying and removing bottlenecks large and small. Throughput over time can dramatically improve as subsequent constraints are found and addressed.
Organizations wanting to maximize their effectiveness have adopted various Agile approaches. DevOps is a natural “next step” for these organizations looking to continue their journey of improvement.
DevOps isn’t a magic wand. Like Agile, it takes hard work, time, and honest support from the organization’s leadership. The payoff for successful transformations is the ability to make much better informed strategic decisions based on constant visibility into the delivery teams’ current state.
For many years at Usetrace our approach has been to build tools and services that enable the entire product delivery team inclusive of Product Owners, Business Analysts, and existing manual QA engineers to effectively incorporate automation in their day to day work. When these team-members are able to contribute to the automation efforts they often more directly reflect the paths of actual users through the system.
By providing tools that record these user paths and then connect the Usetrace automation suites to CI and CD our clients can run the full suite of unit, integration, e2e, as well as user-centric browser based tests through Usetrace on every code change. This dramatic shift allows continual feedback to the team and a moment by moment understanding of the applications readiness for deployment.
While Usetrace is one piece of the puzzle as it relates to the shift to DevOps, more and more organizations are seeing the value of a tool like Usetrace to empower more users of varying backgrounds in their automation efforts. Sign up for a free trial at https://usetrace.com/.