The Automation Trap
Mad rush to quality engineering
Solutions to automate testing have been around for decades. Having spent my 20+ year career in quality management – starting out as a testing intern in Bangalore, India, moving on to automation engineer roles and finally into quality management leadership roles in locations such as Singapore and New York City – I have led efforts including framework developments, building proprietary automation tools and running automation centers of excellence (COEs) for Fortune 500 companies. Automation done right enables higher quality and efficiency, but over the last few years, there seems to be a trend that is leading some firms down a path of blind proliferation of test automation.
Observe how many organizations have transformed quality assurance COEs into quality engineering COEs in a short timeframe. And the number of traditional quality assurance teams that have been reimagined as “all automation’’ teams. Most of these decisions seem to be based on notions about agile development and DevOps and are sometimes simply about having a “cool factor.” As the pendulum swings towards test automation, firms run the risk of being more interested in automating tests than understanding if the right things are being tested. The cost of automation may exceed current and future benefits. Recruitment is impacted as there are more automation jobs than people with the skill.
I am a big supporter of test automation.It is a powerful means to enable better testing. But when it becomes the only strategy, the pendulum has swung too far. Enterprise level automation coverage sounds like an admirable goal. But for organizations with complex and intertwined business lines that have a mix of commercially off the shelf products, distributed applications, and some legacy applications, automating 100% of testing isn’t the optimal solution. The return on investment should ensure the costs justify the means.
At a Crossroads
Over the past ten years as we have assessed automation at BBH we asked questions such as “What automation coverage do wehave relative to what we need?” and“How many manual testers are on the team?” We knew that an ideological argument over automation vs.manual testing was not going to help us and we avoided the “automate everything” trap. We assessed where we were in our own automation journey;we had many strengths:An exceptionally skilled team with deep testing and domain knowledge,
a good set of tools, and a robust framework. We also started with a high proportion of manual testers and faced high expectations for more automation, and so it was clear that increasing the number of people who could automate testing was a key goal. We saw three roads to get there:
• Release the manual testers and hire new automation engineers
• Upskill manual testers so they can automate
• A hybrid approach of hiring automation engineers combined with an upskilling program
Taking the first road mightseemeasy or leave some with a false impression that it would have allowed us to begin immediately. The cost of releasing experienced testers is losing years of domain and testing expertise with a team that has a deep understanding of our business and applications. The second road also had challenges as it can be slower to upskill than hire, and not all manual testers are good candidates for upskilling. We choose the third path as we believed, in addition to onboarding new talent, that ups killing our manual testers would result in a broader skill set on the team with a greater chance of long-term success.
The Upskilling Plan
Almost five years ago when we made decision to include an upskilling program within our testing organization, we developed a phased implementation approach. The first phase entailed upskilling our testers with automation skills. The second phase provided tools to enable non-technical testers to contribute to automation. We launched our test automation upskilling program as a pilot with 25% of ourmanual testers. They included junior and senior testers with different experience levels and roles. The key features of the program included:
• A senior stakeholder as program sponsor
• 12 – 24-month customized learning plan tailored for each individual based on skill, experience, and corporate title
• Mentors that were assigned to each candidate
• Defined goals to enable both the mentee’s and mentor’s success
• Expectations for the commitment required from participants
• A common set of exit criteria for “graduation”
A key component of our communication to the team was demonstrating respect and appreciation of current skills and reinforcing how the program supplements those skills, it doesn’t deem them irrelevant. We hand-selected each person to be part of the pilot, but also made it clear that they could drop out at anytime. This helped make team members feel comfortable participating in the program.
Our pilot was successful and is a program we continue to leverage. Only 5% of candidates have dropped from the program and 95% have remained with us as test automation engineers. We also saw teams who were historically hesitant to be in the program become interested having seen their colleagues’ success.
The combination of our recruitment and upskilling efforts have put us in the position to achieve our optimal automation levels later this year. In addition to achieving our financial automation goals, increasing automation levels across our key applications has the added benefit of de-risking the change process by reducing production issues and increasing agility by making it more cost effective to put in smaller more frequent changes. Just as important, we have highly motivated teams that are gaining new and empowering skills. They feel valued and are appreciative of BBH for enabling this professional growth.