For this edition of Startupon, I spoke with Jon Perl from QA Wolf. If you’re interested in connecting with the QA Wolf team, reply to this email
Software Search: QA Wolf
Company Snapshot
Founded: 2019
Employees: 54
Funding: $20M
Valuation: N/A
Stage: Seed
Locations: Seattle, Remote
Company Overview
QA Wolf is pioneering a new category of quality assurance (QA) solutions — QA-as-a-Service.
Founding Story
Like many engineers, co-founders Jon Perl and Laura Cressman each experienced the stress of shipping high quality code. Previously, Jon was CTO at ZipDrug (acquired by IngenioRx) where a major aspect of his role was helping secure customers’ personal health information. As such, any bugs in the codebase could leak customers’ PHI and put patients at risk, while negatively impacting ZipDrug’s ability to do business. Similarly, Laura was the first engineer at Cityblock Health and whenever there was a bug in the code, it directly impacted whether or not Cityblock would be able to treat a patient. To ensure compliance and strong user experiences, Jon and Laura’s engineering teams were consistently running QA checks and implementing bug-bashes to ensure the code was production ready.
Tell Me More
Founded in 2019, QA Wolf’s goal is to give time back to engineers by providing a zero-effort, automated QA platform. The company helps engineering teams achieve 80% test coverage in four months, then provides ongoing maintenance, bug reporting, and 24 hour QA concierge to ensure each code release is ready to go. The product integrates with companies’ existing internal systems to give a replicated view of the end-user’s experience and software performance. Customers take QA Wolf’s engineers on a short tour of their app, which allows QA Wolf to build a testing plan and begin coding automated test suites.
As of July 2022, QA Wolf is 54 people strong, has saved engineering teams over one million hours of manual QA effort, and services over 50 customers including Gumroad, MakersPlace, and Possible Finance.
Timeout! What’s QA again?
In software, QA is a means of reviewing and auditing software products to verify that the software meets ideal user experience and legal compliance standards. In simpler terms, QA means reviewing code and testing for bugs to make sure the features work as intended, while adhering to predetermined project compliance. Additionally, as developers release code (i.e., add new code to their codebase), it can have unintended impacts on existing features, A.K.A. regressions. QA helps engineers spot regressions before they reach the customer.
Ideally, engineers run an automated test for every feature requirement at least once. To keep track of these automated tests, engineers document the number of tests executed for each feature requirement. To measure testing efficiency, teams calculate their “test coverage”, which is simply the number of features tested divided by the total features being released.
As a rule of thumb, a requirement is covered if there is at least one test executed against that requirement. Imagine there are 10 requirements and 100 tests are created for our code. If those tests were run against 8 requirements, test coverage would equal 80%.
Market Opportunity
Jon and Laura weren’t alone in thinking that there could be a better way to handle QA - you’ll see that there’s a lot of confusion around QA itself on Stack Overflow. Currently, there are few best practices and no gold standard for QA automation tools. It appears that every company is thinking about testing their code differently and only the biggest companies can afford to have full-time QA teams on staff.
For early-stage startups and smaller companies without dedicated QA teams, code reviews, end-to-end regression testing, and ensuring code functionality usually means taking time away from higher-value engineering activities. Internally, there is a constant tradeoff between hiring QA engineers to run the tests yourself vs. outsourcing QA to an expensive consulting firm. Given QA Wolf’s automation capabilities, companies have a single source of truth to hire engineers, automate testing, and outsource when needed, all for a lower cost than traditional measures.
However, even when companies hire “expert” consulting firms to help QA their code, the end product usually isn’t perfect because incentives are constantly misaligned. Consulting firms bill by the hour, not on a value-based metric like test coverage as QA Wolf does. Additionally, consulting firms have multiple customers running tests in parallel, and their clients fail to receive the attention they deserve.
When I spoke with Jon, he mentioned:
“The hardest thing about test coverage is maintaining test coverage. About two-thirds of companies have less than 50% test coverage and 90% of companies have under 75% test coverage.”
As technology has penetrated all aspects of life, QA’s impact on the software development process has historically been undervalued as ~not sexy~. Every business application, mobile game, and Snapchat filter goes through its own QA process to ensure usable features and efficient code. In the fullness of time, as software and technology continue to permeate throughout our work and personal lives, the underlying need for automated and efficient QA increases.
Furthermore, Statista found that organizations spent around 23% of their annual IT budgets on quality assurance and testing between 2012 and 2019. Despite the large allocation to QA, most companies hire low-quality QA professionals, or use SaaS solutions that have a high technical barrier to entry.
Why I Like QA Wolf
QA Wolf’s vision is to “become the ‘operating system for quality’ that companies use to improve the quality of their applications, beginning with automated end-to-end testing.” The product is a QA-as-a-Service application, enabling customers to achieve desired test coverage through automated testing and outsourced oversight. Unlike consulting firms, QA Wolf sets up technology to enhance test coverage and guarantees a result.
QA Wolf simplifies the complex challenges associated with building software today. Specifically the company’s automation product decreases the cost of testing software by consolidating infrastructure and human resources under one product.Additionally, the company aligns values between the client and QA Wolf by charging based on test coverage, not hourly like traditional consulting firms. Additionally, the team at QA Wolf has built a platform that makes it easy to keep track of tests for small startups and enterprise customers alike.
In terms of traction, QA Wolf, customers ranging from Y Combinator startups to multi-billion dollar enterprise companies run over 4,000+ automated tests every day, which equates to over 20 hours of manual testing per customer daily. Similarly, in the past twelve months, the team has scaled from 3 to 54 full-time employees spanning engineering, design, and marketing.
Lastly, unlike traditional consulting firms' staff augmentation offering, QA Wolf’s product enables the company to rapidly iterate on its offering. As the product matures, QA Wolf will continue to differentiate itself by building out-of-the-box datasets of tests across web apps which will allow them to develop new products and services on top of what the company already offers.
QA Wolf’s Hiring Corner
Similar Companies
Reminder
As always, you can find the abbreviated list of companies we’ve already talked about here. Below are links to the previous posts