What does a Platform Quality Assurance Engineer Do?

DevLab is in search of a person to focus on and own the quality of our platform. Agile teams share ownership of the quality of their products. It is challenging to be an expert software engineer and an expert tester. The platform is broad and deep for only the engineers to think through all the testing required. I am writing this article to convey the attributes of a platform QA Engineer.

A platform QA engineer is different from other QA Engineers because they focus on a generalized and commoditized product. Platforms do not typically contain business rules and business logic inherent to a domain. This platform is a machine that delivers a product to build on. The hard part of a platform is maintaining quality as the platform changes over time. Platform QAs perform work to sustain the overall system integrity. A QA on this team would describe themselves with these words: self-directed, autonomous, empowered, comfortable with being uncomfortable, a coach, and a critical thinker.

Communication

Reporting and Communicating Risk

If you are reporting a bug, please do not make me think. Tell me the story of why it is a problem and convey how I should prioritize it. Here is an example of ineffective communication about risk. 80% of test cases passed related to the eventing feature. Uhhhh, what should I do about it? Were they critical test cases? Do I need to investigate further? Who do I need to talk to to fix the problem? Do you see that is not clear or helpful to a business owner? What I want to see is the narrative format. Our eventing sidecar has a problem with dropping messages in a common scenario. Our users are sensitive to this type of issue. I recommend we defer the next release until we can resolve this problem.

  • Problem
  • Severity
  • Possible next step

Stakeholders need information on why the schedule is changing. The product manager sets those expectations.

Debugging/Triaging/Replicating User Issues

Perception of platform stability is key to the success of any platform. Other products might be able to get away with varying levels of stability. In 2021, stability and resiliency are table stakes for any product. The best way to maintain a good perception of your platform stability is to jump on issues immediately as they occur and triage them. Your users want to know that you are aware of the problem. They want to know you have a plan. If there is an estimated time to fix that would be great too. That is not always possible. The value that a QA can add is finding out the scenario that led to the end-user problem. It might be too technical. If the QA can gather the details and present the results it will save everyone a lot of time. If they can also replicate the scenario that occurred, that is awesome.

Critical thinking

Systems Thinking

A platform QA engineer is a systems thinker. Testing a story card for a feature request is not limited to only that feature in isolation. They look at how it fits into the entire system. Are there emergent properties of this feature? What unintended effects could occur due to the addition of this feature? What scenarios could happen that might feel unexpected by a user? Discussion between the product manager, software architect, engineers, and QA can lead to a better product. The fun part of the job is thinking through the possibilities and demonstrating your intellect to find gaps.

Test Coach

Your teammates want to learn from someone that understands quality and looking at the system as a whole. Their day-to-day is not focused on thinking in this mindset. They want to learn and grow in this area. Due to their duties, it is not a focus for them. This person has the opportunity to teach and coach the rest of the team on the testing mindset and lens. This coaching could entail a variety of testing methods. You can use heuristics, test plans, test strategy, exploratory testing, and more.

Owning User Journeys

Devising the Test Strategy

I like this article by Nils Balkow-Tychsen, Testing in Microservice Architectures. It describes the type of test strategy I am looking for in my platform product. I want a test strategy that is cost-effective and efficient. As a new tester, you might think that end-to-end tests for every scenario are the solution. You might learn that end-to-end tests are painful to maintain.

Your goal is broad maintainable test coverage of the product. How you apply different types of testing is up to you. Communicate the why around the test strategy. You will save your colleagues time and waste by explaining what parts of the product are harmless and what parts are risky. Explain how to test components inexpensively and effectively.

Test Data/Test Harness/Test Framework/Test Case Administration

These components keep a product functional and understandable by a team. Automation is not the be-all-end-all of testing. Some scenarios are so tedious that you might as well automate them. Automation frees you up to focus your energy on other tasks. Yet, automation is another thing to maintain. I expect a QA to review tests ala the Marie Kondo “Spark Joy” method. Retire tests no longer deliver value.

  • Test Data
    • What is valid and invalid data to use?
    • What features need what kind of setup?
    • How are we not polluting production with test data?
  • Test Harness
    • Everything that might make various testing activities easier for the team.
  • Test Framework
    • What types of programs make testing systems easier?
    • Wrappers of several APIs to create a test scenario and dispose of the test case when done.
  • Test Case Administration
    • What user journeys are core?
    • What are ancillary?
    • How might we know when a new feature might risk system stability?

Conclusion

This seems like a lot to ask for in a person, right? It is unlikely that one person contains all these attributes. It is more likely that someone will step in and have a few characteristics and grow into the other responsibilities. Or share the load with another QA colleague. The point of this article is to describe the value a platform QA Engineer adds to a product. Systems and frameworks are becoming more resilient and are baking in safer defaults. This will be challenging to testers accustomed to pointing out obvious problems. Grow your skillset to add more value to your products.

  • Think critically about how new features impact the system.
  • Communicate the risk of changes to the business to prioritize.
  • Own the testing strategy and framework of testing to maintain a quality product.

Are you a platform QA engineer? Are you interested in testing platforms? What do you think sets apart someone that tests a platform?