Session Based Test Management Advice

One of the challenges of a blog on consulting is the difficulty of showing what actual consulting looks like. From time to time, though, I receive a consulting request by email which can be answered with a brief email. Brent Plumley recently sent me a request and had given me permission to blog it and my reply, so others may share, and comment.

Brent's Situation

I am currently researching methods to improve our (Session Based Test Management) SBTM debrief/review process in an attempt to make the process more efficient and scalable. Our current process has the Test Manager reviewing 100% of the testing sessions. As the testing team grows, the test manager no longer has the capacity to review all the sessions and as a result a significant backlog session list has accumulated.


- I have several areas that I was hoping you would be able to provide some insight

(1) Testing Session Debrief Trade off

- When a test session is reviewed we consider the following and attempt to achieve a 10/10 score on each.

     (a) Quality of testing performed

          - Did all the risk areas get covered

          - Were the correct test techniques implemented

          - Are there anything missing(different combinations, order of events etc...)


     (b) Quality of Testing Notes

          - Do the notes clearly outline the testing

          - Is the testing reproducible using the notes

          - Does the test properly convey the thoughts and observations

          - etc...



- As a result, the time required to perform the debriefs is quite long.



QUESTION: When performing a session debriefs should we consider a trade off between 'time', 'Quality of Testing Performed' and 'Quality of Testing Notes'. (ie. spent less time, ensure 10/10 on 'quality of testing performed' and 6/10 on 'quality of notes') or, do you think that both (a) and (b) above are important?

(2) Debrief Process

- Even if we can reduce the time spent on session reviews, as the team continues to grow, we will once again be faced with the problem of scalability (more testers = more test sessions = more time needed to review)



- We are currently considering several ways to address the scalability issue


(1) Test Sessions are reviewed by other team members (peer-review process). This will help relieve the strain on the Test Manager of having to review all test session. Also, this will provide opportunity to all test team members to learn from each others unique styles.


ANSWER: This is what I usually suggest to clients in similar situations. The number one job of any manager is to develop the people who work under their mentorship. This is an excellent way to do so.


You can introduce this leadership training opportunity incrementally, with the early leads being supervised by the manager until she finds each person prepared to lead on their own.



(2) Test Manager continues to review the test sessions but depending on the experience level of the tester, only a certain percentage of test sessions will be reviewed (ex: 100% of sessions are reviewed for junior/New testers, and 15% of test sessions for more experienced testers)


ANSWER: Absolutely not. This simply encourages people to game the system by not having their test sessions reviewed. DO NOT DO THIS.


- I have identified several problems with both of the above options. For both options, the test manager may feel out of the loop as much of the testing is not reviewed by them. Ultimately it is the test manager that is responsible for signing off on the testing that is performed. If the session notes are reviewed by peers, or not reviewed at all (only 15% of experienced testers sessions are reviewed), the test manager personally cannot guarantee perfect coverage of testing.


ANSWER: Managers must know how to delegate. If you cannot delegate, you should not be a manager.


Option 2, however, is not a matter of delegating, but of abdicating.


Again, DO NOT DO THIS.


- We are currently leading towards the Peer-Review process. This will ensure that all test sessions are reviewed by someone, which will help guarantee proper test coverage.


ANSWER: I totally agree.


QUESTION: Do you know of any other debrief processes which have been successful for other companies?


ANSWER: Not successful ones.


QUESTION: What method would you suggest for keeping the test manager informed and in the loop? We have thought of bi-weekly team meetings where key points and high level outline of testing performed is presented. Of another option, the test manager will debrief a small sample of the test sessions to get a measure of the quality of testing and debriefing performed by their team.


ANSWER: Forget the sampling. Each person who leads a review should fill out a simple report that everyone, including the test manager, can see. (For more on such reports, see my book, The Handbook of Walkthroughs, Inspections, and Technical Reviews.)


I hope this helps, Brent. It represents half a century of observations on my clients, what works and what doesn't work.



My blog readers, of course, know that no advice works in every situation, though certain mistakes seem to repeat themselves with every new generation. Brent responded by telling me the advice was helpful to him, but let me know how this advice works for you. And ask for answers and clarification.