Project Managers: Identify Your Information Requirements and get the Reporting that You Need
One important, but frequently overlooked element of planning a project is agreeing on the content, format and frequency of status reports that a test manager will provide to the project manager. A project manager can prepare for these discussions with the test manager by assembling a toolkit of information about test status reporting – the contents of this toolkit include example test goals, a set of questions that provide insight into progress in accomplishing those goals, and a set of example test status reports. This article provides example test goals and questions, and gives additional guidance on assembling a set of example test reports.
Test Status Reports - Knowing What Information You Need is More Important Now
Early in my project management career I gave little thought to the specific information I would need to see in status reports from the test managers for my projects. I went from one project to the next with no specific requirements on test status reporting, simply accepting the reports that happened to be provided. Over time, I became comfortable with a relatively simple report of testing progress: one chart showing the progression of test execution statistics and a few graphs showing summary information about system/product defects. I acquired supplemental information about testing progress through weekly (or sometimes daily) discussions with the manager of the test team. This method served me well on a typical multi-year project where we seemed to have ample time to discover and then recover from any difficult situation that arose during the test execution window.
The world of projects is quite different now; there is more business urgency to complete projects on accelerated schedules, integrating a multitude of disparate systems has complicated the work of technology delivery, and the emergence of agile methods is pushing us to more frequent deliveries; as a result the project manager and project team must quickly detect and recover from problem situations during all phases of development, including testing.
Some may believe that success in such an environment is only possible by extending the working hours for every team member (“working harder”), but that is not sustainable. Rather, I think one enabler for success is improved consideration of the types of information that are needed to manage the project (“working smarter”).
In this article, we’ll focus on identifying the types of information a project manager must have in order to know that test activities are proceeding satisfactorily. This is a follow-up to my first article, entitled Creating a Project Manager's Toolkit forTest Reports, that advocates creation of a toolkit of information about testing and test reports. That article suggests that each project manager build a three-part toolkit that can be tailored for use on every project:
- Develop a framework of testing goals. Your framework of testing goals becomes the basis for your alignment discussions during planning sessions with the test manager.
- Enumerate the questions you need answered by a testting status report. Consider your own information needs, ensuring that these all relate to the agreed testing goals.
- Build your toolkit of expected test reports. This will be your own thoughts on the information and format of test reports that you want to see from a test team.
The rest of this article will help you in constructing your framework of testing goals and questions; I present an example set of goals and questions that are broadly applicable to most projects that have independent test teams. Feel free to use these as your own starting point for creating your own goals and questions that are relevant for your type of projects in your company’s culture and environment.
A Set of General Goals for Testing by an Independent Test Team
All too often, a project manager will enter into project planning with a singular focus on schedules – start date and end date negotiations seem to dominate most discussions. Unfortunately, this narrow focus is often with the (sometimes invalid) assumption that the project manager and project team members are all in alignment on many significant elements of the project. Rather than risk the possibility of a misalignment, a better approach to planning is to explicitly discuss and agree on project goals and the goals of each of the contributing teams. For testing, the way to do this is for the test manager and project manager to discuss and come to agreement on test goals, test requirements, test reporting and other facets of verification and validation.
So how can a project manager ensure that the planning discussions (with the test manager) go far beyond the topics of dates and schedules? I think the best technique is to start the planning discussions with a dialogue on testing goals. Without a common view of testing goals, your project could end up with a fabulous test effort that doesn’t contribute sufficiently to overall project success. By starting the planning discussions with a focus on testing goals and reaching agreement on those goals, you will have a foundation for subsequent planning and ongoing management of project activities.
To get you started, here is a basic framework for testing goals along with several general goals that are probably applicable to your project situation. Consider using this list as a starting point as you create the set of testing goals for your project – modify and supplement this framework and list with items are relevant for your projects.
General Set of Testing Goals
SCHEDULE PROGRESS. Ensuring that all test preparation and execution activities start and complete as agreed in the project and test plans. The specific schedule-related test goals are:
- Start and finish, as scheduled, the creation of test specifications.
- Start and finish, as scheduled, execution of the planned tests.
- Start and finish, as scheduled, verification of all problem resolutions.
- QUALITY. Ensuring that the system/application under test satisfies the specified requirements. The specific goals to be satisfied by the test team are:
- Ensure that test coverage exercises the complete scope, as defined for this testing phase, of the system being tested. Here’s what this means: if the system under test is required to perform in a certain way, then the executed tests will verify this correct operation – this includes all aspect of system operation including those in the areas of functionality, reliability, usability, efficiency, maintainability, portability (more on these areas is available from the ISO as standard 9126; an overview is available on Wikipedia).
- Provide timely test execution results that can be used to evaluate and improve the quality of the system being tested.
- At the conclusion of testing activities provide a clear, unambiguous statement on the quality of the system being tested and its fitness-for-purpose.
- Manage all test activities for a reasonable cost to the project.
Questions to Help Determine if Testing Goals are Being Achieved
The general goals we’ve just established for testing are to carry out testing activities as scheduled, ensure adequate quality of the system under test, and perform test activities efficiently. The next step we’ll take in constructing our toolkit is to determine the questions the project manager should be asking of the test team during the course of the project to understand if these goals will be achieved. In essence, the project manager will be examining historical factual data along with various forward-looking projections to determine if the project is on a trajectory to complete successfully.
For simplicity and consistency, when building our list of questions we will use the very same framework as defined for our testing goals. Most of these questions cover topics that the project manager will want to review frequently with the test manager, probably no less often than once a week; in some projects or in certain project situations you may need to review this information daily.
Questions to Assess if Testing Goals Are Being Achieved
- SCHEDULE PROGRESS. Verifying that we are on schedule for completing as planned.
- Relative to the schedule for preparing test specifications, how many new test specifications have been written (or updates made to existing test specifications) and are now ready for the test execution phase of testing?
- How many tests have been executed relative to the schedule for executing these tests?
- QUALITY. Verifying that system under test has sufficient quality.
- Are all testing requirements (e.g., use cases, user stories, enumerated requirements) verified by the executed tests?
- Of the tests that have been executed, how many have passed?
- How many trouble reports have been created as a result of executing tests?
- How many previously created trouble reports are still unresolved?
- As test execution progresses, are the ‘testing complete’ criteria trending towards satisfactory?
- At the conclusion of testing activities: How many test requirements were verified? How many tests passed? How many defects remain unresolved? Have the ‘testing complete criteria’ been satisfied?
- EFFICIENCY. Verifying that testing activities are being performed efficiently.
- How many people are writing tests? What is the productivity of test creation activities (e.g., tests created per person per day)?
- How many people are executing tests? What is the productivity of test execution activities (e.g., tests executed per person per day)?
As you look through this list of questions, see if you can identify any other questions (about progress and results of testing activities) that you find yourself commonly asking the test teams who are on your projects. As you come up with these additional questions, add them to your own version of these questions; as well review your list of goals to see if you’ve intuitively revealed some additional testing goals that are fundamental to your projects.
On Your Own: Preparing to Define the Reports You Need
Now you are almost ready to build your version of the testing reports you would like to see from the testing teams for your projects. Although it may seem like the easiest approach to this, don’t assume that an existing test report for one of your current projects will be sufficient for your information needs. Rather, start from the questions you have listed and create a set of reports that provide the information described by those questions.
As you create your reports, consider not only the specific information you’ll need but also give ample consideration of your options for effectively representing that information. I prefer graphical charts – a well constructed chart can quickly provide details about current status, convey information about historical trends, and give a perspective on future performance; all three aspects of status reporting are essential for a project manager who has responsibility for successfully delivering a project.
I have a few pointers to materials can show you the testing reports that others have created:
- Stephen Kan, from IBM, published a paper in 2001 on the topic of In-process metrics for software testing. This paper, which stirred my interest in the topic, was subsequently included as chapter 10 in his book Metrics and Models in Software Quality Engineering (2nd Edition) - this chapter is available as a sample on the informit.com web site.
- I recommend skimming through Agile Testing: How to Succeed in an Extreme Testing Environment. In this book you’ll find the perspectives of several different authors, ranging from those with decades of technology experience to a college student. This book may help rein back any inclinations you may have of measuring and reporting on absolutely everything having to do with testing.
- I spotted some very good reports in Chapter 9 of Managing the Testing Process.
What’s Next? Creating your own Toolkit.
This article has given you an example to use as a starting point in constructing your own Project Manager's Toolkit for Test Reports; with this information, you are well on the path for creating your version of testing goals and questions. These two items then can become the basis for creating your template of reports that identify the information you would like your testing teams to provide to you.