Testing Blog
Post Release: Closing the loop
Tuesday, October 02, 2007
Posted by Michael Bachman
, Test Engineering Manager
A testing organization's job is not done with the release of a product. As the software development cycle does not end with the release of the product but has an extension into the post-release diagnostics and evaluation. Learning from post-release metrics like product performance, defects, and behavior after it is in production (or in the field) provides valuable input into how to adjust future testing and development techniques. Measuring product defect trends and performance, and analyzing those results, can identify holes in test coverage, prevent bugs, plug gaps in the release cycle or product life cycle, and determine if the pre-release test environment was adequately representative of key customer scenarios.
Here are a few metrics that can help jump-start this effort
Pre- versus post-production defect ratio: This metric measures the ratio of total number of defects found before production divided by the overall number of defects in the product (including the post-release issues). This lets a team measure how many defects are being caught before release. This effort supplements the age old valuable practice of partnering with Product Support to measure incidents/defect rate. The goal is not and should not be to indulge in the blame game of "A defect found after release is test's "fault," " but as a means to find ways to make the product release cycle better. The thing to focus on is to identify what were the causes for the issue - gaps in the release cycle, communication mishaps between product, development, and testing, inadequate test environments, or the overall testability of a product. There may not be a perfect metric, but obvious ones might be: time to resolutions (how long it takes to react to a broken issue), cost to the customer, or cost to customer support. The main point is to agree on a cross-organizational metric, track it, do the root cause analysis, and make the time to change.
Breakdown of defects by component or functional area: In conjunction with monitoring which defects are found in production, categorizing them by functional area and component provides the necessary information to highlight trends and, more specifically, problematic areas. When a problematic component is identified, the test team can fill holes in test coverage, unit test coverage, product usability issues or the life cycle managing that functional area. Also, trending defects by component over time has additional advantages like - this data provides a better sense of the quality of particular components as they age, also provides the information of how effective the changes(if any) that were introduced in the engineering practices resulting in better quality and finally measure of introduction of new functionality caused any de-stabilizing effects to the system. This metric will allow product teams in making informed decisions regarding the product. Potential outcomes are resources allocation changes, feature deprecation or redesign of the features.
Performance measurements (CPU usage, memory consumption, disk load, database load, latency, etc.): Without going into the various load and performance (L&P) measurements one can monitor within a product (since that can be a whole separate article in itself :) ) the product teams should ensure they have mechanisms to measure key and product relevant metrics can be collected. In order to gauge the effectiveness of the test environments these measurements need to obtained both in the test labs and production environments. Identifying these mismatches allows test organizations to correct any topology issues early and before any subsequent releases (or similar releases).
Furthermore analysis of these could expose multiple causes of why the product behavior was different in production than in test labs. Some examples (like we have found at Google) of these issues are : different machine hardware, load mismatch on the system, localized tests not measuring round-trip latency, the number of concurrent users hitting the product simultaneously (your testing team compared to the potential millions of users in production). It is important to remember that, the most important point here to measure and monitor the performance of the product as well as determine the adequacy of the test environment. When a performance issue is found in production, ask yourself "could we have caught that in the test environment?"
Are users or your monitoring systems finding the defects? Having reliable monitoring and debugging systems, logs, and notifications are key in reacting immediately to large defects in production, as well as potentially finding the defect before your users do. Some of the best practices of engineering teams (and those followed at Google) are real-time notifications of exceptions, load, and performance, pager alerts when systems are unavailable, as well as robust logs to help developers and testing debug the system state before and after a crash. There are many open source and commercial solutions to these pieces rather than building in-house solutions. The bulk of the effort in setting up reliable monitoring typically lies in development, but it is key for test teams to assist in identifying the need and also ensure they are utilized in their test environments. This not only allows the testing organization to test the functionality of the monitoring tools, but also alerts testing of defects that might have slipped through and are not directly visible on a front-end.
So, what does this get you? A solid picture of the product's quality and performance trend over time. Measuring lets testing tweak their coverage and environments, as well as analyze how the team works together. Reacting to the findings helps open communication channels between development, production, and testing, and lets them join together to debug and reproduce defects and eventually reduce defects. Every part of the larger team can watch defect trends; help prioritize resources and features, and better increase unit test, system test, and performance test coverage. Getting to a robust, real-time defect, performance, monitoring framework takes effort from all teams, but, in the end, everyone can reap the benefit, especially, and most important, your users.
9 comments
Labels
TotT
104
GTAC
61
James Whittaker
42
Misko Hevery
32
Code Health
31
Anthony Vallone
27
Patrick Copeland
23
Jobs
18
Andrew Trenk
13
C++
11
Patrik Höglund
8
JavaScript
7
Allen Hutchison
6
George Pirocanac
6
Zhanyong Wan
6
Harry Robinson
5
Java
5
Julian Harty
5
Adam Bender
4
Alberto Savoia
4
Ben Yu
4
Erik Kuefler
4
Philip Zembrod
4
Shyam Seshadri
4
Chrome
3
Dillon Bly
3
John Thomas
3
Lesley Katzen
3
Marc Kaplan
3
Markus Clermont
3
Max Kanat-Alexander
3
Sonal Shah
3
APIs
2
Abhishek Arya
2
Alan Myrvold
2
Alek Icev
2
Android
2
April Fools
2
Chaitali Narla
2
Chris Lewis
2
Chrome OS
2
Diego Salas
2
Dori Reuveni
2
Jason Arbon
2
Jochen Wuttke
2
Kostya Serebryany
2
Marc Eaddy
2
Marko Ivanković
2
Mobile
2
Oliver Chang
2
Simon Stewart
2
Stefan Kennedy
2
Test Flakiness
2
Titus Winters
2
Tony Voellm
2
WebRTC
2
Yiming Sun
2
Yvette Nameth
2
Zuri Kemp
2
Aaron Jacobs
1
Adam Porter
1
Adam Raider
1
Adel Saoud
1
Alan Faulkner
1
Alex Eagle
1
Amy Fu
1
Anantha Keesara
1
Antoine Picard
1
App Engine
1
Ari Shamash
1
Arif Sukoco
1
Benjamin Pick
1
Bob Nystrom
1
Bruce Leban
1
Carlos Arguelles
1
Carlos Israel Ortiz García
1
Cathal Weakliam
1
Christopher Semturs
1
Clay Murphy
1
Dagang Wei
1
Dan Maksimovich
1
Dan Shi
1
Dan Willemsen
1
Dave Chen
1
Dave Gladfelter
1
David Bendory
1
David Mandelberg
1
Derek Snyder
1
Diego Cavalcanti
1
Dmitry Vyukov
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Elliott Karpilovsky
1
Elliotte Rusty Harold
1
Espresso
1
Felipe Sodré
1
Francois Aube
1
Gene Volovich
1
Google+
1
Goran Petrovic
1
Goranka Bjedov
1
Hank Duan
1
Havard Rast Blok
1
Hongfei Ding
1
Jason Elbaum
1
Jason Huggins
1
Jay Han
1
Jeff Hoy
1
Jeff Listfield
1
Jessica Tomechak
1
Jim Reardon
1
Joe Allan Muharsky
1
Joel Hynoski
1
John Micco
1
John Penix
1
Jonathan Rockway
1
Jonathan Velasquez
1
Josh Armour
1
Julie Ralph
1
Kai Kent
1
Kanu Tewary
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Bourrillion
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Manjusha Parvathaneni
1
Marek Kiszkis
1
Marius Latinis
1
Mark Ivey
1
Mark Manley
1
Mark Striebeck
1
Matt Lowrie
1
Meredith Whittaker
1
Michael Bachman
1
Michael Klepikov
1
Mike Aizatsky
1
Mike Wacker
1
Mona El Mahdy
1
Noel Yap
1
Palak Bansal
1
Patricia Legaspi
1
Per Jacobsson
1
Peter Arrenbrecht
1
Peter Spragins
1
Phil Norman
1
Phil Rollet
1
Pooja Gupta
1
Project Showcase
1
Radoslav Vasilev
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Sam Lee
1
Sean Jordan
1
Sebastian Dörner
1
Sharon Zhou
1
Shiva Garg
1
Siddartha Janga
1
Simran Basi
1
Stan Chan
1
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Test Engineer
1
Tim Lyakhovetskiy
1
Tom O'Neill
1
Vojta Jína
1
automation
1
dead code
1
iOS
1
mutation testing
1
Archive
▼
2025
(1)
▼
Jan
(1)
Arrange Your Code to Communicate Data Flow
►
2024
(13)
►
Dec
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(1)
►
May
(3)
►
Apr
(3)
►
Mar
(1)
►
Feb
(1)
►
2023
(14)
►
Dec
(2)
►
Nov
(2)
►
Oct
(5)
►
Sep
(3)
►
Aug
(1)
►
Apr
(1)
►
2022
(2)
►
Feb
(2)
►
2021
(3)
►
Jun
(1)
►
Apr
(1)
►
Mar
(1)
►
2020
(8)
►
Dec
(2)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
May
(1)
►
2019
(4)
►
Dec
(1)
►
Nov
(1)
►
Jul
(1)
►
Jan
(1)
►
2018
(7)
►
Nov
(1)
►
Sep
(1)
►
Jul
(1)
►
Jun
(2)
►
May
(1)
►
Feb
(1)
►
2017
(17)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(2)
►
Jun
(2)
►
May
(3)
►
Apr
(2)
►
Feb
(1)
►
Jan
(2)
►
2016
(15)
►
Dec
(1)
►
Nov
(2)
►
Oct
(1)
►
Sep
(2)
►
Aug
(1)
►
Jun
(2)
►
May
(3)
►
Apr
(1)
►
Mar
(1)
►
Feb
(1)
►
2015
(14)
►
Dec
(1)
►
Nov
(1)
►
Oct
(2)
►
Aug
(1)
►
Jun
(1)
►
May
(2)
►
Apr
(2)
►
Mar
(1)
►
Feb
(1)
►
Jan
(2)
►
2014
(24)
►
Dec
(2)
►
Nov
(1)
►
Oct
(2)
►
Sep
(2)
►
Aug
(2)
►
Jul
(3)
►
Jun
(3)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Feb
(1)
►
Jan
(2)
►
2013
(16)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
Jun
(2)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Jan
(2)
►
2012
(11)
►
Dec
(1)
►
Nov
(2)
►
Oct
(3)
►
Sep
(1)
►
Aug
(4)
►
2011
(39)
►
Nov
(2)
►
Oct
(5)
►
Sep
(2)
►
Aug
(4)
►
Jul
(2)
►
Jun
(5)
►
May
(4)
►
Apr
(3)
►
Mar
(4)
►
Feb
(5)
►
Jan
(3)
►
2010
(37)
►
Dec
(3)
►
Nov
(3)
►
Oct
(4)
►
Sep
(8)
►
Aug
(3)
►
Jul
(3)
►
Jun
(2)
►
May
(2)
►
Apr
(3)
►
Mar
(3)
►
Feb
(2)
►
Jan
(1)
►
2009
(54)
►
Dec
(3)
►
Nov
(2)
►
Oct
(3)
►
Sep
(5)
►
Aug
(4)
►
Jul
(15)
►
Jun
(8)
►
May
(3)
►
Apr
(2)
►
Feb
(5)
►
Jan
(4)
►
2008
(75)
►
Dec
(6)
►
Nov
(8)
►
Oct
(9)
►
Sep
(8)
►
Aug
(9)
►
Jul
(9)
►
Jun
(6)
►
May
(6)
►
Apr
(4)
►
Mar
(4)
►
Feb
(4)
►
Jan
(2)
►
2007
(41)
►
Oct
(6)
►
Sep
(5)
►
Aug
(3)
►
Jul
(2)
►
Jun
(2)
►
May
(2)
►
Apr
(7)
►
Mar
(5)
►
Feb
(5)
►
Jan
(4)
Feed
Follow @googletesting
