MEX 2007 Report: Paul Nerger, Vice President of Worldwide Sales & Marketing, Argogroup

MEX CONFERENCE, LONDON — Paul Nerger, Vice President of Worldwide Sales & Marketing at Argogroup, spoke in response to the following manifesto point:

“User experience performance must be measured if it is to be improved. It must be constantly tracked through quantitative and qualitative methods. We think organizations throughout the value chain are failing to recognize the importance of understanding customers because quantifying the return on investment is too difficult”.

Paul started his talk by illustrating Argogroup’s services through case studies. One highlighted a “false event”, a momentary spike in service that results in temporary failures. Typical SMS-based false events include New Year Eve and the Super Bowl. So many people send New Year’s greetings to their family and friends at the stroke of midnight or to trash talk during the last few minutes of the Super Bowl that service spikes, resulting in high levels of data usage. When these patterns are not anticipated by carrier’s operations centers performance of the SMS service is drastically impacted, with message delivery unreasonably delayed. In some instances messages are not delivered at all.

Argogroup’s clients are mainly carriers for whom they measure the “end-to-end” experience. By this Paul means the key performance indicators (KPI) that directly impact the consumer. These can be measured according to data type:

• Voice: mean opinion scores. These are gathered by asking consumers to rate quality on a scale of 1 to 5. The average score represents the mean. A variation is used for video: video mean opinion scores (VMOS).
• SMS: delivery time between sender and recipient
• MMS: compound measure because MMS touches on so many systems.
• Ringers: device compatibility. The number one complaint about ringers is that it doesn’t work on the device used to purchase and download it.
• eCommerce transactions: end-to-end measures, eg. instances of overbilling.

Before identifying KPIs, a company needs to identify its key quality indicators (KQI) from which KPIs are derived. What is most important to your customers about the service or product you provide? What are the key metrics that could be used to produce quantitative and qualitative measures of your service’s or product’s performance? Paul stressed that companies need to “think like users but test like engineers”. The challenge is then how to re-engineer your business to translate the results obtained into actionable tasks that will have a positive impact on your service’s or product’s overall user experience.

In the breakout session following the presentation we were tasked with defining 5 key metrics for a mobile service and consider how these metrics could be used to justify increased investment in user experience. The results showed that many of the measures defined for web-based experiences extend into the mobile realm:

• Latency
• Cognitive load
• Time to complete
• Success/failure
• Satisfaction (predicted vs. perceived):
  • Perceived usability
  • Perceived time
  • Affinity

From the perspective of a user experience professional who advocates the use of user research in throughout the service or product lifecycle, the challenge is not in quantifying the return on investment but getting any research into the process. Research is time-consuming and may result in findings that require significant design, functional or business changes. Clearly it is easier not to include our customers than to deal appropriately with their feedback. But we do so at the risk of delivering a poor user experience and thus open the door to our more astute, customer-aware business rivals.