From the article What Testing Won’t Tell You, by Lynsey Thornton at uxbooth.com, March 25, 2014. Go to the link to see the process in detail.
The Shopify team was having trouble digging through existing analytics on their onboarding process. So they decided to take a research cocktail approach:
- Diary studies. They recruited 12 actual users (assuming 4 would drop out) during the Shopify signup process to keep diaries for 2 weeks. Shopify kept the diary template short so that users would only spend a couple of minutes per day with it, being sure to ask questions that provoked an emotional response such as “Did you make the progress you wanted today,” “How are you feeling,” “Why did you end your session” etc.
- Phone interviews. They occasionally emailed participants to schedule feedback calls, booking appointments via youcanbook.me. At the end of the 2 weeks, they had a summative call, thanked the users for participating and gave each one a $100 voucher.
- Basic analytics. The team spent about 10 minutes per day reading the diaries and pulling key quotes or keywords. This helped them identify patterns earlier and stopped the final analysis from seeming too daunting and time-consuming. They color coded data types for quick assessments. The team realized which tasks were tiresome and found ways to make them more efficient.
The team felt many of their findings would never have shown up via customer support, as users would think they were too small to report or “just me”s–but it was a goldmine for the team!