Notes on Kate Rutter’s Making the Most of Metrics

(From‘s All You Can Learn library)

Kate’s 3 languages:

  1. What are we trying to accomplish?
    • Words
  2. What does that look like?
    • Pictures
  3. How will we measure the work and track progress?
    • Numbers

UX expertise horizons

Screen Shot 2017-07-01 at 5.58.08 PM

“Your metrics will be as unique as your business” —Josh Porter


Number of downloads, average time on site, number of new sign ins etc are not the metrics we need

  • They might be early indicators
  • But they don’t actually answer the question of whether users find value in our product

These are the places we want to go looking for metrics for measurable value for our customers:

Screen Shot 2017-07-01 at 6.10.32 PM

Some metrics and analytics to consider:

  • Click tracking
  • Analytics packages
  • A/B testing
  • KPIs/OKRs
  • Usage stats
  • Conversion

Some frameworks:

PULSE metrics – mostly technical, a little old fashioned now

  • Page views
  • Uptime
  • Latency
  • Seven-day active users
  • Earnings

PIRATES metrics – esp good for start-ups

  • Acquisition
  • Activation
  • Retention
  • Referral
  • Revenue

HEART metrics – from Google research labs

  • Happiness
  • Engagement
  • Adoption
  • Retention
  • Task success

Retention is the one common metric in the previous two frameworks. It’s not enough to get traffic through the funnel— people need to use the product

Screen Shot 2017-07-01 at 6.22.40 PM

Read Lean Analytics for the best book on this, but the authors believe:

  • A good metric measures the usage of your product by a person

Criteria for a good metric:

  • Clear and specific
    • Know what interactions represent desired user behaviors
    • Identify the specific interactions that support that behavior or action
    • Find the numbers to track these interactions
  • Normalized (rate or ratio)
    • Percentages are inherently comparative
    • They balance out big changes in total quantities
  • Comparable
    • Captured at regular intervals
    • Directly comparable over a span of time
  • Actionable
  • Changes your behavior
    • “Don’t just ask questions. Know how the answers will change your  behavior”

Metrics vary in quality of information:

Screen Shot 2017-07-01 at 6.45.59 PM

Habitual usage is probably the most awesome things you can measure

So in this context:

  • What does the user need to do?
  • Where in the interface can they do that?

You can whiteboard the UX stack for this question:

Screen Shot 2017-07-01 at 6.51.55 PM

Everything above usage is about users, everything below is about product. Don’t get stuck too deeply on either side

Where the company purpose and uses intersect is your value proposition

The uses area is where thinking turns into practices and plans

“Your produce must be bridge between your customers and their dreams.”—Christina Wodke

For Facebook, here could be a sample stack:

Screen Shot 2017-07-01 at 6.58.08 PM

You can create these for multiple personas or scenarios, but all the teams must align on what the story of this stack is

Note your key use is at the Uses level.

  • This is the area where you need to define your features around, instrument around and build your metrics around
  • So ask, What can our customer do with our product that they can’t do without it?
  • Use a mad-lib style diagram to answer that question:

Screen Shot 2017-07-01 at 7.05.33 PM

Draw that use case as a simple picture:

Screen Shot 2017-07-01 at 7.08.25 PM

Ask, Who’s our user? Where do they go and what do they do? What do they think/feel/say?

  • Ideate multiple scenarios via 6-up

Screen Shot 2017-07-01 at 7.15.18 PM.png

Make sure to do this sketching as a team! Discuss and pick one winning scenario to take further as a SCAM artist

Then take that scenario and draw it out down to the interface level:

Screen Shot 2017-07-01 at 7.16.24 PM

Now add that with your key metrics to measure:

Screen Shot 2017-07-01 at 7.18.26 PM

Work with your product team to figure out what the key habituation moment is in your product and design to that. Samples:

Screen Shot 2017-07-01 at 7.22.10 PM

These habituation points (where users are more likely to continue use if they can get past this point) are leading indicators of your product’s long-term success

Metric systems

Funnels are one system

Screen Shot 2017-07-01 at 7.24.42 PM

Look to storyboards or hero stories our journey maps to find funnels

Then map the stories to the funnel:

Screen Shot 2017-07-01 at 7.28.46 PM

Measure these funnels over time


Instrumentation is setting up the tech to capture that numbers that matter. Data can be gathered:

  • Non-technical
  • Technical by hand
  • Instrumented with software

About Kate (from LinkedIn): Specialties: visual thinking, graphic recording and facilitation, sketchnoting, teaching/training, public speaking
Also skilled in Lean startup, entrepreneurship, UX for startups, 
Seasoned UX generalist: User experience strategy & design, information architecture, interaction design, design research, business strategy & just enough tech to be dangerous.






Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s