Case Study Banner

Project Summary

One of Amplitude’s main use cases is monitoring core metrics and understanding their trends. Today, there are multiple ways users seek to understand changes in their core metrics within Amplitude.  

The current ways our users monitor metrics and decide if a change is meaningful are very unscientific and full of noise, leading to wasted resources in investigating these fluctuations.

Design
Product Management
User Research
Data Analytics
Machine Learning

Project Details

Role :

Design Lead

Duration :

April - July 2020 (4 months)

Tools :

Figma, JIRA, Amplitude,

Teammates

Case Team Image

Me!

Sr Product Designer
Case Team Image

Andrea

Sr PM
Case Team Image

Kelson

Sr Software Eng
Case Team Image

Mike

Sr Software Eng

Growth

Problem Icon

So how does Amplitude's growth team fit into the rest of the product org?

Imagine growth as the resource utilized to test and validate concepts quickly and iteratively before investing resources full-time in those problem areas.

Growth assisted in meaningful discoveries around risk versus reward in various product teams' bets, before having to commit those to a roadmap.

Solution Image

Problem

Problem Icon
How do we reduce noise in monitoring and understanding users’ core metrics
by highlighting statistically significant deviations in historical data and forecasting potential ones. Users need to see when their data deviates from their expected query results.
Problem Tab Image

Solution

Problem Icon

Users need to see and understand the thing, apply and do the thing, and interpret the thing. That means:

  • Automating user settings: smart defaults
  • Layering in more data without confusion
  • Reducing noise in a loud environment

Our solution offers value to all of Amplitude's personas while focusing on the optimal solution for our project's target persona. We built a tool that empowers users to uncover meaningful fluctuations quickly without reliance on others to complete their jobs to be done. The design also creates trust between the user and how Amplitude's model applies parameters in order to produce accurate results.

Solution Image

Project Kickoff

This stage helps us orient on a path forward with our projects. We solidify confidence in the path we're building to paint an end-to-end picture for optimal feature success. We hypothesized outcomes that might derive directly from the impact features we built to help stakeholders understand the value customers see with the tool.

Features That Scale

Research Image

Anomaly Detection

Product teams should have the ability to detect anomalies in their data. This is the base-level MVP for this project.

Research Image

Monitor Changes

Simultaneously monitor all parent metrics and subgroups without needing to define individual monitors for each permutation.

Research Image

Distinguishable Cause

Users should be able to quickly distinguish what caused an anomaly whether it be instrumentation or user behavior change.

Desired Outcomes

Research Image

Building the Rocket

We laid the foundation for our Analytics team to continue to build upon to maximize the value customers gain from the feature.

Research Image

Product Utilization

A primary use case for sharing within Amplitude is when users find something "insightful". We hypothesized that users who identified anomalies would share frequently.

Research Image

Persona Inclusion

Amplitude isn't just for our data scientists. We want to enable product teams to detect anomalies as they occur, while utilizing fewer resources to gain those insights

Measuring Success

Research Image

Feature Adoption

% of weekly active users that have a “chart compute” event with the anomaly feature turned on.

Research Image

2 Week Retention

The % of users utilizing the feature in week 2. Our proxy was our "compare to past" feature, with a 24% 2-week retention.

Research Image

Sharing Insights

Broadcasted learnings are the number of notebooks and dashboards shared within an org between users. our target was 100 weekly BL.

Research and Discovery

An overview of the UX process applied to this project. Informing on what we started with, who we targeted, how we approached research, what we discovered, and the outcome. There are no specialized roles at Amplitude for a user researcher, so this process is part of the product designer's role expectations.

Story Image
Story Image

Which Amplitude Persona is Our Target Persona?

Initial research done prior to Growth's ownership of this project revealed that "pioneers" were our target personas.

Data scientists can typically spot outliers or anomalies at a high-level glance due to constant involvement in the data, and the role of the aspiring pioneer is out of scope for targeting these types of fluctuations within their roles.

Product managers would benefit most from this enablement, allowing them to utilize fewer resources, and spot statistically significant changes quicker.

User Interviews + Usability Testing

Our discovery phase took 2 weeks. Once we moved into validating design iterations through usability testing, that added another 2-3 weeks of calls, iterating, and re-testing. The research was not linear and was spread out over the course of 3 months.

We held 30+ customer call sessions total (combining discovery calls and usability testing), across 17 organizations across all segment verticals and sizes. These included customers such as Peloton, T-Mobile, Hubspot, and Mozilla to name a few. All calls were conducted via Zoom.

We initially identified Amplitude users who fit our research criteria by analyzing which users engaged with the "compare to past" feature in the last 90 days. Additionally, we received a list of customers who had previously expressed interest in a potential "anomalies" feature via the ticketing system or through CSM's.

Plan Image

Themes Uncovered

The prototype we used in testing revealed three main feature asks that were grouped into high-level themes.

Using the feedback provided, we broke down those key pieces of feedback into themes. This created a bigger picture for our customer asks, not just focusing on the specifics of feature desires mentioned in each call, but what the root of those asks is to holistically apply those concepts throughout the feature.

Research Image

Clear Visuals

With early iterations, we weren’t being clear enough with differentiating future vs past data when forecasting as well as partial data versus expected value.

Research Image

User Friendly Language

The nature of the tool was confusing and complex. we needed our UX writing to speak to users, not robots.

Research Image

Build Trust

The most common inquiry was about how to trust what they’re seeing and understanding which parameters were best to utilize for their insights.

Plan Image

Iterations on Themes

During the testing and iterating phase, we presented a few different ideas with live prototypes in our dev environment for our cohorts to engage with. This helped scope down design iterations by highlighting the trade-offs in each as told through the users' interaction.

It was important to see the "a-ha" moment happen in real-time. Using our north star for the project, we knew we had to create a strong foundation for this tool in order to create a case for prioritizing the continuation of feature builds for added value. If users can't find the feature, understand what it's doing, or trust the model, it will fail to get the adoption and engagement necessary to deem it worthy of dedicating engineering resources to it.

Concept Image

1. How to Achieve Consistency

We needed to ensure we remained consistent with our design system, chart patterns, and any other tools that have a similar behavior expectancy with state changes.

02. Visualizing a State Change

The UI in Amplitude is complicated and at times cluttered. The design didn’t have existing patterns that represented state changes while simultaneously layering more data in. We needed to nail representing clarity behind the behavior of the tool.

Concept Image
Concept Image

03. Build Trust

Users expressed their biggest concern was trusting the tool. They expressed that to adopt it, they needed to trust that the results are accurate, as well as be able to communicate to their stakeholders how they uncovered that output. It raised the question with our customers "How can I trust this tool more than my Data Scientist?"

Finalized Product: The Design Decisions

Consistency, Trust, and State Changes

Story Image

So how did we create an intuitive experience for users with the expected behavior of the feature, and the interaction with the button itself?

We landed on a button that functions as a toggle, rather than an actual toggle. Besides adding a component to our product that doesn't currently exist, it would also create confusion and inconsistencies with our design patterns throughout Amplitude. Color and tag affordances were used to signify the state of the chart.

Selecting it turns it on, automatically defining a default mode. This displays the smart default tags that the mode is applying to the chart. Users can still edit global settings easily through the button or by selecting the tags.

Automation: Smart Defaults to Build Trust

Story Image

Throughout the research, one thing remained consistent: users loved smart defaults.

We used smart defaults to represent the parameters being applied to the chart through the associated “modes”. This clarified confusion around users trusting their selection and how each setting might be perceived (or guessed) "best".

For forecasting the tag defaults to an empty state but provides access to layer in added complexity, without having to open the settings. This was done to preserve the discoverability of the feature.

Why We Went With "Modes"

Story Image

One of the biggest pain points in the tool was how users formed an understanding of what “model” we apply to the chart and how results are being calculated. Each user had a different guess as to what information the model might apply in order to create an output.

Some users thought the standard might be 60 days of data. For others, they thought it meant the confidence bands applied needed to be set to a higher %.

Using modes allowed us to encompass multiple personas while simultaneously reducing confusion for the lesser experienced, allowing flexibility for the more experienced, and building trust in the model for all through transparency and best practice.

Clear Visuals: Reducing Noise

Story Image

We landed on a hovering effect for each segment (line) in order to offer the ability to analyze anomalous behaviors between several related metrics. Up to 10 segments can be applied by default, which is the max amount of information to be displayed before the design became compromised (very messy!)

Hovering solved the problem of needing to show more context on the chart all at once with the confidence bands and forecasting parameters, while also allowing users to investigate and isolate anomalies easily.

I was able to align the design team to agree on reserving one color from the design system to be used only in the case of displaying an anomalous data point to avoid further confusion amongst all the noise and "blue" sea of features in Amplitude.

What's Next

Research Image

Measure Success with Beta

Launch and measure! Building machines takes a while, so it could be quite a bit of time before you see success metrics listed here.

Research Image

Real Time Alerts

Alert users as anomalies happen in real-time. Any anomaly from more than 1 day ago is old news that makes this capability less useful.

Research Image

Root Cause Analysis

Quickly answer users’ first question when they see an anomaly: is this an instrumentation issue or user behavior change?

Reflecting

The high-level objective of anomalies + forecasting was to set the Analytics team up for success through a strong foundation built early and quickly during an uncertain time (covid 2020).

Keeping our process focused meant a lot of questions that surrounded key parts of the original customer feature requests that were not within our scope. This led to time and resources being spent on figuring out ways around the constraints, as it was difficult to pull insights from customers without guiding questions, for them to pinpoint the value of this feature, without being able to tell what caused the anomaly.

The project's highlight was working collaboratively with my teammates on a machine learning project, which was a first for me! I was able to learn more about how we can utilize data to prevent future failures, and how our previous patterns can inform our decisions in a meaningful way.