Transforming User Satisfaction

Project Overview

Background

Keller Williams (KW) is a leading global real estate franchise known for its innovative use of technology to empower agents and enhance client experiences. But at the time of this case study, KW was about 5 years into the journey to transition from real estate brokerage to real estate technology company. 

In this project, I launched Keller Williams' first user satisfaction survey program using Google's HEART framework to evaluate agent satisfaction with their products. This initiative had massive impacts, including a pause on new feature releases so the team could focus on improving the UX of the existing product.

My Role: Lead User Researcher

The Problem: Keller Williams had spent years focused on building as many features as they could to stand up their real estate technology platform, but this created technical and UX debt that was impacting user satisfaction. Complaints were raised through forums and meetings, but there was no systematic way for users to submit feedback about their product experiences. We needed a way to monitor the UX of each product in the ecosystem.

About Command

Command helps real estate agents and their supporting team members manage the day-to-day tasks that they need to do to work in and on their business. This includes managing a large database of current, future, and past clients (the lifeblood of a real estate agent’s business), intelligent and automated task management, marketing and lead generation, and goal setting and tracking. To learn more, see this page from KW: Explore Command

Objectives

• Evaluate the level of user satisfaction for each app within the Command interface

• Identify top pain points and user comments for each app

• Guide roadmaps to create the biggest improvements on user satisfaction

Research Strategy

I wanted to institute a product analytics system with a robust, holistic framework. For this, I chose Google’s HEART framework (Happiness, Engagement, Adoption, Retention, and Task Success). The goal was to implement HEART on all major applets within KW’s core product, Command, so we could compare the performance of each and create specialized action plans to improve them. The main component I was in charge of setting up was for Happiness: user satisfaction surveys.

Research Methods

To measure Happiness, I chose a simple 1-5 user satisfaction rating with a follow-up question for open-ended comments. I worked with the Data & Analytics team and Engineering team to implement the surveys across all major apps within the product. Agent satisfaction is incredibly vital at Keller Williams, so it was important to survey agents in the least disruptive way. These surveys would pop up for a random selection of users and ask them to rate their satisfaction with this product. I kept the surveys live until we had 1000-5000 responses per app.

Analysis

First I compared each app’s average rating, as well as the distribution of ratings (how many people rated each number 1-5). This gave us a general sense of which apps needed the most attention in general to improve the UX. 

The next step was analyzing the qualitative comments. Because the product users are real estate agents who are very invested in the success of KW and its products, there was high engagement on the follow-up question. I was able to analyze 200-300 qualitative responses for each app in the product. 

I identified themes and then categorized them into buckets:

• Feature Requests: Users directly or indirectly asking for new features

• Usability: Comments about difficulty, complexity, and inefficiency

• Stability: Freezing, crashing, bugs, error message, and slow loading

Analyzing the comments this way allowed me to 1) compare the spread of comment types across the apps, and 2) compare the average rating amongst people with certain types of comments. For example, users who left a comment related to stability or usability tended to give the app lower ratings than users who commented with feature requests. 

This analysis turned out to be extremely powerful in understanding how to best improve each app. We were able to use this to create roadmaps for each app which suited their specific problem areas.

Outcomes

The main overall finding from the Happiness survey was that, if Keller Williams wanted to make the biggest impact possible on user satisfaction for their products, they needed to flip their strategy - their priorities should be 1) improving stability, 2) improving usability, and 3) then considering more features - in that order.

After presenting these findings to executive leadership, a decision was made to pause all new feature development for several months while all teams focus solely on fixing the existing product. The insights from the theme analysis were pivotal inputs to the improvement plans for each app. This was a huge win for the teams who had been pushing for years to take some time to stabilize the product - they finally had the firepower to get leadership support.

One of the key apps in the platform, Opportunities, was the focus of a few follow-up studies after the Happiness surveys. As the main app for tracking leads and clients in the sales flow, Opportunities is integral to the daily work of real estate agents. I was able to conduct user testing and internal workshops to further explore how to improve this app. The findings of these follow-up studies combined with the themes identified in the user satisfaction comments were used to inform a product improvement roadmap. 

Around a year after the first Happiness survey analysis, we measured Happiness again. The results showed major improvements to the overall score and the distribution of ratings. The percent of people who rated Opportunities as a 5/5 more than doubled, from 26% to 57%!

Long Term Impacts

The implementation of Keller Williams' first user satisfaction survey program using Google's HEART framework yielded significant positive changes. By prioritizing stability and usability over new features, the team was able to make substantial improvements to the product, leading to a marked increase in user satisfaction. 

This initiative not only enhanced the user experience but also fostered advocacy for UX research and product analytics insights, establishing a regular cadence for measuring and improving the UX.