Rocket Mortgage

Time | 6 weeks

Role | Conversational UX Designer

Elevating Rocket Mortgage's Chatbot Engagement: A Design Transformation

Background
Rocket Mortgage's authenticated client chatbot handles a substantial volume of 60,000+ monthly conversations. Despite this, data indicated that clients were not fully engaging with the bot's FAQs, often mistaking it for a live agent. Redesigning the welcome message became imperative to rectify these issues. Following the redesign, Rocket witnessed a remarkable 17% increase in conversation retention, a 35% surge in text-to-chat engagement, an 8 point decrease in negative CSAT scores, and established a scalable design for streamlined updates.

Challenges Identified

Clients' limited engagement with the bot's comprehensive FAQs and mistaking the bot for a live agent posed significant challenges for the company. Customer satisfaction surveys revealed that users often felt constrained by selecting from quick reply options, overlooking the bot's full NLU capabilities, and mistaking the bot for a human. This led to a decline in success tracking metrics and customer satisfaction.

My Contribution

As the lead designer for this project, I spearheaded the research, testing, and implementation of the new welcome message. Collaborating closely with Quality Engineers I validated designs and rectified issues within IBM's Watson NLU engine, enhancing stability and functionality.

Approach and Process

A thorough analysis of client-bot conversations and surveys highlighted critical issues within the welcome message. Hypothesizing that an updated message would encourage clients to engage with the bot's NLU capabilities and minimize user confusion around the bot being a live agent, I designed a plan backed by multiple data points.

Project Roadmap

• Continued research and satisfaction survey analysis

• Design exploration for welcome message content and small talk dialog

• Internal design critiques and defining KPIs

• QA testing

• Production A/B testing for quick insights

Design Execution

Creating multiple welcome message designs and small talk dialogues, I sought approval from internal design teams and leadership. To enhance accessibility, I incorporated 'small talk' content for users urging them to ask the bot questions outside of the quick reply options, integrating an intent training dataset to cater a wider array of small talk questions.

Since I was also responsible for design implementation, I took to building and testing the new welcome experience in IBM Watson from my BotMock prototypes. In order to validate the success of these designs, I ran 5 day A/B test in production environment.

Results and Impact

Upon receiving very positive test results, I switched over all traffic to the new experience. After full implementation, in just a few weeks we saw results that were striking: a 17% increase in conversation retention, a 35% surge in text-to-chat engagement, and an 8 point decrease in negative customer satisfaction scores. The swift rollout of the updated welcome message to all authenticated guests further solidified these gains.

Conclusion

The redesigned welcome experience not only addressed initial engagement issues but also facilitated advanced NLU support, evidenced by increased small talk queries. Rocket Mortgage now boasts an improved, multi-state design welcome experience, significantly enhancing client conversational experiences with their self-service chatbot. This project underscores the power of strategic design interventions in driving meaningful improvements in user engagement and satisfaction.