Problem Statement & Goals
problem statement
This was part of a larger initiative to reduce call center volume while working at Best Buy. At the time of this project, not much was known about why and how users interact with conversational AI, especially in an ecommerce setting. Conversational design was approached with a similar method as content design and users were openly expressing their dissatisfaction with the feature.
Goal
Leverage an exploratory and generative approach to uncover the foundation of what the customer experience is like while engaging with Conversational AI. Validate assumptions of the customer experience and surface further opportunities for more defined research to iterate on.
Research Process & Tools
1) Team alignment & research project planning
2) execute
1:1 Interviews
dScout
3) Analysis and synthesis
Inductive & Deductive Coding
Miro
4) craft the deliverable
Microsoft PowerPoint
5) deliver
Team Structure
Sr. Product Manager
Product Manager
Lead Researcher (My role)
Research Sr. Manager
Lead UX Designer
UX Designer
Content Designer
Team Alignment and Research Project Planning
Stakeholder and subject matter expert interviews
When starting off with such a broad subject where I don’t consider myself an expert, my first move is to absorb as much information from those around me as possible. My goal was to:
Assess what we currently know about conversational AI users (secondary research, existing internal research, web analytics, etc.)
Identify stakeholder assumptions
Determine driver for the research request (what prompted this request and what primary questions exist?)
Gain a better idea of where this research can make an impact.
This is one of many inputs for scoping the project and helps determine the format of deliverable that can best assist the team.
Project Plan Documentation and gaining stakeholder buy-in
Once I completed a Research Brief I shared it with teammates and allowed time for feedback before moving on. The Research Brief document consisted of basic project information such as:
Background
Project Objectives & Key Questions
Project Outcomes
Approach
Participant Sampling Strategy
Estimated Timeline
Execute
participant sampling strategy
12 consumers distributed into groups based on:
frequently choose to use chat to interact with retail companies.
have used chat but usually choose other methods to interact with retail companies.
are aware of chat but don’t choose to use it.
Other factors were collected to ensure inclusivity and validity of the participant sample such as:
Geographic location
Education level
Employment status
Living Situation
Household Income
Disability status
schedule interviews
Interview tool: dScout
Remote moderated 1:1 interviews
60 minutes interviews
No more than 4 interviews per day (ideally 2-3)
Minimum 30 minute breaks in between interviews
interview Script
Below, you can see a portion of the interview script that I developed. When writing interview scripts, I like to use bullet points and try to flesh out the possible paths the conversation could go down. Not every question is asked in every interview, and while I make sure to guide the conversation to stay within scope, I try to make sure the participant feels in control of their own responses.
note-taking & Stakeholder participation
While I conducted the interviews, I invited stakeholders to watch and take notes. Additionally, once interviews were completed I went back and rewatched recordings to take my own notes.
Analysis & Synthesis
data dump, inductive coding & deductive coding
All notes were combined into one Miro board.
Inductive coding was used to analyze attitudinal data.
Deductive coding was used to analyze behavioral data since we had previous data on customer behavior that we could use to sort information.
Craft the Deliverable
Themes
As themes began to emerge within the Miro board I would often start to map them out on my note pad as I was making sense of how they fit into the larger picture.
DELIVERABLE creation
In order to convey a story filled with dense information, I found it most appropriate to collaborate with my design partners involved in the project in order to allow visuals and metaphors to carry some of the weight in communicating the message.
Findings
Primary advantages of using chat compared to other methods include: Availability, Speed, and the ability to Multitask.
Customer expectations are generally low and typically formed based on their previous experiences engaging with a chat bot.
Users will evaluate their experiences with a chat bot primarily based on task success. The manner in which resolution is achieved remains a vital element of a quality experience.
Conversational AI users expect to be heard, hope to be understood, aspire to feel valued.
Deliver
shareout #1
The first presentation included the project team and direct stakeholders. After presenting the deliverable, I left time for follow up questions and feedback on the deck.
shareout #2
For the second presentation, I opened up the invitation to all associated product teams, design teams, researchers and leaders. In total, the meeting consisted of ~80 individuals.
distribute & Archive
Once the information had been presented twice, I emailed the deck to all internal stakeholders and allowed for company-wide distribution. Additionally, the deck was added to our research repository so that it could be accessed and referenced in the future.
Feedback
“I just heard about the glowing feedback about your research on Conversational AI Platform! I see this as one of the foundational research studies that truly crosses over app and dotcom. I am so proud of how you have grown this past year. You have shown great attitude for learning, growth, ability to take feedback, and independently take on big responsibilities with optimism!”
~Jen (Director, Experience Research)
“I want to give a VERY BIG shoutout to Mitch for his excellent work on the CAIP project. This has been something the App team has wanted for a while, and not only did he manage the whole thing entirely on his own, but he was able to help coordinate and bring the Web team along for the journey too. He also collaborated with his design partners and together they put together an amazing deck full of insights and great stories. Definitely ask him for a copy, and excellent work Mitch! Couldn’t be more proud of this huge accomplishment!”
~Jon (Sr. Manager, Experience Research)