Build A Platform For Researchers And Recruiters To Screen Suitable Candidates For Usability Testing Study In Real Time
OVERVIEW
CONCEPT
UserScout is a chatbot that helps researchers and recruiters in screening out the right candidates for a usability testing study.
INTRODUCTION
UserScout is a bot where companies can customize screener questions for their study. It accepts user responses in real-time to filter out the best candidates for usability testing interviews. Once the users have been selected, it schedules user testing interviews for them, easing the recruiting process. We have created a hypothetical scenario, where Spotify users would be recruited to test the “Hey Spotify” feature (voice assistive feature by Spotify) for listening to music while driving.
DURATION
15 weeks,
Teamwork
SKILLS
UX Research, UX Design,Conversational Design,
Visual Design, Prototype,
UX Testing
MY ROLE
UX Designer
UX Researcher
TOOLS
Miro,
Figma,
Voiceflow,
Illustrator,
Photoshop
ABOUT THE TEAM
The UserScout Conversational UX Design project is a collaborative effort by a group at Center for Digital Experience at Pratt Institute. Team members: Jichen Zhu, Ruhee Shah and Samika Rastogi. We completed the whole process of cooperation from initial research, program planning, to conversational design, prototype design and user testing.
CHALLENGE FACED BY RESEARCHERS
How might we provide researchers and recruiters with a convenient and fast flow of screening out right candidates? The following are the main problems we are facing.
BENEFITS OF A SCREENER BOT
Based on these insights, we further identified some of the benefits a screener bot could provide the users in finding the right fit, extracting key insights from the screener, and making the process more time-efficient.
PRODUCT DETAILS (*Take the example of embedding the Spotify platform)
RESEARCH
INITIAL RESEARCH
Our initial concept was to create a voice bot for screening the right participants for an usability study. With the help of research we formed few assumptions about using the voice bot for our screener.
1. Reaching a wider audience
A. Targeting audience with limited internet access
B. Targeting less tech-savvy users who are not too familiar with surveys or google forms
2. Presenting the screening questions in a conversation
A. Could be easier for users to follow-up with the questions via voice instead of text
WHAT DOES A VOICE BOT BRING TO THE TABLE?
Based on our research, we then developed a survey and a questionnaire to ask questions about our concept.
SUBJECT EXPERT INTERVIEWS
“I prefer a written survey. Usually it’s enough to gather the people we need.”
Preferred Method For Screening
Usually via email messages with links to a screening questionnaire, or through a user testing site like usertesting.com or optimal.
Preferred Method For Scheduling
Via email, a calendar or tools like Calendly that is set up to accept certain time slot selections from the participants.
CHALLENGING IN THE PARTICIPANT RECRUITMENT PROCESS
WHAT UX RESEARCHERS THINK
Google FORM SURVEY
Looking at our initial survey results, 60% of users were not comfortable with being contacted by an automated voice call. While 50% of the users wanted to complete the screening process using a chatbot. Few users also felt intimidated about being contacted by a random voice call and usually didn't trust such a bot for sharing information.
PERSONALITY
INTERACTION GOAL
LOW LEVEL OF PERSONIFICATION
Task-oriented, infrequent interaction: Simple impersonal task which requires no relationship or bond between the system and the user. Collecting users’ data, anonymously, isn’t necessarily a personal task.
Higher level of personification = more human-like behavior: A voice call doesn’t have a backstory as that may be a potential distraction in a task-oriented scenario.
In this scenario, the system builds trust through three key factors: Efficiency, transparency and consistency. Since the communication involved is only collecting information from the user.
POWER DYNAMICS
What power does each party have?
System: Asking screening questions
User: Only providing relevant answers to the system’s question
How intimate does this relationship need to be?
Low level of personification + one-time interaction, hence no intimacy is required.
The interaction is very task oriented where the system has very specific questions for the users and expects specific answers.
How will their relationship change over time?
No impact on the relationship in the long-term since this is a one-time interaction based on efficiency, as well as reliability provided from the system’s end.
CHARACTER TRAITS
The personality traits that this IVR system would have been dutiful, trusting, reliable, organized, task focused. With these traits the users will feel comfortable answering the screening questions and not being judged.
TONE
SAMPLE SCRIPT
AUTOMATED VOICE CALL
**A pop out message asks if you want to participate in a user testing on Spotify. (Voice call/Text screening option)**
TRAINING DATA
INITIAL FLOW DIAGRAM
SCREENER QUESTIONS FLOW
VOICE BOT PROTOTYPE
USABILITY TESTING
WIZARD OF OZ
Prototype: We used the Wizard of Oz prototype. During an automated voice call, we will act as the bot while the participant is the user.
Methodology: We recruited 6 participants by asking around who uses Spotify as our bot is a screener for a user test relating to a new Spotify feature ‘Hey Spotify’. Before we conducted the test, we explained what our bot was and then showed them a short video that as soon as the users open the Spotify app, a popup will appear asking them to participate in a user testing for their new features. If they agree they would be asked if they want to take a screening either by a voice automation call or by an online survey. After that we conducted the testing by us acting as the bot while the participants being the user.
VOICEFLOW
Prototype: After updating our training data and flow diagram, we created a voice bot prototype. We built it in Voiceflow. We let our participants go through the flow and let them experience the voice bot themselves. Before we conducted the test, we explained what our bot was and then showed them a short video that as soon as the users open the Spotify app, a popup will appear asking them to participate in a user testing for their new features. If they agree they would be asked if they want to take a screening either by a voice automation call or by an online survey.
INSIGHT
Most users said they wouldn’t use a voice bot and would instead prefer an online screener, survey, google form etc.
SOLUTION
Hence, we decided to Switch to a Chatbot prototype instead of voice, as that process may seem more natural to the users and also could prove to be Less Time Consuming.
Why Chatbot?
1. A chatbot is also more suitable for our target audience for this screener - Spotify users, with stable internet access, tech savvy and fast paced.
2. A chatbot can still filter participants for the usability study soon after recording their response. This can speed up the recruitment process.
3. A chatbot can be incorporated into the spotify app, without navigating the user to a different flow ( another interface, a different mode of communication )
FINAL SAMPLE SCRIPT
FINAL FLOW DIAGRAM
.png)
FINAL DESIGN
VOICEFLOW WORKSPACE
PROTOTYPE VIDEO DEMO
CONCLUSION
Screener is still a very immature product but has a lot of potentials. There are so many good features waiting to be designed and developed to help UX researchers or companies to manage their user testing screener. However, due to the scope of this class project and the time limitation, those features have to be proposed for future steps. Overall, We felt satisfied with designing our first chatbot product.
For future consideration, the companies can gather information regarding user’s usage patterns and provide that data to the bot for a better and smooth screening process. Also, to enhance the bot’s accessibility, we would like to add a voice call feature for less tech savvy users who are not too comfortable with google forms, surveys or chatbots.
Our work on this project helped us understand many fundamental requirements for conversational UX products. We also learned about various challenges associated with designing an excellent conversational UX product. An example would be if one had to design a conversation flow when designing a complex and tedious conversation flow to cover the content but with an appropriate amount of speech.