Designing and Evaluating A Couples Chatbot for Sexual Health
Research Context
Our societal partner AidsFonds Nederland told us that, contrary to popular belief, young people who have just gotten into new, exclusive relationships are more likely to stop using condoms than young people who have casual sex, even though they are still at risk for sexually-transmitted infections. ​They were interested in how we could use digital health to address this issue.
​
Research on sexual health interventions suggests that it can be better to address the couple together, as sexual behaviours are a result of interpersonal processes between both individuals.
​
Within the context of our own project, the question was: how could we do this using a conversational agent?​
Research Goals
-
How can we use end-to-end user-centred design to create a conversational agent that supports a new, young couple's sexual health?
-
How do real couples from our target group perceive the interaction?
Approach
As with all of the projects during my PhD, I had full ownership of this one - I was responsible for planning and conducting the end-to-end design of a couples-based conversational agent, including both UX research, design and evaluation agent (see below for an outline of the approach I took).
Interviews
Potential End-Users
i.e. Young People
Design Guidelines (1)
Design Thinking Workshop
Experts
i.e. Sexual Health Counselors
Design Guidelines (2)
Paper-and-pen prototype
Acceptance Testing
Potential End-Users
i.e. Young People
Design
User Interviews
Participants
Recruited via the university research pool
Inclusion criteria:
-
Between 18-25 years old
-
Identifies as heterosexual
-
Currently sexually active (i.e. in the past 30 days)
Method
We led with the following prompt: "How do you envision a similar chatbot designed for couples use?"
​
We asked follow-up questions about what the chatbot would (or would not) do, and any barriers that they foresee.
​
Thematic analysis was used to analyse the data, and three themes emerged.
These interviews were conducted as part of a prior study in which another (similar) chatbot was being user-tested.
This served two advantages:
-
Streamlining data collection by not having to recruit a separate sample
-
Interacting with a prototype gave users a frame of reference
Findings
Foreseen Barriers
Challenge of bringing up chatbot with a new sexual partner in an already awkward and delicate context
​Female users felt that their male partners would not be open to using such a chatbot
Concerns about what each person would reveal about the partner and relationship behind the other's back
Male users suggested that the target group might not think they need the chatbot
All users were uncomfortable with separate, dyadic chats​​
Triadic chat
Separate dyadic chats
What Participants Proposed
Design
Design Thinking Workshop
-
Four GGD sexual health counselors from all over the Netherlands
-
Participants were invited to an on-site meeting room at the University of Amsterdam
-
The workshop lasted from 10am-4pm
-
Participants were compensated for their time
-
I created a workshop schedule, and a workshop script as well as a slide deck
-
I moderated the workshop, and a colleague observed the session and took notes
empathise
define
Participants were provided with a persona, our objecives, and the following design problem: "How might a chatbot support a couple's sexual health?"
ideate
select
Participants used brainwriting and the three hat rule to generate and select ideas
prototype
Participants worked together to arrange and extend the selected ideas along an interaction timeline
Final Design
Design Guidelines (1): Young People
-
Goal - Encourage Communication, Not Condom Use
-
Focus - Holistic Sexual Health
-
Set-up - Triadic, Not Dyadic
-
Barrier - Pitching To A New Partner
Design Guidelines (2): Professionals
-
Goal - Educate and Support, Not Counsel
-
Focus - Holistic Sexual Health, Sex-Positivity
-
Features - Inform and Engage
I cleaned up the paper-and-pen prototype, and used it as a guide to develop the dialogue flows.
With the paper-and-pen prototype as the foundation for the interaction, I fleshed out a design that also met the design guidelines from both groups.
Luckily, the counselors directed me to existing resources that I could adapt rather than writing dialogues from scratch.
I used lucidchart to create dialogue flows, working with counselors to ensure the dialogues were appropriate
Evaluation
User Acceptance Testing
Participants
​​Inclusion Criteria:
-
Between 16 and 20 years old
-
Identifies as heterosexual
-
In a new relationship i.e. 6 months or less (self-defined)
-
Willing to participate as a couple
11 couples participated in this study; 10 were used for analysis (1 was excluded due to language barrier)
Method​
​​We used the wizard-of-oz paradigm to evaluate the chatbot.
Participants...
-
Signed up via the research participation pool, and recruited their partner as well
-
Joined an online meeting at the designated date and time
-
Interacted with the chatbot with their partner in the same chat via Rocket.chat (unaware that a human is chatting with them)
-
Participated in individual interviews, where one person waits in the main room while the other is interviewed in a breakout room
The wizard-of-oz paradigm is useful in the early stages of design, and can produce actionable insights in the face of limited resources (as long as measures are taken to mitigate the potential issues surrounding human error)
An example of what participants saw during testing
​​
​
I conducted a two-step analysis given the couples-nature of this design, focusing on both the individual- and couple-level.
​
I chose to conduct hybrid thematic analysis, which allows both data-driven and researcher-driven coding.
​
Note: I like hybrid thematic analysis because it's a focused approach that lets me analyse the data through a goal-specific lens, but allows the emergence of new codes I wouldn't have anticipated.
Analysis​
Transcription
I transcribed each interview using Amberscript, and imported the transcripts into atlas.ti
Open, line-by-line coding
Considering each transcript individually, I generated open codes to capture as much detail as possible
Closed coding using constructs of interest
Now, I go over each transcript again, this time looking for data/open codes that corresponds to the user acceptance constructs of interest, and coding it accordingly.
Closed coding for differences
Then, we considered couples as pairs of transcripts, and for each construct, coded for similarities and differences.
Guiding User Acceptance Framework
1. User Experience
-
Ease of Use
-
Usefulness
-
Engagement
3. Information Quality
-
Relevance
-
Quantity
-
Clarity
-
Language Style
4. Therapeutic Alliance
-
Bond
-
Collaboration
-
Openness
-
Confidence
5. Usage Intentions
2. Interaction-Specific Elements
-
Triadic chat
-
Activities
​​
​
The novel triadic interaction format that we tested was positively received, and participants mentioned several couple-based benefits they experienced as a result of it:
Findings
Learning about my partner
Communicating my thoughts, feelings and opinions
Aligning with my partner on things important for our relationship
However, there were several challenges that need to be addressed in the subsequent iteration: ​
1
Concerns about self-disclosing openly and honestly in their partner's presence
2
Expectations about acceptable behaviour from chatbots interfering with establishing a relationship, or a "therapeutic alliance" with the chatbot
3
Individuals within a couple are more likely to differ than align in how they perceive and prefer to interact with the chatbot
Next Steps + Recommendations
Look Into...
Before going into technical development, I would conduct A/B testing to iron out the three considerations that emerged from concept testing​​​:​
1
Compare a hybrid interaction (including both triadic and dyadic chats) to the original to see if this addresses self-disclosure concerns while still remaining beneficial to users
2
Explore and test solutions for how we can address the preconceptions users have about chatbots
3
For each instance of asymmetry found in couples, explore and test a multi-faceted solution that is likely to fulfill both individuals' needs
At this point, I'd consider using a more top-down approach for evaluation, focusing on the specific constructs and phenomena of interest. This can mean either using highly-focused qualitative methods, or simply surveys to collect the data.
Reflections
What Went Well
challenges
I was able to independently design and test a novel interaction concept to show its promise for further development
The sheer volume of data I got through qualitative methods sometimes made me lose sight of what's important. I found it helpful to have my research goals and questions beside me while sifting through the data
​​I found that I needed more effort than anticipated to recruit both our potential end-users as well as the sexual health professionals. If I had to do this again, I would come up with a recruitment plan much earlier.Â
I successfully advocated for user-centred design methods in an environment that was not fully familiar nor inclined to use them
I got an immense amount of nuanced data from participants through interviews, and this led to both actionable insights as well as deepened my understanding of how users interact with chatbots.