DxD Lab

Key publications

1. How Do You Feel Online: Exploiting Smartphone Sensors to Detect Transitory Emotions during Social Media Use

   Ruensuk, Mintra, Cheon, Eunyong, Hong, Hwajung, and Oakley, Ian

   IMWUT ’20, Dec 2020

   Abs HTML PDF

   Emotions are an intrinsic part of the social media user experience that can evoke negative behaviors such as cyberbullying and trolling. Detecting the emotions of social media users may enable responding to and mitigating these problems. Prior work suggests this may be achievable on smartphones: emotions can be detected via built-in sensors during prolonged input tasks. We extend these ideas to a social media context featuring sparse input interleaved with more passive browsing and media consumption activities. To achieve this, we present two studies. In the first, we elicit participant’s emotions using images and videos and capture sensor data from a mobile device, including data from a novel passive sensor: its built-in eye-tracker. Using this data, we construct machine learning models that predict self-reported binary affect, achieving 93.20% peak accuracy. A follow-up study extends these results to a more ecologically valid scenario in which participants browse their social media feeds. The study yields high accuracies for both self-reported binary valence (94.16%) and arousal (92.28%). We present a discussion of the sensors, features and study design choices that contribute to this high performance and that future designers and researchers can use to create effective and accurate smartphone-based affect detection systems.

2. Understanding University Students’ Experiences, Perceptions, and Attitudes Toward Peers Displaying Mental Health–Related Problems on Social Networking Sites: Online Survey and Interview Study

   Kim, Taewan, and Hong, Hwajung

   JMIR Mental Health ’21, Dec 2021

   HTML PDF
3. MAMAS: Supporting Parent–Child Mealtime Interactions Using Automated Tracking and Speech Recognition

   Jo, Eunkyung, Bang, Hyeonseok, Ryu, Myeonghan, Sung, Eun Jee, Leem, Sungmook, and Hwajung Hong,

   CSCW ’20, May 2020

   Abs HTML PDF

   Many parents of young children find it challenging to deal with their children’s eating problems, and parent–child mealtime interaction is fundamental in forming children’s healthy eating habits. In this paper, we present the results of a three-week study through which we deployed a mealtime assistant application, MAMAS, for monitoring parent–child mealtime conversation and food intake with 15 parent–child pairs. Our findings indicate that the use of MAMAS helped 1) increase children’s autonomy during mealtime, 2) enhance parents’ self-awareness of their words and behaviors, 3) promote the parent–child relationship, and 4) positively influence the mealtime experiences of the entire family. The study also revealed some challenges in eating behavior interventions due to the complex dynamics of childhood eating problems. Based on the findings, we discuss how a mealtime assistant application can be better designed for parents and children with challenging eating behaviors.

4. In Helping a Vulnerable Bot, You Help Yourself: Designing a Social Bot as a Care-Receiver to Promote Mental Health and Reduce Stigma

   Kim, Taewan, Ruensuk, Mintra, and Hong, Hwajung

   CHI ’20, Apr 2020

   Abs HTML PDF

   Helping others can have a positive effect on both the giver and the receiver. However, supporting someone with depression can be complicated and overwhelming. To address this, we proposed a Facebook-based social bot displaying depressive symptoms and disclosing vulnerable experiences that allows users to practice providing reactions online. We investigated how 55 college students interacted with the social bot for three weeks and how these support-giving experiences affected their mental health and stigma. By responding to the bot, the participants reframed their own negative experiences, reported reduced feelings of danger regarding an individual with depression and increased willingness to help the person, and presented favorable attitudes toward seeking treatment for depression. We discuss design opportunities for accessible social bots that could help users to keep practicing peer support interventions without fear of negative consequences.