Striking a Balance Between Machine Learning & Local Context When Addressing Challenges for Youth Online

By Sarah Degnan Kambou (ICRW), Latisha Chisholm (Bard High School Early College DC), Priyanka Bhalla (Quilt.Ai), Happy Awide (ICRW), Chisina Kapungu (WomenStrong International), Biva Rajbhandari (Quilt.Ai), Brittany Hinkle (ICRW) & Adam Odomore (ICRW)

As students return to in-person or hybrid classes this fall, it is vital for schools to support their emotional and physical well-being. The long-term disruptions from COVID-19 and impact on the students’ mental and physical state has been far-reaching. 

In Washington, D.C., given systemic discrepancies in affluence and infrastructure between city wards, the impact from the pandemic has fallen more heavily on areas where the residents are predominantly Black, Indigenous, and People of Color (BIPOC). Historically, BIPOC communities in the District of Columbia have suffered from higher unemployment rates, lower wages, lower incomes, and much less savings to fall back on, as well as significantly higher poverty rates than more affluent and predominantly White areas of the city. The pandemic has only exacerbated these discrepancies in health, economics, and overall well-being.

Municipal authorities and school administrators in the District are wrestling with maintaining a safe and healthy environment for students, teachers, and administrators. And as the District works through protocols around vaccination status, rapid testing, masking, and social distancing, there is a pressing ancillary issue: how best to support students as they transition back to in-person from a remote learning experience.

What We Have Been Working On

Looking for pathways forward,  the International Center Research for Women (ICRW), the Connected Schools Program at Anacostia High School, and Quilt.AI formed a unique public-private partnership that could leverage quantitative, qualitative, and digital research to discover what’s top of mind in the COVID era for students and their parents. 

Through this partnership, we focused initially on one of the District’s four areas most impacted by socio-economic disparities: Ward 8, home to the highest concentration of African American residents. Collectively, we set out to understand the experience of older adolescents and their families in the ward during remote learning and to solicit their input on what they hope to find in terms of programming and support as they return to school.

From October  2020 to March 2021, ICRW and the Connected Schools Program at Anacostia High School conducted formative research with parents, guardians, and youth aged 14-18 years to understand the psychosocial, health, and well-being needs of adolescents and their families. In June 2020, Quilt.AI, a global artificial intelligence firm, joined the partnership to help document the digital ecosystem of students and families dealing with the daily COVID grind.

A Closer Look: Search Behavior & Machine Learning

We gathered important insights from our conversations with parents, guardians, and youth. However, we wanted to dig a little deeper, and we did this by exploring online search behavior and social media posts, and ran that data through a machine learning model – to give us broad insights across Washington, DC, as well as more specific insights for Ward 8.

Broad-based online searches across the District provided important context for our more focused analysis of Ward 8. For the latter, we focused on social media posts in Ward 8 and ran the information through a custom machine learning model – providing us with more detailed information on the specific lived experiences of adolescents in four under-served wards of the District, including Ward 8.

Over 3 million searches were analyzed, with keywords sorted into thematic categories such as mental health, sexual and reproductive health, violence, and remote learning. Meanwhile, the custom machine learning model was further tailored to understand the local context, as other mainstream AI (artificial intelligence) models would misinterpret emotions due to lack of culturally appropriate training data. For example, captions typically used in the Black community to express confidence, self-love, and self-care, through standard AI models are commonly misclassified by the machine as either sensual or erotic.

In order to address these challenges, the research team created two new classifications: 1) confidence, 2) sexuality and romance. Culturally appropriate examples for these categories were selected from online public platforms and then used as training data to develop our own modified machine learning model. The team, additionally, aimed to reduce the probability of bias by ensuring the research team was diverse, both in terms of gender and race. The purpose was to increase the accuracy and effectiveness of the machine learning model, centering cultural context and intersectionality through the project. With refined definitions and contextual insights, the tailored Machine Learning model more accurately sorted data to align with local vernacular culture and forms of expression.

The custom machine learning model was first trained with a sample of close to 2,700 posts from adolescents in the specific wards, to be able to understand their sentiments and engagements both pre and during COVID-19. Subsequently, 13 classifications were created for analysis that capture the different emotional sets  posts fell within, while further training each classification with a total of at least 150 text examples. 

What We Learned

There were three important learnings from the social behavior analysis and machine learning model.

1. Digital data can paint a vivid picture of the lived experience of students and their families during the COVID lockdown. Analyzing online searches and content consumption by youth during the first and second waves of the pandemic illuminated patterns of interest and awareness – providing further insight into attitudes, beliefs, views, and perspectives on self and others, school and work, and life in general. It’s important to note that involvement by people from the community is essential in interpreting results to ensure bias is further mitigated.
2. Youth mental health must be a core focus during and after COVID. In comparison to all other categories, mental health had the highest average search volume both pre-pandemic and since. While our team restructured the machine learning model, we developed a more comprehensive list of search terms associated with mental health that are likely to be searched by teens. Previously, “overcoming anxiety” and “self care” were not being captured. However, those terms grew in popularity during the search growth’s peak. These findings imply that teens are not only knowledgeable of their psychological experience, but that they are also seeking ways to improve it.
3. There is an increased need for safe spaces for online expression. Interestingly, before the pandemic, confidence was found to be the most showcased emotion. However, confidence, identified as an emotion through social media posts on Twitter and Instagram, dramatically decreased during COVID – demonstrating the negative impact of the pandemic on the students’ online expression. With an increase in online engagement, there’s a heightened need for online spaces where youth can safely express themselves – free from judgment and harassment. 

Recommendations

Our overarching goal in this project was to find evidence to inform solutions to the emerging challenges that face youth online. By analyzing online behavior of teens in Washington, D.C., we gained a deeper understanding of the range of experiences for high school students during both the first and second waves of COVID-19. Research like this can help inform effective school-level interventions. Taking the insights gained and implementing solutions will be the next step for schools and communities to support youth.

For now, here are our recommendations: 

• Create safe, inclusive online spaces. Adolescents are organically trying to build collective resilience amongst themselves and their peers. This needs to be fostered by creating and promoting focus groups, discussion forums, etc. online where adolescents can freely share their feelings and thoughts. 

• 

  Ensure access to and knowledge of digital tools. Adolescents should be able to navigate the digital landscape effectively and safely. This means giving them access to tools and resources they need to support their growth in the digital space. 

• Make education fit for students. Equip students with tools to better understand current events and to promote self care and mental health support. It is important to enhance cognitive, creative, social and emotional skills that students both need and want to navigate complex situations.
• Document stories: Adolescents look for inspiration online which is why it is important to create more context relevant content for them in the form of stories of how their peers cope with issues they face, resources that keep them going etc. Initiatives like these can be run as campaigns and promoted through adolescent-centered social media spaces.