May 3, 2013

Making Qualitative Data Count

By Lisa Jones

Qualitative data. Many organizations have it, but don’t know what to do with it. From one line quotes to multi-page surveys, qualitative data can speak volumes about an organization’s impact. The question is, how can organizations synthesize those volumes into substantiated statements that offer a depth of insight into their impact that numbers alone cannot match?

Chris Raine, founder of HSM
Chris Raine, a Young Social Pioneer, YouthActionNet’s partner in Australia, and founder of Hello Sunday Morning, has done just that. Hello Sunday Morning (HSM) is a blogging website that encourages people to undertake a period of sobriety (typically 3, 6, or 12 months) and use this time to actively reflect on the role that alcohol plays in their life. The HSM participants voluntarily participate, blogging about their journey as they experience the highs and lows of a substantial lifestyle adjustment. At its core, HSM is a movement towards a better drinking culture—a platform for individuals to create meaningful change in their lives while being part of a larger community of others taking up the same challenge. 

Soon after founding HSM in 2009, Chris sought out solutions for program evaluation. Since the program leaves it to the individual to decide what amount of alcohol consumption works in their lives, he needed to find a way to measure the overall affect HSM had on a person’s well-being—not just the number of drinks they did or did not consume.  Chris knew he had a treasure-trove of valuable insights—stories of transformation, and stories of slip-ups, documenting the experience with deep personal reflection and honesty. In 2011, HSM had 2,190 blog posts from 1,768 individuals—a total of 846,676 words. The daunting task was determining how to use them. 

To synthesize the content of the posts, HSM enlisted the help of a team of students and professors from University of Queensland. The team used software called Leximancer to identify common concepts occurring throughout all of HSM’s blog posts. The process to determine common concepts began with an analysis of words that appear frequently, either together or apart. Next, the software mapped how concepts were related to one another within the text. The final result?  A map of blog content grouped according to the larger themes of culture, life experiences, and both personal and collective change.

Once the content of the posts was mapped, Chris was able to use categorical data associated with each post (such as number of days the HSM blogger had undertaken the challenge) to draw conclusions. By running a search for posts at particular stages of the challenge against the themes that had been mapped, the team learned that HSM bloggers’ posts change thematically over the course of their HSM experience:

 In the earlier parts of an HSM experience, an HSMer is more likely to describe their drinking practices, be conscious of peers’ reactions, be focused on individual goals, and seek advice from fellow HSMers. As their HSM experience progresses the thematic content of their blog shifts, first documenting efforts to make personal change and reflecting on their own drinking practices, to then reflecting on drinking culture and in turn offering their own advice and strategies for change to other HSMers.1

Through the blog post analysis, the research team was able to prove that the blog facilitates peer-to-peer collaboration—participants use it to document personal reflections, and discuss shared experiences collaboratively. Also, the qualitative data revealed that the HSM bloggers create “an imagined community characterized by particular narratives, norms, and values.”2 

In the effort to substantiate bold claims such as these, Chris and his team used multiple strategies—not just the blog analysis. They administered surveys to randomly-selected participants at the beginning and end of their experience, organized small group discussions, and sent out email questionnaires. However, they encountered difficulties with these other methods such as low response rate, incomplete responses, or variances in when surveys were completed by different participants. In the end they discovered an important lesson: Data was easier to collect when it was ‘naturally-occurring’ or ‘built in’ to the HSM process. Using this learning, HSM built the qualitative questionnaire into the sign up process. 

Throughout their evaluation endeavor, HSM learned the value of qualitative data, but also learned the limitations of data that is collected for evaluation alone, rather than as part of the larger journey. The data in the blog was not written with evaluation in mind—posts are heartfelt, vulnerable, and capture how the participant is feeling at pivotal moments throughout their HSM experience. 

You can’t rate a participant’s sober 21st birthday story from 1-5, or calculate the importance of her finding time to volunteer on the weekends, but you can begin to understand the commonalities in a shared experience—the struggles, motivations, and turning points. You can take that information and use it to increase program efficiency and empathy, attract more people to participate, and continue to synthesize their collective voices into data that informs a larger discussion on drinking culture. With 10,000 new posts each month and counting, Hello Sunday Morning is well on its way to revolutionizing people’s relationship with alcohol, and Sunday mornings, in Australia and beyond.

Source (1,2):
Hamley, Ben and Dr. Nicoholas Carah. One Sunday at a Time: Evaluating Hello Sunday Morning. Queensland, 2012.
Photo of Chris by triple j 
Share this post
  • Share to Facebook
  • Share to Twitter
  • Share to Google+
  • Share to Stumble Upon
  • Share to Evernote
  • Share to Blogger
  • Share to Email
  • Share to Yahoo Messenger
  • More...