By Lisa Jones
Qualitative data. Many organizations have it, but don’t know what to do with it. From one line quotes to multi-page surveys, qualitative data can speak volumes about an organization’s impact. The question is, how can organizations synthesize those volumes into substantiated statements that offer a depth of insight into their impact that numbers alone cannot match?
Qualitative data. Many organizations have it, but don’t know what to do with it. From one line quotes to multi-page surveys, qualitative data can speak volumes about an organization’s impact. The question is, how can organizations synthesize those volumes into substantiated statements that offer a depth of insight into their impact that numbers alone cannot match?
 |
| Chris Raine, founder of HSM |
Chris Raine, a Young
Social Pioneer, YouthActionNet’s partner in Australia, and founder of Hello Sunday Morning, has done just that.
Hello Sunday Morning (HSM) is a blogging website that encourages people to
undertake a period of sobriety (typically 3, 6, or 12 months) and use this time
to actively reflect on the role that alcohol plays in their life. The HSM participants
voluntarily participate, blogging about their journey as they experience the
highs and lows of a substantial lifestyle adjustment. At its core, HSM is a
movement towards a better drinking culture—a platform for individuals to create
meaningful change in their lives while being part of a larger community of others
taking up the same challenge.
Soon after founding HSM in 2009, Chris sought out solutions
for program evaluation. Since the program leaves it to the individual to decide
what amount of alcohol consumption works in their lives, he needed to find a way
to measure the overall affect HSM had on a person’s well-being—not just the
number of drinks they did or did not consume.
Chris knew he had a treasure-trove of valuable insights—stories of
transformation, and stories of slip-ups, documenting the experience with deep
personal reflection and honesty. In 2011, HSM had 2,190 blog posts from 1,768
individuals—a total of 846,676 words. The daunting task was determining how to
use them.
To synthesize the content of the posts, HSM enlisted the help
of a team of students and professors from University of Queensland. The team used
software called Leximancer to
identify common concepts occurring throughout all of HSM’s blog posts. The
process to determine common concepts began with an analysis of words that
appear frequently, either together or apart. Next, the software mapped how
concepts were related to one another within the text. The final result? A map of blog content grouped according to the
larger themes of culture, life experiences, and both personal and collective
change.
Once the content of the posts was mapped, Chris was able to
use categorical data associated with each post (such as number of days the HSM
blogger had undertaken the challenge) to draw conclusions. By running a search
for posts at particular stages of the challenge against the themes that had
been mapped, the team learned that HSM bloggers’ posts change thematically over
the course of their HSM experience:
“In the
earlier parts of an HSM experience, an HSMer is more likely to describe their
drinking practices, be conscious of peers’ reactions, be focused on individual
goals, and seek advice from fellow HSMers. As their HSM experience progresses
the thematic content of their blog shifts, first documenting efforts to make
personal change and reflecting on their own drinking practices, to then
reflecting on drinking culture and in turn offering their own advice and
strategies for change to other HSMers.”1
Through the blog post analysis, the research team was able to
prove that the blog facilitates peer-to-peer collaboration—participants use it
to document personal reflections, and discuss shared experiences
collaboratively. Also, the qualitative data revealed that the HSM bloggers
create “an imagined community characterized by particular narratives, norms,
and values.”2
In the effort to substantiate bold claims such as these,
Chris and his team used multiple strategies—not just the blog analysis. They administered
surveys to randomly-selected participants at the beginning and end of their
experience, organized small group discussions, and sent out email
questionnaires. However, they encountered difficulties with these other methods
such as low response rate, incomplete responses, or variances in when surveys
were completed by different participants. In the end they discovered an
important lesson: Data was easier to collect when it was ‘naturally-occurring’
or ‘built in’ to the HSM process. Using this learning, HSM built the qualitative
questionnaire into the sign up process.
Throughout
their evaluation endeavor, HSM learned the value of qualitative data, but also
learned the limitations of data that is collected for evaluation alone, rather
than as part of the larger journey. The data in the blog was not written with
evaluation in mind—posts are heartfelt, vulnerable, and capture how the
participant is feeling at pivotal moments throughout their HSM experience.
You
can’t rate a participant’s sober 21st birthday story from 1-5, or
calculate the importance of her finding time to volunteer on the weekends, but
you can begin to understand the commonalities in a shared experience—the
struggles, motivations, and turning points. You can take that information and use
it to increase program efficiency and empathy, attract more people to
participate, and continue to synthesize their collective voices into data that informs
a larger discussion on drinking culture. With 10,000 new posts each month and counting,
Hello Sunday Morning is well on its way to revolutionizing people’s relationship
with alcohol, and Sunday mornings, in Australia and beyond.
Source (1,2):
Hamley, Ben and Dr. Nicoholas Carah. One Sunday at a Time: Evaluating Hello Sunday Morning. Queensland, 2012.
Photo of Chris by triple j
