Evaluation Not everything that can be counted counts, and not everything that counts can be counted.
--Albert Einstein  So you've planned and implemented your social marketing campaign, congratulations! Now it's time to figure out if your program is meeting it's intended objectives. In other words, did it work?
Assessment and evaluation are like opposite sides of the same coin. In planning your social marketing campaign and gathering data, you have laid the groundwork for your evaluation. Evaluation is based on the process and outcome objectives developed in your program plan and on the assessment responses gathered from your priority population.
The CDC has created a framework which lays out how to conduct an evaluation of any public health project. This CDC resource link goes into great detail about each step and lists many more resources for developing an effective evaluation plan. Another resource from the CDC is the Thinking Like a Marketer National Training Collaborative for Social Marketing. Also, the CDCynergy can be very helpful.
Depending on the size and budget of your social marketing campaign, the evaluation can be done with existing staff or with an outside agency. One of the benefits of having an outside evaluator is that they can identify the outcomes independently and then compare their findings to the established objectives. This unbiased approach supports objectivity and allows for the inclusion of unintended outcomes. One drawback to using an outside agency (in addition to the extra cost) may be that they are not familiar with the priority population.
Below are a few different types of evaluations that are useful in social marketing campaigns:
- Process - documenting how the program was developed; counting the steps along the way. Process evaluation helps you improve as you go.
- Outcome - results of the program (related to the outcome objectives)
- Case Study - in-depth interviews with individuals or agencies who were impacted by the program
- Cost-benefit analysis - how much did the program cost in relation to the cost savings or changes that resulted from the program
- Prevention - evaluation of what did not happen, but what might have if the program did not exist
Once you choose one or more types of evaluation from above, choose from the different evaluation designs for the collection of data: - Pre-post - collect baseline data before the program implementation and then again right afterward
- Retrospective - collect data only after implementation. Ask questions for reflective responses about reactions that occurred before and after the intervention
- Time sequence (pre-post and follow-up) - this is a pre-post evaluation with another data gathering point later on (e.g. 3 months later)
There are various methods for collecting evaluation data. Remember to use the same methods in your pre and post collection when applicable: Some examples of quantitative data - Observed data collection (i.e. medical records, databases of clinic visits)
- Self-reported data collection (i.e. surveys of clients re: behavior or intention for behavior change)
Some examples of qualitative data - Focus groups
- Open-ended survey questions
- Key informant interviews
Online social marketing campaigns can also benefit from evaluation tools like.... Web Analytics - a free way to keep track of website activity. This tool will measure, collect, and analyze website data including the total number of times people visited your site, unique page hits, and the amount of time spent on the website. You can register for free web analysis tools at www.google.com/analytics or www.getclicky.com. While web analysis does not tell you what people are doing with the information on your site, it does tell you if people are seeing your message, how long they visit the site, what they are looking at and how they go there. This type of evaluation can inform programmatic improvements and help to determine if you are reaching your priority population. Some innovative ways to collect quantitative and qualitative data using the internet and other computer technology are via online focus groups and interactive chats, online surveys and data anlysis, palm-held computers for high-tech street data collection, etc. What's out there that worked
The following examples of research show effective evaluations of recent social marketing campaigns designed to improve sexual and reproductive health. Click on each to see a summary of their findings:
Staying Alive - a global campaign made to promote
interpersonal communication about HIV and postive beliefs
about HIV prevention POWER for Reproductive Health - a campaign to increase
awareness of and change attitudes toward the use of
female and male condoms
Tu No Me Conoces - a campaign to promote HIV risk awareness
and testing among Latinos living on the U.S.- Mexico border Remember, you can’t evaluate everything! Prioritize which outcome factors to evaluate based on available resources and the information needed to further your program goals. Before asking questions, decide which data you will use the most. For more information on methods and selecting the right method for your organization, click here for the Free Management Library's Guide to Program Evaluation.
 See Sally Evaluate!
 Find Resources for Evaluation at the Social Marketing Toolbox
Staying Alive
In 2002 MTV launched a global multicomponent HIV prevention campaign, "Staying Alive," reaching over 166 countries worldwide. An evaluation of this campaign focused on three diverse sites: Kathmandu, Nepal; São Paulo, Brazil; and Dakar, Senegal. Data were collected before and after campaign implementation through population-based household surveys. Using linear regression techniques, the evaluation examined the effects of campaign exposure on interpersonal communication about HIV and beliefs about HIV prevention. We found a consistent positive effect of exposure on interpersonal communication across all sites. However, there were differences among sites with regard to whom the respondent talked about HIV. We also found a consistent positive effect of exposure on HIV prevention beliefs across sites when interpersonal communication was simultaneously entered into the model. Finally, in two sites we found a relationship between interpersonal communication and HIV prevention beliefs, controlling for exposure. However; the effects differed by the type of person the communication was with. Similar results in three diverse sites provide ecological validity of the findings that "Staying Alive" promoted interpersonal communication and influenced young people's beliefs about HIV prevention in a positive way, evidence for the potential of a global media campaign to impact social norms. MTV's "Staying Alive" global campaign promoted interpersonal communication about HIV and positive beliefs about HIV prevention. Geary CW, Burke HM, Castelnau L, Neupane S, Sall YB, Wong E, Tucker HT, AIDS Educ Prev. 2007 Feb;19(1):51-67 back to top POWER for Reproductive Health To evaluate effects of a 6-month social marketing campaign on awareness of, attitudes toward and use of female and condoms for 15-25 year-old-women. Using a time-space sampling methodology, we conducted a cross-sectional survey of 3407 women at pre-campaign in 12 western U.S. neighborhoods on female and male condom awareness, attitudes, and use. Six of the 12 study neighborhoods were randomly selected to receive the POWER social marketing campaign designed to impact condom knowledge, attitudes, and use. The campaign was followed with another cross-sectional survey of 3,003 women in all 12 study neighborhoods on condom knowledge, attitudes, use and awareness of POWER materials. We compared pre-and post-campaign surveys to determine the efficacy of POWER and conducted post hoc analyses on post-campaign data to determine if exposure to POWER was related to higher levels of positive attitudes and norms towards condoms and condom use. We found no differences between neighborhoods with and without the POWER campaign with regard to our primary outcomes. To diagnose reasons for this null effect, we examined outcomes post hoc examining the influence of POWER exposure. Post hoc analyses show some evidence that exposure to POWER was associated with condom use. In the context of the nested trial, this raises concerns that post test only evaluations are limited. Establishing the efficacy of a social marketing campaign is challenging. This group randomized trial showed a null effect. Social marketing campaigns may need to have more media channels and saturation before they can show behavioral effects. Using a nested design with randomization at the community level and probability sampling introduces rigor not commonly seen in evaluations of social marketing campaigns. POWER for reproductive health: results from a social marketing campaign promoting female and male condoms. Bull SS, Posner SF, Ortiz C, Beaty B, Benton K, Lin L, Pals SL, Evans T. J Adolesc Health. 2008 Jul;43(1):71-8. Epub 2008 May 12. back to top Tu No Me Conoces
Increased incidence of HIV/AIDS in Latinos warrants effective social marketing messages to promote testing. The Tú No Me Conoces (You Don't Know Me) social marketing campaign promoted awareness of HIV risk and testing of Latinos living on the California-Mexico border. The 8-week campaign included Spanish-language radio, print media, a Website, and a toll-free HIV-testing referral hotline. They documented an increase in HIV testing at partner clinics; 28% of testers who heard or saw an HIV advertisement specifically identified the campaign. Improved understanding of effective social marketing messages for HIV testing in the growing Latino border population is warranted.
Promoting HIV risk awareness and testing in Latinos living on the U.S.-Mexico border: the Tu No Me Conoces social marketing campaign. Olshefsky AM, Zive MM, Scolari R, Zuniga M. AIDS Educ. Prev. 2007 Oct;19(5):422-35. back to top NEXT PAGE PREVIOUS PAGE |