Impacts and outcomes (part two): Tools to use
11 June 2013 – Kate Preissler
In my last post, I talked about some of the trainings and conceptual frameworks that help me to measure the impact of my programs at the Trustees of Reservations. I wanted to expand on the idea of measuring success and highlight some of the tools I have used for various programs and organizations that I have worked with.
These days I start by organizing the items we want to measure into two equally important categories: Organizational Goals and Engagement Impact Goals. Although I did not always explicitly make this distinction, looking back over the engagement activities I’ve worked with at different organizations, I can see that this distinction was more often present than not.
Photo by Kate Preissler for The Trustees 2012
Organizational Goals are the goals rooted in the needs and sustainability of the program or organization. This might include recruiting and engaging volunteers, generating revenue and admissions fees, increasing visitor numbers, increasing donations, and supporting membership. In addition to the fact that we need these outcomes to sustain our work, they also serve to indicate the value and quality of our work. People choose to volunteer, to donate, to visit, or to become members because something we have done has resonated deeply enough with them that they want to be a part of it. These organizational goals are both practical and affirming for our work.
I use the term Engagement Impact Goals to think about those results that extend beyond our organization. They are the goals that speak to what we want to change in the people who we reach out to and to the change we want to enact in the world. These are the exciting and motivating goals but they are also sometimes the hardest to clearly define and put onto paper. Goals of this sort that I have worked toward in the past have included empowering youth, teaching specific content, training individuals in hard skills, increasing literacy, bridging communities, encouraging healthy lifestyles, building teamwork, increasing communication skills, and more.
Once my list of goals is complete, I put together a set of tools that will get me the information I need to determine whether or not we are making progress on those goals. Most of the organizations where I have run programs have utilized shoestring budgets and overstretched staff, but we’ve always found ways to integrate evaluation into the program and once in a while even have some fun with it. Here are some of the activities I have done to help me understand the success and impact of my work:
Photo by Kate Preissler for The Trustees 2012
Basic Data Collection: Collecting this information can be as simple as having sign-up sheets for programs, having a person with a number clicker at the door, or utilizing registration materials. Whether it is an Excel spreadsheet, an Access database, or something more complex, we always have a central location for collecting a basic set of information. Because I work on the program side, my metrics are usually items like: number of participants, number of programs, number of member attendees, hours of program time, and number of volunteers.
Surveys: This is the classic form of evaluation and still effective for a variety of purposes. I have used them to collect visitor satisfaction information, to assess the effectiveness of our marketing, to do program evaluation, collect demographic information of participants and more. Online survey tools like www.surveymonkey.com make it very easy to collect responses and compile data. On the other hand, I just as often use paper copies being passed out at programs or volunteers circulating and taking responses verbally.
Pre- and Post- Program Impact Questionnaires: For prolonged programs such as camps, multi-day workshops, school programs, and clubs I have used pre- and post- questionnaires. The change in answers between the first round of questions and the second can be used to indicate gains in content knowledge and changes in attitude or behavior. I have used this tool most effectively with youth programs, for instance to measure gains in knowledge about the ecology of Boston Harbor in kids who were participating in a sailing program.
Interviews: With Flip video cameras, audio recorders and even iPhones, it is simple and affordable to interview participants to get a much more personal sense of how they have been impacted by programs and activities. It’s also a great way to collect quotes and personalized anecdotes that can be integrated into reports. It can be as effective for catching reactions from one-time visitors as it is for longer-term participants. Last year at The Trustees we interviewed our Youth Corps participants at the beginning and end of the programs and were able to see just how much their attitudes towards the environment had changed. (You can see the piece we created with those interviews here.)
Social Media: Social media outlets like Twitter and Facebook can provide simple points of measure for popularity (“likes” and “followers”) but they can also be used to track some of the longer-term effects of programs by starting special groups for participants of programs or activities. Creating groups for alumni of programs, interns, and volunteer groups can allow you to periodically check in with questions, follow up, and information regarding the influence of your programs. I’ve used this method at camps and summer sports programs and am still able to see the progress of some of our participants even though I have moved on to other jobs.
Measuring Physical Contributions: In many instances there may be a way to measure the tangible outcomes of your activities. One way I have used this is with volunteer activities. In addition to counting how many volunteers are working or how many hours they have put in, I have measured exactly how much work they have done. (For instance cleared 2 miles of trails, catalogued a certain number of pieces in a collection, or led a particular number of programs). Other physical metrics I’ve used include how much space is made available for community and partner use, how much food a farm or garden contributes to the local food system, or how many educational or training publications (like these) have been bought, distributed or downloaded.
These are just some of the most basic ways that I have measured the success of various programs I have been a part of. There are many more creative ideas and methods floating around in the world. I would love to hear from others as to what they are doing to measure and articulate their success. Leave a comment below with your ideas.
NEMA is also hosting a workshop on this topic on June 19th at the Currier Museum of Art in Manchester NH. Perhaps I will see you there!
~Kate Preissler is the Western Region Engagement Manager for The Trustees of Reservations, coordinating points of contact between the public and about 40 historic, ecological, and recreational sites in the Berkshires, Pioneer Valley, and Central Massachusetts. The Trustees is a private organization whose mission is to permanently preserve properties of exceptional scenic, historic and ecological value in Massachusetts for public use and enjoyment. Kate holds a B.A. from Bates College in English and History and graduated from the UMass Amherst Public History Masters Program in 2010.