Impacts and outcomes: Learning to measure success

08 February 2013 – Kate Preissler

The ultimate challenge: How can we measure what a place means to a visitor?

We are just passing that time of year when my team at The Trustees of Reservations  transitions out of our high season and into the relative quiet of the winter. With a busy program season for our historic homes from around April to October and a budget planning season starting in December, I have about two months in the fall to review all of the data that we have collected. I use the information to determine where I will allocate staff and resources for the next year, what the key promotional opportunities might be, and what projects will become priorities for our winter fundraising efforts.

Being able to do this thoughtfully and comprehensively is critical to my programs and, more importantly, to the people we serve.

If you are new to this type of work, getting a sense for how other organizations are measuring and expressing their impacts—how they communicate success—can give you a sense for what your board, funders, and colleagues might be expecting. I have benefited greatly from being part of networking groups focused on evaluation, and by having partners willing to share their evaluation tools. It’s not necessary to find people doing the same types of programs, nor even other history programs. I’ve found great inspiration and overlap from colleagues in completely different fields of non-profit work. There are also innovative people like Nina Simon who are finding and sharing inspirational ways to blur the lines between evaluating to better understand visitor’s experience, and making feedback opportunities part of an enhanced visitor’s experience.

Evaluation training programs have also been invaluable to my work. Two frameworks in particular have informed the evaluation plans I am building for our region and may be of interest to others:

In 2006 I participated in a year-long Logic Model and Evaluation training with the United Way. This training focused on the foundational concepts and tools for understanding how to measure for program outcomes. At the core of this framework is identifying and tracking quantifiable data points, or “indicators,” that can be used as evidence that your activities are having the effects that you intended. This type of evaluation provides us with the basic information that underlies most of our decisions.

Throughout the season we track basic some basic indicators, numbers that help us gauge our success. These always include the number of visitors to our site, number of participants attending a program, number of people becoming members, and amount of revenue generated. Sometimes we use surveys or feedback cards with questions such as “What was the most interesting thing you learned today?” or “What would you like to know more about?” to collect information related to the content delivered at programs. We also use survey information to understand how well our marketing efforts are working and who we are attracting and engaging. The key is to choose indicators that are directly connected to what you want most to understand about your efforts.

All of this quantifiable information is essential to assessing my programs; helping me to make logistical decisions and improve my program quality. When it comes to philanthropy, however, another layer of evaluation has been proving just as important. Donor culture now has seen a shift to a much more outcomes-driven type of evaluation – assessment that focuses on being able to show the specific change (the “outcomes”) your work is making in the world. For example, in my work we might consider a program that teaches land owners about different options for legally protecting their land from future development a success if it attracts a high turnout, but to  evaluate its impact we have to understand more about what decisions people make after the program. Did any of them take positive steps to preserve their land? Are any of them looking at and valuing their own land differently as a result of the information they received regarding the importance of protected land? These kinds of questions speak to program “outcomes.”

This shift in how to understand our work has been happening over the last several decades but is becoming even more pronounced with the new generation of philanthropists. The excellent Millennial Impact Report shows that younger philanthropists (aged 18-35) are very concerned with understanding the specific impact being achieved by the organizations they are supporting. They are more often motivated to give time or money when they have specific information detailing the achievement of the mission outcomes. It is not always enough to count participants and call it a success without also illustrating what difference we are making – how are we enacting positive change?

With this in mind, in 2011 I went off to the Center for Whole Communities to participate in their week-long Whole Measures training. Whole Measures is a comprehensive, values-based tool used to demonstrate and communicate the impacts of community-based work. It was designed for land conservation organizations, but can and has been used by those working in other fields including human services and public history. This methodology focuses on what I think of as measuring the un-measurables: showing compelling results in those qualitative areas of our work that defy quantification.

Knoll Farm in Vermont where I spent an immersive week with the Center for Whole Communities, learning about transformational leadership and measuring what matters.

One of the values we discussed was “The Power of Story.” There are currently many programs and activities that harness “The Power of Story” to interesting and valuable ends. Usually there are still numerical outcomes to measure, but a lot of times the power of story is something that is very hard to assess at an individual or a community level. To borrow an example outside of my organization (and with which I have no affiliation), StoryCorps is currently one of the most famous organizations utilizing and upholding the value of “The Power of Story,” across the nation. StoryCorps’s primary mission is to collect and preserve individual stories—something we can count—but its immense popularity has as much to do with how those stories affect radio listeners, the emotions they bring out in interview participants, and the sense of worth and importance it can bring to people who feel voiceless and forgotten. This is something much harder to quantify. The Whole Measures training challenged us to try to spend time thinking about how our work reflects our values and how it strengthens those values within the communities we serve.

I consider myself still at the beginning of my journey of being able to fully understand and communicate the impacts of our work, but the combination of training in both data-based and values-based measurements has given me a broad toolkit of strategies and has helped me to be more creative when planning and assessing my engagement activities. What strategies do you use to showcase the success of your programs externally? What measures do you use to determine the success of your efforts?

~Kate Preissler is the Western Region Engagement Manager for The Trustees of Reservations, coordinating points of contact between the public and about 40 historic, ecological, and recreational sites in the Berkshires, Pioneer Valley, and Central Massachusetts. The Trustees is a private organization whose mission is to permanently preserve properties of exceptional scenic, historic and ecological value in Massachusetts for public use and enjoyment. Kate holds a B.A. from Bates College in English and History and graduated from the UMass Amherst Public History Masters Program in 2010.