At Spark Policy Institute (Spark) and ORS Impact (ORS), we have been doing systems building work for over a decade. When the Collective Impact approach came along, it created a level of clarity for many people, both about what it means to change systems as well as providing clear insights about how to do so.
And now, six years into the CI approach creating momentum and excitement, many systems change leaders find themselves asking the questions:
“Does the Collective Impact (CI) approach directly contribute to systems changes that leads to population changes? When does it contribute and in what ways? And most importantly, what does that mean for our work?”
We at ORS Impact and Spark Policy Institute are excited to have the opportunity to answer the first two questions in partnership with 30 collective impact sites in the US and Canada as part of the Collective Impact Research Study. Our goal is to provide the type of information that can help with the third question – putting the learning into action.
Research and evaluation of CI, particularly when testing the efficacy of the approach, must be approached differently than program evaluation or a more-straightforward descriptive study. It is not sufficient to collect self-report data about activities and changes occurring in the system, even with a verification process, without having a clear understanding of the types of changes that matter and the types of impact desired.
Consequently, our approach will consider how the external environment and CI initiatives have evolved over time and support an understanding of the causal relationship between CI efforts and their outcomes. As part of the study, we will seek to understand the range of ways CI is experienced and perceived, the implications of these differences on its effectiveness, and the implications for how the approach is deployed and supported.
Together, Spark and ORS bring extensive expertise in the study of complex initiatives. We know communities, organizations, and funders, and we know what it means to fully participate in a long-term initiative that involves multiple individuals, organizations, and systems moving toward a common goal of change. We also bring a healthy skepticism about the approach and how the five conditions and principles come together to drive systemic change.
We are also acutely aware of the need for a credible, actionable study. We will be following rigorous research practices and providing a high level of transparency around our methods. To that end, we want to share some high-level attributes of our study and lay out some of the content we will be providing along the way.
Research Study Phases
ORS and Spark are approaching this research in a multiphase process that will allow us to use multiple methods that will add rigor and enhance our ability to make useful comparisons across disparate sites while focusing on answering the primary causal question. Our research will occur through three phases:
- Develop a set of analytic rubrics that will provide the foundation for all our research activities. These analytic rubrics will be grounded in the conditions and principles of CI, as well as approaches for tracking systems changes, equity and population-level changes.
- Examine extant data, review documents, and collect new high-level data across a broad set of ~30 CI initiatives to understand more broadly how CI initiatives are implementing the conditions and principles of the approach and their systems change outcomes and population-level impacts. As you may have seen in outreach from the CI Forum, we used an open nomination process to help ensure our sample for this stage is broad and diverse in its initiative issue areas, origins, and funding sources.
- Dive more deeply in a focused group of 8 CI initiatives initially evaluated as part of the first phase of site analysis to better understand the conditions that support or impede population success. Our goal in this phase is to examine the implementation of the CI approach and more deeply understand the degree to which different causal explanations can be supported in different contexts and with differing levels of success in achieving population outcomes. We are using a method called process tracing, which is a qualitative analysis approach that helps understand causal inferences by interrogating rival hypotheses to explain changes observed (we will describe process tracing in detail in a future blog post).
Future Blog Topics
To continue in our efforts to bring transparency to this work, we will be blogging each month about this study, presenting our methods and specific rubrics we will be using as well as providing examples and lessons learned. Please check back each month for blogs on the following topics.
- Early June: Design details and list of sites being included in the study.
- June and July: Three-part series discussing the rubrics being used for this study: CI, systems change, and equity.
- August: A description of process tracing and an example.
- September: Key lessons from untangling cause and effect of CI and population outcomes.
- October: A case study example from one site evaluated.
- November/December: Key findings from the study.
- January: Final report release via the CI Forum.
We encourage you to share any of your insights about CI in the comments section below!