Blog Viewer

Understanding and Measuring Systems Change


By Kathy Booth, INCLUDES Coordination Hub Co-Principal Investigator

In August, members of NSF’s Eddie Bernice Johnson INCLUDES National Network (National Network) met in Washington, DC, to share strategies for broadening participation in STEM and learn how our efforts are addressing systems change. At the opening plenary, members of the INCLUDES Coordination Hub (Hub) outlined the relationship between the INCLUDES Theory of Change, strategies associated with building collaborative infrastructure, and the conditions for systems change, with a focus on how the INCLUDES Initiative will measure outcomes in each domain. This post shares that framing and provides resources you can use to identify how your work addresses both underlying structures and individual accomplishments.

The INCLUDES Theory of Change is predicated on forming a coalition of the willing. The National Network, with the support of the Hub, creates both virtual and in-person spaces for like-minded people to find one another, share their experiences, and learn from one another. This is the community that is choosing to build something together. The Hub envisions that one way to measure success for community engagement is the growth of the National Network; another could be how frequently National Network members co-create opportunities for engagement with the Hub.

The INCLUDES Initiative adopted collaborative infrastructure as a framework to nurture connections. Collaborative infrastructure includes the concepts of:

  • Shared vision: a common agenda that reflects a collective understanding of the broadening participation challenge and links to existing research, promising practices, and/or the previous and ongoing activities of partnering organizations
  • Partnerships: primary organizations and additional partners that come together locally, regionally, nationally, by disciplinary focus, or by other multisector categories
  • Goals and metrics: well-defined, relevant goals and measurable objectives and outcomes, including progress indicators
  • Leadership and communications: internal and external communication plans and strategies for developing leadership capacity within and among all partnering organizations
  • Expansion, sustainability, and scale: plans for expansion, sustainability, and scale that contribute to broadening participation in the nation’s scientific and engineering workforce

You can think of collaborative infrastructure as the framing for a house — a strong foundation with anchors that connect the support beams to that foundation, which ensures the community is helping to design solutions that dismantle inequity. The Hub measures how projects are building collaborative infrastructure through surveys of National Network members.

Creating a strong collaborative infrastructure is in service to systems change. Systems change is the blueprint that ensures we are building a sturdy, more inclusive space that supports a sense of belonging for diverse groups. If we focus on modifying the infrastructure, which plays an enormous role in who gets to participate and how, we will help many more people enter and stay in STEM. It means creating a graceful entrance ramp rather than attaching a lift to a steep set of stairs — or designing open, well-lit spaces where it is possible to work in community, rather than a warren of dark rooms that isolate individuals. Systems change means we are designing for everyone, sustainably, rather than asking people to fit into systems that can be narrow and exclusionary.

There are many ways to define systems change. In our case, the Hub is mapping the work of the National Network to the FSG Water of Systems Change framework. FSG defines systems change as shifting the conditions that hold a problem in place. It names six interdependent conditions that must be addressed to create sustainable change.

What the conditions of systems change might look like in practice to broaden participation in STEM

A coalition of organizations is seeking to strengthen transfer pathways from community college construction programs to engineering majors at 4-year institutions. First, the coalition developed a set of shared objectives related to explicit changes, including altering practices by working together to provide mentoring opportunities for community college students, changing policies by altering articulation agreements so that math courses taught in community college construction programs count toward STEM bachelor’s degree requirements, and adjusting resource flows by devoting more funds to onboarding transfer students. At the semi-explicit level, the coalition strengthened relationships by convening 2- and 4-year faculty to build a sense of shared purpose. The coalition addressed power dynamics by creating a cross-education system coordinating committee in which community colleges are on equal footing with 4-year partners with regard to decision-making. Finally, to address implicit mental models, the coalition worked with both educators and employers to make a case for why people who build knowledge in applied settings are uniquely qualified for engineering jobs, as is the case for community college students who work in the building trades.

Structural changes are the types of work that many of us think about first when we seek to broaden participation:

  • Policy: governmental, institutional, or organizational rules, regulations, and/or priorities
  • Practices: activities, procedures, guidelines, or informal shared habits   
  • Resource flows: how money, people, knowledge, information, and/or other assets such as infrastructure are allocated and distributed

However, the progress we may make through discrete efforts is less likely to become embedded without relational changes:

  • Relationship and connections: the quality and kinds of connections and communications among actors in the systems, especially among those with differing histories and viewpoints
  • Power dynamics: the distribution of decision-making power, authority, and formal and informal influence among individuals and organizations

Finally, shifts are more likely to endure once transformative change is attained because the way we understand problems or communities often defines which solutions seem possible. Transformative change refers to shifting:

  • Mental models: deeply held beliefs and assumptions, and taken-for-granted ways of operating that influence how we think, what we do, and how we talk

It is important to think about what we choose to measure and how we measure it in the context of systems change. If we are focused primarily on supporting students to enter STEM majors in college and our efforts stop there, we may leave in place hostile conditions in the academy or the workplace. While we may see enrollment or hiring numbers increase, we may see far fewer individuals graduating or staying in those jobs. That is why it is important to pair actions and numbers on student engagement and outcomes with actions and measures of relationships, power, and mindsets. Those measures may be numbers, but they may also be documented through survey responses, case studies, focus groups, and narratives. In addition, we can use data to better understand the underlying structures that are holding inequities in place and determine where to focus efforts. Therefore, the Hub will be gathering initial information about systems change efforts from surveys of National Network members, paired with analyses that describe how those efforts were implemented and stories that share the experiences of people who benefited from those actions.

If you’d like to explore how the INCLUDES Initiative is supporting systems change further, here are several resources:

  • The Hub developed a 10-page brief on “Demystifying Systems Change” that maps out the connections between collaborative infrastructure and the FSG Water of Systems Change framework.
  • The Hub developed a Systems Change Worksheet you can use to identify how your projects address the different conditions of systems change.
  • FSG has posted the recording of a 1-hour webinar that provides additional examples of how the Water of Systems Change framework can inform practice.