A client/partner spotlight from McREL
While most grant-funded projects—and their associated program evaluations—require flexibility, the Trinity School Design Network (TSDN) project, a multiyear cohort-based principal fellowship program, is an example of the importance of flexibility, adaptability, and perseverance. In five years, the project—supported by McREL’s research and evaluation team—survived a global pandemic, major personnel changes, funding cutbacks, and revised goals, and still wrapped up with a group of fellows who have grown as leaders and become agents of change in their communities.
“It’s a tribute to both their resilience and flexibility that they were able to complete the project with some really positive outcomes,” said Dr. Jeanette King from McREL, who served as an external evaluator of the project.
The disruptions also required forging new relationships mid-project, which can often spell disaster. However, through commitment, patience, and hard work, these relationships blossomed. And in the case of the program leaders and McREL’s evaluation team, it grew into a working dynamic that can be a model for other education projects that want the insights that can come from an external evaluation partnership but are unable to afford a traditional program evaluation.
“We’ve had a really good partnership with McREL.”
“Our partnership with TSDN is really a beautiful story about the creation of a new evaluation process that, even with limited resources, was still able to inform their project team, participants, and stakeholders about the project’s designs, processes, and outcomes,” said McREL evaluator Sayansk Queiroz Da Silva.
“We’ve had a really good partnership with McREL,” said Dr. Vangie Aguilera, director of TSDN. “They understood our struggles and took it all in stride. They were very accommodating and hands on, and they managed the data in a very organized and structured way that was very supportive.”
Project Background
Launched in 2019 and housed at Trinity University, TSDN aimed to promote broader systemic change by increasing the number of innovative school opportunities for marginalized students across the San Antonio area. The curriculum and learning experience was developed to be intense and rigorous to help fellows attain the critical skills and competencies necessary to design new schools with a lens focused on equity, empowerment, and innovation, according to Dr. Aguilera.
However, in a tale as old as 2020, in the first full year of the five-year project, the pandemic hit. Instead of realizing the dream of opening new schools, they watched as existing schools shut down and went remote. To put it lightly, “it was very bad timing for the project to start,” said Queiroz Da Silva.
With the realization that opening new schools in the current climate was not realistic, the project needed to shift. The original project leaders went back to the project’s funders, Walton Family Foundation, and changed the focus to transformational leadership. The participating fellows already were distinguished, change-oriented principals, and supporting their efforts to bring even more innovative changes into their existing schools was much more achievable than opening new schools, explained Dr. Aguilera.
However, since they were no longer opening new schools and the number of cohorts shrank from five groups to three, the grant funding was revised. Then amidst the pandemic upheaval, there was an abundance of staff changes across the board, leading to disengagement among participating fellows as the expectations and goals shifted. In 2021, as the third and final cohort of 10 fellows was finishing its first year of the program and getting ready to launch their school innovation projects, Dr. Aguilera joined the project as the new director, followed later by Dr. Avilés as the new coach/mentor.
“This program has been so effective and impactful, even though there were those kinds of interruptions, but it takes time to accept change,” Dr. Avilés explained. “It takes time to build trusting relationships. At the beginning, I was struggling to figure out how I can best work with the principal fellows to support them, show them I am advocating for them, and they can trust me,” she said. “But now, they have become innovative agents of change and transformational leaders.”
The fellows launched their designs in 2022–23 and received two years of coaching and school development before the project ended in May 2024.
McREL’s Responsive Changes to Program Evaluation Drives Engagement
While Dr. Aguilera and Dr. Avilés were building relationships with the existing fellows, McREL began working with the TSDN team to re-envision the program evaluation process in light of the changes to the project scope and funding. When the grant structure changed, the evaluation budget was trimmed to about 30% of what it had been. However, the assessment questions and the expectation to meet the needs of the client and its funder’s expectations remained. In other words, the McREL staff needed to do some extensive overhauling of the evaluation processes with limited resources.
At the beginning of the project, prior to COVID, McREL had outlined a traditional program evaluation model that relied heavily on student achievement data, state assessment data, interviews, and a lot of surveys administered to the leaders, the leaders’ supervisors, teachers, students, and family members.
However, when COVID hit, survey participation declined, interviews became more challenging to schedule and hold, and state assessments were suspended. During the interviews that did happen, the evaluators were hearing rich, informative stories that weren’t being captured in the data. “When rethinking how to do the evaluation, we knew that sharing these stories would be critical,” said King.
The McREL team also knew that in-person meetings remained an important piece of the evaluation puzzle even on the tighter budget. While the number and frequency of in-person meetings had to be reduced, a McREL evaluator still met in-person with the project leaders, conducted in-person focus groups with the fellows (in place of individual fellow interviews), and made school site visits. King, Aguilera, and Avilés got creative and found ways to meet up at conferences around the country that they were all attending for other purposes.
“We have this strong relationship and partnership, and they’re learning from us
as much as we’re getting the data from them.”
From this retooling came a more participatory model of evaluation, which also offered a quicker transfer of information between the fellows, the project’s leadership, and McREL’s evaluators. “We were able to build trust and shared purpose and it evolved into the project it is today, which is sort of a partnered evaluation. It’s cooperative. They’ve had to do a lot more evaluation lifting on their side, and we’ve been giving them just-in-time feedback all the time about how to do the work with less funding. We have this strong relationship and partnership, and they’re learning from us as much as we’re getting the data from them,” King said.
The TSDN folks started collecting data and uploading it to a shared folder, which was then analyzed by the McREL evaluation team. Part of the new process was expanding everyone’s definition of data. For example, when looking at family engagement, rather than basing the data on a survey that no one was taking, they looked to tell the story in other ways. The principal fellows started to share descriptions and photos of family events, along with attendance data. One school had a Parent Advisory Council that met regularly, so those meeting’s agendas became evidence.
“We looked beyond the traditional method of data collection,” King said. “The TSDN team and the fellows started to see data everywhere—which is how we live here at McREL, we see data everywhere— but I think that’s really helped them, and I think they will carry that forward.”
This new process took a couple months to get used to, but it was worth it. “It became more like a partnership than a client relationship,” said Queiroz Da Silva. “We’re working together to get this done and to see how these redesigns have affected the schools and the students, and that change really helped improve engagement.”
Since data was being collected on a continuous basis, information could be shared more frequently, further enhancing trust. “By making the TSDN team part of our data collection team, it helped strengthen our relationship, because they learned things from us in real time. We didn’t wait until the annual report to tell them that participants were feeling trust issues, and that they need to address it,” said King.
Evaluation with a Lowercase ‘e’
The new version of the annual report presented easy-to-read tables and figures that directly highlighted what the team collectively believed to be the essential pieces of the story. It highlighted the status of each outcome, used a lot of visualizations, and included key quotes. The report’s creation became participatory, with Dr. Aguilera and Dr. Avilés looking it over to ensure the story was complete. At times, they would offer suggestions of when to take questions back to the cohort to find out more information about a specific outcome.
“We wanted their input on how the report would make sense for them, their funders, and their stakeholders,” said Queiroz Da Silva. They had conversations about how the report would look and meet their needs, he said.
“This slimmed down, streamlined, equal partnership model of doing evaluation— maybe with a small e instead of a capital E—could benefit other innovative education projects or smaller projects that have limited room in their budget for evaluation,” said King.
Project Outcomes
The five-year TSDN program wrapped up in May 2024, and 4 of the 10 principal fellows in Cohort 3 have carried out their school redesigns and accomplished their goals, Dr. Aguilera said. China Cardriche redesigned her school, Compass Rose Legacy Secondary, to be a community school focusing on entrepreneurship. Dr. Megan Correia launched TriPoint Academy, San Antonio’s first free and public micro school, which is a community-designed school serving 50 students in small, mixed-age group pods. Rosie Hidalgo redesigned her school, Kriewald Road School of the Arts, to embed fine arts into core instruction. And Dr. Kelle Lofton redesigned Franz Elementary into Franz Leadership Academy as part of her district’s innovation efforts.
“We started with a very talented group of educators, and they all have stated that their leadership skills have gone to the next level,” said Dr. Avilés. “They all have shared that what they learned, the relationships and the networks that they built have been very impactful in their professional life.”
The McREL team also enjoyed the work, the partnership, and the project’s success. “I’m glad I was able to participate in helping them address the needs of those students in those communities and also help them navigate through those changes and the evaluation redesign,” said Queiroz Da Silva.