Many organizations and nonprofits seek to make a positive impact on the people they serve by providing programs and services. Significant resources, including time, staffing, and money, are invested in providing programs and services intended to bring upon certain outcomes. While observation can be an effective tool in assessing the successful fulfillment of outcomes related to a program or service, other methods of research and evaluation can provide data that can support program efficacy and guide improvement.
Kappa Alpha Order (KA), a national social fraternal organization, aims to provide meaningful educational experiences to its members through various leadership development programs. Having witnessed the positive effects of the programs on attendees, KA wanted to further understand how its programs were promoting learning among members. In 2017, KA embarked on a research journey to develop insights on how its programs were working and identify areas it could improve to elevate experiences for learners. Over the course of the last three years, KA has conducted intensive studies on two of its leadership development programs, Emerging Leaders Academy (ELA) and the Number Is Leadership Institute (NLI). Both programs aim to equip leaders with important knowledge and skills to assist in the membership experience. ELA engages rising leaders and prepares them to fulfill leadership roles within the chapter and throughout the campus. NLI involves leadership training for the Number Is, or presidents, from each active chapter and provisional chapter.
In this interview, Anthony Graziani, Assistant Executive Director for Chapter Services for Kappa Alpha Order, shares about KA’s research initiatives in the last few years and how Plaid has assisted along the way.
Plaid: What motivated KA to engage in research on its educational programs?
Anthony: “We have always known that our educational programs are beneficial and effective in preparing our members for leadership roles both within and outside of KA. In the past, we had used satisfaction surveys to measure the effectiveness of our programs which provided some great feedback and information about how well-received our programs are. However, surveying participants on how well they liked a program provides very little proof of how effective your programs are in meeting the program’s learning outcomes. We were interested in having verifiable research data that would give us some direction on the actual efficacy of our programs. It is easy to say anecdotally that your programs are effective but it is an entirely different thing to be able to back up that claim with actually third-party research data.”
Plaid: How has KA used research to promote its programs’ efficacy and assist in program improvement?
Anthony: “The study findings have been summarized for both the fraternity and foundation boards as well as shared in our magazine. The article in our magazine and summary report is now included in our expansion materials. We have reviewed the research findings in great detail to identify which aspects of our programs are most effective and which learning outcomes are not being fully addressed within our curriculum. We made several adjustments to our curriculum to ensure that we are effectively educating and achieving all of the programmatic learning outcomes.”
Plaid: What have been some of the greatest benefits of conducting research on KA’s programs?
Anthony: “It has challenged our staff to think critically about how we approach our program curriculum to focus on the effectiveness of the program instead of how well our members like a program. Just because participants enjoy the program doesn’t mean that it is effective in meeting the learning outcomes. The research project has raised the bar for our staff and challenged us to make adjustments to curriculum that has been recycled for several years. It’s easy to get into a rhythm of recycling the same curriculum year after year. This has allowed us to evaluate which aspects need to remain and which aspects need to be revisited. “
Plaid: How have your organizational stakeholders responded to the findings?
Anthony: “Everyone who has reviewed the results has been very impressed with the sophistication in which KA has approached its program review and feels more confident in both their financial and time commitment to the programs. Campus partners have also been impressed with our interest in doing this level of program review and excited to see that our members and their students are actually coming away from our programs with real leadership development improvement.”
Plaid: What findings from the research are your favorite or that you find most interesting?
Anthony: “In the NLI Post-Program assessment report, there was a statistically significant improvement for every learning outcome. That level of assurance made me feel confident that we are hitting the mark in every area of the program and that our Number I’s are leaving the program with the knowledge and education they need to be successful in their term.”
KA is one of the first in the fraternal world to make such an ongoing effort towards engaging in research on their programs. The insights the research has garnered has given KA greater insight and confidence in the work of its staff to create meaningful experiences and the investment of the organization to support such programs. Further, the research KA is conducting contributes to a larger body of research on the role of fraternal leadership experiences in the college student experience.
Plaid is honored to support KA throughout its research initiatives over the last three years. From assisting in the process of developing effective research designs to providing analytic support, Plaid’s partnership with KA has allowed meaningful insights that promote and prove the important experiences KA provides its members. Plaid’s team of researchers bring backgrounds in both quantitative and qualitative research to support clients in research design, data collection, analysis, and reporting. To learn more about how Plaid can support your organization’s research and assessment efforts, email info@beingplaid.com.
