The University is committed to developing the strongest possible evidence to inform the evaluation of the impact of the intervention strategies. The Access Participation Evaluation Framework is the cornerstone of the strategic approach to evaluation. It provides the tools for the design of the activities, assess the impact of this work, and add to the sector knowledge base on what works (and does not work) to achieve equality of opportunity in higher education.
Access and Participation Evaluation Framework
The Framework consists of two resources: the Evaluation Guidance and Toolkit and the APP Knowledge Hub, which are available to internal and sector colleagues. Support is provided to help colleagues to use the guidance and to develop their evaluation practice.
The resources draw on OfS materials and the knowledge and frameworks developed by sector networks, including NERUPI, TASO NEON and FACE.
Theory of Change methodology is central to the Framework and underpins the approach to evidence and impact evaluation. The intervention strategies have been designed using the Theory of Change to plan, monitor and evaluate activities. The framework will be used to strengthen institutional evaluation activity, including activities and services not directly linked to the plan.
Evaluation Guidance and Toolkit
The evaluation guidance and toolkit provide resources, templates and guidance to support colleagues at the start of their evaluative journey and to develop their evaluative expertise. The guidance unpacks Theory of Change, the different types of evaluation (including levels of evidence), examples of evaluation methods and methodologies, guidance on analysis, reporting, dissemination, GDPR and ethics.
The toolkit signposts datasets that can be used for evaluation and monitoring purposes (including HESA and GO data) as well as tracking data from the East Midlands Widening Participation Research and Evaluation Partnership that is key to evaluating the impact of outreach activity.
The University has also developed an APP student characteristics dataset to allow monitoring of which students access and engage with services and activities, target student groups, and evaluate impact of activity by student characteristics. This dataset will help to monitor the effectiveness of the intervention strategy activities.
APP Knowledge Hub
The APP Knowledge Hub is a themed repository where colleagues can record and share evidence/evaluation findings; monitor the progress and impact of the intervention strategy activities; share emerging reports, literature, research and best practice; learn about projects and activities taking place across the institution and the sector more broadly; and facilitate collaboration. The resource is curated from contributions from internal and sector colleagues. The University will publish the findings of its APP evaluation activity on the APP Knowledge Hub.
Evaluation Principles
The following principles underpin the approach to evaluation and impact reporting:
- Evidence-based culture: The University has an ambitious commitment to create an evidence-based and impact focused culture across the institution. Colleagues and students will be supported to develop their evaluative skills and practice. Through the framework, support will be provided to embed high quality, mixed methods evaluation within practice. Type 1/2 levels of evidence (including the use of comparator groups) will be embedded across activity linked to the intervention strategies. Opportunities will be taken to embark on a journey towards conducting Type 3 levels of evidence collaboratively, with other providers with the support of TASO, when/if an appropriate opportunity can be developed.
- Co-creation: The University will create and conduct research and evaluation with the Union of Students and the wider student body. Where relevant, a participatory methodology will be adopted for evaluation and research design to analyse and disseminate findings.
- Collaboration: Engagement and co-operation with sector partners will be sought, to pool resources and expertise to conduct a high standard of evaluation and share evaluation practices and findings with sector colleagues. As members of a NERUPI Collaborative Enhancement project the University will help train a pool of impartial sector evaluation peer reviewers. On completion of the programme, the group will work together on collaborative evaluation projects and be critical friends. The group will be available as a resource for sector colleagues to consult regarding their evaluation activity and planning.
- Curation of evidence: The University continues to grow an institutional evidence-base (Knowledge Hub) of quantitative and qualitative research and evaluation findings (generated both internally and externally), examples of sector best practice, third-party reports and conference materials. The strongest possible evidence informs the creation of our intervention strategies and supports the review and refinement of institutional practice. Colleagues with responsibility for intervention strategy activity engage with the latest research and learning in their fields and apply scholarship and development to their practice.
- Monitoring: Evaluation of differential outcomes for target student groups is embedded within the University’s module and programme continual monitoring processes reporting into academic governance and committee structures to secure accountability for the outcomes of the plan. This allows the active monitoring of the impact of intervention strategy, respond to emerging evaluation findings. and assess the overall progress being made towards the objectives and targets. The committee structure includes College APP groups, which report to the University APP Steering Group (led by a member of the University Executive team), which reports into both Academic Board and the University Executive.
- Dissemination: The University contributes actively to sector network events, conferences, and webinars through presentations on the APP evaluation framework and evaluation and research findings, linked to the APP. In addition, findings are published through several channels, including conference publication chapters and blogs. In future this work will expand to meet the commitment to the publication of APP evaluation and research. The University is fully engaged with TASO and will be a committed contributor to the network to fully support and engage with them in the role that they will have in disseminating APP evaluation evidence.
- Research: Understanding the experiences of students and the risks that they face is key to the University’s strategy. The APP has been co-created with students and it draws upon the outcomes from research into sense of belonging and lived experience conducted in partnership with the Union of Students. The University has also worked in partnership with several institutions (through NERUPI) on a research project led by the University of Kent on culturally sensitive curriculum scales. Internal research into ‘areas of interest’ identified through evaluation will continue and be supported by opportunities to work in collaboration with research and evaluation projects across the sector.
Students can bid for funding to conduct student led research into barriers to equality of opportunity as a strand within the University Research Scholarship Scheme. Providing a means to undertake either individual or collaborative research to inform the direction of the APP work. Funding for a PhD studentship has been ringfenced to investigate issue as specifically associated with the Black awarding gap and evaluate interventions to address this through curriculum design and delivery.