Education Scotland as a Learning Organisation: Evaluating Professional Learning
The evaluation of professional learning and the planning of professional learning go hand-in-hand. The evaluation needs to be actively planned and prepared for at the inception of the professional learning. This is especially true if the need is to evaluate at a deeper level as this is more complex and time consuming.
There are several models for evaluating the impact of professional learning. Two popular ones are Kirkpatrick (2016) and Guskey (2002).
These frameworks are very similar, but Guskey’s framework has a more detailed focus on the learning of the people involved (participant, colleagues and learners) whilst Kirkpatrick has a bigger emphasis is more on behavioural change and wider setting impact.
Incorporating different aspects of these models provides a robust structure for evaluating professional learning, ensuring that both immediate and long-term outcomes are effectively measured and used for continuous improvement.
Both models cover different levels of evaluation from the surface level evaluation that would be done during the professional learning or soon after, and the deeper evaluation that would be done over time. It is important to note that this does not indicate a hierarchy of evaluation levels, all levels are valid and required for different purposes. However, it is also important to note that not every level of evaluation will be possible or necessary in every professional learning opportunity.
Both models also acknowledge that as the levels of evaluation become deeper, the learning being evaluated also becomes deeper and more transformative (Korthagen, 2017). Meaning that as the levels become deeper, the potential impact of the professional learning is not only cognitive but also touches on individual’s emotions, beliefs, values and motivations.
The professional/personal impact becomes more intertwined. At the deepest evaluative level this expands out to explore the transformative, ‘domino effect’ impact on whole learning communities and the people and systems that exist in them.
The amalgamation of these evaluation models is exemplified in the diagram below. Further detail on the key questions, methods and types of professional learning these models can apply to are exemplified in the accordions below.
The diagram shows an iceberg of participation from surface evaluation being the immediate impact on individual to deep evaluation; the long-term impact on learning community. As you move through the following five levels the learning becomes more transformative and evaluation becomes more complex:
Level 1: Participant Reaction & Engagement
Measure participants' initial reactions to the professional learning experience, including their engagement, satisfaction and perceived relevance.
Level 2: Initial Participant Learning
Evaluate the initial knowledge, skills and attitudes gained by participants as a result of the professional learning experience.
Level 3: Setting Application & Support
Evaluate how effectively participants apply their new knowledge and skills in their work environment, considering the support provided by their setting.
Level 4: Participant Behavioural & Practice Change
Evaluate the degree of participant behavioural change and improvement in professional practice resulting from the professional learning experience.
Level 5: Setting impact & systemic change
Evaluate the broader impact of the professional learning on the setting and people involved i.e. the learners & colleagues, including systemic changes, and long-term benefits.
Key Questions
How satisfied are participants with the professional learning experience? (content/facilitation/venue/platform)
Did they find the content relevant and useful to their professional needs?
Were they actively engaged during the sessions?
How likely are you to share your learning experience with others?
Possible Methods
Surveys and questionnaires immediately after the session.
Digital feedback forms, quick digital polls, or immediate interviews.
Observation of participant engagement during the sessions.
Social Media engagement stats.
Types of professional learning this level can apply to
This level of evaluation can be done for all kinds of professional learning, whether face-to-face or online, as it explores immediate reactions to the learning and it’s initial, perceived impact by the individual taking part.
Key Questions
What new knowledge or skills did participants acquire?
How have participants' attitudes towards this content changed? (i.e. confidence)
Do participant’s feel that they could/should implement this new learning into their practice?
Possible Methods
Surveys and questionnaires
Pre- and post-learning assessments (e.g. quizzes, tests).
Self-assessment tools or reflective writing.
Competency checklists or learning portfolios.
Types of professional learning this level can apply to
This level of evaluation can be done for all kinds of professional learning, whether face-to-face or online, as it explores immediate reactions to the learning and it’s initial, perceived impact by the individual taking part.
Key Questions
Are participants applying what they have learned in their professional practice?
What challenges do they face in applying new knowledge/skills?
How does the participant’s educational setting support the application of learning (e.g. resources, time, encouragement)?
Possible Methods
Follow-up surveys.
Interviews/Focus groups with participants and/or wider colleagues.
Observations in the workplace.
Case studies or examples of applied learning.
Practitioner enquiry/collaborative enquiry projects.
Action Plans.
Types of professional learning this level can apply to
This level of evaluation requires medium-long-term follow-up after the learning opportunity.
This follow-up could be in the form of additional learning sessions or longer-term evaluation.
This level of evaluation begins to explore the implications of the learning beyond the individual that took part.
Therefore it is unlikely this level can be achieved in a one-off learning opportunity.
Key Questions
How has participants' behaviour and/or practice changed as a result of the learning?
What evidence is there of professional growth in the participant?
Has the individual’s values/beliefs changed as a result of the learning?
What evidence is there of improved professional practice?
Possible Methods
Observation of participants' practice over time.
Longitudinal learning/reflective journal.
Feedback from peers, supervisors and learners.
Interviews.
Types of professional learning this level can apply to
This level of evaluation requires sustained and continuous involvement in the professional learning and multiple points of evaluation over time.
This level begins to explore changes to individual’s beliefs and values which is much more complex to evaluate than their actions (level 3).
Therefore it is unlikely this level can be achieved in a one-off learning opportunity.
Effective programmes of professional learning, for example, can achieve this level of impact.
Key Questions
What systemic changes have been implemented as a result of the learning?
Has the professional learning impacted colleagues and/or learners?
How has the professional learning contributed to educational setting’s goals (e.g. The improvement plan, improved learner outcomes etc)?
What is the long-term impact on the educational setting’s culture and/or practices?
Possible Methods
Case studies or success stories highlighting long-term impact.
Strategic Change Initiatives/Collaborative Enquiries.
Longitudinal interviews or focus groups with participants, colleagues, learners and wider setting community.
Analysis of relevant performance data, such as attainment data or attendance.
Types of professional learning this level can apply to
This level of evaluation requires sustained and continuous involvement in the professional learning and multiple points of evaluation over time.
This level also includes the impact of the learning on different groups of other people and not just the participant themselves which will require long-term collaboration between the learning community and the person doing the evaluation.
Therefore it is unlikely this level can be achieved in a one-off learning opportunity.
Effective programmes of professional learning, for example, can achieve this level of impact.
