Department of Natural Resources and Environment, Box 3100, Bendigo Delivery Centre, Vic, 3554
Abstract
Farm$mart Action Learning Sets, an initiative of the Farm$mart Quality Staff Development group, were the result of a search for an effective method for implementing self-directed learning with the aim of improving staff competency.
The aim of the evaluation project was to determine if the Farm$mart Action Learning Sets were a successful vehicle to improve staff skills and ultimately if it would be worthwhile to pursue this type of staff development again in the future. The purpose of this paper is to document the outcomes of the evaluation project.
Bennett’s hierarchy of evaluation criteria was used for the investigation and the results presented are drawn from eleven out of fourteen responses gained within the set timeframe.
A snapshot of the results shows that:
Improved knowledge in chosen skill area, group benefits and making new networks were positive aspects of Action Learning Sets. Frustrating or negative aspects of Action Learning Sets were travel, the facilitator’s role, comfort level and small group. Many found that the learning journal was useful for reflection however it was not useful for those who could not force themselves to record their learnings. 91% participants increased their skills in a chosen competency area. Greater than 70% participants see Action Learning Sets as a valuable way to improve their ability in a particular skill area. Greater than 73% felt they reached their learning outcome and almost 30% achieved more than they expected to. 80% would participate in another Action Learning Set. 90% would use what they have learnt in their role. Participants enjoyed the opportunity to gain knowledge from others in their group and have gained a greater awareness of what learning style is preferred. As an additional outcome of participating in an Action Learning Set, participants have gained a greater awareness of Action Learning.
ALS provided Farm$mart staff with a good learning environment allowing the majority of participants to improve a chosen core competency required for their job. The Sets provided strong support and enlisted a sense of commitment from group members to achieve their project objectives. An added benefit of involvement in ALS was the ability to form strong relationships with staff they would not normally work with and gain knowledge from others beyond the scope of their own project.
To sum up the investigation, I believe Action Learning Sets are a successful vehicle to improve staff skills.
Introduction
Action Learning Sets (ALS) are based on the action learning cycle and the underlying principle of learning by doing. The Sets consist of a small group of no more than 5or 6 participants who meet regularly and frequently. Each member has a self-selected task to undertake, preferably one that they are passionate about. When the group meets, each member is given their own air space or time for talking. Other group members primarily use a questioning approach during the session, giving no advice and few solutions, thus not judging but supporting and challenging everyone. The Sets focus on the learning that occurs while the task is undertaken, as much as task itself. Action Learning Sets probably work best with set facilitator to encourage the process.
Farm$mart1 Action Learning Sets, an initiative of the Farm$mart Quality Staff Development (QSD) group, were the result of a search for an effective method for implementing self-directed learning with the aim of improving staff skills. Farm$mart facilitators (staff) were encouraged to become involved to enhance their skills in specific competencies as well as to enhance their understanding of action learning.
The Farm$mart QSD group recognised the importance of evaluating the process and the outcomes of this ALS program. Thus, the aim of the evaluation project was to determine if Farm$mart ALS were a successful vehicle to improve staff skills. It was hoped that the appropriate evaluation method would help the QSD group to decide if the program should be continued, identify ways to improve its goals and delivery and record the programs accomplishments. The purpose of this paper is to document these outcomes of the evaluation project.
Methods
Bennett’s hierarchy of evaluation criteria was used to provide a framework to target outcomes and then to assess the degree to which these targets are reached. This method of evaluation was determined to be the most useful method to evaluate the ALS process and the program outputs as well as benefits or changes for individuals. The hierarchy of questions, criteria and evidence are presented in table 1.
Fifteen evaluation questions (refer table 1), closed and open, were developed to collect the evidence required at each level of the hierarchy. The survey questions, a combination of pictures and text, (refer to appendix 1) were mailed in advance to142 of the 18 ALS participants. Responses were gathered mostly by phone, with the balance received via return mail. Two weeks was allowed for data collection.
Table 1. Bennett’s hierarchy of evaluation criteria, evidence and evaluation questions.
Level of Evidence |
Evidence |
Evaluation Questions |
END RESULT |
Will not be evaluated |
|
PRACTICE CHANGE |
Participants use their new skills. |
15. What have you changed/done differently as a result of the learnings in your ALS? How have you used your learnings since starting the ALS? Where have you used your new skills? |
KASA CHANGE |
Knowledge – Participants increased their competency/skill level/ability in their chosen skill area. [80% achieve increase in ability in chosen skill area.] |
14. What did you learn about your learning style? |
REACTIONS |
80% Participants report favourably on the ALS. |
7. What was BAD or FRUSTRATING about your ALS? 6. What was GOOD about your ALS? |
PEOPLE INVOLVEMENT |
All participants attended 80% of meetings. |
2. How many meetings did you attend? 3. What stopped you from attending? |
ACTIVITIES |
One three hour facilitated meeting each month for six months. |
13. What was the skill area you were hoping to improve? |
INPUTS |
Four Groups, four facilitators |
Eleven of fourteen surveys were completed in the set timeframe. Once received, quantitative data were listed and qualitative data were tabulated. Insights were derived by comparing the expected evidence (refer table 1) with the actual evidence received from the returned surveys.
Results
The results presented in the next section are drawn from 11 responses gained within the set timeframe of two weeks and are layed out in ascending order of the hierarchy, one level at a time.
INPUTS: the resources allocated to the project
Table 2. Comparison of evidence expected and the actual situation at the Input level of evaluation.
Expected Evidence |
Actual Evidence |
Four groups |
Four groups |
Four facilitators |
Four facilitators |
18 participants |
18 participants |
One project per participant |
One project per participant |
Learning journal |
Learning journal |
Table 3. Number of participants undertaking a project that aims to improve their competency in that skill area
Number of Participants |
Skill area |
1 |
Understanding program attributes |
4 |
Strategic Planning |
1 |
Adult Learning |
1 |
Action Learning |
4 |
Group Facilitation |
0 |
Marketing, Planning & Promotion |
1 |
Evaluation |
1 |
Development of community & professional networks |
3 |
Personal performance and development |
1 |
Management of change |
0 |
Technical knowledge |
Insights:
- More participants chose the area of Strategic Planning & Group facilitation
- Technical knowledge & Marketing, Planning, Promotion was not the focus of any project undertaken.
- Some participants chose more than one skill area for their project.
ACTIVITIES: various events or strategies used to inform or educated the audience
Table 4. Comparison of evidence expected and the actual situation at the Activities level of evaluation.
Expected Evidence |
Actual Evidence |
One three hour facilitated meeting each month for six months |
One three hour facilitated meeting each month for six months |
Reporting back session at Lorne |
Reporting back session at Lorne |
PEOPLE INVOLVEMENT: involvement of participants in the program
Table 5. Comparison of evidence expected and the actual situation at the People Involvement level of evaluation.
Expected Evidence |
Actual Evidence |
All participants attended 80% of meetings |
90% of participants attended 80% of meetings |
All participants get involved in the session at Lorne |
All participants got involved in the session at Lorne |
Insights:
- Although only 90% of participants attended the required 80% of meetings, the reasons for not attending varied from illness to length of trip to other obligations.
REACTIONS: degree of positive or negative interest in the program, their acceptance of leaders and attraction to methods
Table 6. Comparison of evidence expected and the actual situation at the Reactions level of evaluation.
Expected Evidence |
Actual Evidence |
80% participants report favourably on the ALS |
80% participants report favourably on the ALS |
80% used a learning journal |
73% used a learning journal |
80% filled in Part A and Part B |
82% filled out Part A and Part B |
Insights:
- Improved knowledge in skill area, group benefits and making new networks were good things about ALS.
- Travel, the facilitator’s role, comfort level and small group were some of the frustrating things about ALS.
- The learning journal was useful for reflection.
- It was not useful for those who cannot force themselves to record their learnings.
- Participants tended to fill out Part A and B to focus and clarify their project.
KASA CHANGE: Knowledge, Attitude, Skills, and Aspirations that influence the adoption
Table 7. Comparison of evidence expected and the actual situation at the KASA change level of evaluation.
Expected Evidence |
Actual Evidence |
Knowledge: 80% participants increased their competency/skill level/ability in their chosen skill area |
91% participants increased their competency/skill level/ability in their chosen skill area |
Attitude: Participants see ALS as a valuable way to improve their ability in a particular skill area |
>70% participants see ALS as a valuable way to improve their ability in a particular skill area |
Skills: Participants feel they have more skills in the chosen skill area and can use them in their role. |
91% participants feel they have more skills in the chosen skill area and can use them in their role. |
Aspirations: Participants intend to improve other areas on their skill wheel through ALS |
|
Insights:
- Participants enjoyed the opportunity to gain knowledge from others in their group.
- Participants gained a greater awareness of what learning style is preferred.
PRACTICE CHANGE: Change in patterns of behaviours that influence the Social, Environmental and Economic conditions.
Table 8. Comparison of evidence expected and the actual situation at the Practice change level of evaluation.
Expected Evidence |
Actual Evidence |
|
Participants used the greater awareness of learning styles in role |
General comments
Below are general comments expressed by participants in the evaluation outside the scope of the evaluation questions:
- Our group was very powerful
- Learnt a lot about myself
- Learning was deep and profound
- Learnt a lot more from being in an ‘unchosen’ group
- Some people seemed not to enjoy ALS
- ALS was worthwhile for me but I had to be very committed
- It was valuable to understand how ALS work
- Disappointed with the role the facilitator played – too stand offish, acted like they were the fountain of all wisdom but wouldn’t share
- ALS was great with respect to getting to know people
- Are learnings from participation in ALS an effective use of time, money & resources?
Conclusion
- The Action Learning Sets are a successful vehicle to improve staff skills.
- ALS provided Farm$mart staff with a good learning environment allowing the majority of participants to improve a chosen core competency required for their job.
- ALS provided strong support and enlisted a sense of commitment from group members to achieve their project objectives.
- An added benefit of involvement in ALS was the ability to form strong relationships with staff they would not normally work with and gain knowledge from others beyond the scope of their own project.
- Many participants commented on the benefit of also learning about action learning, how they learn and how they can apply this to their role in working with groups and adult learning.
- Some of the difficulties to overcome would be distance of travel for group members, clarity about the size of the commitment that is required, ensuring the facilitator creates a comfortable learning environment and ensuring participants are clear about what they would gain from ALS.
To sum up the investigation, I believe Action Learning Sets are a successful vehicle to improve staff skills.
References
- Bennett, C. (1976) Up the hierarchy. In Journal of Extension, Vol 4., pp18-26 March/April, U.S.A.
- Proc. 3rd National Property Management Planning Forum (1999). 22-25 March, Mildura, Vic.
- Targeted outcomes of programs (TOP) Indicators and evaluation, 2000. Web site. http://deal.unl.edu/TOP/indicators.html
Appendix 1
1 Farm$mart is the Victorian Property Management Planning (PMP) Program
2 Four ALS participants requested to be omitted from the evaluation.
3 Numbered in order of how the questions would be asked.