Abstract

Farm$mart Action Learning Sets, an initiative of the Farm$mart Quality Staff Development group, were the result of a search for an effective method for implementing self-directed learning with the aim of improving staff competency.

The aim of the evaluation project was to determine if the Farm$mart Action Learning Sets were a successful vehicle to improve staff skills and ultimately if it would be worthwhile to pursue this type of staff development again in the future. The purpose of this paper is to document the outcomes of the evaluation project.

Bennett’s hierarchy of evaluation criteria was used for the investigation and the results presented are drawn from eleven out of fourteen responses gained within the set timeframe.

A snapshot of the results shows that:

Improved knowledge in chosen skill area, group benefits and making new networks were positive aspects of Action Learning Sets. Frustrating or negative aspects of Action Learning Sets were travel, the facilitator’s role, comfort level and small group. Many found that the learning journal was useful for reflection however it was not useful for those who could not force themselves to record their learnings. 91% participants increased their skills in a chosen competency area. Greater than 70% participants see Action Learning Sets as a valuable way to improve their ability in a particular skill area. Greater than 73% felt they reached their learning outcome and almost 30% achieved more than they expected to. 80% would participate in another Action Learning Set. 90% would use what they have learnt in their role. Participants enjoyed the opportunity to gain knowledge from others in their group and have gained a greater awareness of what learning style is preferred. As an additional outcome of participating in an Action Learning Set, participants have gained a greater awareness of Action Learning.

ALS provided Farm$mart staff with a good learning environment allowing the majority of participants to improve a chosen core competency required for their job. The Sets provided strong support and enlisted a sense of commitment from group members to achieve their project objectives. An added benefit of involvement in ALS was the ability to form strong relationships with staff they would not normally work with and gain knowledge from others beyond the scope of their own project.

To sum up the investigation, I believe Action Learning Sets are a successful vehicle to improve staff skills.

Introduction

Action Learning Sets (ALS) are based on the action learning cycle and the underlying principle of learning by doing. The Sets consist of a small group of no more than 5or 6 participants who meet regularly and frequently. Each member has a self-selected task to undertake, preferably one that they are passionate about. When the group meets, each member is given their own air space or time for talking. Other group members primarily use a questioning approach during the session, giving no advice and few solutions, thus not judging but supporting and challenging everyone. The Sets focus on the learning that occurs while the task is undertaken, as much as task itself. Action Learning Sets probably work best with set facilitator to encourage the process.

Farm$mart¹ Action Learning Sets, an initiative of the Farm$mart Quality Staff Development (QSD) group, were the result of a search for an effective method for implementing self-directed learning with the aim of improving staff skills. Farm$mart facilitators (staff) were encouraged to become involved to enhance their skills in specific competencies as well as to enhance their understanding of action learning.

The Farm$mart QSD group recognised the importance of evaluating the process and the outcomes of this ALS program. Thus, the aim of the evaluation project was to determine if Farm$mart ALS were a successful vehicle to improve staff skills. It was hoped that the appropriate evaluation method would help the QSD group to decide if the program should be continued, identify ways to improve its goals and delivery and record the programs accomplishments. The purpose of this paper is to document these outcomes of the evaluation project.

Methods

Bennett’s hierarchy of evaluation criteria was used to provide a framework to target outcomes and then to assess the degree to which these targets are reached. This method of evaluation was determined to be the most useful method to evaluate the ALS process and the program outputs as well as benefits or changes for individuals. The hierarchy of questions, criteria and evidence are presented in table 1.

Fifteen evaluation questions (refer table 1), closed and open, were developed to collect the evidence required at each level of the hierarchy. The survey questions, a combination of pictures and text, (refer to appendix 1) were mailed in advance to14² of the 18 ALS participants. Responses were gathered mostly by phone, with the balance received via return mail. Two weeks was allowed for data collection.

Table 1. Bennett’s hierarchy of evaluation criteria, evidence and evaluation questions.

Level of Evidence	Evidence	Evaluation Questions
END RESULT	Will not be evaluated
PRACTICE CHANGE	Participants use their new skills.	15. What have you changed/done differently as a result of the learnings in your ALS? How have you used your learnings since starting the ALS? Where have you used your new skills?
KASA CHANGE	Knowledge – Participants increased their competency/skill level/ability in their chosen skill area. [80% achieve increase in ability in chosen skill area.] Attitude - Participants see ALS as a valuable way to improve their ability in a particular skill area. Skills - Participants feel they have more skills in the chosen skill area and can use them in their job/role. Participants feel they are higher on the skill wheel scaling for that skill area. Aspirations - Participants intend to improve other areas on their skill wheel through ALS.	14. What did you learn about your learning style? 13. How did ALS contribute to your skill development? 12. How valuable to you was participating in ALS to increase your ability in your chosen skill area? 11. Do you feel capable to use what you have learnt in your role as Farm$mart facilitator? 10. Given the opportunity, do you think you would participate in another ALS? 9. Do you feel that your skill wheel has expanded in your chosen skill area? How much? 8. Do you feel you have: not reached; reached; exceeded; your learning objective?
REACTIONS	80% Participants report favourably on the ALS. 80% used learning journal. 80% Participants filled in Part A & B.	7. What was BAD or FRUSTRATING about your ALS? 6. What was GOOD about your ALS? 5. Did you fill in Part A-Learning Plan and Part B –Learning outcomes? Why or Why not? 4. Did you use a learning journal? In what way was it useful/not?
PEOPLE INVOLVEMENT	All participants attended 80% of meetings. All participants participated in workshop at Lorne.	2. How many meetings did you attend? 3. What stopped you from attending?
ACTIVITIES	One three hour facilitated meeting each month for six months. Reporting back workshop at Lorne.	13. What was the skill area you were hoping to improve?
INPUTS	Four Groups, four facilitators 18 participants in total Learning journal. Learning contract

Eleven of fourteen surveys were completed in the set timeframe. Once received, quantitative data were listed and qualitative data were tabulated. Insights were derived by comparing the expected evidence (refer table 1) with the actual evidence received from the returned surveys.

Results

The results presented in the next section are drawn from 11 responses gained within the set timeframe of two weeks and are layed out in ascending order of the hierarchy, one level at a time.

INPUTS: the resources allocated to the project

Table 2. Comparison of evidence expected and the actual situation at the Input level of evaluation.

Expected Evidence	Actual Evidence
Four groups	Four groups
Four facilitators	Four facilitators
18 participants	18 participants
One project per participant	One project per participant
Learning journal	Learning journal

Table 3. Number of participants undertaking a project that aims to improve their competency in that skill area

Number of Participants	Skill area
1	Understanding program attributes
4	Strategic Planning
1	Adult Learning
1	Action Learning
4	Group Facilitation
0	Marketing, Planning & Promotion
1	Evaluation
1	Development of community & professional networks
3	Personal performance and development
1	Management of change
0	Technical knowledge

Insights:

• More participants chose the area of Strategic Planning & Group facilitation
• Technical knowledge & Marketing, Planning, Promotion was not the focus of any project undertaken.
• Some participants chose more than one skill area for their project.

ACTIVITIES: various events or strategies used to inform or educated the audience

Table 4. Comparison of evidence expected and the actual situation at the Activities level of evaluation.

Expected Evidence	Actual Evidence
One three hour facilitated meeting each month for six months	One three hour facilitated meeting each month for six months
Reporting back session at Lorne	Reporting back session at Lorne

PEOPLE INVOLVEMENT: involvement of participants in the program

Table 5. Comparison of evidence expected and the actual situation at the People Involvement level of evaluation.

Expected Evidence	Actual Evidence
All participants attended 80% of meetings	90% of participants attended 80% of meetings
All participants get involved in the session at Lorne	All participants got involved in the session at Lorne

Insights:

• Although only 90% of participants attended the required 80% of meetings, the reasons for not attending varied from illness to length of trip to other obligations.

REACTIONS: degree of positive or negative interest in the program, their acceptance of leaders and attraction to methods

Table 6. Comparison of evidence expected and the actual situation at the Reactions level of evaluation.

Expected Evidence	Actual Evidence
80% participants report favourably on the ALS	80% participants report favourably on the ALS
80% used a learning journal	73% used a learning journal
80% filled in Part A and Part B	82% filled out Part A and Part B

Insights:

• Improved knowledge in skill area, group benefits and making new networks were good things about ALS.
• Travel, the facilitator’s role, comfort level and small group were some of the frustrating things about ALS.
• The learning journal was useful for reflection.
• It was not useful for those who cannot force themselves to record their learnings.
• Participants tended to fill out Part A and B to focus and clarify their project.

KASA CHANGE: Knowledge, Attitude, Skills, and Aspirations that influence the adoption

Table 7. Comparison of evidence expected and the actual situation at the KASA change level of evaluation.

Expected Evidence	Actual Evidence
Knowledge: 80% participants increased their competency/skill level/ability in their chosen skill area	91% participants increased their competency/skill level/ability in their chosen skill area
Attitude: Participants see ALS as a valuable way to improve their ability in a particular skill area	>70% participants see ALS as a valuable way to improve their ability in a particular skill area
Skills: Participants feel they have more skills in the chosen skill area and can use them in their role. Participants feel they are higher on the skill wheel scaling for that skill area.	91% participants feel they have more skills in the chosen skill area and can use them in their role. >73% reached their learning outcome and almost 30% achieved more than they expected to. All participants feel they are higher on the skill wheel scaling for that skill area.
Aspirations: Participants intend to improve other areas on their skill wheel through ALS	80% would participate in another ALS

Insights:

• Participants enjoyed the opportunity to gain knowledge from others in their group.
• Participants gained a greater awareness of what learning style is preferred.

PRACTICE CHANGE: Change in patterns of behaviours that influence the Social, Environmental and Economic conditions.

Table 8. Comparison of evidence expected and the actual situation at the Practice change level of evaluation.

Expected Evidence	Actual Evidence
Participants use their new skills	Participants used the greater awareness of learning styles in role Participants have used their new skills to help themselves flex Participants have used knowledge gained to help others move through the entire action learning cycle 90% would use what they have learnt in their role

General comments

Below are general comments expressed by participants in the evaluation outside the scope of the evaluation questions:

• Our group was very powerful
• Learnt a lot about myself
• Learning was deep and profound
• Learnt a lot more from being in an ‘unchosen’ group
• Some people seemed not to enjoy ALS
• ALS was worthwhile for me but I had to be very committed
• It was valuable to understand how ALS work
• Disappointed with the role the facilitator played – too stand offish, acted like they were the fountain of all wisdom but wouldn’t share
• ALS was great with respect to getting to know people
• Are learnings from participation in ALS an effective use of time, money & resources?

Conclusion

• The Action Learning Sets are a successful vehicle to improve staff skills.
• ALS provided Farm$mart staff with a good learning environment allowing the majority of participants to improve a chosen core competency required for their job.
• ALS provided strong support and enlisted a sense of commitment from group members to achieve their project objectives.
• An added benefit of involvement in ALS was the ability to form strong relationships with staff they would not normally work with and gain knowledge from others beyond the scope of their own project.
• Many participants commented on the benefit of also learning about action learning, how they learn and how they can apply this to their role in working with groups and adult learning.
• Some of the difficulties to overcome would be distance of travel for group members, clarity about the size of the commitment that is required, ensuring the facilitator creates a comfortable learning environment and ensuring participants are clear about what they would gain from ALS.

To sum up the investigation, I believe Action Learning Sets are a successful vehicle to improve staff skills.

References

1. Bennett, C. (1976) Up the hierarchy. In Journal of Extension, Vol 4., pp18-26 March/April, U.S.A.
2. Proc. 3^rd National Property Management Planning Forum (1999). 22-25 March, Mildura, Vic.
3. Targeted outcomes of programs (TOP) Indicators and evaluation, 2000. Web site. http://deal.unl.edu/TOP/indicators.html

Appendix 1

¹ Farm$mart is the Victorian Property Management Planning (PMP) Program

² Four ALS participants requested to be omitted from the evaluation.

³ Numbered in order of how the questions would be asked.