1Department of Primary Industries, LandLearn Program PO Box 3100 Bendigo Delivery Centre 3554 http://landlearn.netc.net.au Email Lydia.Fehring@dpi.vic.gov.au
2Department of Primary Industries, LandLearn Program Locked Bag 3000 Box Hill 3128 Email Jenny.Pettenon@dpi.vic.gov.au Jessica.Connor@dpi.vic.gov.au
3Department of Primary Industries, LandLearn Program Private Bag 1 Tatura 3616 Email Ann.Fagan@dpi.vic.gov.au Kathryn.Goyen@dpi.vic.gov.au
LandLearn is an education program provided by the Victorian Department of Primary Industries (DPI) aimed at assisting teachers to integrate sustainable agriculture and natural resource management into the curriculum to encourage student learnings around those themes. To coincide with the end of the previous five years of funding, LandLearn engaged an external evaluation consultancy to undertake a formative evaluation of the program.The aim of the evaluation was to plan the next phase of LandLearn, guide and inform decision making and to look for the intermediate indicators of success. That is, we wanted to know that we were contributing to long-term behaviour change by measuring the extent that our audience was using and valuing LandLearn beyond our initial contact. An unexpected outcome of the evaluation was using the Most Significant Change (MSC) technique as a method of evaluating the program. Through the use of MSC, LandLearn was able to successfully collect qualitative data on the long-term impact of the LandLearn program. This presentation will demonstrate the value of the MSC technique and the way LandLearn has used project evaluation and learning for continual improvement.
Three key learnings: (1) Using the Most Significant Change technique was a particularly powerful method where traditionally there are challenges to measure and demonstrate short-term impact. MSC also engages key stakeholders, investors and partners in a process that is fun, enjoyable, interesting and a learning opportunity for them. (2) Undertaking the evaluation allowed staff to look critically at how we do business – it confirmed the way that we do things, supported our assumptions and provided us clarity in planning and defining our next steps; and (3) The MSC technique has considerable value in evaluating the long term practice change of programs, especially when working with the intermediate users (teachers).
Most Significant Change technique, strategic planning
The purpose of this paper is to present an example of a monitoring, evaluation and learning framework for programs contributing to long term attitude and behaviour changes. The design for the review was influenced by the steps used in Appreciative Inquiry and developed for LandLearn under the expert guidance of evaluation experts, Clear Horizon1. The aim of the evaluation was to investigate the achievements and strengths of the LandLearn program to inform future strategic direction. It was also aimed at guiding LandLearn staff to use monitoring and evaluation data to make decisions and better target audiences while being more accountable to investors.
LandLearn staff recognised the need to evaluate the program, not only because an internal review in 2004 suggested it but also to contribute to organisational learning. Evaluation can be used as a tool to improve decision making, therefore ensuring programs better meet the needs of clients and investors. Evaluation is directed at improving all aspects of LandLearn and it puts LandLearn in a position to make decisions based on good management rather than being reactive. It can also highlight what is working well and where the gaps are.
The key project outcomes that LandLearn required from the evaluation included:
- Identification of the key success factors of the current LandLearn program;
- Recommendations about how best to encourage school communities to value learning and teaching about sustainable agriculture in the curriculum;
- Informing the strategic planning of LandLearn beyond June 2005 to maximise opportunities and build on the current program;
- Advice on future evaluation by recommending intermediate indicators to monitor implementation over time.
The evaluation needed to evaluate more than key program outcomes. It also needed to provide guidance to support the program staff, managers and investors to:
- Determine and define a strategic approach for increasing the use of sustainable agriculture as a theme in schools’ curriculum;
- Determine the appropriate level of project resourcing;
- Plan strategic alliances;
- Define the on-going funding arrangements; and
- Develop a better understanding of the ‘target markets’ for this project.
An important aspect of evaluating the LandLearn program was understanding that the ultimate learning and behaviour change outcomes LandLearn contributes to are long term in nature, which means that over time the LandLearn evaluation strategy needs to measure intermediate indicators that monitor whether the underlying strategy is on track.
What is LandLearn?
LandLearn is a state wide education program provided by the Department of Primary Industries. The LandLearn program provides practical training, support and resources for teachers to incorporate studies of ecologically sustainable agriculture and natural resource management into school curricula. This is done in the context of:
- A holistic approach to environmental education;
- Current curriculum standards and other education priorities, including the National Strategy for Environmental Education;
- Scientific research and the promotion of science in schools;
- Partnerships with related education programs.
LandLearn, previously known as Food and Agriculture in the Classroom (FAITC) was launched in 1994 to assist students and teachers to appreciate and value agriculture for its contribution to the economy and as a possible career. The program has evolved and grown to currently promote and support the formal education sector and school communities across Victoria to:
- Understand and respect the place agriculture has in their lives and in school curriculum;
- Recognise the need for sustainable natural resource management practices to produce quality food and fibre;
- Support and empathise with farming communities;
- Reflect these values and outcomes in the curriculum and learning experiences in the school and local community.
LandLearn works closely with teachers to support and resource them to incorporate these themes into their curriculum by delivering targeted and relevant teacher professional development with whole schools, clusters of schools or at conferences and workshops. LandLearn develops hands-on integrated curriculum material which is linked to the Victorian Essential Learning Standards. This material is trialled by LandLearn staff and participating teachers across the state. Landlearn also delivers directly to students on a semi-regular basis depending on the school’s commitment to environmental and sustainability education (ie: if the visit is part of a larger program/topic/theme within the school). Landlearn employs six staff who work across the state and are located at Tatura, Box Hill and Bendigo. Some staff are teacher-trained (or in training) and others have a passion for education and training.
Past LandLearn Evaluation
Since 1998, LandLearn has collected data on key program activities, the reactions of users and within the constraints of available resources and skills, data on changes in KASA (Knowledge, Attitudes, Skills & Aspirations) of intermediate users (usually teachers or other educators). However, measuring and evaluating the level of practice change by individual teachers and in school communities was beyond the scope and resources of LandLearn staff. Despite this LandLearn staff developed an evaluation strategy in 2002 which helped focus their evaluation resources on teacher professional development, and overall the evaluation data and anecdotal feedback indicated that LandLearn had achieved outcomes described for project funding.
In late 2004 LandLearn underwent a DPI project review. The purpose of the review was to evaluate progress and performance of the 2001 – 2005 project, to guide future investment and provide insights for the development of future initiatives.
The important conclusions from the review included:
- The work of LandLearn has made a significant impact in schools throughout Victoria. It has fulfilled a need and much has been learnt by DPI. LandLearn is an important project as it fills a gap. This work is not being done by others.
- A significant contribution has been made to the culture and curriculum in hundreds of schools across the state.
- There is significant potential for further development of the program, which would build on what has been achieved. It is important before this is done, however, that a strategic plan for the future is developed.
This initial review recommended that:
The resources currently being put into evaluation are insufficient to determine the long-term direction for LandLearn. An independent evaluation should be carried out to determine:
- whether the approach taken by LandLearn can be improved, and
- the optimum scale for the LandLearn project.
These findings ultimately led to LandLearn engaging an external evaluation consultancy to undertake a formative evaluation of the program which commenced in April 2005.
The aim was to understand the current achievements, the strengths and challenges of the program to inform the design of the future LandLearn. The evaluation was implemented in four progressive stages: Discover, Dream, Design and aDapt. The processes involved in each of these are detailed below.
Discover phase consisted of documenting existing LandLearn achievements and understanding the best of what has been achieved. This phase saw Clear Horizon conduct 19 semi structured interviews where key informants were encouraged to report their stories of their involvement with LandLearn using the Most Significant Change technique (discussed below). The interviews were conducted with a sample of past LandLearn staff, DPI managers, Department of Education and Training and with representatives from schools. Two group interviews were also conducted with teachers, the end-users, to explore perceptions of the program including strengths, barriers, achievements and future possibilities through more in-depth discussion.
This phase determined that LandLearn had been very successful in achieving its objectives and a number of groups were identified as being beneficiaries of the program including teachers, students, parents, wider community and industry. This phase also highlighted that the commitment and enthusiasm of staff is central to the success of LandLearn and the quality of resources and support provided to schools and identified a number of areas for improvement including the program’s profile and positioning within DPI.
Dream phase was an ‘energising exploration’ of the possibilities for LandLearn. A one-day forum of representatives from a range of key stakeholders, including teachers, investors, DPI management and collaborators enabled participants to ‘dream’ about and explore ‘what might be’. Analysis of the stories of Most Significant Change collected during the Discover phase to identify key successes and success factors stimulated creation of a vision of what future success looks like. This broad ranging vision for the future of LandLearn encompassing community engagement, school and educational extension and an expanding networking and coordination role provided the basis for the LandLearn team to move forward with the evaluators to the next phase.
Design phase saw the findings and the stories from the Discover phase and the vision from the Dream phase used in a workshop where LandLearn staff, facilitated by Clear Horizon experts, identified outcomes and future priority areas for LandLearn. This process then generated and refined a program logic model with next users identified and their ideas of success described.
aDapt phase finally developed a 7-stepped evidence- based monitoring evaluation and learning framework in accordance with the revised program logic. A monitoring, evaluation and learning framework is the systematic collection of information which is used and helps guide LandLearn to make strategic and operational decisions and to be accountable to investors.
The Most Significant Change Technique
The Most Significant Change technique (MSC) is a qualitative monitoring tool used to monitor and evaluate projects from the start to the finish (Davies & Dart 2005). Developed by Rick Davies through his work in Community Development in Bangladesh in 1994 it is also known as the ‘story approach’. Unlike conventional evaluation methods, MSC does not rely on quantitative measures but is designed to collect stories of significant change. It uses stories that provide insight into what people value, and illustrate the way in which individuals make sense of situations.
The MSC approach is a useful tool for evaluating education programs, as conventional monitoring and evaluation tools often do not provide sufficient data to demonstrate program impacts. It is particularly useful for programs focussed on social change which is lengthy and complex in nature.
In addition, the MSC technique is perfectly suited to monitoring that focuses on learning rather than just accountability. In LandLearn’s case, data had always been collected on the numbers of participants engaged and their responses to delivery (happy sheets), and while staff recognised the need to collect more meaningful data, they were lacking the resources, tools and skills required to implement this. In the past LandLearn found it very difficult to follow up with teachers to investigate the implementation and impact of the program after initial contact. The MSC technique gave LandLearn the opportunity to bring teachers together to share their own experiences and opinions, have them valued and to contribute to planning the future directions of LandLearn.
How was MSC used in the evaluation of LandLearn?
The MSC technique was employed in the Discover phase of the evaluation to capture stories from teachers of the most significant change precipitated by the LandLearn program. The stories can come from anyone directly involved in the project. In LandLearn’s case the question asked of teachers was:
“What is the most significant change that has taken place (ie: in agriculture or natural resource management education in the class/school) over the time you have been involved with the LandLearn program, and why is it significant?”
The stories are then documented and filtered by asking another group to consider them. In the LandLearn evaluation it was the large group of stakeholders in the Dream phase that analysed the stories to develop a vision of what success looked like. They were asked:
“Which of these changes are the most significant of all?”
The involvement of staff in this process was essential as it helped them to make sense of extensive information collected from many participants over time. It also assisted staff in recognising significant changes in participants and will help them identify and collect stories in the future.
The involvement and contribution of LandLearn’s key clients in the Discover and Dream phases demonstrated LandLearn’s commitment to addressing client needs and valuing their input. An unexpected outcome of this participation was the creation of stronger partnerships with many of these key clients and some ownership of the future directions of LandLearn.
LandLearn staff were amazed by the stories collected and the process renewed their enthusiasm for their work. It inspired a sense of worth – that what they do does make a difference.
Working through the process to articulate the program logic, then set up a monitoring, evaluation and learning framework ensured ownership of the process. Staff are enthusiastic about evaluation now, as the story process provides a means of collecting meaningful data over time; and they know why and how the information is collected and that it will be used to improve LandLearn.
Below is an example of a MSC story collected in the Discover phase of the evaluation process. The LandLearn team hopes that you enjoy it.
Most Significant Change Story
Last May 2004 I attended a workshop that was held by the Royal Botanic Garden Education program and LandLearn. It was my first introduction to LandLearn. During the course of the day I was most impressed with the way the workshop was conducted. The staff of LandLearn was extremely well organised, the activities we did were terrific and very easy to do with the students. At the end of the day I walked out feeling very inspired about working with our students in our community garden which was in the process of being constructed.
At the next staff meeting I briefed the staff about the workshop I had attended and we decided that early in 2005 we would invite the LandLearn staff to talk to us about horticultural activities for our students.
Early this year Kathleen and Jenny conducted a workshop for us at a staff meeting and the teachers said that it was one of the best workshops we’ve had. We did the very same activities that students would do. Kathleen and Jenny were inspiring and made many practical suggestions about what we could do with our students, both in the classroom and in the community garden. They provided all the equipment and materials needed for these activities and were extremely well organised. The wonderful thing about these activities is that we don’t need expensive equipment. Materials and equipment suggested by them are cheap to buy and easily available. They also brought in suggested books and other resources for us to have a look at.
(as spoken by a teacher from an urban primary school)
Evaluation does not only have to be about crunching numbers and producing tables and statistics that indicate quantitative ‘success’. Much can be learned by evaluating other elements, the more subtle impacts of a project or program and the intermediate steps in programs working towards long term change.
LandLearn has found that an externally facilitated, expert driven, appropriately funded evaluation is extremely beneficial as it provides the time and space for the staff involved to be part of the process rather than driving it themselves.
Some of the unintended impacts were the most powerful. For LandLearn this included:
- Strengthening partnerships with clients;
- Promotion of LandLearn to investors, collaborators and partners from DPI, Department of Education & Training, Department of Sustainability & Environment, industry and the education sector;
- Teachers’ experiences of LandLearn and the enthusiasm generated by that;
- Validation that LandLearn is successfully delivering a meaningful, credible program.
Some of the things the LandLearn team intends to do differently as a result of this evaluation include:
- Identify and collect stories of “Most Significant Change” as staff recognise and come across them in their work;
- Address some of the gaps identified in the evaluation – including targeting DPI staff to work with on collaborative projects;
- Target client groups more easily as there is now clear identification of different client groups and their particular needs;
- Regularly communicate the successes of LandLearn to key stakeholders and internally to DPI management and staff;
- Reflect on learnings, opportunities and success more often.
Davies R. and Dart J (2005) The ‘Most Significant Change’ (MSC) Technique – a guide to its use. www.clearhorizon.com.au . Accessed November 2005.
Goodburn, B. (2005) Review of the LandLearn Program Final Report. Clear Horizon, Hastings
1 Clear Horizon - www.clearhorizon.com.au