Abstract

From its inception, the National Cotton Extension Network has had close links with industry stakeholders, providing mechanisms for continual feedback and priority setting for extension and education programs. A culture of evaluation, planning and continual improvement is being further developed within the extension network to ensure extension and education programs are “hitting the mark” for stakeholders and to provide accountability to funding agencies.

Media summary

Keywords

Evaluation, Outcomes, Participation, Cotton, Extension Skills, IPM

Introduction

A saying widely used by irrigation extension officers through to business managers tells us “You can’t manage what you don’t measure”. Considering this, as extension professionals, how well do we measure our achievements, the impacts and outcomes of our work and identify areas for improvement? Are we gathering the information we need in order to develop our own extension careers and our extension programs effectively? This paper outlines some of the approaches and activities used by the National Cotton Extension Network in moving towards an outcomes focussed, evaluated approach.

Background

The management of a cotton crop requires a high level of knowledge and timely decision making. The cotton industry has developed a strong demand for information and learning avenues to develop its knowledge and skills base. Since the modern cotton industry began to grow in Australia in the 1960s there have been strong linkages between industry and research. The Australian Cotton Growers Research Association (ACGRA), formed in 1972, provides grower input into planning, priority setting and critical review of research. ACGRA advise Cotton Research and Development Corporation (CRDC) project investments and host the biennial Australian Cotton Conference.

Since 1993 the Australian Cotton Cooperative Research Centre and its predecessor have enhanced research prioritisation and linkages across agencies and with industry.

There are a wide range of mechanisms by which growers and consultants contribute to setting priorities for research and extension. Regional reference groups in each valley and for specific programs help to set priorities for local and national extension programs. Grower learning groups such as area wide management groups also identify priorities, needs and interests for extension and research. Cotton Consultants Australia host technical forums and are involved with many research and industry committees.

Many researchers, growers, agronomics consultants, agribusiness and other industry participants take an active role in the knowledge system. Agronomic consultants are a key part of the cotton industry, particularly in the area of insect management and their focus is now expanding to whole farm agronomic management.

Key Objectives for Cotton Extension and Research

The Cotton Research and Development Corporation and the Australian Cotton Cooperative Research Centre provide the primary leadership for research and extension in the Australian cotton industry. CRDC’s core outcome is for “a more sustainable, profitable and competitive cotton industry providing increased environmental, economic and social benefits to regional communities and the nation”. CRDCs five year strategic plan highlights for Program 1, People and Knowledge, an aim towards “Improving the capacity of industry and the community to use the knowledge and innovations gained through research and development. A continuing culture of innovation in the cotton industry, which creates viable rural communities” (CRDC 2003). Similar themes of uptake of research and knowledge apply across all programs.

Similarly, the Australian Cotton CRC aims to “enhance the development and growth of the Australian cotton industry through the application of collaborative research, education and the adoption of sustainable farming systems”. Through a coordinated national network providing extension, support and educational support to the cotton industry, it strives for benefits of widespread adoption of sustainable management practices, increasing the industry skill base and an industry perceived to be a responsible corporate and community citizen. Each of the partner agencies also work towards similar objectives of sustainable production.

In both organisations these objectives are underpinned by specific targets. Measuring the contribution of extension programs towards achieving these objectives is becoming increasingly important.

The National Cotton Extension Network

Under the leadership of the Australian Cotton CRC, the National Cotton Extension Network provides a close link between research, industry initiatives and the consultants and growers. The uptake of research is a high priority for the industry with CRDC investing approximately 10% of its R&D expenditure in extension programs (CRDC 2003) to supplement the state department extension services. In particular, this includes Cotton Industry Development Officers (regional extension positions focussed on cotton production), national extension coordination, a technology resource centre, decision support and education. These positions partner with NSW Agriculture District Agronomists, Cotton Seed Distributors’ Extension Agronomists and specialists in the areas of Water Use Efficiency, Irrigation and Integrated Pest Management (IPM) to form the National Cotton Extension Network. Each member of the network is active through their local role and as part of one of the national extension focus teams – Insects, Disease & Weeds, Environment, Farming Systems and Water. Senior members of the extension team provide leadership of these focus teams. This leadership role provides a career development opportunity for team members as well as encouraging the sharing of knowledge and experience.

Four new environmental extension specialist roles will soon expand the capabilities of this network in the natural resource management area. Cotton Australia, the peak industry body, leads the implementation of the industry’s Best Management Practices program (BMP).

Key extension methods

Knowledge is developed and shared through a range of extension approaches which can be broadly grouped as:

• Trials and Demonstrations;
• Groups;
• Comparative Analyses;
• Education;
• Information Transfer; and
• Computer-based Decision Support Packages

Information resources are made available to the industry free of charge through the Cotton CRC Technology Resource Centre and local extension staff. These include the COTTONpaks (compendiums of information about a technical issue), CottonTales newsletters (1 page weekly-fortnightly regional news and research by fax or email), Research updates, identification guides, pocket guides and the Cotton CRC Website.

Computerised decision support packages are available for insect and nutrient management (CottonLOGIC) and water management (newly released HydroLOGIC) and a searchable CD of all printed resources. Information is also available via field days and seminars.

Area wide management groups, educational courses and comparative analyses are some of the avenues for communication and development of ideas.

The changing face of extension

This forum challenges us to “extend extension: beyond the traditional boundaries, methods and ways of thinking”. With so many changes in the extension context, traditional approaches need to be queried, adapted, abandoned or progressed.

Recent years have seen significant changes in the profession of extension. The tradition of an extension officer who would spend 20-30 years in a single region, developing their local knowledge and delivering that to the growers in their community is rapidly disappearing. In its place we see a young, mobile workforce, with temporary contracts and many extension professionals moving on to new projects after 3-5 years. Extension workers are coming in from and moving out to a wide diversity of roles. The extension role is also now shared across many players including researchers and consultants.

How do we know if we are hitting the mark?

With such a high commitment by industry to the extension network, it is important to ask whether this is returning an enhanced uptake of research. Evaluating the impact of extension programs is important for a range of reasons, particularly for accountability; on-going improvement and for achieving change. Increasingly research funders are focusing on the “triple-bottom line” (CRDC 2003; Roth 2003) and the related contributions of research and extension programs.

Evaluation Frameworks

There are multiple levels at which extension programs are targeted and similarly can be evaluated. At the higher levels, measures of actual impacts are the ones that can most clearly demonstrate return on investment but they can lose their value if too many assumptions are needed to attribute change to research and extension inputs. Lower level evaluations often provide valuable feedback for enhancing extension activities but don’t measure the actual impact of these programs. In practice, evaluations at different levels are useful for different purposes. Evaluation frameworks have been introduced as a means to both plan extension and evaluation activities and to bring together the results of evaluations at a range of levels. This recognises that short, simple evaluations are often the best means for learning and to gather feedback on a certain issue or tool. Use of a carefully designed evaluation framework can provide a means for activity evaluations to contribute to a broader impact evaluation. As an example, Table 1 depicts some of the evaluations conducted by the Cotton CRC in relation to Integrated Pest Management (IPM) against Bennett’s hierarchy (Bennett 1977). Several of these cross multiple levels.

Table 1. Evaluation activities at multiple levels related to IPM in the Cotton Industry.

Bennett’s level	Evaluations conducted
Outcomes	- Analysis of total pesticide usage data - Water quality data - Record of social complaints
Practices	- Benchmark Survey	Attitudes to IPM - Focus groups 1997 & 2001			Regional Tipping Out Trial Impact Survey
KASA*	- IPM Short Course pilot + follow up interviews		AWM groups phone survey	Insecticide Usage Comparative Analysis Evaluation
Reactions	- Cotton Tales Surveys - Information Resources Survey - Field day evaluations
Participation	- IPM Short Course Numbers - Numbers of AWM groups - Cotton CRC website hits
Activities	- Project reports
Inputs & Resources	- Annual reports - Project Budgets

* KASA = Knowledge, Attitudes, Skills and Aspirations

A wide range of tools and frameworks have been used for extension evaluation in Australia. (Dart, Petheram et al. 1998). Some of these are quite prescriptive and an instant “turn-off” for many extension staff. Some of the key considerations in selecting a framework to use for the cotton extension program were:

• Useful for extension evaluation AND planning.
• Flexibility for a wide range of issues and evaluation methods.
• Simple to use and to communicate.
• Aligned to the key outcomes of our funding bodies.
• Help extension staff to review, learn and improve.
• Encourage staff to be mindful of outcomes and impacts when planning extension activities.
• A framework open enough to stimulate creative approaches to evaluation.

The framework we have introduced (Figure 1) is a combination of the LOGIC model (Taylor-Powell 2001), using Bennett’s Heirarchy to help define the logic (Bennett 1977). Its simplicity allows great flexibility in choosing evaluation methods to suit the issues and the evaluator. It also fits very well with how our primary funder, CRDC report on how we have contributed towards R, D&E targets and impacts. Bennett’s is a useful tool to help break it down further and understand the value of the different levels of extension impact in leading towards the triple bottom line impacts.

Figure 1 Frameworks for cotton extension evaluation

The frameworks were introduced to the extension network at the annual planning workshop in August this year. As the team worked through them for various issues, they recognised that evaluation tools are also extremely useful for planning extension programs. This is consistent with the Targeting Outcomes of Projects (TOP) framework (Rockwell and Bennett 1995) that suggests that you work down Bennett’s hierarchy for planning and up the hierarchy for evaluation.

Developing an evaluated outcomes culture

Evaluation in extension could be considered like going to the dentist - one of those jobs everyone knows is important but avoids doing. Building an evaluation culture is not only about the skills to evaluate, it’s also a shift in focus from outputs to outcomes and impacts and a continuous learning and improvement cycle.

Where were we?

Prior to 2001, a few major evaluation exercises had been undertaken and a small handful of extension evaluations had been reported on. Evaluation was rarely included in extension planning workshops. Several extension officers commented that they regularly gathered feedback for their own, informal evaluation. This is a great habit for personal learning and reflection. However, as much of this tacit knowledge was rarely recorded, or even communicated with peers, much opportunity has been lost to be able to measure impact of extension programs over time. As staff leave, they take this tacit knowledge with them and the capacity of the new extension team members to learn from the experiences, or draw on the history of success or failure of different extension approaches in given regions is limited.

What have we done?

Building evaluation capability is best approached through multiple angles, from top–down and bottom-up (McDonald, Rogers et al. 2003). Similarly, the skills and motivation for extension evaluation have been introduced to the Cotton Extension Network through complementary approaches including training, encouragement and practical experience. The appointment of a new National Cotton Extension Coordinator in 2001 coincided with an increased focus from the CRDC board on evaluated impacts of extension. Building the focus and capability for evaluation has been central to the Extension Coordinator role.

Our overall philosophy regards evaluation as a learning and planning exercise that also provides accountability reporting. Integral to this approach is team participation rather than purely external evaluation. Some partnerships with external evaluators to study industry wide changes have developed skills and brought in “fresh eyes” as detailed further below. As with many CRDC research programs, external reviews are conducted with the extension program as a whole reviewed in 1995 and another external review is planned.

Some of the activities include:

• Training – as a group or individually through CRRI-Q courses.
• Participation in evaluation research eg IPM Focus groups, Knowledge Management.
• Evaluation “tasks” eg each team member presented an evaluation activity at the annual extension workshop.
• Reporting frameworks modified to base on evaluation framework.
• Focus team planning and reporting.
• Annual planning workshops.
• Reinforcement from board members.

Evaluation of selected extension activities conducted by individuals and shared with their colleagues has built confidence in evaluation skills, enhanced communication of planned outcomes and approaches to extension and provided valuable data on the effectiveness of different extension approaches. Maintaining the commitment of individuals to evaluate without the time pressure of presenting this to their peers is difficult.

Over the past three annual extension and planning workshops, the concepts of evaluation and outcomes have been gradually built upon as depicted in Table 2.

Year	Evaluation trends
2001	Individuals presented activity evaluations Team vision and mission revisited and redeveloped
2002	Skills session on planning for outcomes and outputs Planning frameworks introduced
2003	Evaluation frameworks introduced Skills session – planning and evaluating TBL outcomes Focus teams applied evaluation frameworks to plans Logic based reporting format developed Focus teams identified key indicators and evaluation needs

Table 2. Evaluation and outcomes development at annual extension planning workshops

Where are we now?

The last few years have seen a change in the general approach. Evaluation is now closely linked with planning and is largely considered as a core operating activity. The concepts of working towards outcomes and evaluating these are well ingrained in the extension team.

The evaluation frameworks introduced this year aim to draw together the multiple levels of evaluation. The focus teams are taking the lead in monitoring and understanding the effectiveness of their activities at all levels. It is also hoped that this will provide some direction and achievement for the focus teams which had been struggling to deliver outcomes.

Team participation in industry wide studies

Whole of industry studies have been undertaken of changes in industry attitudes towards IPM (1997 & 2001) and knowledge management (2003). In keeping with the participative evaluation and learning approach, these were undertaken by the extension team under the guidance of independent consultants. This model provided team members with the chance to build skills through first hand experience whilst, with the guidance of an experienced social researcher, gathering valuable information. Team members were also able to gain a closer understanding of the issues than could have been achieved through simply reading the report. Several have made changes directly as a result of the direct feedback and the team as a whole have greater ownership of the results.

There are also limitations of this approach including the cost and time of having so many involved, potential bias, potential difficulties in collating data across so many interviewers and the risk of missing some key points due to previous perceptions. In general, the approach fits well with our objectives of achieving a mix of accountability reporting, learning and on-going improvement through evaluating.

Industry Attitudes Towards IPM

Integrated pest management approaches to reduce the reliance on pesticides have been one of the primary focuses of cotton research and extension. Focus groups were undertaken in 1997 and 2001 to better understand the issues impacting on the adoption of IPM in the industry with a view to identifying priority areas for extension and measuring the impacts. They explored the social, economic and technological aspects of IPM. In response to the initial focus groups study in 1997, IPM guidelines were developed, an IPM short course was developed and is now being delivered, economic comparisons of IPM were conducted and the Area Wide Management concept was extended.

Key findings from the 2001 study were (Coutts, Christiansen et al. 2001):

• The principle of IPM is widely accepted throughout the cotton industry as an integral part of farm management.
• IPM was viewed in a holistic fashion with an acceptance that it was the appropriate use of a range of available approaches and tools to reduce insecticide use and improve insect control, farm productivity and sustainability. There was some variation between regions in the emphasis placed on the various tools for IPM and resistance management.
• Growers in many regions are taking increasing responsibility for insect control decisions, with decision making moving toward a partnership between grower and consultant.
• Growers seem to be more educated and interested in insect management and are using their knowledge to set the type of pest management approaches they want.
• There was strong support for the extension role and the need for on-going information dissemination and education programs about IPM in the industry.
• Research was widely supported by industry stakeholders.
• There was strong support for Area Wide Management (AWM) Groups across the industry, a desire for increased involvement of non-cotton growers, and a preference for smaller, more informal groups.
• The introduction of ‘Bollgard^®II’ was viewed by all groups as the next major advance in IPM with a comprehensive management strategy needed.
• The future for IPM was seen to be with a fully integrated farming systems approach involving cooperation with other crop management systems.
• Regions vary in the degree to which they have practically implemented IPM. Those more interested in IPM in 1997 were generally more advanced with IPM in 2001.

These focus groups were conducted during a three-day intensive workshop for the cotton extension team together with some extension staff from other industries. In the first day, Jeff Coutts, then with the Rural Extension Centre, trained participants in the design and conduct of the focus group technique and assisted with the overall design. The second day saw participants form pairs to travel widely to various regions to conduct focus groups of growers, consultants and researchers. These groups had been pre-arranged by the local extension officer. The next day participants wrote up their reports around emerging themes and regrouped to debrief and share learnings. Jeff Coutts then worked through the individual reports to identify key themes and differences across the industry sectors and regions.

Extension officers who participated in this study have commented that as well as learning the focus group technique and building their skills in it by actually applying it, being involved in the process gave them a greater depth of understanding of the issues and influenced their own extension practices more than had they simply received a report.

Knowledge Management

Water management has greatly increased in focus in the industry over the past few years. Water use efficiency is being identified as a key management, research and extension priority by industry, governments and regional natural resource management bodies. In planning extension and research programs in this area, it is valuable to understand how industry sectors access information and develop their knowledge to make decisions. This is the subject of a project being undertaken by the Cotton CRC with the National Program for Sustainable Irrigation. The project aims to develop extension and capacity building frameworks that match the preferred learning styles of cotton and grain irrigators and their consultants. This study has been very strongly supported by industry participants who are keen to see a focussed irrigation extension and research effort.

Applying a similar model of extension participation and learning as was used with the IPM focus groups, the extension team were trained in interview techniques before conducting interviews across cotton growing regions, converging by phone to compare findings and refine the questions, doing more interviews, debriefing key findings and then returning home to conduct a few more interviews. A modified convergent interviewing approach was used with techniques such as card maps and force field analysis used in the interviews. Professor Victor Callan from the University of Queensland Business School is leading the training and data analysis for this project, providing a unique perspective for interpretation. A few interviews were conducted prior to the workshop to test-run the process and develop the questions.

It was particularly valuable during the workshop to have extension practitioners and a supply chain manager from other industries involved. They were able to provide different perspectives to the discussion to help reduce internal bias and share ideas from their industries. They also provided the team with valuable feedback about our own collaboration style.

Extension participants commented that participation in the workshop and conducting interviews provided them with a greater understanding and ownership of the issues as well as experience with interview technique. Responses to a reflective question “what did you get out of being involved that you wouldn’t have gained from reading the report?” varied but had many common themes as grouped in Table 3.

It is particularly interesting to note the wide range of different skills and understanding that was developed through this three-day workshop. That is, in addition to the funders’ primary objective of understanding growers’ learning styles, participants also developed their understanding of the issues affecting water management, developed their skills and developed their sense of team with their colleagues.

Table 3. Benefits that extension team members indicated they gained from being involved with the Knowledge Management Research project that they didn’t feel they would have gained from simply reading the report.

Understanding of issues	More knowledge of the issues and priorities in different regions; Felt the emotion and frustration of water reform; Opportunity to become conversant with irrigation issues; Differences in priority between different people.
Understanding of industry learning styles and decisions	Good appreciation of how growers develop ideas and make decisions; Realisation that growers are at so many different levels – no one package keeps everyone happy; Impact of uncertainty of water reform as a barrier to investments to improve WUE; Growers’ perspectives, how they make decisions and access information; Growers have intent to improve WUE – they either don’t know how to or can’t afford to progress; Level of awareness, understanding and acceptance by consultant agronomists of water impacts.
Skills	Appreciated benefits of formalised interviews and following replicated questions for transferability to local area; Confidence, skills and technique in approaching interviewing to develop outcomes; Specific instruction, training and experience in conducting interviews; First hand experience in [modified] convergent interview technique – see what worked and identify areas for improvement; Convergence would have had more impact in different circumstance; Use of the card technique helped to think from a different angle; Interview methods; Understand how information is gathered for this type of study; How a relatively quick process can quickly retrieve a lot of valuable data; Lack of involvement in planning and design limited learning; Valued the different perspectives of participants from other industries; Good to get out of my comfort zone; What I think I know may not always be right.
Team development	Understanding of the different attitudes of team members to some issues; Build confidence in process by doing it together; Understand the common things all face - building sense of team; Understand the issues faced by colleagues in other regions; Ownership of results & motivation to read the report.

Future

The extension network has developed to be a strong team that collaborates closely over its 16 locations and state boundaries. A potential risk of this is that as the extension group have developed to be as strong team, this may have been at the cost of linkages with research in some cases. Evaluation has been increasing in the extension program. Research will also be faced with a need to move towards evaluated outcomes.

Evaluation of full research and extension programs is the next stage. Bringing together research and extension to identify shared outcomes could be a good starting point to build R, D & E evaluation and enhance linkages. There may be an opportunity to embed some learning elements in a Benefit – Cost Analysis of the Cotton CRC soon to be undertaken by an external consultant. This may be an opportunity to assist research and extension disciplines to reflect on their achievements and identify key priorities for a new CRC.

A part of this approach will be to refocus the National Extension Coordinator position as an extension and evaluation specialist to be a resource in developing the extension, participation and evaluation skills of both the extension network and the research community.

Conclusion

Evaluating the outcomes of extension and research programs is becoming increasingly important for funding agencies. It is also an opportunity for extension programs to learn, to change and continually improve. Moving towards evaluated outcomes signifies a change in organisational culture that takes time, skills and motivation. This change is perhaps somewhat easier to appreciate and relate to for extension practitioners than for researchers.

Evaluation on it’s own is not a priority task of most extension professionals. Building skills, capacity and motivation for review and innovation helps in embedding evaluation and outcome reporting in the overall extension approach. Regular encouragement by funders and managers of the value of the extension role, reinforced by the importance of demonstrating outcomes can be partnered with training and development opportunities.

By using evaluation frameworks we hope to enhance the thinking about evaluated outcomes when extension activities are planned. We are also hoping that these will enable us to measure a higher level understanding of program impact by drawing on the manageable sized evaluations that extension individuals conduct for their own learning and improvement.

References

Bennett, C. F. (1977). Analysing impacts of extension programs., Washington DC Extension Service, US Dept of Agriculture.

Coutts, J., I. Christiansen, et al. (2001). Changes in Attitudes to Integrated Pest Management in the Cotton Industry: 1997-2001 and Attitudes to Area Wide Management. A Focus Group Study October / November 2001. Narrabri, CRDC and REC, UQld.

CRDC (2003). Five year strategic research and development plan 2003-2008. Narrabri, Cotton Research and Development Corporation.

CRDC (2003). Triple Bottom Line Highlights. Cotton Research and Development Corporation Spotlight.

Dart, J., R. J. Petheram, et al. (1998). Review of evaluation in Agricultural Extension, RIRDC Publication No. 98/136.

McDonald, B., P. Rogers, et al. (2003). "Teaching People to Fish? Building the Evaluation Capabilty of Public Sector Organisations." Evaluation 9(1): 9-29.

Rockwell, K. and C. Bennett (1995). Targeting Outcomes of Programs (TOP), University of Nebraska - Lincoln; Cooperative State Research Education and Extension Service, USDA. 2003.

Roth, G. W. (2003). Measuring the sustainability of cotton production systems. World Cotton Research Conference - 3, Capetown, South Africa, In Press.

Taylor-Powell, E. (2001). The Logic Model: A Program Performance Framework. Third annual University of Wisconsin-Extension, Cooperative Extension program evaluation conference June 18-21, 2001, Madison, Wisconsin, University of Wisconsin - Extension.