Previous PageTable Of ContentsNext Page

Using Participatory Evaluation for better NRM Engagement across different partners. A case in progress

Valerie Sapin

AgSIP Coordinator Department of Primary Industries and Fisheries, PO 23 Kingaroy Queensland 4610.
Web site: www.dpi.qld.gov.au/agsip Email: valerie.sapin@dpi.qld.gov.au

Abstract

The Sustainable Agriculture State-level Investment Program (AgSIP) was designed during an intensive 18 months of comprehensive consultation with five industry peak bodies, eight regional groups, three state agencies and two community conservation groups.

The turning point of this design phase was the use of a Multi Criteria Decision Support System to help these parties select a range of strategic projects statewide. Choosing the selection criteria and their respective weightings allowed people to better articulate their vision for NRM work. This process did not reconcile all expectations but it certainly helped cement support and commitment towards the funding and delivery of twenty AgSIP sub projects.

Twelve months (and 8 million dollars) later this process was used again at a much higher level to engage the regional group collective (RGC), all State-level Investment Programs (SIPs), the QLD joint steering committee (JSC), Qld industry and conservation groups in the selection of future Impact Assessment Criteria for AgSIP.

This article will present the process used and the results that ensued. Interim results show varying and quite contrasted stakeholder expectations. It also shows various interpretations of how success in NRM management could and should be measured.

The following propositions will be put forward for discussion:

NRM Engagement includes active involvement in NRM project design and NRM program evaluations.

  • Continuous evaluation is crucial to guiding strategic and operational NRM engagement efforts
  • Evaluation can both initiate and sustain engagement on the proviso that it provides and maintains a common forum for dialogue
  • And finally different professional roles engender different value systems in NRM, and thus potentially engender different measures of success.

Key words

capacity building, evaluation, impact assessment, program logic, assumptions.

Introduction

One of the many merits of Evaluation of NRM projects is that it helps people:

1. Articulate and make NRM values and expectations explicit

2. Unearth assumptions behind those values and expectations

3. Negotiate what success/performance should look like

4. Report, analyse and draw new learnings

5. And be accountable for the funding received.

Unfortunately, very few comprehensive reports evaluating NRM projects or environmental policy are available in Qld and/or in Australia. This is quite surprising and shows very little rigour and or accountability seeing the amount of funding environmental issues receive1 and the relative maturity of evaluation as a professional discipline worldwide.

When the Sustainable Agriculture State-level Investment Program (AgSIP) was designed in 2003, one of the first needs that emerged from an initial industry review was the need for comprehensive evaluations of past and or current industry NRM projects. Apart from the Rural Water Use Efficiency Initiative (RWUEI), and the Cotton BMP surveys and environmental audits, very few industry and agency projects had undertaken any comprehensive impact assessments of their programs. Some R&D Corporations such as the Sugar Research Development Corporations (SRDC) do have evaluation plans in all their project proposals but very few reports seem available to the wider public. The problem is that this dearth of impact evaluation in many NRM projects fosters much uncertainty and emotional debate about their intrinsic value and or actual NRM outcomes2 (Cawley et al 2003, p 6). It is thus encouraging to see that as far as the National Action Plan for Salinity and Water Quality (NAPSWQ or NAP) is concerned, Queensland, New South Wales, Western Australia and Victoria have developed monitoring and evaluation plans3.

However, these evaluations may be difficult to carry out because

  • Specific criteria for achieving NAP Reef or NHT outcomes are yet to be refined negotiated and made unequivocal.
  • The realism of some of the NRM plans and their Management Action Targets needs to be reassessed.
  • Regions, agencies and industry peak bodies need to dramatically build their capacity and allocate specific financial and human resources towards project impact assessments
  • and finally, most of the evaluations so far seem to concentrate on institutional effectiveness rather than outcome or output effectiveness.

1- Developing impact criteria and performance indicators to clarify intent and expected benefits

One of the reviewers of the Natural Heritage Trust observed that unless there are clear and unambiguous statements about what is intended to be achieved over a given time frame, with the resources available, then it is not possible to assess the results (Howard 1999 p 54).

AgSIP has tried to ensure that its different objectives, planned outcomes and performance indicators are clear and that evaluation activities help shed light on the intent of the project over time. In order to do this AgSIP in collaborations with other State level Investment Projects and in consultation with regional bodies developed specific impact assessment categories, key deliverables and associated performance indicators4. These expected impacts, planned deliverables and performance indicators allowed clearer visioning of what was and still is a complex and ambitious project. It also reduced the ambiguity behind what some of the projects were to deliver, and helped plan their future impact assessments. Having a common impact assessment list of criteria does not mean however, that everybody agrees on what is most important within those categories. Different AgSIP baseline studies demonstrate that value systems and roles do influence the type of impact assessment categories or performance indicators people concentrate on5.

2- Using evaluation for “Reality check” on programme assumptions

AgSIP performance indicators are similar to some of the Management Action Targets in the recently produced regional NRM plans. These Management Action Targets (Mats) are milestones that represent the first steps (1-5 years) towards reaching (10-20 years) resource condition targets (RCTs). Achieving the RCTs ensure the region meets the long term (20-50 years) aspirational targets (ATs) of sustainable and preservation of the region’s rich natural resource assets for future generations (source Country to Coast 2005, p7).

Some of these MATs will be very hard to appraise without a baseline and rigorous qualitative and quantitative evaluation skills. Others are going to be even more difficult to appraise because as with some of the Resource Condition Targets (RCTs) and some of the MATs, the logic and causality of the assumptions behind AgSIP objectives, deliverables and performance indicators are yet to be proven (Derrick, 2004 p 9). An example of this is that one of the main assumptions behind AgSIP is regional bodies’ capacity and or willingness to take up AgSIP results and recommendations. This has yet to be proven!

An additional danger with writing vague and unrealistic project objectives or not critically reviewing the assumptions behind these objectives is that it often engenders commitment to an unattainable purpose and mixed perceptions over the success or failure of a programme. One way to minimise this is to develop specific time-bound and negotiated deliverables and performance indicators. Another solution is fostering and encouraging robust dialogue between people with different value systems and ensuring that regular external and internal scrutiny takes place. Active and comprehensive engagement actually demands continuously examining the logic and pertinence of projects from different viewpoints. Having critical steering committees or boards, and publishing the results of comprehensive baseline studies and or annual impact assessments does help.

3-Building regional Industry and agency evaluative capacity

AgSIP coordinators have observed that apart from a few rare individuals there is a widespread lack of skills and confidence and sometimes a lack of interest altogether in evaluation processes. There is nevertheless a need for the different parties engaged in the regional NRM process to come together and to learn from past and forthcoming evaluations content and process. Such a forum for continual learning accountability and improvement could go a long way to improve engagement levels. Future participation in the commissioning steering and or co-analysing of these evaluations could be an indicator of the maturity and level of partnerships in the regional NRM process. Seeing the current level of mistrust and wariness between the different industries regions and governments currently engaged in the regional NRM process, building trust and power sharing arrangements will take much longer than initially expected! Joint evaluations may help to accelerate this.

4-Institutional effectiveness versus impact effectiveness

Queensland’s monitoring, evaluation and reporting framework focuses on the measurement and assessment of:

  • Condition and trend in land, water, vegetation, biological and cultural resources, and landscape health
  • Performance of programs, strategies, policies and structures that support and promote the sustainable use, conservation, and rehabilitation of these resources.

Apart from some forthcoming evaluations on weeds, sustainable agriculture and reef plan, most of the evaluations to date have been on institutional arrangements, governance and other themes such as the planning and implementation of the regional processes.

Current work is underway to adapt the Land and Water audit to the new regional planning context (Grundy per com June 2005) but it will take some time to assess the actual effectiveness of the NAP, reef and latest NHT funding. Local information networks such as Integrated Area Wide Monitoring Systems are also being trialled. Their cost effectiveness in improving resource condition and change in landholder practices is also yet to be documented and fully assessed. AgSIP will endeavour within its own limited timeframe to report on practice change and or resource conditions where appropriate. Most of its impact assessment however will focus on its contribution to regional investment strategies, its generation of new knowledge and tools and its brokering of new partnerships. All this implies that impact effectiveness will require long-term efforts.

Conclusion

AgSIP has tried to engage its partners in a higher level of dialogue level by agreeing on future impact assessment categories, and encouraging AgSIP staff to negotiate performance targets with regional bodies and industry. It is also encouraging industry and regions to be involved in checking the assumptions behind some of the project objectives and to co-design future evaluations and co-interpretation of results. Trying to engage partners in articulating specific NRM impacts and objectives helps share the responsibility, risks and potential successes and failures of project outcomes. There is an urgent need however for:

  • More organisation leadership in impact assessment
  • Increased skills and confidence in impact assessment
  • Compulsory resourcing of evaluation in all budget proposals.

NRM Evaluation like NRM Engagement is a two way process. If Regional bodies and agencies want landholders to provide tangible evidence of their minimal impact on the environment, it only seems fair then that the same organisations also comprehensively evaluate and disseminate the results, success, failures and actual impacts of their own projects!

Acknowledgement

AgSIP is funded by the National Action Plan for Salinity and Water Quality (Australian and Queensland Governments) and is led by the Queensland Department of Primary Industries and Fisheries.

References

Cawley S., Anderson E, Sapin V (2003), Sustainable Agricultural Production Strategic Investment Proposal.

Country to Coast a Healthy Sustainable Future, Integrated Natural Resource Management Plan of the Burnett Mary region accessed on September 15th 2005 http://www.burnettmarynrm.org.au/c2c/volume2/Volume%202%20NRM%20PLAN_06Feb05.pdf

Derrick Jim, 2004. The development of monitoring and evaluation system for natural resource management programs. Paper presented at the 2004 international evaluation conference organised by the Australasian Evaluation Society, Glenelg, Adelaide October 2004.

Howard et al, 1999. Mid-Term Review of the Natural Heritage Trust Review of Administration http://www.nht.gov.au/review/mtrfinrpt/pubs/administrative.pdf

accessed on September 5th 2005.

Natural Heritage Trust Annual Report 2003-04 chapter 2 monitoring and evaluation for NRM programs, http://www.nht.gov.au/publications/annrpt0304/chapter-2-evaluation.html and http://www.nht.gov.au/publications/annrpt0304/chapter-4-qld.html#evaluation for the Qld chapter accessed on September 5th 2005.

National Action Plan for Salinity and Water Quality and Natural Heritage Trust Regional Programs Summary Report 2003-04 Commonwealth of Australia 2005
Published by the Departments of the Environment and Heritage and Agriculture, Fisheries and Forestry http://www.nrm.gov.au/publications/regional-report/03-04/03-04-summary/index.html accessed on September 5th 2005.

1 As at 30 June 2004, governments had approved some $517 million of regional focused investment through these (NAPSWQ and NHT) programs (source 2003-2004 annual NAP and NHT report)

2 Kelly in her recent PHD thesis about participation in Rangelands Natural Resource management quotes Foucault saying “People know what they do , they frequently know why they do what they do but what they don’t know is what they do does (Foucault 1982 cited in Kelly 2005 p 189).

3 Source NAP 2003-2004 Annual report

4 All AgSIP projects have key deliverables and an evaluation plan on how they will report on impact. See AgSIP baseline report and or project deliverables on www.dpi.qld.gov.au/agsip

5 V Sapin AgSIP baseline phase 2 report in preparation.

Previous PageTop Of PageNext Page