Johnstone Centre, Charles Sturt University
PO Box 789, Albury NSW
Australian Journal of Environmental Management (1998) 5(2): 109-119
Abstract
There has been very little published about evaluations of natural resource management programs in Australia. This situation reduces information transfer between program managers and evaluators and increases the likelihood that past mistakes will be repeated. In this paper the authors reflect upon their recent experience with evaluations of the Murray-Darling Basin Commission’s (MDBC) Riverine Environment Investigation and Education program and the Department of Primary Industries and Energy (DPIE) Farm Forestry Program. While the programs varied in the soundness of their program logic, they were making a difference in terms of accomplishing important program goals. However, about a quarter of the projects in both programs performed poorly. Concerns about program logic and its implementation, and with program and project management are explored where these have wider relevance to natural resource policy development, program management and evaluation. The authors also suggest some of the key elements for evaluations of natural resource management programs.
Introduction
There has been very little published, certainly in refereed journals, about evaluations of natural resource management programs in Australia1. In part this situation reflects evaluator concerns for client confidentiality and the potential for evaluation findings to be controversial. Many evaluators are private consultants who are reluctant to explain their methods and have little commitment to publication of findings. This situation reduces information transfer between program managers and evaluators and increases the likelihood that past mistakes will be repeated. This paper attempts to address that gap and we hope it will stimulate more active reporting and discussion of evaluations of natural resource management programs in Australia. We have recently undertaken evaluations of the Murray-Darling Basin Commission’s (MDBC) Riverine Environment Investigation and Education program (Riverine I&E)2; and the Department of Primary Industries and Energy (DPIE) Farm Forestry Program (FFP)3.
The MDBC Riverine I&E program began in 1989 and has funded projects addressing generic issues requiring coordinated action or research to overcome impediments to achieving Natural Resources Management Strategy (NRMS) objectives4. Funding is provided equally by the Commonwealth and each of the state partners. Since 1990-1991 around $2.5m per year has been allocated to Riverine Environment I&E projects. Our evaluation examined forty eight completed projects with a value of $7.4m. Riverine I&E projects may focus upon overcoming knowledge gaps, raising awareness or enhancing governmental cooperation. However, the emphasis is upon effecting change in natural resources management: hence projects must have clear linkages with natural resource managers and the Integrated Catchment Management (ICM) projects which MDBC funds. ICM projects may involve some I&E components, but their real focus is upon implementing improved management practices in the Basin. Riverine I&E projects have addressed topics related to water quality, river flows, aquatic flora and fauna, wetlands, riparian lands and vegetation and floodplain planning.
The FFP is managed by the federal Department of Primary Industries and Energy (DPIE) and aims to promote commercial wood production on cleared agricultural land that will provide timber for industry, stimulate regional development and contribute to the achievement of other natural resource management objectives at regional or catchment scales5. FFP projects attempt to accomplish these objectives by increasing awareness of farm forestry, establishing positive landholder attitudes towards farm forestry, providing training that will develop farm forestry skills and developing linkages between landholders and industry. The FFP involved Commonwealth expenditure of $3.7m on twenty seven projects covering twenty two community development projects and five strategic projects over the period 1993-1996. Additional Commonwealth funds of $2m were committed for the FFP in 1995/96. Projects funded have included a number of regionally based farm forestry development initiatives and strategic projects that provided training for extension advisors and investigated commercial arrangements for farm forestry.
In the following sections we provide a brief overview of program evaluation and discuss the specific approaches adopted in evaluating the Riverine Environment I&E and Farm Forestry programs. We then provide a summary of the findings of both evaluations, with particular emphasis on project and program effectiveness, that is, on the achievement of objectives rather than assessment of efficiency using cost-benefit analysis. We relate our findings to natural resource management policy issues and discuss important implications for program managers, investigators and evaluators.
Evaluation Methodology
Program evaluation: an overview
Cook and Shadish explained that program evaluators seek “knowledge about the value of social programs and their constituent parts, knowledge that can be used in the short term to make the programs more responsive to the social problems they are meant to ameliorate.”6 Evaluation therefore has an action focus which distinguishes it from basic research which attempts to contribute to theory development and the discovery of knowledge for its own sake7. Whilst applied research attempts to identify solutions to social problems, Patton suggested that evaluation tests the effectiveness of specific interventions with the focus upon obtaining information of use to stakeholders attempting to solve those problems8. Evaluation is therefore closely related to policy research9.
Increased understanding of organisation and political decision making and greater awareness of the difficulties of influencing decision making has forced evaluators to adopt more limited expectations and to renew their efforts to make evaluation relevant. These efforts to increase evaluation usage have focused upon addressing the needs of multiple stakeholders10; moving beyond summative assessments of the achievement of program goals to formative assessments that enlighten stakeholders of client needs, program implementation -including side effects - or program logic11; working at the local scale to improve local practice12; focusing upon incremental rather than bold program changes13; collaborating with key stakeholders, particularly with program managers, throughout the evaluation process14; and moving beyond the formal presentation of evaluation reports to regular and informal contact with potential users and publicising findings through the mass media15.
As Prosavac and Carey suggested, “A program is not ready to be evaluated until its theoretical basis has been developed.”16 Evaluators can turn to a number of sources in their effort to unravel program theory: they can approach program staff, clients and other stakeholders for their views; they can review literature on the program under scrutiny or similar programs; they can examine program documentation; and they can observe program operation17. It is also important to distinguish the program as implemented from the program as planned18. Unraveling program logic is an important step in identifying intermediate objectives that can be used to assess program impact19.
Developments in the theory of knowledge creation have illustrated the advantages of qualitative methods and suggested the dichotomy between positivist and naturalist paradigms was unproductive20. Qualitative methods are increasingly perceived as a valid approach that enables evaluators to explore the field in order to identify important research questions; adapt to changing circumstances during research; tap the real meaning of situations for stakeholders; explore program implementation; and unravel causal explanations likely to explain program success or failure21. These changes have combined to influence the emergence of contingency theories of evaluation where “the legitimacy of a method or concept depends on the circumstances.”22
Evaluation approach adopted
The evaluation briefs (see below), evaluation funds (around $50,000 in each case), time available (from three to six months), and the experience and skills of the evaluators were critical influences shaping the approaches adopted. Each evaluation was undertaken by a team of two: a social scientist experienced in program evaluation and familiar with natural resource management structures and issues in Australia; working on the Riverine I&E evaluation with an ecologist with experience in the evaluation of research programs; and for the FFP evaluation, working with a farm forester with experience in Australia and New Zealand.
With the FFP the evaluators confronted the task of assessing the impact of a program in its infancy. The FFP had received limited funding and relied upon voluntary approaches to effect the adoption of technology requiring major changes in landholder behaviour. Given the above and the long-term and uncertain economic returns from farm forestry it would be unrealistic to expect FFP projects to have effected improvements in catchment condition or farm and regional viability. For these reasons ‘trees in the ground’ was also considered an inappropriate measure of program impact at that time. Indeed, examination of program documentation suggested successful FFP project outcomes would include landholders making informed decisions not to adopt farm forestry23. This point reinforces the importance of understanding program logic or “the causal ‘logic’ which underpins the program.”24
Strategies adopted to ensure evaluation findings for both programs were useful and likely to be implemented included: refining evaluation briefs through a series of meetings with program staff; regularly working in the Canberra offices of the lead agencies to examine project files and discuss preliminary findings with agency staff, evaluation steering committee members, or agency advisory bodies; presenting draft reports for comment by agency staff and other stakeholders; adopting a two volume report format to meet the needs of different stakeholders; and with the FFP evaluation, interviewing over eighty program participants/stakeholders to obtain feedback about projects/program.
A mix of quantitative and qualitative approaches was adopted for these evaluations, with relatively greater emphasis upon qualitative approaches using interviews and document studies.
The Riverine Environment I&E program
The evaluation task was to review forty eight completed projects representing $7.4m of MDBC funds (Table 1). Project effectiveness was assessed against four criteria: project performance on project objectives; whether findings could be generalised to other geographic areas; the degree of linkage with other agencies; and participation of stakeholders in a project. Rating schemes were developed for each criterion and scores allocated by the evaluators on the basis of information in project files. The rating scheme for project performance is shown in Table 2. Four sets of projects (23 of 48 projects) were selected for in-depth review. These projects provided coverage of key Riverine I&E topics; topics where a large investment of MDBC funds had been made; the major geographic concentrations of Riverine I&E projects; representation across the states; exemplars of “good” and “poor” projects; and projects with an onground or Integrated Catchment Management (ICM) rather than an I&E focus.
The Farm Forestry Program
The FFP brief was to evaluate the appropriateness, efficiency and effectiveness of the Farm Forestry Program (Table 1). The phased approach adopted by the evaluation team involved a desk top review of all FFP projects using agency files, followed by an in-depth study of important farm forestry topics through an assessment of projects in four key regions and interviews with eighty five stakeholders. Project performance was assessed against project contribution to achievement of specific program objectives, the extent of project findings and the extent program managers discharged their administrative responsibilities (Table 3). A summative assessment of project contribution to overall program objectives was also made (Table 2).
Findings
Program logic
The Riverine Environment I&E program was underpinned by sound program logic. The Murray-Darling Basin Commission has limited capacity to prescribe the management practices of private land managers and the state agencies which have constitutional jurisdiction over natural resource management. The MDBC influences natural resources management by funding I&E and ICM projects, providing leadership for strategy development and by disseminating information. The MDBC approach of targeting Riverine I&E funds to projects addressing problems which transcend local issues, require coordinated action or research and require explicit linkages between research and management outcomes, is therefore sound.
By comparison, there were some problems with the logic underpinning the Farm Forestry Program. By linking awareness raising, attitudinal change and behavioural change the FFP program logic contributed to excessive emphasis within projects upon activities related to awareness raising. Given the dominance of economic rationalism and a small program budget it was reasonable that the FFP should focus upon extension/communication strategies to effect changes in landholder behaviour. Programs such as the FFP that attempt to effect voluntary changes in landholder behaviour through activities that raise awareness, increase knowledge and improve management skills can be effective25. FFP program documentation also linked increasing awareness, the development of more favourable attitudes to farm forestry, and the adoption of farm forestry26. Research suggests attitudinal change is relatively slow and links between attitudes and behaviour in the context of landholder adoption of conservation or farming practices is unclear/weak27. Where innovations are expensive, unproven, complicated or contrary to accepted farming ways - as is often the case with farm forestry - adoption of new technologies will be slow28. Efforts to effect behavioural change need to address the underlying reasons for non-adoption29. For farm forestry this may involve feasibility studies to establish financial viability, education and training of landholders with accompanying technical support, the payment of financial incentives, legislative changes or the provision of infrastructure.
Whilst it is reasonable to assume markets for farm forestry products do/will exist30, assumptions that farm forestry will enhance farmer and regional viability are somewhat problematic. Returns will be greatest for communities where processing facilities are located31, and individual landowners will frequently be entering agreements with large corporations enjoying regional market monopolies. These corporations can be expected to take the largest share of profits32. However, it may be unreasonable to expect a small program to overcome difficult structural issues. It can also be argued that FFP extension and communication activities are an appropriate means of addressing these issues.
Program effectiveness
Eighty percent of FFP projects and seventy five percent of Riverine I&E projects were considered to have performed satisfactorily in that they received scores of three or above out of five for performance (Figure 1 and Table 2). Riverine I&E projects had generated many important research findings; there were examples of project findings having important impacts upon management practices; there were many examples of excellent project management; and a majority of projects produced findings relevant to a basin-wide audience. The Riverine I&E program was also deemed to be effective in that projects covered a broad spectrum of the issues outlined in various MDBC strategies.
The FFP had made a distinctive and important contribution to the emergence of farm forestry as an identifiable and legitimate farm enterprise in that most projects had made a strong contribution to raising the awareness of farm forestry and establishing linkages amongst key farm forestry stakeholders. There were also examples of outstanding projects that had made important contributions to the achievement of program objectives. A number of FFP projects reported important findings of relevance to a wider regional/national audience. Information in project files indicated that at July 1995 the FFP had funded the establishment of 1,300 hectares of farm forestry, mainly as demonstration sites, and had stimulated the establishment of another 500 hectares.
Project outcomes
For any number of reasons it would be unreasonable to expect every project in a portfolio to perform at a high level. Program managers and their advisory boards should also be encouraged to take some risks in funding projects with innovative methodologies or unproven project management. For these reasons project managers should not be harshly criticised if some projects fail. Notwithstanding these comments, and the overall evaluation finding of Riverine I&E and FFP program effectiveness, a large number of projects in both programs underperformed and the overall return from these programs was less than could reasonably be expected. Twelve of the forty eight Riverine I&E projects, with a total value of $1.7m, received very poor project performance scores of one or two (Figure 1): six projects had flawed methodologies or failed to provide evidence of undertaking proposed activities and another six projects provided no evidence of any real findings related to project objectives. In many cases, these projects had not produced a final report. Six of the twenty six FFP projects, with a total value of $773,000, received poor project performance scores (Figure 1): two projects had accomplished few project objectives even at a minimal level, and another four projects accomplished most objectives but at only a minimal level of performance.
Program and project management
To a large extent the deficiencies outlined above resulted from poor program and project management. Many of the projects in both programs were managed by researchers in state agencies who frequently had limited research training or experience33, few linkages with major research centres and little commitment to publication of findings in peer reviewed journals34.
There was insufficient scrutiny of initial Riverine I&E project proposals. Projects were assessed by program managers and steering committees largely representing the interests of state agencies, with little review by scientific peers35. Strong commitment by state representatives to effecting changes in onground conditions may explain the allocation of scarce Riverine I&E funds to what were really Integrated Catchment Management (ICM) projects. Five projects, with a total value of $841,000, were ICM projects in that their primary focus was upon onground work or planning activities in a specific locality, as opposed to investigating a problem to provide findings with basin-wide application.
Reporting formats for both programs focussed heavily on project activities. There was no systematic approach to gathering or storing information about products (extension materials, reports), findings (what was learned) and impacts (how management by agencies, private landmanagers or businesses had changed). For example, Riverine I&E project files were frequently incomplete. Examples of field audits by program managers were encouraging but infrequent and MDBC staff were reluctant to impose sanctions to challenge the culture of poor reporting. In part this problem may reflect ongoing tensions in federal/state relations. A large number of Riverine I&E projects were managed by state agencies who are partners to the MDB initiative. Staff from these agencies appear to resent being accountable to a federal body. An important example from the FFP was where a large proportion of funding was allocated to the establishment of demonstration sites but there were no protocols for establishing and monitoring these sites. This meant there had been no systematic attempt to gather the lessons learned from existing sites.
Awareness raising activities such as preparing newspaper articles, establishing demonstration sites, developing notes for landholders and establishing regional farm forestry networks had taken most of the time and resources of many regional FFP projects. The important tasks of completing regional feasibility assessments, preparing handbooks and developing landholder skills and knowledge were frequently dropped from the activities of regional FFP projects. For example, eighty six percent of FFP projects provided evidence of some impact in contributing to awareness of farm forestry but only forty four percent of projects provided evidence of impact upon learning or training about farm forestry. The FFP approach of developing strategic or overview projects offered a more appropriate model for accomplishing these complex tasks. Unfortunately, the strategic projects that were undertaken suffered from poor integration with regional projects. For example, an excellent strategic project provided training for extension advisers but there was no evidence of graduates contributing to regionally based training of landholders.
Our evaluation of both programs identified important gaps in project coverage and poor coordination with similar programs as additional factors constraining program effectiveness. For example, most Riverine I&E projects were focussed in the south and east of the Murray-Darling Basin: only seven investigation sites in the area north of the Murray River and west of longitude 147°E, which just west of Wagga Wagga. The forty eight Riverine I&E projects provided good coverage of the key NRMS issues, but investigations of some critical issues such as algal blooms (two projects) and water pricing and allocation (one project) had been limited. Other gaps highlighted by the Riverine I&E evaluation included knowledge of the direct and indirect effects of European carp on MDB resources; the role of grazing in changing the structure and productivity of floodplain vegetation; the management of effluent from intensive rural industries; and socio-economic research examining the adoption of new technologies, for example in the irrigation industry. Given the emphasis upon education within the Riverine I&E program it was noteworthy that about five percent of the total program budget had been allocated to education activities. Apart from the need for a systematic approach to landholder training, noteworthy gaps in FFP project coverage included the issue of managing remnant vegetation on private land for farm forestry; investigation and trialing of low-rainfall farm forestry in eastern Australia; social factors affecting farm forestry adoption, including landholder industry linkages; and establishment of regional projects in Tasmania and the Hunter Valley of New South Wales. At the time of the evaluation, farm forestry projects were being undertaken at the federal level by the Murray-Darling Basin Commission, the Department of Primary Industries and Energy and the Rural Industries Research and Development Corporation. There was little coordination between these federal programs and considerable overlap in the activities. For example, both the MDBC and DPIE were funding projects to develop regional farm forestry strategies.
The MDBC had not developed a coherent strategy for disseminating research findings or implementing management recommendations flowing from I&E projects. Many MDBC staff were technical experts with little understanding of communication or extension theory and staff managing both programs were over committed with the number of projects they managed or other duties they performed. A large number of examples were highlighted where lead agencies might profitably invest additional resources to publicise important findings, replicate a localised case study or coordinate a workshop to discuss contentious findings. For example, a Riverine I&E project investigating the nutrient loads carried by irrigation drains in the Shepparton region established that loads of filterable reactive phosphorus (FRP) from these drains amounted to twenty six percent and forty percent of the annual load in the Murray River, with impact highest during the irrigation season. There was no evidence that these findings had been widely disseminated, published in peer reviewed journals, or any consideration given to replicating the study in other districts.
Publication and peer review
In both programs there was a general lack of peer review or public discussion of project findings. Fifteen of the thirty five (60%) publications flowing from FFP projects were unpublished reports. Of the ninety six Riverine I&E publications, fourteen (15%) were published in peer reviewed journals, twenty seven were non-refereed published papers or reports, eight were conference papers, seven conference abstracts and there were forty non-refereed unpublished reports unavailable to the wider community. A number of project teams displayed little understanding of social science research methodologies. For example, information obtained from landholder surveys with unsatisfactory response rates of around 20% underpinned critical decisions in a number of regional FFP projects.
Conclusions
Program evaluation is a critical element of successful program delivery and policy development36. It is also a complex and difficult task which requires adequate funding. Funds allocated for the Riverine I&E and FFP evaluation represented between one and two percent of total program funds. Despite this limitation and others that may have been imposed by the evaluation briefs, the skills and experience of the evaluation teams or the methodologies adopted, these evaluations provided important lessons for policy development, program management and program evaluation.
Around twenty five percent of projects in both programs performed poorly and the overall return from these programs was far less than could reasonably be expected from investments of $7.4m in the Riverine I&E program and $3.7m in the FFP.
Some flaws were identified in FFP program logic and major inconsistencies between program logic and program implementation were found in both programs. Excessive emphasis in the FFP on awareness raising activities arose, at least in part, due to the assumption that awareness raising would lead to changes in attitudes and in turn, effect behavioural changes. Research suggests attitudinal change is relatively slow and links between attitudes and behaviour in the context of landholder adoption of conservation or farming practices is unclear/weak. Where innovations are expensive, unproven, complicated or contrary to accepted farming ways - as is the case with farm forestry - adoption of new technologies will be slow. Policy makers developing natural resources management programs need to move beyond this focus upon effecting attitudinal change and address the underlying reasons for non-adoption of management practices.
Australia’s natural resource management agencies have limited capacity to effect desired changes in the management practices of private landowners or other agencies. Effective strategies for collecting, disseminating and implementing important research findings would seem critical to program success. At the time of these evaluations, there was little evidence that either the Riverine I&E or the FFP had developed comprehensive or sophisticated approaches to these important tasks. Both evaluation teams also identified important knowledge gaps, particularly for socio-economic research. To some extent these deficiences reflect the legacy of an Australian natural resources management structure dominated by biophysical scientists. The FFP evaluation also highlighted issues of poor coordination between government agencies. By early 1995 farm forestry projects were being funded by DPIE, MDBC and the Rural Industries Research and Development Corporation (RIRDC), as well as various state forestry and conservation agencies. There was considerable overlap with a number of these projects and agencies may have been as much concerned with staking a claim in an emerging federal priority as with undertaking important work. There was an obvious need for these players to coordinate farm forestry priority setting, project calls and selection, and forums to review important projects.
Despite the many strengths of Riverine I&E and FFP project management, including a willingness to undertake evaluation, poor program management was an important factor affecting program performance. Riverine I&E and FFP program managers have addressed some of these issues, but they are issues likely to afflict other natural resource management programs. Program managers’ time was invariably over-committed. In part this was because these programs had expanded without a corresponding allocation of funding for project management. Some projects had poorly developed objectives or methodologies in that activities proposed were unlikely to result in useful outcomes, and some were overly ambitious. To a large extent these problems reflected poor scrutiny of project proposals. Most project proposals were assessed by program managers and/or steering committees with little scrutiny by scientific peers. The influence of state agency personnel on these steering committees may explain the allocation of scarce Riverine I&E resources to projects with an ICM or onground focus. Many projects in both programs were developed and managed by state agency researchers who had limited research training or experience, and few linkages with major research centres. Indeed, there was only limited peer review or public discussion of project findings. Examples of field audits were encouraging but infrequent and program managers were reluctant to apply sanctions to improve reporting. Reporting formats that focussed upon reporting activities meant most projects failed to provide evidence of producing useful products or findings, or of project impact upon natural resource managers. A large proportion of FFP project funding was allocated to the establishment of demonstration sites. Again, protocols for establishing and monitoring demonstration sites had not been developed and there had been no systematic attempt to gather lessons learned.
The limitations of using ‘trees in the ground’ as a measure of FFP program effectiveness highlighted the difficulty of evaluating natural resource management programs in their infancy. Evaluators should explore program logic to determine if it is consistent with current theory and is supported by empirical studies. They should also determine whether the program operates as planned and whether key stakeholders have a shared understanding of program rationale. Once articulated, program logic will assist with the development of intermediate indicators of program success. Project assessment should then focus upon the extent projects have appropriate objectives, sound methods, have undertaken activities as outlined and had products, findings and impacts likely to contribute to the achievement of important program objectives (real outcomes). Evaluators should also seek to identify gaps in project coverage and cases where additional investment would be worthwhile to identify, disseminate, further investigate or implement important findings.
The aims of most of the projects in the Riverine I&E and FFP programs were to contribute knowledge required for ecosystem management. Effective adaptive management of ecosystems requires peer reviewed publication of results37. Unfortunately, in the case of the Riverine I&E and the FFP, most investigators had little commitment to dissemination of findings. Indeed, there was only limited peer review or public discussion of project findings.
Evaluation findings are only one source of information used by program managers and policy makers and it is highly problematic to attribute changes in programs to specific evaluation findings. In the period since these evaluations the FFP has been considerably expanded and the Riverine I&E program management substantially revised. Suggestions for program improvement that flowed from these evaluations have been adopted as part of these changes. For example, the MDBC has established a forum to discuss and disseminate project findings38; there is better coordination of farm forestry amongst the agencies; and topics such as landholder-industry linkages in farm forestry and the impact of carp on riverine environments are being addressed. These developments are encouraging and may provide some validation of the evaluation approach used. Whilst there have been improvements in program management, the recent Australian National Audit Office (ANAO)39 evaluation of key natural resource management programs identified inadequate project reporting and limited evidence of project and program outcomes as important factors preventing federal agencies satisfactorily accounting for public moneys expended. This is an important challenge for future natural resource management programs.
Table 1
Summary of Evaluation briefs
for Riverine Environment I&E and Farm Forestry Programs
Riverine Environment I&E Program |
|
|
|
|
Farm Forestry Program |
|
|
|
Table 2
Rating schemes for assessment of overall project performance
for Riverine Environment I&E and Farm Forestry projects
Riverine Environment I&E Program |
0 Objectives unlikely to produce outcomes relevant to key NRMS issues identified in project proposal. |
1 Objectives relevant to key NRMS issues identified in project proposal, but activities inappropriate and unlikely to result in useful outcomes or no evidence of activities/outcomes. |
2 Objectives relevant to key NRMS issues identified in project proposal, activities appropriate and likely to result in useful outcomes, but outcomes inadequate in terms of products and findings or little/no evidence of outcomes (eg, no final report). |
3 Objectives relevant to key NRMS issues identified in project proposal, activities appropriate and likely to result in useful outcomes. Products and findings useful in resolving key NRMS issues identified in project proposal. However project did not address management outcomes. |
4 Objectives relevant to key NRMS issues identified in project proposal, activities appropriate and likely to result in useful outcomes. Products and findings useful in resolving key NRMS issues identified in project proposal and project addressed the issue of management outcomes. However no evidence of impact upon management outcomes. |
5 Objectives relevant to key NRMS issue identified in project proposal, activities appropriate and likely to result in useful outcomes. Products and findings useful in resolving key NRMS issues identified in project proposal. Project addressed impact of investigations upon management outcomes and documented evidence of impact upon management practices in the MDB. |
Farm Forestry Program |
1 Few objectives accomplished. |
2 Most objectives achieved at a satisfactory level. |
3 Some objectives achieved at a high level. |
4 Most objectives achieved at a high level. |
5 All objectives accomplished at a high level. |
Table 3
Other rating schemes used in the assessment of Farm Forestry projects
Contribution to raising awareness; Contribution to learning/training; Contribution to impact (adoption) on key stakeholders |
0. Not applicable. |
1. No evidence of impact. |
2. Yes, strong likelihood of an impact. |
3. Yes, evidence of some impact. |
4. Yes, considerable impact in project area. |
5. Yes, extensive impact in project area. |
Enhanced linkages between stakeholders |
1. No evidence of impact. |
2. Yes, evidence of informal linkages. |
3. Yes, formal linkages. |
4. Yes, formal linkages, including with community groups. |
5. Yes, formal linkages, including extensive links with community groups. |
Project administrative performance |
1. Irregular reporting. |
2. Regular reporting with adequate financial statements |
3. Regular reporting, adequate financial statements and limited information about project outcomes (findings, products, impacts). |
4. Regular reporting, adequate financial statements and adequate information about project outcomes (findings, products, impacts). |
5. Regular reporting, adequate financial statements and comprehensive information about project outcomes (findings, products, impacts). |
Extent of project findings (from research, monitoring or simply project activities) |
0. Not applicable. |
1. None. |
2. Yes but not very important. |
3. Yes and locally important. |
4. Yes and important findings with generic application. |
5. Yes, with generic application and impact on managers elsewhere. |
Figure 1
Distribution of overall performance scores (see Table 2) for
projects in the Riverine I&E (dark bars) and FFP (light bars) programs
1 see for example, C O’Faircheallaigh and B Ryan (eds), Program evaluation and performance monitoring (Centre for Australian Public Sector Management, Griffith University, Queensland, 1992).
2 A Robertson and A Curtis, Review of completed Murray-Darling Basin Commission Riverine Environment Investigations and Education projects. Volumes 1&2 (Johnstone Centre Charles Sturt University, Albury, 1995).
3 A Curtis and D Race, Review of DPIE National Farm Forestry Program projects. Volumes 1 & 2 (Johnstone Centre Charles Sturt University, Albury, 1995).
4 MDBC, Murray-Darling Basin Commission Annual Report 1993-1994 (Canberra, 1994).
5 Department of Primary Industries and Energy, Farm Foresty Program - terms of reference (DPIE - Forests Branch, Canberra, 1995).
6 T D Cook and W R Shadish, “Program evaluation: the worldly science” (1986) 37 Annual Review of Psychology, 193-232, p 193-194.
7 E J Prosavac and R G Carey, Program evaluation: methods and case studies (Prentice Hall, New Jersey, 1992).
8 M Q Patton, Qualitative evaluation methods (Sage, Newbury Park, California, 1990).
9 A Majchrzak, Methods for policy research (Sage, Newbury Park, California, 1984); W R Shadish Jr, T D Cook and L C Leviton, Foundations of program evaluation: theories of practice (Sage, Newbury Park, California, 1991).
10 Cook and Shadish, op cit n 7; M Q Patton, Practical evaluation (Sage, Newbury Park, California, 1982); M Scriven, Hard-won lessons in program evaluation (Jossey-Bass, San Francisco, 1993).
11 Cook and Shadish, op cit n 7; T T Chen and P H Rossi, “Evaluating with sense: the theory-driven approach” (1983) 7 Evaluation review 283-302; J Cronbach, S R Ambron, S M Dornbusch, R D Hess, R C Hornik, D C Phillips, D F Walker and S S Weiner, Toward reform of program evaluation (Jossey-Bass, California, 1980); Patton, op cit n 8.
12 R E Stake, “Program evaluation, particularly responsive evaluation” in W B Dockrell and D Hamilton (eds), Rethinking educational research (Hodder and Stoughton, London, 1980).
13 Shadish et al. op cit n 9.
14 M Q Patton, Creative evaluation (Sage, California, 1987).
15 Cook and Shadish, op cit n 7.
16 Prosavac and Carey, op cit n 7, p 32.
17 H T Chen, Theory-driven evaluations (Sage, Newbury Park, California, 1990); P H Rossi and H E Freeman, Evaluation: a systematic approach (Sage, Beverly Hills, California 1985).
18 Patton, op cit n 8; Prosavac and Carey, op cit n 7.
19 Chen, op cit n 22; Rossi and Freeman, op cit n 22.
20 Patton, op cit n 8; Patton op cit n 10.
21 M Q Patton, How to use qualitative methods in evaluation (Sage, Newbury Park, California, 1987); Patton, op cit n 8.
22 Shadish et al. op cit n 7, p 315.
23 DPIE, op cit n 5.
24 B Ryan, “Evaluation in the commonwealth government: a critical appraisal” in C O'Faircheallaigh and B Ryan (eds), Program evaluation and performance monitoring (Centre for Australian Public Sector Management, Griffith University, Queensland, 1992), p 73.
25 F Vanclay “The social context of farmers’ adoption of environmentally-sound farming practices” in G Lawrence, F Vanclay and B Furze (eds), Agriculture, environment and society (Macmillan, Melbourne, 1992): A Curtis and T De Lacy, “Landcare in Australia: does it make a difference” (1996) 46 Journal of Environmental Management 119-137.
26 DPIE, op cit n 5.
27 J Mueller, Measuring social attitudes (Teachers College Press, New York, 1986); F Buttel, O Larson and G Gilespie Jr, The sociology of agriculture (Greenwood Press, New York, 1990); Vanclay, op cit n 25; A Curtis and T De Lacy, “Landcare in Australia: beyond the expert farmer” (1996) 13 1 Agriculture and Human Values 20-31.
28 E M Rogers, Diffusion of innovations (Free Press, New York, 1983); S Chamala, “Group effectiveness: from group extension methods to participative community Landcare groups” in S Chamala and K Keith, (eds), Participative approaches to Landcare: perspectives, policies and programs (Australian Academic Press, Brisbane, 1995), pp 73-92.
29 Vanclay, op cit n 25.
30 AACM International Pty Ltd, Centre for International Economics and Forestry Services, Commercial farm forestry in Australia - Development of a strategy framework: a resource book (Rural Industries Research and Development Corporation, Canberra, 1996).
31 G T McDonald, “Reafforestation in Australia” in A Mather (ed), Afforestation - policies, planning and progress (Belhaven Press, London, 1993).
32 D Burch, R Rickson and R Annels, “The growth of agribusiness: environmental and social implications of contract farming” in G Lawrence, F Vanclay and B Furze (eds), Agriculture, environment and society: contemporary issues for Australia (Macmillan, Melbourne, 1992).
33 B Finlayson and T McMahon, “Funding and conduct of environmental research” in L Cosgrove, D G Evans and D Yencken (eds), Restoring the land: environmental values, knowledge and action (Melbourne University Press, Melbourne, 1994), p 98.
34 Ibid, p 99.
35 Ibid, pp 97-98.
36 S Dovers, “Information, sustainability and policy” (1996) 2 Australian Journal of Environmental Management 142-156.
37 N L Christensen, A M Bartuska, J H Brown, S Carpenter, C D’Antonio, R Francis, J F Franklin, J A McMahon, R F Noss, D J Partson, C H Peterson, M C Turner and R G Woodmansee, “The report of the Ecological Society of America committee on the scientific basis for ecosystem management” (1996) 6 Ecological Applications 665-691.
38 R J Banens and R Lehane, 1995 Riverine Environment Research Forum (Murray-Darling Basin Commission, Canberra, 1997).
39 Australian National Audit Office, Commonwealth Natural Resource Management and Environment Programs: Australia’s land, water and vegetation resources. The Auditor-General Performance Audit Report No. 36: 1996-97 (Australian Government Publishing Service, Canberra, 1997).
