Introduction

Why do we need Decision support tools?

Rural and natural resource management is concerned with achieving the integrated, productive and sustainable use of biological, physical, social and financial capital at diverse geographic and temporal scales. As our understanding of the natural resource base has improved, the need for integrated approaches to management has been increasingly widely appreciated. As a consequence, decision-making in rural, natural and environmental resource management has become a more complex process. The intensification of agricultural production and more recent emphasis on holistic environmental management has meant that managers are increasingly expected to address more complex issues (including negative externalities as well as issues associated with productivity) such that a broader domain of information needs to be considered. As the complexity of the decision-making task increases, resource managers (whether farmers, agri-business, Government Agency staff or other managers) are increasingly unlikely to have the necessary expertise, and, therefore, capacity to make resource management decisions that integrate the range of issues that demand consideration (Walker, 2000).

Have they helped us so far?

This increasingly complex environment for resource use and management has necessitated the development of new skills, methods and tools to consider new information and apply new ways of thinking to consideration of that information. As a consequence, research has played an increasingly active role in preparing resource managers to achieve high quality decision-making processes and outcomes. This ‘decision support’ research has included:

Development of effective access to the broad range of technical data, knowledge and process information that might be relevant to decision-making;

The development of new ways of analysing potential strategies for resource use and their implications

The development of tools or methods that ‘package’ these new approaches to make them accessible to the resource manager, and

A role in building the capacity of land managers and their advisers to bring these advances into existing and evolving decision-making processes.

How do we proceed from here?

Given that decision support seeks to improve the quality of decision-making processes and outcomes, the provision of decision support needs to be thought of more broadly than the development of decision support systems. It is, in part, a scientific and technical undertaking but, given that it seeks to change decision-making processes and, therefore, the decisions made, it is influenced by institutional, social, policy and political context. “If you change the way you make decisions, you will change the decisions you make” (Attributed to Jim McNeil by Slater, 1995). Seeking to change the way that people make decisions about resource use and management is not a consequence-free academic exercise, it is an initiative that bears significant responsibilities (Walker 2000).

What this means is that a bad decision support tool, and by this I mean an aid that produces wrong or misleading information, is worse than no tool at all. A farmer that acts on wrong or misleading information supplied by a decision support aid will at best be loath to trust information supplied from such sources in the future, and at worst will diminish the economic and environmental condition of his farm. Due to the critical nature of the environmental imperative facing our agricultural systems, the relationship between the providers of rural extension and farmers cannot afford to be compromised in this way. It is partly for this reason that a need has arisen for a way to systematically evaluate potential decision support tools.

Assessing Decision tools.

The aim of this paper is to discuss the Decision Support Tool Assessment framework. The products of the Farm Forestry and Revegetation team at the Department of Agriculture Western Australia target frontline revegetation advisers. The decision support tools and systems to be analysed using this framework are expected to be used by our clients – the frontline revegetation advisers - to assist them in their efforts to successfully extend revegetation information to land managers and community stakeholders. To successfully analyse the tools, we need to keep in mind the fact that they will be used as an information bridge for the passage of revegetation information between these groups.

A decision support tool can be presented in a number of forms, such as

• Simple text based guidelines
• Flow diagrams or decision trees
• Spreadsheet based systems with calculations
• Software based systems with a complex of numeric values and decision trees
• Optimising systems such as MIDAS

Output from the use of the tool might fall into a number of categories, such as:

• Simple conceptual
• Single choice action
• Single solution numeric value
• Statistical options
• Conditional multiple options
• Ranking of options

The framework will need to incorporate a filter to ensure the inclusion of a strong experiential and empirical component to the knowledge of the Decision Support System (DSS), reflecting conservative decision behaviour of even innovative farmers. By this I mean that stakeholder input will be important to help make outputs relevant, and acceptable to farmers.

Extending this stakeholder involvement concept to the actual use of the tool allows the process to integrate stakeholder perspectives, and ensures all participants see the problem in the same way. While a decision support tool will be based on common definitions, and therefore will be able to describe a given problem in a way that is understood by most people, it should also allow users to define the problem in their own terms, enabling them to build on their existing problem solving framework.

This is particularly important when the problem to be resolved involves a considerable amount of technical information. Some decision support systems or tools may be more useful if they are viewed as a source of knowledge, rather than as an analysis tool, which may require not only significant data to use, but also acceptance that the form of analysis was the most appropriate (Parker, 1999). In other words, the outcomes can be dependent on the process of facilitation as much as the utility of a decision support system. A tool may be useful just as a source of information to add into a group discussion. Bear in mind here, that DS tools vary enormously – from tools with lots of data and very little analysis at one extreme, to ‘empty’ tools which contain no data, but have lots of built in analyses capacity at the other extreme.

DSS tool Assessment framework

The Rural Extension Centre at the University of Queensland designed a “Change Analysis Framework” (Clarke, et al. 1997) as an aid to the design, management and evaluation of extension processes. The DSS assessment framework discussed in this paper used the CAF as a starting point for its design. The framework below is suggested as a method of systematically analysing existing and potential decision support tools. This framework has been developed specifically with farm forestry and revegetation tools in mind, but could be more widely applicable.

A potential DS tool or system will need to be assessed against each step in the process. This will ensure that it is tested against each relevant criterion when deciding whether it will be worth the cost of development and promotion. The framework is phrased as a series of questions to be asked about the tool under scrutiny. The list is not exhaustive. It is a set of prompts to facilitate discussion. The response to these questions can be reported in a standardised format. Ranking of the importance of the response to each question will be a subjective process to be done on a case by case basis.

Many of these questions should be answered explicitly in the manual accompanying the tool. As well as analysing existing tools, the following process would be useful to follow when developing a new tool.

See Figure 1 for a diagrammatic representation of how the following criteria or steps in the process fit together.

Criterion 1: Context

In what situation will the tool be used? It could be one or more of the following:

• Single ‘Do it yourself’ user
• One to one advice
• Group Discussion
• Strategic decision making
• Negotiation
• Learning aid
• Field day display

Other questions which help establish the context would are:

What is the target audience or user group?

Does the tool fill an identified knowledge gap?

If not, is the knowledge developed useful?

Is there an existing equivalent?

Criterion 2: Objectives

What does the tool help with?

Eg Oil Mallee profitability

Tree water use

What is its intended role?

Eg Should I or Shouldn’t I?

General information

Or what area of expertise is involved?

Eg Hydrology

Tree physiology

What is the type and level of output?

Eg Economics ($ etc)

Graphics

General information

How can the output be used in decision making?

Once these questions have been answered, put together a list of stakeholders and experts to assess and analyse the tool. Combine the ‘who’ information from the step above with the ‘what area of expertise’ field from this step, to select individuals or groups who can help determine whether this tool will be useful to enough of our clients to warrant the expense required to develop and promote it. Perhaps ask all stakeholders and relevant experts including farmers, extension specialists, DS specialists and researchers to study the tool. Some may not need to be involved in discussion, but just asked for their appraisal.

Criterion 3: Principles and Assumptions

Approach this stage on two levels. Firstly, examine the tool in relation to the principles of the project.

Are the principles congruent with its purpose and context? For example, if one of the project principles is to ensure stakeholder involvement at every stage of the decision-making process, does the tool allow for this in its current form?

Secondly, examine the tool in relation to the assumptions it is based on.

Is the tool designed around realistic assumptions? Most tools, especially those developed using scientific research will be based on certain assumptions and generalisations about the natural, social and political environment. Assess the suitability of these assumptions for the type of decision to be made. For example, a DS tool on the economics of Oil Mallees will need to include some consideration of whether or not the political environment is likely to be conducive to the development of a market for the product. In other words, the Oil mallee DS tool may be based on an (unstated) assumption that politics will not be an impediment. Our job at this stage is to assess whether or not that assumption is reasonable. If not, then the tool is not useful.

There are two types of assumptions – internal ones on which the model is based, and external ones, which it is the user’s responsibility to check from time to time. For example, consider cannabis instead of Oil mallees. A DS tool to decide whether or not to grow Cannabis may contain internal assumptions about what things you need to know to help calculate profitability. But as well, there are many external assumptions. When the DS tool was designed, cultivation of cannabis may have been legal, but now it is not. Therefore, the tool is no longer useful, because something outside the tool has changed.

The most common external assumption is that everything outside the model stays unchanged – the sun keeps rising every day, weather follows a predictable pattern, demand for primary products continues, etc. Instead of developing a long list of assumptions to check, it might be best to look for things that have changed since the DS tool was developed, and see if they effect its validity or usefulness.

Criterion 4: Equipping

• What is the type and level of input data required?
• Does the end user have the skills and resources required?
• What hardware and software are needed to operate the tool?
• What are the operating requirements (Time etc)?
• What are the requirements for analysing or interpreting the outputs?

Bear in mind that the tool will be used by our clients (revegetation front-liners), or our clients in conjunction with their clients (land managers). A decision support tool may require certain inputs that the user will need to provide. One or both of the parties will need to have the necessary information available to them.

A second requirement will be that the user has the skills to effectively work the tool. This may include not only understanding the inputs required, but some level of understanding of the rationale behind the process. This will be necessary to put the results in context in relation to the underlying assumptions. The user will also need to be able to understand the output from the tool in order to make use of the information extracted. This problem can be approached from a couple of angles. If the users are not equipped to use the tool we can either discard the tool, make it easier to use, or provide some training. For example, some useful decision support tools are based on spreadsheets, but many clients may not have the skills needed to use them. Providing training in spreadsheet use could help many of our clients use the tools more effectively.

Are the resources available to run the process?

Commitments of time and money will be necessary to get the tool to a useable stage, as well as getting the tool used. For example, uptake of a useful decision support tool may be hindered if the product is not fully developed, or adequately promoted. Take these expenses into account when deciding whether or not a tool is worth developing.

Criterion 5: Organising

What’s the best way make it available and get it used?

Decide whether distribution of the tool will be economically viable. Compare the benefits to be gained from its deployment against its distribution cost. Then, if you decide to proceed, organise the distribution logistics. Although we are dealing with an extension product, apply adult learning principles when planning its promotion.

Criterion 6: Communicating

Decision support (DS) tools are an avenue for communication of information or knowledge and ideas.

Is the tool likely to help pass revegetation information from our clients to theirs? Will it facilitate discussion on relevant issues?

Knowledge-based DS systems can be a valuable method of capturing farmer experience, and those systems that can accumulate this knowledge will become progressively more useful.

Can the outputs be integrated with other more qualitative information for whole farm decision-making?

Are the results quantitative (numerical)? Or quantitative (“rule of thumb”)? Differences between farmer wisdom and model outputs provide an opportunity for discussion and perhaps new insights on such issues.

Users can be sceptical of recipes, so generating a result interactively by allowing them to add or delete selection criteria may make a tool more user friendly.

Users will appreciate an interface that provides a maximum amount of information from a minimum of inputs, as well as being able to manipulate variables to see what happens.

Criterion 7: Performance indicators

How will we know whether this tool is performing the task it was designed for, or if it is useful to our target audience?

Is it possible to agree on performance measures before it is released for field assessment? If so, document them.

If a tool is used by our clients, we may assume that they find it useful. But are the outputs of the tool accurate? Some field testing will be necessary amongst our client group to collect feedback on performance, but this may not be enough. For tools that provide economic analysis of revegetation options, test them on examples where the results are known, to see if the model gives realistic results. Alternatively, does the tool provide examples of acceptable and realistic ranges for the output values. Even if the results are not accurate, the tool may still be useful if its results are within a certain percentage or order of magnitude of real life examples.

Criterion 8: Observing

Once the tool is released and (hopefully) being used by our clients, monitor its use and efficiency. The use of a valuable tool may be below expectations because of a lack of backup support or a need for periodical updates. This kind of efficiency monitoring will be help make sure we get the most out of a tool we have invested resources into developing, but will also involve an ongoing cost. Make an assessment of what level of ongoing commitment is required to get the most out of the tool. If the maintenance required is regular or expensive, is the tool worth promoting at all? Think about the operational life of a tool, or decide at what point to review it. For example, the economics of oil mallees spreadsheet will need to be re-examined once the oil mallee processing plant is operational.

Criterion 9: System practice

Critical thinking – why wouldn’t it work?

Are there any significant barriers (for example political or social) that would prevent or affect the rate of uptake within our target group?

Play devil’s advocate, and try to look at the tool in question from a different perspective. Try to find reasons why the model would fail either through its content or its promotion, and explore whether these points pose a serious risk to the successful extension of the tool. Once potential problems are listed, they can be explored in the following step, then the tool can be reinserted into the framework at the appropriate step.

Criterion 10: Creating

How could it be improved or tailored to satisfy an identified need?

Make the necessary changes, repeat the process.

If at any step, a tool is considered to be not worth pursuing, then explore what changes could be made to overcome the perceived problem. The tool can then be put through the framework again, to see whether it would be viable if the changes were made.

Even after a decision support tool has been through this process, keep looking for improvements. There are no perfect tools or models, and there will always be many avenues to explore for improvements.

Conclusion

The steps in this framework are intended to guide analysis of a decision support tool or system by a group of relevant experts and stakeholders including (where appropriate):

• scientists to comment on the validity of the basic model,
• extension experts to identify any problems with the extension of the model,
• our clients to comment on whether or not the tool fits their needs, and whether or not they have the skills to use it,
• land managers to comment on the usefulness of the output.

The efficacy of the process will be governed by the choice of people chosen to execute it. To guarantee a useful analysis, someone needs to take responsibility for putting a DS tool through the process, and the right people need to be asked for their input.

References

1. Clarke, R. et al (1997) a Framework for the design, management and evaluation of extension processes. Proceedings 2^nd APEN Conference Albury, NSW pp621-627.

2. Lawrence, P., Shaw, R., Lane, L. & Eisner, R. (2000) Participatory Multiple Objective Decision Making Process: Emerging approaches with new challenges. In Proceedings of Symposium: Watershed Management 2000. 20-24 June 2000, Colorado USA

3. Parker, C. (1999) Decision Support Systems: Lessons from past failures. Farm Management. V10 No5, pp 273-289

4. Slater, R.W. (1995) Presentation to Decision Support 2000. In M. Power, M. Strome and T.C. Daniels (Eds) Proceedings of Decision Support 2001, 17^th Annual Geographical Information Seminar and Resource Technology ’94 Symposium, Toronto, September 1994. Published by the American Society for Photogrammetry and Remote Sensing, Bethesda, Maryland. Volume 1 pp16-18

5. Walker, D (2000) Providing Decision Support for Natural Resource Management , http://nrmtools.tvl.tag.csiro.au/

6. Walker, D & Zhu, X. (2000) Decision support systems for rural resource management. Proceedings of a specialist workshop (in press) “Deepening the basis of Rural Resource management” I.N.S.A.R. The Hague, February 2000

Figure 1 Tool Analysis Framework