Previous PageTable Of ContentsNext Page

Evaluation of research, development and extension: how and why?

David Morrison

Agriculture Western Australia
3 Baron-Hay Court, South Perth, WA 6151

Summary. Government agricultural research and development (R&D) organisations are operating in an environment of increasing competition for funds and changing expectations from stakeholders and clients. They are meant to become more business like and market focussed but with, in real terms, less funding. Changed circumstances and expectations require new skills and strategies. An analytical framework for evaluation of R&D and principles for allocation of scarce resources, is outlined. Examples of its practical application are provided, showing how information from evaluation can support decisions on resource allocation at several levels in R&D providing organisations. The case is made that this information is also likely to be increasingly valuable in retaining funding from Government and winning funding from rural industry research corporations, and that it can support a cultural shift in the organisations.

INTRODUCTION

Australia’s publicly funded agricultural R&D providers are operating in a new environment characterised by changing expectations from stakeholders and clients, less funding, and more competition for funds.

Both Rural Industry Research Corporations (RIRCs) and Governments are calling for R&D providers to be more business-like in their management. They are demanding that new, market focussed activities are adopted and that providers achieve a greater bang for their buck. A cultural shift is called for to enable organisations to implement that change.

At the same time Governments are increasingly questioning all expenditure and in some cases agriculture is being singled out for particular attention because it stands out as a large item of industry support. Public R&D providers must compete with other agencies for Government funds and with each other for funds from the RIRCs.

It is difficult for R&D providers to cope with the rapid change. Strategies which have appeared to be good enough in the past are now clearly inappropriate. For example, reductions in budget have often been dealt with by small cuts across the board, sometimes confined to operating costs only. With significant cuts to funding and also demands to take on new activities, this strategy is now more inadequate than ever.

Many R&D providers have traditionally invested very little in analysis to support direction setting and resource allocations with historic allocations being the main guide to future allocations. This may also have been good enough in the past, but is increasingly inadequate.

This paper outlines an analytical approach to the evaluation of R&D and principles for allocation of scarce resources. It explains how they can be used as part of a system to meet the needs of R&D providers as they face their present challenges.

ANALYTICAL APPROACH

The analytical approach outlined in this section involves questions about the kind of project that the R&D provider should consider taking on. If it should be considered, the question of whether it is worth doing is then addressed by benefit-cost analysis (BCA). Issues of why BCA should be used and some basic standards of how it should be used, are presented.

(i) Simple decision rules

This involves asking a series of questions on whether an R&D activity should be carried out, or not.

If it should be done, who should do it? The issue here is whether public R&D providers should do it (or at least arrange for it to be done), or whether it should be left to the private sector. The public sector should have a role only if there is market failure. This criterion is based on whether there is any valid reason why the private sector cannot adequately carry out the task. The market can ‘fail’ where charging for the service is difficult, so that there is not the financial initiative for the private sector to deliver the service. The concept is explained further in Appendix 1.

Conclusions drawn on the basis of this criterion would be either:

  • There is market failure, so that a public R&D provider possibly should have a role;
  • There is not market failure, so that it should be left to the private sector.

If market failure does apply, it does not mean that the Government should do it. There is still a need to address the question:

Should it be done?

This is addressed by answering the sub-questions of:

  • Are benefits likely to exceed costs? and where funds are scarce,
  • Are benefits likely to exceed the value of the alternative use of resources?

If the answer to both these questions is yes, the work should be undertaken by Government or Government should make arrangements to overcome market failure. An example of the latter is the Rural Industries Research Council (RIRC) system set up by the Commonwealth Government.

(ii) Quantitative evaluation - BCA

The private sector has to deal with the same should it be done questions when it makes investment decisions. Private sector decisions to increase funding of some activities, and cut back on the funding of others, are based largely on the returns received or expected on the investments in the different activities. Its planning involves projections of revenues and costs of activities. These are brought together in formal investment analyses. The organisations are dynamic, with decisions being made on the basis of revenue and cost information and non-performing investments being phased out to fund potentially high return new areas. How well a firm performs depends to a very large degree on how well it chooses what to cut and what to expand.

BCA can provide some of the same support for public R&D providers. Unlike the private sector, public R&D providers do not receive the revenue generated from most of their activities but need to estimate the benefits industry is receiving or will receive. BCA techniques applied to agricultural R&D are very similar to those used in the private sector for investment analysis. A difference is that the software developed for agricultural R&D is customised to represent components of agricultural R&D investment.

There are many differences in the detail of BCA for different projects, however key information requirements for analyses are common to many different kinds of agricultural R&D. All analyses need to start with the question: what is the estimated difference in cost and revenue on the farm, (or firm) with and without the R&D being conducted?

Since the outcome may not be known with certainty (an exception is ex poste analysis) it may be necessary to specify more than one plausible outcome from the R&D. If so, not only do the outcome and its financial consequences need to be identified, but the probability of that outcome occurring needs to be estimated.

The scale of the potential benefits from R&D needs to be estimated. For example, if the unit being considered is the farm, how many farms are potential beneficiaries of this kind of research?

Not all of the potential beneficiaries will adopt findings of research, and those who do will not all adopt it at the same time. Thus there is the need to specify how many are likely to adopt, and by when.

R&D costs need to be included for each year.

Steps in the process of BCA calculation are described briefly in Appendix 3.

The BCA calculation for R&D, by accounting for the impact(s) on net income, probability of the impact(s), scale of impact, rate of adoption and R&D costs, integrates the relevant information for determining whether work should start, be continued or cease. It uses a mathematical computation consistent with well established standards of investment analysis to generate estimates of the return on the R&D investment. Three measures of this return which are used are the net present value (NPV), the benefit cost ratio (BCR), and the internal rate of return (IRR) (Appendix 3).

(iii) Why BCA?

BCA provides the support to decision making and direction setting, similar to that achieved in the private sector. However, there are other ways it can support the current needs of R&D providers, through supporting cultural shift and helping focus attention on outcomes.

The process of conducting BCA can contribute to cultural shift in an organisation and help achieve better thinking about R&D. Undertaking BCA properly requires that analysts think like an investor, and use the jargon of investment. It also requires them to focus thinking on the outcome of the R&D, and how much change to the industry can be claimed to result from it. However, until researchers feel they have a stake in the analysis, and the process it supports, it cannot be effective in influencing decisions or contributing to a cultural shift. Unambiguous leadership from the top of organisations is also required.

Conducting the BCA also makes clear which factors are likely to determine the success of the project, and which factors are most limiting the rate of returns that can be generated from the project (e.g. is it scale, profitability, adoption, probability of success?). It also makes clear the outcome which must be achieved for an activity to succeed (eg. the yield or quality improvement necessary for benefits to exceed costs) for an activity to succeed. Armed with this information, a researcher may be stimulated to rethink the emphasis of a project or the directions of research. When this kind of thinking has been developed, researchers have clearly gone beyond arguing for the status quo, and are contributing to a flexible and dynamic organisation.

Why the rigour of BCA analysis instead of softer alternatives? One alternative to BCA which can be used to support allocation decisions is a scoring system. By comparison with BCA scoring systems do not have any well established standard on the parameters to be scored nor on how scores should be weighted. Scoring is subjective, so that scores cannot be disputed and challenged in the way that the assumptions of BCA can. While it is concluded that BCA is a better support to decision making, the two are not necessarily mutually exclusive.

Most researchers and research leaders will give some thought to the benefits and costs of their R&D however this is not usually formalised. Collection, estimation, and argument about these data is, in itself, useful.

While this discussion emphasises the contribution to change, BCA also will help to identify which existing activities are highly valuable, and which should be retained or expanded.

(iv) Appropriate and Inappropriate use of BCA

It is important that analysts and those interpreting the results of analyses to support their decision making, are aware of the limitations of the analysis. The main limitation is that the data are never perfect. Garbage in means garbage out. BCA can be conducted cynically so that inputs can be manipulated to achieve a required answer. The software packages are very efficient for calculating the inputs needed to achieve a BCR greater than a real or imagined hurdle rate.

This limitation is not a case for discrediting BCA, but for using it properly. Important aspects of using it properly are adhering to appropriate standards for the conduct and calculation of BCA and also for information presentation to the user.

Analysis and presentation should make it easy for users (especially decision makers) to assess the credibility of the results. A minimum standard for this purpose is:

  • transparency of input assumptions and outputs (if you cannot see in, put it in the bin);
  • provision of break-even, and sensitivity analysis for parameters which are uncertain and are likely to have a big impact on results.

Given these standards, cynical analysis to achieve a required answer can be expected to damage the credibility of the analysts and their organisations. It must also be clear to those using the results of BCA, that the purpose of the analysis is to support decision maker’s judgement and not to replace it.

As well as having in place standards for transparency of assumptions, it is important at times to subject assumptions to external scrutiny.

ALLOCATION PRINCIPLES

Once analysis and judgement have identified existing low return activities and opportunities for high return new funding, reallocation of resources can proceed. Where an organisation does not have a tradition of doing this, it may find significant opportunities for reallocation.

Decisions occur at different levels: for example, the allocation between projects that may make up a sub-program and then allocation between sub-programs which may make up a program, and finally allocation between programs which may make up the organisation.

At each level the same principles apply of freeing up resources from low return activities and reallocating to high return activities.

This is simple at the level of funding or not funding a project but can be confusing at higher levels of aggregation. Take for example a case of determining the allocation between two sub-programs, where each sub-program consists of a number of existing projects but also has opportunities for high returns to new activities or acceleration of existing projects. If one sub-program has, on average, a BCR of 4 and the other a BCR of 1, this does not mean the program leader should shut down the second sub-program and allocate all of the resources to the first. The average return information is not a good basis for decision making. It is information on returns at the margin which is most important

The principle which applies is best seen in an illustration. Figure 1 illustrates the distribution of BCRs of activities in two hypothetical sub-programs. The area of the pie chart represents the amount of money spent on activities within the sub-program, and the segments of the pie chart are shaded to represent the BCR for the existing use of that segment of funds. Thus sub-program 1 shows nearly half its funds allocated to activities estimated to have a BCR of less than 1 (the most lightly shaded segment). However it also has a substantial amount of activity yielding moderately high returns and some yielding very high returns. Simply closing down sub program 1 would be inappropriate given moderately high and high yielding activities (dark shadings). Sub-program 2 has only about 10% of activities with a BCR less than 1. In addition to this information it is known that sub-program 2 has new high return investment options for about an additional 20% of its resources and sub-program 1 for about 5%.

No Analysis

 

BCR < 1

 

BCR 1-1.5

 

BCR 1.5-3.0

 

BCR 3-5

 

BCR >5

 

Figure 1. Pie chart representation of rates of return to hypothetical programs.

On the basis of the information presented, the decision should be to terminate or phase out funding from low BCR activities. The funds released by this would go to the high return new opportunities within each program and also there should be substantial reallocation from program 1 to program 2.

In theory reallocation from one sub-program to another should occur until, at the margin, all sub-programs have the same rate of return. In practice the accuracy of the BCA data and the extent of coverage of BCA do not warrant reallocation with such precision, and the judgment of decision makers may modify the ordering of activities so that allocation is not solely determined on the basis of benefit cost ratios. For example, a decision maker may back off a decision to radically reallocate resources because there is not a large difference between the BCR of the present and proposed R&D and there is some uncertainty about achieving the outcome of the proposed R&D. Judgement is also needed to allow for objectives other than those represented in the BCA and to allow activities which have not been analysed, to be put into a priority ordering. Where the judgement is made without an analysis it is useful to consider the R&D activities with respect to the BCA components outlined in part (ii) of the analytical approach section.

Reallocation of resources may, for example, require staff to be reallocated from areas they specialise in to other activities they will have to learn or be trained for. Ideally any lesser performance of staff in the new role and the cost of retraining should be included in the BCA. If it cannot be, then this should be allowed for in the decision maker’s judgement.

In the real world, the scarcity of resources available for BCA means that all activities cannot be analysed. Analyses should focus on providing information which is of most relevance to decision making. Thus it is important to analyse those activities which it is thought may have low rates of return or at least are not obviously high rate of return activities. Proposed new activities which would require significant funding, should also be analysed and analysis used to support decision making on major new strategic directions.

As with the doing of the BCAs, their use in the process of resource allocation can also support cultural change. Reallocation procedures require the R&D providers to become flexible and dynamic institutions, by formalising the search for new high return opportunities and inquiring to find the low return existing activities to fund these opportunities.

DECISION MAKING

The following provides examples of BCA use at several levels in an R&D providing organisation. Some examples are taken from Agriculture Western Australia.

(i) Individual research or research leader level. Here it helps by strengthening and supporting applications for funding, screening ideas, deciding which of the existing areas of work to cut back on, and helping achieve a cultural shift. Many of the good ideas for research projects come from researchers or are picked up by researchers through their involvement with industry. It is helpful for them to subject these ideas to the rigour of BCA and the way of thinking that goes along with this. Easy to use BCA software can be made available to researchers and training can be provided.

One of a number of examples of researcher driven evaluation is pasture research by Agriculture Western Australia. This work has gone beyond BCA screening proposals one by one and has involved developing some broader principles for directing investment pasture R&D. It is based on a series of analyses using BCA and farm models to help clarify on which soil classes the pasture breeding and development work should take place, and which characteristics of pasture it is most valuable to be breeding for (1; 2). These analyses have been influential because the research leaders see it as theirs rather than one undertaken by an external group of economists.

Funding applications with a credible BCA attached are increasingly likely to influence the success or failure of applications. The most effective BCA is one that is presented as an integral part of the case for funding, with key assumptions appended. At present, with some RIRCs it may not yet make much difference whether an application has a BCA or not, or whether the analysis is done well or not. However, the RIRCs are gradually gaining a better understanding of how to use BCA and its importance. Increasingly panels and boards are seen as decision makers investing millions of dollars in R&D so that some members of each panel/board may in future be expected to have skills in interpreting analyses as a basic tool of their trade.

(ii) Program Manager level. BCA supports decisions on resource allocation within and between sub-programs. The appropriate principles are as described in the previous section. Benefit cost information needs to be presented to program managers in ways that they can most easily understand and interpret.

For example in the case of Agriculture Western Australia’s wool program, low return activities were identified such as the lice eradication program and some sheep reproduction research, while high return opportunities for developing higher quality wool, were identified and resources were reallocated accordingly. On the basis of judgement by the program leader and the results of analysis, a major reallocation of resources occurred within the program.

Figure 2 shows the shift from 1992, when a large part of the program was yielding low returns (most lightly shaded segment) and none had a BCR above 5, to the present situation of very little in the poor return category and around 20% of the program achieving a BCR greater than 5. All the portfolio’s industry programs have been charged with the task of seeking out higher return opportunities and reallocating resources accordingly.

Wool Program 1992

Wool Program 1995

No Analysis

 

BCR < 1

 

BCR 1-1.5

 

BCR 1.5-3.0

 

BCR 3-5

 

BCR >5

 

Figure 2. Benefit cost ratios for wool program activities in 1992 and 1995.

Program managers need to be more than passive recipients of a pie chart summary. They should be thinking about the strategic directions of their programs and should be asking for analyses relating to this. They also need to receive additional information which supplements the pie charts such as key assumptions and sensitivity and break-even analysis. Tables 1 and 2 are an example of part of a summary output provided to program leaders. These tables are small excerpts from the wool and oilseeds full program summaries. Similar summaries were provided to all program leaders.

Table 1 provides an example of the relevance of the additional information. It shows that the ectoparasite control sub program has low benefit cost ratio (BCR = 0.9) which warrants further examination to see whether its continuation is justified. This BCR is not sufficient information on which to base a decision to cease or continue the sub program. Although it is low, uncertainty of data means that the sensitivity and break-even information are needed. These show the extent to which the sub program must achieve a reduction in prevalence of ectoparasites or a reduction in pesticide use for it to break-even. When presented in this way it is easier for a program manager to assess whether this program target is likely to be achieved. The break-even value then can become a target against which research achievement can be monitored. If further research showed achievement of the target to be unlikely, the research would be terminated. This makes the value of BCA much greater than simply presenting benefit cost ratios.

Table 1: Information to support program leader’s decisions - ectoparasite control for wool program.

Planned Achievement
(March 1994)

Best bet returns

Range of returns (2/3 of possible returns fall in range)

Key sensitivities and break evens

Ectoparasite control

NPV -$1.4 M
BCR 0.9

IRR 4%

NPV -$15M up to $9 M
IRR up to 17%
BCR 0.3 up to 1.3

Results very sensitive to how much the sub program reduces parasite prevalence. Eight per cent reduction needed to break-even. Twenty five per cent of benefits are derived from more efficient use of pesticides. Pesticide use needs to be reduced by $2M per annum (i.e. 35%) assuming no prevalence reduction to break-even.

Table 2 provides an excerpt from an analysis of a fertiliser research sub-program of the oilseeds program. It identifies the key adoption and yield increase assumptions. A break-even adoption of 4 per cent of present producers would seem to be quite achievable.

Table 2: Information to support program leader’s decision - fertiliser research for oilseeds program.


Planned Achievement


Best bet returns

Range of returns (2/3 of possible returns fall in range)


Key sensitivities and break evens


Fertiliser research canola.

NPV - $2.3M

IRR 146

BCR 7.5

NPV - $1 M to $3.3 M

IRR 72% to 194%

BCR 4 to 10

Assumed that 30% of canola producers will adopt recommendations resulting in an expected yield increase of 0.2 t/ha over 5 years. Adoption of fertiliser recommendations must be greater than 4% of producers by 1996 for this to break-even.

(iii) Executive/CEO level. Information provided here includes program pie charts aggregated to the program level, and notes to supplement these. Fig. 3 shows a sample of some programs analysed in 1994 for Agriculture Western Australia. Notes that go with these explain which activities correspond to each coloured segment and the focuses of the rate of return being as high or low as estimated. Key assumptions and sensitivity analyses are presented.

At a glance, the Figure 3 would suggest reallocation of resources from programs such as fruit and pigs to oilseeds. In the case of the programs presented here the Executive needs to consider key assumptions in the analyses to assess their credibility and also to have information on what the return on additional funds allocated to the oilseeds program would be. In this case the return on investment for new opportunities in oilseeds was also found to be high and the analyses judged to be credible. Subsequent to these analyses the oilseeds program has been amalgamated with legume crops and in Agriculture Western Australia’s 1995 budget allocation process, funding for legumes and oilseeds was increased in relation to the other industry programs.

The availability of information in Figure 3 to the Executive provides an additional incentive for Program Managers to reallocate resource within their programs. If they do not, large blocks of low return resources in the program will undoubtedly attract the attention of the Executive and are likely to lead to a reallocation of resources away from the program.

No Analysis

 

BCR < 1

 

BCR 1-1.5

 

BCR 1.5-3.0

 

BCR 3-5

 

BCR >5

 

Figure 3. Benefit cost ratios for some major programs

Information is also presented to government inquiries and central agencies to make the case that agricultural R&D is a good investment. BCRs are a new part of the Agriculture Western Australia’s standard reporting to Parliament and the Auditor General. Understandably, the Auditor General is requiring that reporting is balanced, showing low return activities as well as high return ones.

BCA is also a part of the key argument for retaining the level of Government funding which has been put to several recent State and Commonwealth inquiries. The principal line of argument is that agricultural R&D is a good investment yielding on average a BCR conservatively estimated to be above 3. Examples of such high return activities as development of the lupin industry (yielding net benefits greater than $25 million per year), development of two new apple varieties (BCR = 7) and prevention of the spreading of pests (BCR = 10) are used to illustrate potential returns. This line of argument is of critical importance to agriculture which is perceived in some quarters as a poor investment, unlike the new sunrise industries.

System of Prioritisation and Direction Setting

BCA needs to be part of an integrated decision making process which involves input from industry and Ministers for Agriculture and which also brings in strategic planning.

If industry and the Minister are a part of the process, they are much more likely to have a common view with the organisation’s leadership on the directions it should take. In the case of BCA analyses for major decisions, industry and the Minister should have access to key assumptions. The different perspective they have means that they may challenge and question assumptions, leading to a more credible analysis and improved decision making.

Strategic planning should complement the BCA. Broad strategic planning such as SWOT analyses are likely to be helpful in developing thinking about the new opportunities for R&D, and the broad directions that should be taken. BCA then provides detailed analysis to follow up the ideas and broad directions generated from strategic planning. BCA can be used as part of the strategic planning process. Major proposed shifts in direction should be subjected to BCA before they are accepted as a new strategic direction.

Investment analysis similar to BCA is also a key part of strategic planning for most large private sector companies. Large companies often undertake long term sophisticated analyses in which they look at different future growth scenarios and how a range of long term investment strategies perform across these scenarios. There is usually a ‘best bet’ scenario.

CONCLUSIONS

A changing environment and changing expectations require that R&D providers become more like private sector enterprises. BCA can contribute by supporting decisions on allocation of resources, helping achieve a cultural shift in these organisations and helping to win or maintain funds. In order to deliver these benefits some simple standard practices including transparency of analysis and sensitivity analysis must be met. Decision makers should be trained in interpretation of results so that they understand BCA’s strengths and limitations and how to use it, with their judgement, to support decisions.

Researchers in a position to influence or propose the direction that their work takes, should conduct BCAs to screen ideas and support major internal or external submissions for funds. They need to be able to use BCA as an integral part of the case they are making rather than simply an appendix. Most researchers already consider, informally, aspects of the benefits and costs of what they do or propose to do, so that the additional rigour of a BCA simply builds on this.

Those in positions to set broader directions in R&D providing organisations, need to be able to interpret BCA findings and understand principles of resource allocation. These should be part of the skills of R&D leaders in the new environment.

In spite of the case for BCA, its effective adoption in some R&D providing institutions is slow. There is a need to identify and address constraints to adoption. They may include some of the following:

  • Limited understanding of how it should be used;
  • Lack of drive from the top of the organisation;
  • Lack of a champion to implement it;
  • Inertia and comfort with the status quo in the organisation;
  • No one available to deliver the analysis; and
  • Lack of a client focus amongst those able to deliver the analysis.

BCA needs to be seen in perspective, as part of a prioritisation and planning system. The optimal use of it also needs to be considered. Although many R&D providers would now appear to be under investing in it, the quality of the BCA and its use is a more sensible goal than quantity. Users need to consider the ‘with’ and the ‘without’ for BCA.

ACKNOWLEDGMENTS

Neil Thomson for implementing of this work so professionally and significantly value adding in the process. Dr Bill Collins for helpful comments on the paper. Graham Annan for his support through developing the figures for the paper.

REFERENCES

1. Abadi Ghadim, A. K. and Morrison, D. A. 1992. An economic perspective on pasture research priorities. In: Proc. 6th Agronomy Conf. pp. 190-193.

2. Gittinger, J. P. 1982. Economic Analysis of Agricultural Projects. 2nd Edition.

3. (John Hopkins Univ. Press: Baltimore, USA).

4. Ewing, M.A. and Pannell, D.T. 1987. In: Temperate Pastures: Their Production, Use and

5. Management. (Eds. J.L. Wheeler, C.I. Pearson and G.E. Robards). (CSIRO: Melbourne) pp. 583-585.

APPENDIX 1

MARKET FAILURE

The market failure criterion is important and warrants further explanation. It can best be understood by describing how it applies to agricultural R&D and protection activities. An agricultural R&D project can be widely applicable to hundreds and even thousands of farmers. Each farm is a small business unit, incapable in itself of cost effectively mounting a major R&D program. Yet there are many problems or opportunities which many farmers have in common, and which can be addressed by a single R&D program.

Once the research has been conducted, it is very simple and inexpensive to make information from it available to an additional farmer is very small for most agricultural R&D. In these circumstances, the private sector is likely to under-invest in R&D. One reason is that it is difficult to exclude people from the benefits of much of the R&D. This means that it is difficult for charges to be imposed.

If it does invest, the private sector is likely to charge more for the service than the marginal cost of providing it to an additional farmer. This would discourage maximum use of information from R&D which can be made available widely, at a very low cost. Thus the private sector cannot be counted on to provide this kind of research without Government intervention and if it does provide it, it may restrict its dissemination.

Similarly activities protecting agriculture from pest outbreaks cannot be mounted individually by farmers because they do not have the incentive, nor the resources for an eradication or containment campaign. The market place will not solve the problem because the scheme would not work with farmers choosing whether or not to buy the service. However costs of pests spreading from one property to others can be extremely high, with high BCRs for prevention of spread.

The criterion of market failure also brings into focus the services that the public sector should not provide. In agriculture a clear distinction is made between information which is generally available to industry from R&D conducted, and that which is further customised to meet the particular circumstances of an individual grower. Charging farmers for a customised service is simple, the costs of providing it are relatively high per farmer, and it is not widely applicable. Therefore it should be provided by the private sector.

APPENDIX 2

STEPS IN R&D EVALUATION

Procedures for BCA can be broken down into a number of steps representing components for typical agricultural R&D evaluation. The following is a typical set of steps:

Identify and describe best estimate or estimates of outcomes from the R&D and the ‘without scenario’.

The starting point of these analyses, is to identify what is the difference, with and without the R&D being conducted. This can be difficult and may require some training or support from an experienced analyst able to probe and ask the right questions. One common error is to confuse the without situation with no change from the present. There usually are influences which will be causing changes in productivity and product quality irrespective of the R&D being evaluated. These must be represented in the without situation.

Representation of more than one plausible outcome of the R&D is often appropriate for dealing with uncertainty of future outcomes. A probability needs to be estimated for each outcome. The simplest example of this is two outcomes, success and failure. However, more outcomes can be specified. For example, it may be specified that an agronomic project has a 0.3 probability of achieving a 20% increase in yield, a 0.5 probability of achieving a 10% increase in yield, with failure having a probability of .2.

Estimation of net benefits per unit for each industry outcome. The unit in this case could be a hectare, an animal, a farm, etc. In the case of the above example it would be a simple gross margin of the yield increase, in other cases a more sophisticated analysis may be required.

Calculate potential benefits. This is done by scaling up from the benefit per unit to whole industry which is potentially affected. Frequently units are not homogeneous. It is then necessary to represent more than one kind of unit (e.g. different kinds of farms or different types of soil) and scale up each by multiplying by the number of units each represents.

Account for probability of realising outcome. Where more than one outcome is specified, weight the benefits of each outcome by multiplying by the probability of success of the outcome;

Account for adoption. Specify parameters for adoption over time. Accounting for adoption in the calculation converts potential benefits in each year to estimates of actual benefit;

Account for R&D costs. Enter R&D costs for each year;

Convert dollars of different years to a common value. Discount all benefits and costs, bringing them to the present year’s value. The discount rate is included in calculations to represent the differences of value of dollars in one year versus the next. It is not simply a case of allowing for inflation, but representing the opportunity costs, or the alternative uses to which the money could be put.

Calculate the various measures of return. These are the net present value (NPV) benefit cost ratio (BCR) and an internal rate of return (IRR). See Appendix 3.

APPENDIX 3

MEASURES OF BENEFIT COST PERFORMANCE

All three measures, net present value, internal rate of return, and benefit cost ratio, have strengths and weaknesses. Net present value (NPV) is mathematically the soundest measure. However it is not a useful measure for comparing projects of different sizes, where capital available for R&D is scarce, because bigger projects tend to have higher NPVs than smaller ones. The internal rate of return (IRR) can, under some configurations of benefits and costs, provide misleading results. However, it does allow comparison of projects of different size, and it avoids the need to specify a discount rate (see Appendix 2 for discount rate). The BCR is a more reliable estimate and is also suitable for comparison of projects of different sizes.

It is desirable to use all three measures, and most available software will generate all three. In our use we have found BCR to be the most useful and easily applicable.

It is useful for an R&D provider to have in-house expertise on these matters and access to credible texts such as Gittinger (2).

Previous PageTop Of PageNext Page