Previous PageTable Of ContentsNext Page

Measuring impacts of an holistic farm business management training program

Donald Cameron and Shankariah Chamala

School of Natural and Rural Systems Management, University of Queensland

Abstract

This paper reports on the development of 4 new indices to measure impacts of an extension program on farming family participants. A case study approach within an action research framework incorporating qualitative and quantitative domains was adopted to explore the impact on Queensland farmers of a farm business management extension program. Four new indices were developed to quantify changes perceived by participants.

Two of these new indices, Management Constructs Change and Management Objectives Change, provided evidence of statistically significant changes in participant beliefs about, and attitudes towards farm business management. Although highly correlated with each other, these changes were unrelated statistically to any of seven other commonly used biographical or psychometric indices employed, including level of formal education.

The third measure, the Bennett Change index, provided statistically significant evidence that attitudinal and behavioural changes were more frequent in participants with less formal education, but also more frequent in participants who had high urbanisation and self-directed learning index scores.

A fourth measure, Values Change index, derived from the Management Change index, condensed objectives change data into values domain-oriented scales, and showed that attitudinal change occurred across a wide spectrum of the recognized human values domains.

It is concluded these new measures, whether unchanged or with context-relevant modifications, have potential as aids to program impact evaluation, in a range of agricultural and wider applications, where personal psychological issues are considered useful adjuncts to direct behavioural measures of change.

Introduction

The purpose of this paper is to report on a number of new instruments developed as aids to evaluation of the Property Management Planning (PMP) extension program in Queensland. The PMP program is a nation-wide initiative with the fundamental objective of improving the adoption and use of property management plans by the farm sector. This program has been previously described widely (e.g. Letts 1997, Cameron and Chamala 1999), so salient details need only brief treatment. The program consists of seven or eight themed workshops, known as the 'integrated workshop series' (IWS), delivered to groups each of five to ten family management teams, over a period of 8 to 12 months, and structured around a strategic planning process. The program (i) is whole-systems in orientation; (ii) operates at whole-farm scale; (iii) employs principles of strategic planning and action learning; and (iv) employs principles of adult or facilitative learning. For these reasons Property Management Planning (PMP) is considered an holistic farm business management extension program. There are many methods of evaluating program effectiveness; however, there is a dearth of measures to capture the impact on the family of such a comprehensive extension program as is presented in the IWS.

This paper is one of a series describing the approach taken to documenting program impacts. The focus here is on describing efforts to objectively quantify impressionistic or qualitative data gathered through semi-structured interviews supplemented with completion of constructed opinionaires. Construction of four indices is described: Bennett Change; Management Objectives Change; Management Constructs Change; and Values Change.

Methods

The subjects of the evaluation were 23 families and 46 individuals in the first four groups in central and southern Queensland to complete the IWS Semi-structured interviews were conducted on-farm with all family management team members who had attended part or all of the workshop series. The main thrust of the interview was documenting changes perceived by participants to be partly or fully attributable to the IWS. This approach overcomes the cause & effect assumptions researchers/ evaluators make through conventional statistical methods. To augment semi-structured interviews, respondents were asked to complete two questionnaire instruments designed to capture perceptions of changes to management objectives and management constructs.

Bennett Change Index

The schema selected for studying program impacts was the seven-levels-of-evidence ‘hierarchy’ devised by Bennett (1975) for extension program evaluation. The upper three levels were most pertinent to this evaluation. They are, respectively: Level (5) benefits through changes in knowledge, attitudes, skills and aspirations (KASA); Level (6) changes to practice implemented; and Level (7) resulting outcomes.

The Bennett Change Index was constructed by allocating to one of these categories each change identified by participants, and devising an index score based on the sum of all changes identified by each family team. There was no attempt to differentially weight changes at different levels of the hierarchy.

Management Constructs Change

This rating instrument was developed to capture program impacts in terms of changes in participants’ beliefs about ‘good’ management. Its development followed a two-phase process, in a method based on Personal Construct Psychology (Kelly, 1955), and adapted by Ilbery and Hornby (1983) and Briggs (1985).

Phase one — construct elicitation

This phase had three stages. In the first stage, participants were asked to think about the questions, ‘What makes a good manager? What is it that good managers do, that makes them better or more successful than other managers?’ To aid this process, respondents were taken through a two satge process. First they were asked to identify (to themselves only) individuals who represented the following categories:

  • own farm/self
  • long term resident of the district
  • newcomer
  • a very good farmer;
  • a bad farmer;
  • a farmer to ask for advice;
  • a neighbouring farmer.

The second stage involved a process, first described by Kelly (1955), of considering these exemplars in groups of three (triads), and deciding how two differed from the other one in terms of some attribute. This ensured that respondents not only generated their own management constructs, but also defined their own poles of each construct. For example, from the statement that two farmers were ‘well organised’ and another was ‘disorganised’ the construct ‘organising ability’ could be developed, with polarity defined as well organised at one extreme and disorganised at the other.

In the third stage, management construct statements were reviewed for commonality in theme, and aggregated where possible. From a total of 82 elicited management statements, the positive poles of 25 different constructs were distilled. These were grouped into four categories: attributes good managers have; actions good managers take; attitudes good managers have; and cognitive skills good managers demonstrate (see Appendix 1).

Phase two — questionnaire preparation.

The 25 constructs were listed in a questionnaire where each could be rated for its importance to management, on a Likert scale from 1 (unimportant) to 7 (extremely important). Respondents were asked to rate for importance each construct twice: ‘now’ (at the end of the program), and ‘before the program commenced’. This allowed demonstration of two attributes of respondents’ attitudes towards management: (i) the relative rating, at the end of the program, of the importance of each construct, and (ii) the possible impact of the PMP program, through facilitating perceived changes in relative rankings of constructs.

The Management Objectives Change index was developed from a questionnaire made up of 60 objective statements covering the full spectrum of management responsibilities. Statements were drawn from several literature sources. These included: Kadlec (1985), whose 30 item listing of management objectives drew on a large body of American research; Keith (1986), who drew on six recent Australian studies for his 28 point ‘land concern’ instrument; Reeve and Black (1993), who developed a 75 item instrument to assess Australian farmers’ attitudes to environmental issues; and McGregor et al. (1995), who cited four recent UK surveys in developing their 39 point questionnaire for investigating UK farmer decision making. Participants were asked to rate these sixty possible objectives twice, for before and after completing the PMP program, on a scale ranging from -1 (opposed to my beliefs), through 0 (unimportant), to 7 (of supreme importance).

The Values Change Index was constructed to explore the possibility that changes in participants’ perceptions of the relative importance of various management objectives were associated with deeper restructuring of their values sets. If so, this would provide evidence of significant and long-term program impacts. In the widely accepted normative view of management behaviour, management actions proceed, through a series of iterative processes, from explicit and implicit objectives or goals. Goals are related to attitudes, which in turn are functions of an individual’s belief system or values set (Rokeach 1968). ‘Values’ are defined as: ‘Concepts or beliefs about desirable end states or behaviours that transcend specific situations, guide selection or evaluation of behaviours and events, and are ordered by relative importance’ (Schwartz 1994).

A theme of the values literature is the identification of a number (typically 7 to 11) of motivational domains, universal across all cultures, into which values may be categorised (Hall 1986; Schwartz and Bilsky 1987; Schwartz 1994; Colins and Chippendale 1995). Another common theme is that some values/goals/objectives will be more important than others, and therefore they will exist in an hierarchical form. Consequently, individuals may rate or rank values/goals statements for relative importance. That such rankings can change, accompanied by congruous long-term attitude and behaviour change, has been demonstrated experimentally by Rokeach (1973) Grube et al. (1994) and Waller (1994).

In view of this background, the assumptions underlying the construction of the objectives questionnaire, the Management Objectives Change index, and the Values Change index are:

  • Value orientations and related goals/objectives will be expressed in an individual’s approach to farm business management.
  • Values and goals/objectives can be ranked or rated for relative importance.
  • Objectives rankings and ratings are subject to change in response to life experiences including education and training.
  • Scales may be constructed from summing ratings scores of related objectives, in order to summarise expression of preference for particular orientations.
  • Measurable changes in management objectives preferences over time represent changes in underlying values orientations.

Orientation scales

Embedded within the list were nine possible scales, each of which could be related to a values ‘domain’ of Schwartz (1994) or ‘cluster’ of Hall (1991) and Colins and Chippendale (1995). The intention was not to canvass exhaustively the full spectrum of possible domains, or of the 125 or more identified values, but to sample a range of issues known to be of importance to practising farm managers, and to attempt to relate perceived attitude changes towards them to accepted theory. The proposed objectives scales and statements, together with their respective domain orientations, are shown in the relevant section below.

Results and discussion

Bennett Change Index

Identified changes attributable to FutureProfit were totaled for each individual, to produce their Bennett Change Index score. Results presented in Table 1 show collation of individual scores into group totals. These results demonstrate the ability of participants to identify wide-ranging benefits from program involvement, and thereby the utility of the Bennett schema in documenting impacts. Individual scores were subsequently used in correlational analyses with a suite of other psychometric and biographical indices not reported here, where statistically significant positive associations were found with scores for self-directed learning, urbanisation and conceptual ability.

Table 1. Changes* in each Bennett category

Group

Level 5 changes

Level 6 changes

Level 7 changes

Total changes

 

Knowledge

Attitudes

Skills

Aspirations

Practices

Outcomes

 

1

24

19

23

12

17

7

102

2

10

6

9

8

17

1

51

3

10

14

7

8

13

0

52

4

26

33

23

3

38

0

123

Grand Total

70

72

62

31

85

8

328

* Group scores show total changes identified by individuals within each group, classified according to the hierarchy of Bennett (1975).

Management constructs change

The effects of the program across a spectrum of management constructs is evident in Tables 2 and 3, which show respectively the most notable impacts on construct ratings, and the change in perceptions of most important constructs (for parsimony only the top 5 are shown in each case).

Table 2. Mean Pre- and Post-program ratings, and change in ratings, for management constructs, all groups (maximum rating=7)

Management construct

Pre-PMP

Post-PMP

Change a

think and plan

5.31

6.31

1.00

***

get involved in community

4.88

5.65

0.77

***

professional approach to farming

5.85

6.54

0.69

**

aware of major constraints

5.62

6.31

0.69

***

able to work with people

5.38

6.04

0.65

***

MEAN of all 25 constructs

5.47

5.90

0.43

 

a : *** P<.001, ** P<0.01

Table 3. Highest rating management constructs, pre- and post-PMP

Top 5 management constructs, pre-PMP

Mean rating

Top 5 management constructs, post-PMP

Mean rating

manage for the future

6.24

manage for the future

6.60

look after the family

6.08

willing to listen, learn and change

6.58

willing to listen, learn and change

6.04

professional approach to farming

6.54

make decisions quickly

5.96

look after the family

6.38

accept change as a challenge

5.88

think and plan

6.31

Management objectives change

The impact of the IWS on management objectives was measured as for management constructs above. Data was aggregated within and across groups. Results are shown in Appendix 2, in which objectives are shown in their values domains, discussed below. Results show that the instrument was able to capture participants’ perceptions of the ways in which the program had contributed to reorientation of their approaches to management. The major changes identified, including a heightened awareness of the importance of planning, were congruent with program objectives.

Values orientation change

Documenting change in values orientation proceeded trough a two-step process: (1) validation of domain scales embedded in the list of objectives statements using Reliability Analysis, and (2) determination of changes in scale means, attributable to the IWS. In the first stage, exploratory scale construction led to the satisfactory combination of 59 of the 60 statements in eight domains scales, each containing between four and twelve statements, and meeting the scale reliability criteria of (a) minimum item-to-total correlation coefficients above 0.3, and (b) an alpha value above 0.7 (de Vaus 1995, p.256). Several of these scales were renamed to reflect the most dominant item or items (those having the highest item to total correlation) in each.

Summary information relating to these eight scales, incorporating 59 of the 60 objectives items, is presented below in Table 4 and Appendix 2. Scale component items and mean rating changes are shown. The only objective excluded ultimately was ‘provide a good education for children’.

Data presented in these two tables suggest that the program impacts extended beyond narrow, business-oriented issues to include stimulus of other values domains including environmental concern (Maturity), Prosocial, and Enjoyment. The only scale not to show substantial stimulus was Traditional. In view of both the statistical significance levels, and the changes in scale means, all scales apart from this one were influenced to a similar level.

Table 4. Summary data for objectives/values scales

Objectives/
Values scale

Equivalent Schwartz domain

α value

No. of items

Scale mean change

signif a

Progressive

Achievement

0.7762

12

0.520

***

Business control

Self-direction

0.8071

8

0.650

***

Economics/success

Achievement

0.7857

9

0.526

***

Traditional

Tradition maintenance

0.7804

5

0.237

*

Ecority

Maturity

0.7026

8

0.622

***

Recreation

Enjoyment

0.7028

6

0.816

***

Conservative

Security

0.6714

7

0.602

***

Collaborative

Prosocial

0.6781

4

0.566

***

a two tailed paired sample t test: *** significant at p<0.001, ** significant at p<0.01, * significant at p<0.05

Conclusions

The main purpose has been to report on the development and application of new approaches to documenting impacts of an extension program, rather than the impacts themselves, which have been reported previously. Semi-structured interviews with participants in PMP/FutureProfit presented strong impressionistic data that the program had made real impacts on individuals, families, their approach to management, and on the farm businesses they managed. The new indices described here have helped to document those impressions in meaningful ways that are derived from and in turn add to existing knowledge about meaningful human activities in a farm management context.

Five specific conclusions are drawn:

1. The training program made tangible impacts on individuals, families, their approach to management, and on the farm businesses they managed.

2. Although lack of formal education is seen as a likely impediment to joining a post-formal education/training program, it is not an impediment to gaining benefit from such training once an individual or family is involved. Indeed, those with less formal education appear to benefit as much or more than those with more education.

3. There is an important role of 'gatekeeper' for respected local identities in leading by example and encouraging training-avoiders into training programs.

4. Group culture has a powerful influence on learning experiences and program impacts on individuals.

5. The new indices developed have potential application for measuring impacts of other adult education and extension programs. In some rural management contexts the measures could be applied with little or no change. The concepts and general approach could be applied, with context-relevant modifications, in a range of wider applications, where personal psychological issues are considered useful adjuncts to direct behavioural measures of change.

References

  1. Bennett, C. (1975). ‘Up the hierarchy’. Journal of Extension, March/April, 7-12.
  2. Bennett, C. (1977). Analyzing Impacts of Extension Programs, Extension Service, United States Department of Agriculture, Washington D.C.
  3. Brannen, J. (1992). ‘Combining qualitative and quantitative approaches: an overview’. In J.
  4. Brannen (Ed) Mixing Methods: Qualitative and Quantitative Research, Avebury, Aldershot, 3-38.
  5. Briggs, J. (1985). ‘An exploratory study of farmers’ choice of crops in Central Sudan’. Trans. Inst. Br. Geogr. N.S. 10:170-180.
  6. Cameron, D. C. and Chamala, S. C. (1999). ‘Futureprofit in Queensland: the positive impact of an adult education program on sustainable farming’ Proceedings 12th International Farm Management Congress, Durban, (Contributed Papers), International Farm Management Association.
  7. Cary, J.W. (1993). ‘Three eras of extension in Australia: from public folly to private good’. In J. Coutts et al (Eds) Proceedings Australia Pacific Extension Conference, Surfers Paradise, October 12 – 14, Vol I, 71-75.
  8. Colins, C. and Chippendale, P. (1995). New Wisdom II: Values-Based Development. Acorn Publications, Brisbane.
  9. De Vaus (1995). Surveys in Social Research. Allen & Unwin, St Leonards.
  10. Ekins, P. (1995). ‘Economic policy for environmental sustainability’. In C. Crouch and D.
  11. Marquand (Eds) Reinventing Collective Action. The Political Quarterly Publishing Company, Blackwell Publishers, Oxford, 33-53.
  12. Giles, A. K. and Stansfield, M (1990). The Farmer as Manager, George Allen and Unwin, London.
  13. Grube, J.W. et al. (1994). ‘Inducing change in Values, attitudes and behaviours: Belief System Theory and the method of Value Self-Confrontation’, Journal of Social Issues 50 (4) 153-173.
  14. Hall, B. P. (1986). The Genesis Effect: Personal and Organizational Transformations, Paulist Press, New York.
  15. Ilbery, B.W. and Hornby, R. (1983). ‘Repertory grids and agricultural decision making: a mid-Warwichshire case study’. Geographisker Annaler 65(B) 77-84.
  16. Kadlec, J.E. (1985). Farm Management: Decisions, Operation, Control. Prentice Hall, Englewood Cliffs, New Jersey.
  17. Keith, K. (1986). Human Factors Behind Soil Conserving Land Management by Grain Farmers in Southern Queensland. Unpublished M.Agr.Sci thesis, University of Queensland, Brisbane.
  18. Kelly, G.A. (1955). Principles of Personal Construct Psychology. Norton, New York.
  19. Land Management Task Force Report, (1995). Managing for the Future, Report of the Land Management Task Force, DPIE, Canberra.
  20. Letts, M.A. (1997). ‘A humanistic approach to facilitating change in agriculture.’ Proceedings 11th International Farm Management Congress, Calgary, (Contributed Papers), International Farm Management Association.
  21. McGregor, M. J., Willock, J., Dent, B., Deary, I., Sutherland, A., Gibson, G., and Grieve, R. (1995). ‘Edinburgh study of decision making on farms: links between psychological factors and farmer decision making’. In R. M. Bennett, (editor). Proceedings 10th International Farm Management Congress: Contributed Papers. International Farm Management Association, Reading, U.K. (153-166).
  22. Mues, C., Roper, H. and Ockerby, J. (1994). Survey of Landcare and Land Management Practices 1992-93, ABARE Research Report 94.6, Canberra.
  23. Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. (2nd edition). Sage Publications, Newbury Park, California.
  24. Reeve, I. J. and Black, A.W. (1993). Australian Farmers’ Attitudes to Rural Environmental Issues. The Rural Development Centre, University of New England.
  25. Rokeach, M. (1968). Beliefs, Attitudes, Values. Josey Bass, San Francisco.
  26. Rokeach, M. (1973). The Nature of Human Values. Free Press, New York.
  27. Schwartz, S. H. (1994). ‘Are there universal aspects in the structure and contents of human values?’ The Journal of Social Issues. 50 (Winter), pp. 19-45
  28. Schwartz, S. H. and Bilsky, W. (1987). ‘Towards a universal structure of human values’. Journal of Personality and Social Psychology 53(3), pp. 550-562.
  29. van Beek, P., Claridge, C. L. and Frank, B. (1998). National Evaluation – Property Management Planning: National Report. Centre for Integrated Resource Management, University of Queensland.
  30. Waller, J. E. (1994). ‘Dual processing in value self-confrontation: the role of Need for Cognition’. Journal of Social Issues 50 (4) pp. 175-188.
  31. Grube et al. (1994).

Appendix 1. Management constructs elicited from farm managers

What good managers do

1. are always organised and ready to go, timely with operations

2. don’t skimp on inputs (go for high yields, even when prices are low)

3. get involved in community, interact with other farmers

4. have price & production benchmarks in head

5. keep in touch with current industry/market/political climate

6. keep in touch with lender, know cash position, don’t overspend

7. keep up to date with new ideas and technology

8. take fewer risks because of the way they farm

9. take time to think and plan

10. understand their land, maintain resources, manage for the future (conservative, not exploitive)

Attributes good managers have

11. ability to work with people

12. are aware of major limiting factors affecting farm and business performance

13. expertise, breadth of knowledge, skills that can’t be taught, coming from long experience in farming, proven performance over a long period

Attitudes good managers have

14. do the best you can with what you have

15. like a challenge, accept change in a positive way, want to improve

16. look after the family

17. prepared to use and pay for professional or scientific advice, services

18. professional approach, see farming as a business

19. willing to listen and change, prepared to learn

Cognitive skills of good managers

20. able to make decisions, quickly when necessary

21. able to react to the immediate situation without losing control of longer term plans

22. able to stick with decisions, not change suddenly

23. able to take calculated risks

24. analytical, logical

25. innovative ability

Appendix 2. Objectives clustered through Reliability Analysis into values domains, and mean change in Likert ratings for objectives and values domains

Value/Objective

Mean rating change

t test sig.

Progressive

   

seek new ways of doing things on the farm

1.000

diversify activities on-farm

0.895

***

improve the productivity of the farm

0.711

***

have the best crops/livestock in the district

0.553

**

achieve/exceed local/industry production/price benchmarks

0.526

*

achieve or exceed production/price targets I set myself

0.500

*

maximise efficient use of all resources

0.474

*

adopt modern varieties, techniques & equipment

0.474

*

lead with new ideas

0.316

maintain improvements

0.316

have up to date machinery

0.289

win at shows

0.184

Mean

0.520

***

Business control

   

become involved in farm business management group

1.105

*

develop a long term plan for the farm

0.947

***

achieve development plans already set

0.816

**

minimise operating costs

0.711

***

maximise profit

0.658

**

maintain tight control of budget

0.289

increase size of farm business

0.026

Mean

0.650

***

Ecority

   

utilise resources sustainably

0.974

***

minimise chemical use

0.684

**

contribute to repairing environmental damage on our farm

0.579

***

improve visual/aesthetic appeal of farm

0.579

**

prevent pollution

0.579

**

encourage wildlife

0.553

*

leave the land as good as I found it

0.526

**

contribute to repairing environmental damage in the district

0.500

**

Mean

0.622

***

Recreation

   

have annual holidays off farm

1.263

***

have outside interests

1.132

***

have recreation time with family

0.763

***

have recreation time individually

0.711

***

improve quality of life

0.421

develop other skills outside farming

0.421

Mean

0.816

***

Traditional

   

stay in farming whatever happens

0.474

**

continue the family farming tradition

0.368

*

pass on the farm

0.211

operate on day-to-day basis

0.184

work independently

-0.053

Mean

0.237

*

Economics/success

   

improve living standards

0.737

**

have a comfortable living

0.658

***

contribute membership/leadership to industry organisations

0.579

*

achieve recognition as a top farmer

0.579

*

have respect in the community

0.579

*

have off-farm investments

0.553

*

control of the farm business and its assets

0.395

provide employment

0.368

expand business to make room for children

0.289

Mean

0.526

***

Conservative

   

minimise tax paid

1.263

***

minimise debt

0.947

***

maximise any legislative entitlements received

0.711

**

plan retirement

0.711

***

minimise risk

0.368

eliminate debt

0.342

have off-farm employment

-0.132

Mean

0.602

***

Collaborative

   

improve family communication

0.921

help other farmers

0.737

***

contribute group membership/leadership to community affairs

0.579

**

work with others

0.026

**

Mean

0.566

***

GRAND MEAN

0.574

*

a two tailed paired sample t test: *** significant at p<0.001, ** significant at p<0.01, * significant at p<0.05

Previous PageTop Of PageNext Page