Previous PageTable Of ContentsNext Page

Coming to grips with community capacity: using Goal Attainment Scaling as a measure for Local Area Plans.

Rebecca Lukies

Department of Primary Industries, Private Bag 1, Ferguson Rd, Tatura Victoria 3616. www.dpi.vic.gov.au E-mail rebecca.lukies@dpi.vic.gov.au

Abstract

Many of us working as extension officers have seen community capacity building in action, however, when we are required to describe what it looks like, the answers are not always easy to report on. Most natural resource management projects focus on measuring activities that are easy to count, like the number of trees planted, and gloss over the more difficult area of measuring changes in community capacity. Local Area Planning in the Shepparton Irrigation Region is a community driven planning approach that relies strongly on partnerships between community and agencies to develop and implement a plan that addresses a range of social, economic and environmental issues.

A recent review of the Local Area Plan evaluation process found that previous evaluation work focussed on on-ground works and that little had been done to measure community capacity building. The Local Area Planning Implementation Team tackled this issue and developed a Goal Attainment Scale to create a shared understanding of what community capacity building looks like. This will assist them to develop a suitable evaluation approach. This presentation will focus on the process that the Local Area Planning Team went through to develop their shared understanding of community capacity and how they hope to use this information to evaluate the impact of their work into the future.

Key learnings include: (1) A shared understanding of community capacity is vita; (2) A Goal Attainment Scale is enhanced by broad involvement in its development; (3) Developing a Goal Attainment Scale can be a challenging and enlightening experience.

Key Words

Outcome scale, Participative approach, Catchment management

Introduction

Local Area Planning has been adopted in the Shepparton Irrigation Region (Figure 1) as a ‘bottom up’ extension approach to encourage communities and landowners to plan for the future in close collaboration with agency staff and the Shepparton Irrigation Region Implementation Committee. A primary aim of Local Area Planning is to accelerate the implementation of the Goulburn Broken Catchment Management Authority Regional Catchment Strategy, which is the strategic plan that guides natural resource management within the region and links all natural resource management projects to catchment-wide targets.

Figure 1: Distribution of the Local Area Plan areas across the Shepparton Irrigation Region, Victoria. (Barker, 2005).

The Shepparton Irrigation Region Implementation Committee is the community based group that directs the implementation of the Regional Catchment Strategy in the Shepparton Irrigation Region and works in partnership with a range of agencies, local government and community. The first Local Area Plan was launched in November 2000 with the most recent finalised in April 2005.

There are eight areas within the Shepparton Irrigation Region that have been targeted through the Local Area Planning approach. They were selected using a Geographical Information System with the aim of targeting planning and on-ground works into areas where a significant impact could be made. Therefore the eight areas were chosen because they had significant natural features which were at risk from factors such as elevated water tables, a high level of community activity and good uptake of existing extension programs.

Community Planning Groups were then established in each of the eight areas. These groups worked with agency facilitators for approximately two-years to identify issues of concern and appropriate actions to address them. The draft Local Area Plans were then reviewed by the broader community and government agencies and following further consultation, were endorsed for implementation. A community celebration was held to launch each of the eight plans and the groups are now working towards implementing these plans with the support of a Department of Primary Industries facilitator and a paid community member employed by the Goulburn Murray Landcare Network. Local Area Plans have increased the community ownership of issues described in the Regional Catchment Strategy by enabling them to identify how their local issues relate to the bigger picture.

An integral part of implementing the Local Area Plans has been the development of evaluation plans to assess their effectiveness in meeting the program objective of accelerating the uptake of the regional catchment strategy. Key Evaluation Questions focus on the collection of information to the practice change level of Bennett’s Hierarchy (Rockwell & Bennett 2004) and focus on the level of works in each of the areas.

A review has recently been conducted on the effectiveness of this evaluation process. The review identified that current methods were adequately collecting information that documented activities however a gap was found in recording changes to community capacity. All facilitators working with Local Area Planning groups agreed that they had witnessed changes in community capacity occurring, however describing these changes and how they might be documented, was much more difficult to define.

Community capacity was discussed in broad terms by Local Area Plan Facilitators as the communities’ capacity to manage and deal with change, although further describing what attributes the community would display to demonstrate this ability was much more difficult to define.

The concept of community capacity is a complex one. This paper describes how the Local Area Planning Team has used a Goal Attainment Scaling technique to develop a shared understanding of community capacity and describes the potential of the approach to further evaluate community capacity.

Approach and Findings

There are a number of approaches available to assess the effectiveness of programs however many have focussed on more quantitative information such as the number of trees planted or hectares of land protected, with less emphasis placed on qualitative aspects such as community capacity (McDonald & Kefford 1998).

Goal Attainment Scaling has been used by the Local Area Planning Team to help develop a shared understanding of what community capacity looks like and as a tool to measure progress towards building community capacity.

What is Goal Attainment Scaling?

Goal Attainment Scaling is an evaluation technique that involves the development of an outcome scale to measure an individual or groups progress towards achieving identified goals. Goal Attainment Scales are generally developed to focus on the goals that are targeted for change by a specific program and at its simplest involves setting a goal, implementing a program, determining whether the goal has been met and then using this information to change future activities (Kiresuk & Lund 1978).

Goal Attainment Scaling has been commonly used in the mental and general health fields to assist therapists and patients assess their progress towards achieving goals. Goal Attainment Scaling is now also commonly used in the fields of education, rehabilitation, medicine, corrections, nursing, social work and chemical dependency (Kiresuk, Smith & Cardillo 1994).

Kiresuk, Smith and Cardillo (1994) describe the following nine-step process as a training guide to assist in the development of a Goal Attainment Scale.

Step 1 - Identify the issues that will be the focus of the treatment.

Step 2 - Translate the selected problems into at least 3 goals.

Step 3 - Choose a brief title for each goal.

Step 4 - Select an indicator for each goal.

Step 5 - Specify the expected level of outcome for the goal.

Step 6 - Review the expected level of outcome.

Step 7 - Specify somewhat more and somewhat less than expected level of outcomes for the goal.

Step 8 - Specify the much more and much less than expected levels of outcome.

Step 9 - Repeat these scaling steps for each of the three or more goals.

When developing a Goal Attainment Scale it is intended that the goals and outcomes set through this process are clear and consistently measurable. This then allows others to use the Goal Attainment Scale to decide on a score, even if they have not been involved in its development.

The following example Goal Attainment Scale, figure 2, was developed for use in the field of mental health and provides a good example of goals and outcomes developed using the above nine step process.

Mental Health Goal Attainment Scale

Level of Attainment

Scale 1
Career Planning

Scale 2
Control of Anger

Scale 3
Self-Esteem

Much less
-2
than expected

Has not chosen any preferred fields

Less than 25% of time

Generally negative regard (feels worthless)

Somewhat less
-1
than expected

One or more fields chosen but no planning

At least 25% of time

More negative features than positive

Expected level
0
of outcome

Selected one or more fields with plans for achieving at least one

Controlled anger at least 50% of time in last 2 weeks (self report)

Feels that his positive and negative features are about equal.

Somewhat more
+ 1
than expected

Has followed through with plan (interview etc.)

At least 65% of time

More positive features than negative

Much more
+ 2
than expected

Acquired job in a selected field.

At least 85% of time

Generally positive regard for self

Figure 2: An example of a mental health Goal Attainment Scale (Kiresuk, Smith & Cardillo 1994).

As a typical Goal Attainment Scale, Figure 2 lists the three goals that the patient is working towards along the top of the table: career planning, control of anger and self-esteem. Under each of the goals there is a description of the outcomes that could be observed at different levels. These outcomes range between much less than expected at –2, to much more than expected at +2. The expected level of outcome is identified in the middle or 0 row and is the level that is the most likely to occur.

Goal Attainment Scaling as an approach to define and evaluate community capacity in the Local Area Planning Project.

Early in the development of a community capacity Goal Attainment Scale, facilitators working in the Local Area Plan program identified that it would be important to involve a broader project team to consolidate the different views of community capacity and establish a shared understanding for the term.

It was therefore vital that when “identifying the issues that will be the focus of the treatment” (Kiresuk, Smith & Cardillo 1994) a shared definition of community capacity is developed and that all members of the newly formed project team have the opportunity to contribute.

A Technology of Participation ‘Workshop Method’ (Participative Technologies 2003) was used to ensure that all project team members had the opportunity to provide input into the shared understanding of community capacity. This process involved everyone in the team writing down on pieces of paper the words that described community capacity for them. These words were then discussed and pinned on a wall and those with similar attributes were grouped together and given a name to describe them. The names developed became the goals along the top of the Local Area Plan Goal Attainment Scale (Figure 3) which relate to steps 2 and 3 of the nine step process describe by Kiresuk, Smith and Cardillo (1994).

As well as naming each of the goals, the project team also used the information generated from the ‘Workshop Method’ to develop an aim for each of the goals to describe what it would achieve. This contributed an additional step to the nine step process identified by Kiresuk, Smith and Cardillo (1994), however, was important for the group as it assisted them in the development of the shared understanding of community capacity. This ensured that when establishing the outcomes for each of the goals, all participants were clear on what the particular goal aimed to achieve.

In keeping with the nine step process, the project team also selected indicators or key elements for each goal and began to describe outcomes. Kiresuk & Lund (1978) suggest that the expected level of outcome be developed first, however the project team found that it was easier to think of both extremes first and began by identifying the much more than and much less than expected levels. They were then more easily able to come back and identify the expected level of outcome and complete the process by identifying the somewhat more and somewhat less than expected levels.

The Goal Attainment Scale took approximately four meetings of two-hour duration to complete. Members of the project team facilitated the development of the Goal Attainment Scale with particular guidance from those who were experienced in facilitating similar evaluation processes in the past and had prior exposure to Goal Attainment Scaling. A final review meeting was also held to discuss the scale and how it would be used to evaluate the Local Area Planning project into the future

Figure 3 illustrates the result of the Goal Attainment Scaling Process used by the Local Area Plan project team and shows clearly the goals along the top of the scale, aims and range of expected levels explored through the Goal Attainment Scaling Process.

Figure 3: Goal Attainment Scale for community capacity building in Local Area Plans of the Shepparton Irrigation Region.

 

Growing Knowledge

Motivation for involvement

Commitment & Participation

Values

Common purpose

Managing Change

Responsiveness

Aims:

We are using, sharing and building the collective knowledge and skills of the community

Our Local Area Plan captures peoples interest and stimulates a desire to be involved.

Our programs & activities cater for a range of community needs including altruistic and practical.

We respect each other’s perspectives, roles and responsibilities.

We understand the link between individual purpose and common purpose. (Ability)

We work together to manage change.

We are able to identify & positively address new and emerging issue

Most favourable expected

We have an active network that ensures the knowledge and skills within the community are used and built upon. There is a desire to increase knowledge and skills.

We have a big picture vision that has passionate and enthusiastic people, involved.

We organise and successfully run diverse activities, which cater for a large range of community needs.

We understand and respect each other’s roles and responsibilities, building upon these for the benefits of the community.

We plan and implement projects that demonstrate the links between individuals and wider catchment.

We have the confidence to instigate change within our catchment.

We are able to identify & positively address new & emerging issues & build them into the Local Area Plan document.

More than expected

We use the knowledge and skills within the community and are willing to build upon this for most issues.

We have a big picture vision that encourages passionate and enthusiastic people, to have a desire to be involved.

We organise and successfully run activities, which generally cater for the needs of the community.

We understand and respect each other’s roles and responsibilities.

We understand the links between individual and catchment wide projects and plan projects to strengthen the links.

We work together to encourage independence and build confidence to manage change.

We are able to identify & positively address new & emerging issues.

Expected level of success

We are aware of some skills and knowledge within the community but only use this on an ad-hoc basis. The community is willing to improve their knowledge and skills of some issues.

We are aware of the big picture, and there is generally enthusiasm and passion to be involved.

We organise events that attempt to cater for the needs of the community, but sometimes miss the mark.

We usually respect each others perspectives, roles and responsibilities, however occasionally conflicts may arise.

We understand the links between individual and catchment wide projects and attempt to plan projects accordingly.

We are developing confidence & independence to work together & promote change.

We have some ability to identify new & emerging issues & respond in a positive manner.

Less than expected

We are aware of some skills and knowledge within the community but choose not to use them.

We are aware of the big picture, but lack the motivation to be involved.

We attempt to organise events, however they rarely cater for the needs of the community.

We have limited respect for each other’s perspective, roles and responsibilities, which causes regular conflict.

We attempt to understand the links between individual and wider catchment projects however rarely use them.

We are developing our confidence but have difficulty working together to manage change.

We have some ability to identify new & emerging issues.

Least favourable expected

We’re not aware of the knowledge and skills and don’t seek this information from the community.

We’re are not concerned by the big picture and lack the motivation to be involved.

We rarely organise events that cater for the needs of the greater community.

We create conflict by not respecting each other and this is detrimental to the group and the community.

We don’t understand the links between individual and catchment projects.

We have limited confidence in our ability to manage change.

We have limited ability to identify new & emerging issues & will at times respond negatively.

Definitions:
Community: Wider catchment community, landowners.
We: LAP group, including everyone attending the LAP meeting (eg. representatives from local government etc.)

Discussion

The Goal Attainment Scaling process has added value to the Local Area Planning project in a number of ways, with the most significant being as a tool for developing a shared understanding of ‘community capacity.’ The process has allowed members of the project team to discuss and describe what community capacity building means to them and then develop specific goals and aims to provide a clear picture of what the project hopes to achieve. Whilst the Goal Attainment Scale will provide a valuable evaluation tool to measure success in achieving these goals, the evaluation process will also be enriched by the shared understanding and ownership the team now has for community capacity in the Local Area Planning program.

Involving a diverse project team in the development of the Goal Attainment Scale has also been important to the process and added rigour through the discussion of different ideas and opinions. Project team members had slightly different views of what community capacity looked like for them, which meant that by involving a diverse team, the hard questions could be asked and discussed. This ensured that everyone had a full understanding of the outcomes before they were added to the scale. By incorporating these diverse views a much richer definition was developed for community capacity than would have resulted if an individual or less diverse group had developed the Goal Attainment Scale.

Using a participatory approach also ensures that all project team members have ownership over the content of the Goal Attainment Scale and feel positive about moving forward and using it to evaluate community capacity. The project team now has a shared understanding of the contents of the Goal Attainment Scale, what community capacity looks like and a clear view of the goals that they are all working towards. This is particularly important for those working in the day-to-day facilitation of the project as they now have the confidence that their funders and managers share the goals they are working towards. A participatory approach has been a great vehicle for those involved to share their stories and views on community capacity and helped cement their understanding of the term and how it applies to the day to day work they do with Local Area Planning groups.

Subtle changes were made to the 9 step process identified by Kiresuk, Smith and Cardillo (1994) on an adaptive basis. Whenever the group had difficulty with a section, the process was altered to help overcome this, enabling the Goal Attainment Scale to move forward. This need to adapt the process however may not be surprising as Kiresuk, Smith and Cardillo (1994) explain that “ Goal Attainment Scaling has been successfully used in many settings that, in one or more ways, did not follow our recommendations.” This would suggest that, depending on the field it is used in, adaptations can be made to the general nine-step process and still ensure the integrity of the Goal Attainment Scale.

The development of an aim to describe what each of the goals means to the project team was perhaps the most significant change this project made and added value to the Local Area Plan Goal Attainment Scale. By using the information generated through the workshop method to develop the aims, the project team felt that all their ideas had been included. The project team also consistently referred back to the aims when they were developing the various outcomes in the scale as they helped to refocus on the goals and assisted to visualise what they could expect to see at the various levels of the Goal Attainment Scale.

One of the main hurdles faced when developing the Goal Attainment Scale was ensuring that all of the project team understood the concept and how the scale could be used. It quickly became evident when discussing the approach that some team members felt uncomfortable with the fact that they were not sure where the process was heading and how it might be used once it was developed. It is a credit to those involved that they were willing to put these concerns to one side and trust the process. To some extent this issue was overcome by regularly revisiting the objectives behind developing a Goal Attainment Scale, however it is only now that the goal attainment scale is finished that all members of the project team can see its potential for understanding and evaluating community capacity.

There was also some concern over the potential to compare group performance based on the score they received using the Goal Attainment Scale. It was recognised that this was a risk however the message that a Goal Attainment Scale is about monitoring the change from one level to another, rather than comparing scores, was made very clear by the project team when discussing how to evaluate using the scale.

It is a limitation of the Goal Attainment Scaling Technique that scores cannot be aggregated to compare between groups. However in the context of our work, Goal Attainment Scaling was seen as valuable in helping us understanding our goals, and the extent to which they are reached, and had we wanted to compare groups, another technique would have been used.

Another real challenge faced by the project team was agreeing on what the expected level of outcome would be for each of the goals. Although the literature explains that the ‘expected outcome’ is the outcome that is most likely to occur, it was felt that this may vary between groups and in different situations. An expected level was eventually agreed upon for each of the goals however it should be noted that the there is a need for these outcomes to be reviewed with reference to the different Local Area Planning Groups the Goal Attainment Scale will be applied to.

The ‘picture’ of community capacity that has been developed using the Goal Attainment Scaling process is relevant to those involved in the Local Area Planning in the Shepparton Irrigation Region. It was never intended to be a general definition of community capacity that other groups could necessarily apply to their situation.

However, what will be relevant to others is the process that the project team have used to develop their scale and the adaptations, and learnings they have made along the way. It is hoped that these learnings may assist other projects to go through a similar process, to develop a shared understanding of the goals they are working towards, and help them to develop evaluation strategies to address them.

Conclusions

The challenge of how to assess a change in community capacity is one that will be familiar to many extension and community based programs and one that was particularly important to Local Area Planning.

While evaluation strategies are currently in place to measure the impacts of the program, a gap was identified in the information available to evaluate a change in community capacity. The challenge for the project team was to not only find a technique that would enable them to evaluate community capacity, but to also develop a definition of community capacity for the project.

The Goal Attainment Scaling Process has been used by the Local Area Planning team to develop this shared understanding of community capacity with the potential for the scale to be used as an assessment tool to assess any change in the capacity of the Local Area Planning communities.

While the Local Area Planning team is currently in the process of testing and refining the Goal Attainment Scale as an evaluation tool, the benefits received from developing it are far greater than this. Each member of the team has a shared vision of the goals they are working towards and a clear aim of what each goal will achieve. All those involved in the process feel ownership over it and are excited by the prospect of identifying and evaluating community capacity, now that they have a clear picture of what they are looking for.

Although at times the process has been challenging, the outcomes of developing a Goal Attainment Scale have been well worth the effort and hopefully the experiences and learnings of this process will help others to develop their own.

Key learnings include: (1) A shared understanding of community capacity is vita; (2) A Goal Attainment Scale is enhanced by broad involvement in its development; (3) Developing a Goal Attainment Scale can be a challenging and enlightening experience.

References

Barker S (2005). Shepparton Irrigation Region Local Area Plans, Department of Primary Industries, Tatura, Victoria

Kiresuk T & Lund S (1978). Goal Attainment Scaling. In C. Attkisson, W. Hargreaves, M. Horowitz, & J. Sorensen (Eds.), Evaluation of Human Service Programs. New York: Academic Press.

Kiresuk T Smith A & Cardillo J (1994), Goal Attainment Scaling: Applications, theory and Measurement, Hillsdale: Lawrence Erlbaum.

McDonald B & Kefford B (1998) Exploring a New Approach to Evaluation Proceedings of the IIR Conference, Performance Measurement for the Public Sector, Sydney, Australia.

Participative Tehcnologies. Technology of Participation Course Notes, June 2002 – March 2003, Melbourne.

Rockwell K & Bennett C (2004) Targeting Outcomes of Programs (TOP) http://citnews.unl.edu/TOP/english Accessed 24th November 2005

Previous PageTop Of PageNext Page