Get Together FaST Evaluation Report

John Farhall

La Trobe University

Psychology Department

 

 

The 'Get Together FaST' (GTF) Project was a large scale training and service development initiative. Its general aim was to further the implementation of Victorian Mental Health Branch policy as outlined in the 'In Partnership' document (Human Services Victoria, 1996). More specifically, its purpose was to assist mental health services staff to work more collaboratively with families and other carers of those with mental illness and disability.

 

The project commenced in mid 1997, with planning, recruitment, establishment of advisory groups and commencement of consultation with the field. Training was conducted between February and November 1998. The project was implemented in three 'streams': Adult, Aged and Child and Adolescent Mental Health (CAMHS) Services, each having designated training consultants and developing related, but distinct, materials and training experiences.

 

A full description of the aims, strategies, consultation processes and training programs is available in the main Project report available from the Bouverie Centre and the Mental Health Branch. Please contact Brendan O'Hanlon at b.o'hanlon@latrobe.edu.aufor more information.

 

 

The evaluation strategy for FaST aimed to assess any impact of the project at multiple levels ranging from the intra-personal (did the project change staff beliefs about working with families?) to the statewide (did the project lead to changes in patterns of contact with families?). This section outlines the rationale and methods for the main evaluation components. Most measures were collected via Pre-training and Post- training questionnaires completed by participants in the 44 training programs. In addition, the project devised questionnaires (or structured interviews) for other parties such as managers, carers, consumers and the trainers to complete. All questionnaires are reproduced in the Appendix.

 

Participation in any training and service development project is a prerequisite for success. Without appropriate levels and representativeness of participants, the goals of the Get Together FaST project would be unlikely to be achieved, even if the plans and training content were excellent. Thus, participation was evaluated by:

 

A simple measure of success of a training project is its acceptance by, and impact upon, the target participants. Clearly, if the training process does not engage participants, or if the content is perceived to be unsatisfactory, benefits of training are less likely to emerge. The Post-training questionnaires (See Appendix) included questions seeking overall satisfaction ratings, opportunities to comment on best and worst aspects of the program and feedback ratings on key elements of the program.

 

A simple way to measure of the impact of an intervention is to ask the participants to rate themselves on relevant outcome variables before the intervention and then to rate themselves again after the intervention. It is then possible to determine whether any change in rating is likely to be greater than that expected by chance. This method was used to measure self-rated changes in family sensitive practice and self-rated experience, skill and knowledge. For each stream, the 'Self-rated family sensitive practice' score was derived by summing ratings on items which asked the extent to which participants currently put into practice the objectives of the training. These objectives were clarified with the trainers and formulated into statements for the questionnaires. The skill, knowledge and experience ratings enabled us to study the number of people who changed their ratings following training.

 

The Get Together FaST training aimed to influence opinions and attitudes about families and family work. Within the broad context of government policy promoting "partnerships" with family carers ((Human Services Victoria, 1996)), the training encouraged sensitive understanding of the point of view of family carers. This was addressed at many levels; the inclusion of family carers in the training program, the language used by trainers, the concepts and models utilised (particularly the 'trauma model'), the emotive messages of the FaST videos and so on. The intention was to encourage a higher level of 'family sensitive' practice in attendees, at least in part by promotion of more family-sensitive attitudes. It was therefore a central aim of the evaluation to determine whether any intended attitude change occurred.

 

We know of no validated measure of 'family sensitive practice', yet a key evaluation question is the extent to which participants may view families and family work more sensitively or more favourably as a result of the training. This evaluation therefore built upon the work done previously in the Northern Area Pilot Study (Farhall, 1996). In that study, a questionnaire was devised for workers in adult mental health services (the Opinions About Family Work Scale) drawing upon the published literature on professionals' attitudes towards families and also utilising themes raised in a focus group of family carers and advocacy workers. A key difference in this study was the inclusion of staff from Aged Mental Health Services and Child and Adolescent Mental Health Services. After attendance at the Aged Advisory group and discussion of the adult version with the trainers, an expansion of the adult version was considered to be equally relevant of carers of aged consumers and adult consumers, thus the same expanded version was used in both streams (See Appendix). This 'Opinions About Family Work Scale - Adult/Aged Services Version' included 26 of the original 30 items, supplemented by 14 items suggested by a further review of the literature, discussions within Reference Groups and feedback from the trainers. These new items included statements addressing difficulty or complexity of work with families, and included acknowledgement of issues facing carers of aged consumers. The CAMHS Version had 22 items in common with the adult version and an additional 16 items which introduced themes of treatment needs of the parents, emotional disturbance in the family and the family's need to be informed about services for their child or adolescent. These items were derived from a review of the literature, and from discussion with the CAMHS stream trainers.

 

For this study, measures of attitude were derived in two ways for each version of the Opinions About Family Work Scale. A simple 'Pro-family' score and 'Difficulty of family work' score were computed by totaling relevant item ratings. Taken at face value, most of the individual questionnaire items were either positive or negative statements about families and family work. Ratings on these were combined for each individual to give a 'Pro-family' score. The 'Difficulty score' was also computed by totaling scores on items describing difficulty and complexity in working with families (See Section 2.5 for list of items allocated to Pro-family and Difficulty measures).
The second method of attitude measurement was based on Factor Analysis of the underlying themes in each questionnaire. Data sets from each version of the questionnaire were subjected to a principal components analysis, with four underlying attitude factors identified in each case. Each person was then given a score reflecting their level of attitude on each of the four identified factors.

 

Self-report measures are a useful source of information about acceptability of an intervention and about the participants' perceptions of change. Attitude measures are a more powerful way of determining whether change may have occurred. However, supplementation of these by use of sources other than the participating staff is desirable, to bring in 'objective' sources of evidence for change.

 

 

One source of more objective observations of change is the superiors of the staff attending the training. To tap this source of observation, a 'Managers/Team leaders/Coordinators' Questionnaire was devised. Consultation with the local convenors for each area training program provided a mailing list of managers, team leaders and coordinators of participants who had attended GTF. These managers, team leaders and coordinators were then sent a questionnaire and invitation to participate in the GTF evaluation. Some had attended the training and some had not. The managers questionnaire (See Appendix) primarily sought information from managers about what feedback had been given by staff about the training, as well as observations by the manager about how GTF may have affected their organisation.

 

The GTF project was conceptualised as a service development initiative as well as a training program. Syndicate activities were the main element of the service development aspect of GTF, although the pre and post-training consultations also included discussions beyond the domain of training. Training participants formed syndicate groups that developed a project brief or proposal as part of the training program. Projects were to be aimed at furthering Family Sensitive Practice in the workplaces represented in the syndicate.

 

The evaluators followed up both the topics and scope of these project proposals and the proportion of proposed projects actually implemented. Information about the projects was gathered from the 'Managers/Team Leaders/Coordinators Questionnaire' and the Project Brief Profile sheet which facilitators completed at the post training consultation with each area mental health service.

 

 

Increases in the numbers or duration of contacts with family members was not a primary outcome intended by the project, but was considered to be a likely outcome on the basis of the Northern area pilot program.

 

Although participant feedback questionnaires requested information from staff about numbers of family members seen over the past week, these data were known at the outset to be a very limited opportunity to pick up any change in service patterns, for the following reasons. These questionnaires

a)      only pick up changes in family contacts by staff over the period of the training, whereas any intention to change work pattern may take weeks or months to be evident

b)      rely on honesty of self reports and the memory of staff about their recent family contacts, rather than on independent measures.

 

For the above reasons, these self report data were supplemented by information from the statewide database of mental health service activity (PRISM).

 

Information about staff contacts with family members was made available by the Department of Human Services from the PRISM database. To preserve confidentiality, identifying information was anonymised prior to the data file being made available. The data file included information about the number and nature of contacts recorded by adult and aged services staff with their clients and family members, both together and separately. The data comprised a one-month period prior to implementation of the FaST training and a one-month period following the training. PRISM data for CAMHS was incomplete and therefore not able to be used. Comparison data for the same months of the previous year were also examined to identify any pre-existing trends in family contacts.

 

The government policy that led to funding of this initiative promotes partnership with families in both planning and delivery of services. Although the content of the training was intended to reflect this aim, the processes of the project were also expected to be inclusive of families. Part of the evaluation therefore focussed on the degree of involvement of family carers in planning and training delivery, and their experiences and recommendations for the future. Participation of consumers is also an established element of government policy for mental health services. Their participation in this project was also covered by the evaluation.

 

Participating carers and consumers were contacted by the research assistant for a structured feedback interview regarding experiences of participation, satisfaction with the project and recommendations for future service development (See Appendix for Questionnaires).

 

 

The Bouverie staff and external trainers who implemented the program are in a unique position to comment on the strategies and effectiveness of various elements of the project. The research assistant conducted structured interviews with each trainer at the conclusion of the project.

 

 

As an externally funded evaluation project conducted by La Trobe University staff, the evaluation required, and obtained, approval from the University's Human Ethics Committee. In addition, all participating mental health service organisations were contacted to seek appropriate approvals for the evaluators to invite participation of training participants and managers in the evaluation. This led to the project obtaining approval via Research Ethics Committees in two organisations and approvals via Quality Assurance Procedures in 25 organisations. Approval for the voluntary participation of training participants from non-government organisations (NGOs) was covered by approval of the area mental health services as auspices for each training round. Organisational approval for the participation of NGO managers was usually taken by the senior manager of the service on receipt of the invitation to participate.

 

 

Three separate but overlapping questionnaires were developed for each stream and completed by participants on arrival at the first training session. The questionnaires covered background information (profession, service type, etc.), self-ratings of skill, knowledge and experience, self-ratings relating to family sensitive practices (objectives of training), self-reported contacts with families, and the appropriate version of the 'Opinions About Family Work Scale'.

 

The 'Post-training Questionnaire' sought feedback about the training program (overall rating, ratings of key elements) as well as repeat ratings of skill, knowledge and experience, self-ratings relating to family sensitive practices (objectives of training), self-reported contacts with families, and the appropriate version of the 'Opinions About Family Work Scale'. This questionnaire was given to all participants who attended the last module of training. In addition, people who were not present for the last session were also sent a questionnaire, although the return rate for this group was lower.

 

 

These were sent to the managers of all training course participants within a two month period after the training had been completed in each respective area. The questionnaire sought information from managers about any feedback staff had given about the training, as well as observations by the manager about how the FaST project may have affected their organisation.

 

 

Anonymous client and family contact data was made available by Mental Health Branch. Analyses sought to determine whether, at a statewide level, the number or proportion of contacts by staff with family members rose between February and September 1998, the period during which most of the training and follow-up interventions occurred.

 

 

Six to eight weeks following completion of training in an area mental health service, the training consultants met with the area co-ordinator (and sometimes other senior staff) to review the FaST project and to discuss any further needs. At each of these meetings, the training consultants brought along a list of the projects initiated by participants from that area mental health service. The current status or outcome of these projects was recorded and compiled for evaluation.

 

 

All carers and consumers involved with the FaST project were invited to participate in a structured feedback interview covering experiences of participation, satisfaction with the project and recommendations for future service development.

 

 

Informal interviews were conducted individually with the trainers who implemented the Get Together FaST project.

 

Unless otherwise stated, all relevant tests are two-tailed and a p value of less than 0.05 is accepted as significant. Where multiple independent statistical tests occur, there is a danger that some will achieve statistical significance by chance. To guard against making Type I errors in these situations, a Bonferroni adjustment was made to the acceptable p value by dividing the p value by the number of independent significance tests. This adjusted threshold level of statistical significance is stated when required.

 

When conducting ANOVAs, subgroups of staff were not included in analyses if the number of participants fell below eight.

 

The full report is available from the Bouverie Centre. For further information, please contact Brendan O'Hanlon either on 0385 5100 or by email at b.o'hanlon@latrobe.edu.au.