‘What I want to know is, will it make a difference to my son?’ is what James’ mother asked me when I was introducing the Brothers of Charity evaluation system called ‘How Are We Doing?’ She went on to explain that her son had a profound learning disability and she was interested in a system that would come up with practical suggestions that would help her son. One action plan that arose from the evaluation of her son’s service was a plan to have regular massage sessions–something that James loves.
The ‘How Are We Doing?’ system was first introduced as a pilot system within the Brothers of Charity Services in 1994. A task group of John Dunne, Roy McConkey and Rosemary O’Connell was set up to design an evaluation system which was suitable for our services. They developed a system that involved service users, parents and staff; a system that was self-evaluative in nature with an outside review built into it. Our system is primarily a developmental evaluation which is owned by the main stakeholders in our services. It also highlights and celebrates achievements and positive elements of the services as well as deficiencies and challenges.
The system involves the following six stages:
The service team draws up a vision statement that states its aspirations and gives a sense of where the service is going. This forms a major criterion by which the service will judge its own work.
Information is gathered on how the service operates. The major stakeholders (service users, their families, co-workers, staff members, and support staff) are consulted. The major achievements and challenges for the service are identified.
Visit of outside reviewers
Two outside reviewers are invited to assist in evaluating the service. They provide an objective view, listening and naming what they see, validating what is good and offering challenges to the service. This provides an independent perspective on the operation of the service.
The review gathering
Service users, their families and staff members come together to review the findings of the evaluation, identify challenges, celebrate achievements, and plan goals and future directions.
The report sets out the main themes of the evaluation, the results of the various information-gathering exercises and the major achievements and challenges. It incorporates an action plan that identifies goals for future action.
The recommendations are reviewed on an ongoing basis and formally by the service a year later. A full evaluation takes place every three years. Other modes of evaluation may take place within this period as we use a mix of tools and approaches.
The system is being gradually introduced into all Brothers of Charity services. The feedback from the services to date has been largely positive. The evaluations highlight the numerous challenges that they face and allow them to prioritise where they need to put their energy. However, we still have a lot to learn and develop in our attempts to review and evaluate our services.
What have we learned since the system has been first introduced?
A formal review of the system was carried out by a firm of auditors, M.K. Brazil & Co., and a number of changes are being introduced to the system.
- We have streamlined the system and, where possible, simplified it. We have developed standardised questionnaires, interview schedules and documentation that can be be adapted and customised for each service.
- The evaluation co-ordinator plays a key role in each region. He or she must both lead and drive and at the same time support the service. Each region must ensure that sufficient time is available to the evaluation co-ordinator to carry out this function. As with all evaluation systems, success depends on implementation.
- Staff have requested that we develop clear standards of good practice to measure themselves against. Our first set of national standards for our adult services is being piloted at present.
- Implementation of the action plans is a major, ongoing challenge. This is the litmus test of whether the system works or not. In most evaluations, half of the action plans depend on additional resources, while the other half are within the capability of the service to enact. The ongoing success of the system depends on the degree of commitment from both senior management and frontline staff.
To date ‘How Are We Doing?’ has given us opportunities to take stock and be proud of our achievements and to listen and plan for the future. It has highlighted many challenges for our services. Now we must ensure that ‘How Are We Doing?’ becomes part of an ongoing cycle and an integral part of our services.