- Volume 63 , Number 4
- Page: 552–8
Use of facility-based assessment in the evaluation of a comprehensive Leprosy training program in Nepal
ABSTRACT
A facility-based assessment (FBA) was done to evaluate a comprehensive leprosy training program in Nepal. The training course was developed to prepare the Basic Health Services staff for integrated leprosy work. FBA is a coordinated set of data collection activities designed to determine the extent to which patients are properly diagnosed, treated, and cared for in the treatment facility. During the present evaluation, the data collection activities included: observation of health worker performance, exit interviews with leprosy patients, interviews with health workers, inventory of essential equipment and supplies, and collection of routine statistical data. The objectives of the training course were used as guidelines for the evaluation. Surveys and observation visits were done repeatedly and compared with a (untrained) control group. Different actors involved in leprosy care were used as informants, and different data collection methods were used which enabled cross-checking of the information. Part of the data collection activities was already routinely carried out. FBA proved to be a very useful and effective tool for the évaluation of a leprosy training program. Those parts of the training which need extra attention during the course itself as well as during refresher training and supervision visits became obvious.RÉSUMÉ
On a réalisé une évaluation sur base des services (EBS) pour évaluer un programme global de formation pour la lèpre au Népal. Le cours de formation était développé pour préparer le personnel des services de santé de base à un travail lèpre intégré. Ce type d'évaluation est un ensemble coordonné d'activités de récolte de données dont le but est de déterminer dans quelle mesure les patients sont correctement diagnostiqués, traités et pris en charge dans le centre de traitement. Pour la présente évaluation, les activités de récolte de données comprenaient : observation des performances des travailleurs de santé, inventaire de l'équipement essentiel et des approvisionnements, et collecte des données statistiques de routine. Les objectifs du cours de formation ont été utilisés comme guides pour l'évaluation. Des enquêtes et des visites d'observation ont été réalisées de manière répétée et comparées avec celles réalisées pour un groupe témoin (n'ayant pas suivi de formation). Différents acteurs impliqués dans les soins aux lépreux ont été utilisés comme sources d'information et différentes méthodes de collecte de données ont été utilisées qui permettent la vérification croisée des informations. Une partie des activités de collecte des données clait déjà réalisée en routine. Cette évaluation sur base des services s'est avérée être un outil très utile pour l'évaluation d'un programme de formation pour la lèpre. II devint clair quels étaient les modules de la formation qui nécessitaient une attention supplémentaire pendant le cours lui-même ainsi que dans les cours de rafraîchissement et les visites de supervision.RESUMEN
Se hizo una amplia exploración tendiente a evaluar un programa de entrenamiento comprehensivo de la lepra en Nepal. El programa de entrenamiento se realizó con el fin de preparar al personal de los Servicios Básicos de Salud para el trabajo integral en la lepra. La exploración incluyó una serie de actividades planeadas para determinar cómo es que el paciente es diagnosticado, cómo se instituye su tratamiento, y cómo es atendido médicamente. Durante la exploración se observaron las actividades de los trabajadores de la salud, las entrevistas con los pacientes, se hicieron entrevistas a los trabajadores médicos y paramédicos, se hicieron inventarios del equipo esencial y de los insumos médicos, y se revisaron las estadísticas de rutina. Para hacer la evaluación se tomaron en cuenta los objetivos del curso de entrenamiento. Las visitas de exploración y de observación se hicieron de manera repetida y los resultados se compararon con los obtenidos en estudios donde se involucró personal no entrenado. Se recurrió a diferentes informantes y a diferentes métodos de colección de datos; esto hizo posible que la información obtenida pudiera compararse de manera cruzada. Parte de las actividades de colección de datos ya se hacían de manera rutinaria. El estudio exploratorio demostró ser una herramienta útil y efectiva para la evaluación de programas de entrenamiento sobre la lepra. Aquellas partes del entrenamiento que necesitaban atención extra fueron identificadas en el curso mismo o durante las visitas posteriores de supervisión.There is an urgent need for research on many issues in health care, particularly the training and motivation of peripheral health workers (H). Much of the attention on health services in developing countries has been directed toward how to expand the services to meet previously uncovered groups. Less attention has been given to the quality of the services already provided (8). Health Services Research provides a means to analyze and interpret the masses of data routinely available and information collected through specially designed studies. It can be used for rational decision making in health management (13).
There are three types of Health Services Research: cost-effectiveness research; operations research and interaction research (14). The first type aims at optimizing efficiency. The second type is relevant, e.g., where the primary health care (PHC) structure is relatively fixed and static and where a vertical project must be adapted to it. The third type studies, e.g., the interaction of the PHC system and a vertical project to determine how both may be modified to achieve efficient integration (14).
In 1987 the government of Nepal decided to provide Integrated Basic Health Services all over the country at one and the same time (12). The existing vertical projects were asked to provide supplies and equipment for field level activities, technical supervision and also to organize a special training program for the integrated districts (10).
The Leprosy Control Programme (LCP) was run as a separate, vertical program until 1987. In the Western and Mid-Western Regions of Nepal, the LCP is run as a joint project by the International Nepal Fellowship (INF) and His Majesty's Government (HMG/N) and is supported by the German Leprosy Relief Association. The set-up of this project, the prerequisites for successful integration of the LCP into the Basic Health Services (BHS) and the extent to which these have been met in Nepal, have been described before (12).
In order to prepare the BHS staff for integrated leprosy work, a comprehensive leprosy training course was developed by INF and the HMG/N Leprosy Control Section. This 6-day course is given district-wise to groups of maximal 12 BHS staff of the same level. The objectives of this ongoing course are to enable the BHS staff to: a) make a clinical leprosy diagnosis; b) classify patients for the right treatment category; c) make arrangements to collect skin smears as appropriate; d) treat patients with multidrug therapy (MDT) and arrange for defaulter training; e) do basic nerve function assessment; f) do basic recording on the Leprosy Patient Card, the Daily Register and the Clinical Register; g) recognise leprosy complications and refer them to a referral center or district leprosy supervisor; and h) give health education in leprosy to target groups.
It was felt necessary to evaluate this new and ongoing training program in order to make adaptations in the program for future trainees and to pay attention, during refresher training and supervision visits, to the issues which were not satisfactorily known or carried out by already trained staff.
For the integration of the Leprosy Control Programme into the BHS in Nepal the second type of Health Services Research, operations research, is probably the most relevant since the structure of the health services is more or less fixed.
One of the methods that can be used for the evaluation of a training program is the Facility-Based Assessment (FBA) method (4). This method uses a coordinated set of data collection activities designed to determine the extent to which patients are properly diagnosed, treated and cared for in the treatment facility. A single application of the FBA method can produce useful datafor decision making, while repeated surveys can be useful for program evaluation. It can be used to measure trends in health worker performance and service quality. However, time and experience are needed to carry outthe different activities.
This paper presents an example of the use of FBA in the evaluation of a leprosy training course (CLT) for Basic Health Services staff in Nepal.
METHODS
The district in which the first CLT was given in 1991 was selected as the pilot district. This district is situated in the flat southern plains of Nepal which have a relatively good infrastructure and better developed health services than the hill and mountain districts. Since there had been no opportunity to do a pre-training questionnaire survey, a neighboring district, also in the plains, was selected as the control district. In both districts the Mobile Team conducted leprosy clinics in some of the Health Posts (HPs). For comparison, a district in the hills without a Mobile Team, was selected and included in the assessment.
For the evaluation of the CLT course the aims and objectives of the training were used as a guideline. The data collection instruments used were: observation of health worker performance; exit interviews with leprosy patients; interviews with health workers; inventory of essential equipment and supplies, and routine statistical data.
Observation of health worker performance. The performance of the health workers was regularly observed by the Regional Leprosy Supervisors during their routine supervision visits to the HP. For this purpose an existing checklist was adjusted. Table 1 shows part of the checklist which was used for the evaluation.
Exit interviews with leprosy patients/interviews with health workers. The opinion of the care as perceived by the leprosy patients was obtained via a questionnaire survey. The patients were interviewed after their visit to the HP. Part of the English translation of the questionnaire is shown in Table 2. Questions with answer categories, based on answers given by patients during extensive pretesting of the questionnaire, were asked as open questions. Depending on the answers given, the interviewers picked out the matching categories. The respondent therefore did not know the expected answers.
The HP staff were interviewed to assess their knowledge and their opinion of the CLT course. Table 3 shows part of this questionnaire in English.
Both interviews were done by two independent interviewers who were introduced to the basics of leprosy and trained in interview techniques. The questionnaires were pre-tested in nonselected districts. According to a predetermined schedule the interviewers visited the HPs first to interview the staff. Thereafter they came back on the leprosy clinic day to interview the patients. As many leprosy patients as possible were interviewed. Also, staff who had not been present during the first visit were interviewed at that time.
The interview survey for leprosy patients and HP staff was done four times. In the pilot district it was held 6 months and 1 year after the completion of the CLT course. In the control district and in the hill district the survey was done before the CLT course was conducted. All of the interviews have been done by the same interviewers and according to the same schedule.
Inventory of essential equipment and supplies. The equipment and supplies necessary for leprosy care were checked during the supervision visits by the Regional Leprosy Supervisors using a checklist. The items checked included stock of antileprosy and antireactional drugs; stock of dressing materials; stock of health education materials; display of health education materials; and storage of items.
Routine statistical data. Routine statistical data can be used to measure trends. Important are registered prevalence, new patients found, disability-, child- and multibacillary proportion among new cases. Data were available from monthly reports, aggregated from patient cards. These data are being monitored over time and are not taken into account in this paper.
The data from the observation checklist and the two interviews have been entered into a microcomputer database and analyzed in Epi-Info, a software package designed for field-based epidemiological datahandling (6).
RESULTS
The interview samples for patients in the four questionnaire surveys ranged from 24% to 38% of the number of patients registered for treatment. The interview sample of staff based in the HPs ranged between 51% and 66%. During the first year of the handover of leprosy care to the HP staff in the pilot district, a total of 21 supervision visits (In an earlier article on this subject 12 only supervision visits held in the same period as the interview survey were included in the analysis.) were made to HPs that were previously run by the Mobile Team. These were compared to five supervision visits to the control district before the CLT had taken place.
The results of the FBA survey are examined in light of the objectives of the CLT course. There was much more involvement of the HP staff in the leprosy care in the pilot district than in the control and hill district. During the supervision visits in the control districts, no HP staff involvement was observed.
Clinical diagnosis. The interviewed staff have valued the subject "clinical diagnosis" as too short and difficult during the CLT. The field-based health workers were not often mentioned by the patients as a source of information for hearing about the leprosy clinic day, although casefinding was mentioned as a field activity by most interviewed field-based staff. Quite a number of the interviewed staff said they had diagnosed new leprosy patients in the previous 3 months, in the pilot as well as in the control districts.
Classification. "Classification" was also regarded as a difficult subject in the CLT. The classification observed in the pilot district was said to be competently carried out. Almost half of the staff in the pilot district who treat patients said they had classified patients for a treatment category. In the control districts very few staff said they had classified patients.
Skin smears. Only staff in the pilot district are involved in taking skin smears. Half of the observed skin-smear taking was not done competently.
Treatment and defaulter tracing. The patients seemed to be satisfied with the care they get at the HP, according to each of the interview surveys. When asked in an indirect way, however, more patients in the pilot district than in the control district seemed to be satisfied with the care of the HP staff (12). The improvements wished by the patients usually concerned facilities not related to leprosy care, such as medicines for other ailments and vitamins. There were quite a few patients (25%) in the HPs previously covered by the Mobile Team who said the physical examination used to be much better.
Patients seemed to understand the side effects caused by the antileprosy drugs, although the staff did not often mention that they tell their patients about it. Most patients do not know for how long they have to take medicines. The interviewed staff confirmed this finding: they hardly mentioned the duration of the treatment.
During observation visits in the pilot district antileprosy drugs were available but antireactional drugs were not always present, not even with the district leprosy supervisors. During the visits later that year all drugs were available. Dressing materials were not adequate during most of the visits. Unfortunately, data on the availability of drugs and equipment in the control district were not properly registered during the observation visits. Almost all patients in all surveys said they knew the dose of drugs they have to take at home and mentioned the right dose.
Defaulter tracing was not often mentioned as an activity of the field-based health workers, despite the fact that irregular patients were a problem in all health posts.
Basic nerve function assessment. Basic nerve function assessment was done by most of the staff observed in the pilot district. In most cases it was done competently, although it was mentioned as a difficult aspect during CLT.
Basic recording. During observations most recording was done properly, except for smear results and late patients.
Complications and referral. During almost half of the observation visits, treatment for complications was not properly done in the pilot district. Referrals were observed to be properly done in most cases.
Health education. Specific health education to leprosy patients was observed during most visits. As mentioned before, information regarding treatment was not often given according to the staff themselves.
DISCUSSION
Facility-level assessment methods have been used in different programs using different sets of data collection methods (1-4, 7-9, 11).
The FBA method according to Bryce (4) is a useful tool for evaluation of health worker performance (4, 5). If done repeatedly, it can be useful for program evaluation to compare over time as well as with a control group. Evaluating health worker performance is often mentioned as the objective of facility-based methods (3, 8). Assessment methods that are facility based are also useful in assisting program managers to identify problems that can be addressed by personnel training, effective supervision, better distribution systems, and health education (8). The aspects can be verified from different sources (patients, health care providers and supervisors) and through different data collection instruments, so a number of cross checks can be built in to ensure validity and reliability of the information obtained (2, 7).
Observation- and survey-based activities, carried out mainly in health care facilities, can also provide information for assessing staff performance after extensive training (1). In an assessment of training on acute respiratory infections, the use of different data collection methods was found to be effective (15).
The data available from the observation visits and interviews reveal some interesting issues about the CLT course. Most aspects of care were observed to be properly carried out, although dressing and skinsmear taking were less satisfactorily performed. Diagnosis, classification and nerve function assessment were often mentioned as subjects which were too short, difficult or puzzling during the CLT course. From the interviews with patients it became apparent that they considered the care from the HP staff better after the training than before. More often drugs were adequately available in the HPs where the staff had been trained.
The FBA method uses different actors involved in leprosy care to get information: the patients, the health staff and the supervisors. This ensures that the issues arising are looked at from different perspectives and are cross-checked as much as possible. The different techniques of data collection also make it possible to cross-check the information. The responses of the health workers can be compared with the observations of the supervisors. The interviews reveal the knowledge of the health workers; whereas the actual practices and health worker performance are checked during the observations. Although the questionnaire survey required extra manpower and resources, for the observation visits no extra resources were needed. Also, processing of the data from the supervision checklist was already done by the supervisors themselves before this evaluation. The surveys and observation visits were done repeatedly and compared with a control group. Because of this, it was very useful as a tool for the evaluation of the course, which is ongoing and also includes refresher courses. The FBA method has proved to be applicable in the evaluation of a leprosy training course. The parts of the training which need extra attention during the course itself, as well as during refresher training and supervision visits, became obvious.
REFERENCES
1. ANKER, M., GUIDOTTI, R. J., ORZESZYNA, S., SAPIRIE, S. A. and THURIAUX, M. C. Rapid evaluation methods (REM) of health services performance: methodological observations. Bull. WHO 71(1993)15-21.
2. ASKEW, I., TAPSOBA, P., QUEDRAOGO, Y., VIADRO, C, BAKOUAN, D. and SEGBO, P. Quality of care in family planning programmes: a rapid assessment in Burkina Faso. Health Policy Plan. 8(1993)19-32.
3. BJÖRCK, M., JOHANSSON, R. and KANJI, N. Improving the quality of primary care services in Angola. Health Policy Plan. 7(1992)290-295.
4. BRYCE, J., TOOLE, M. J., WALDMAN, R. J. and VOIGT, A. Assessing the quality of facility based child survival services. Health Policy Plan. 7(1992)155-163.
5. CHASE, E. and CARR-HILL, R. The dangers of managerial perversion: quality assurance in primary health care. Health Policy Plan. 9(1994)267-278.
6. DEAN, A. G., DEAN, J. A., BURTON, A. H. and DICKER, R. C. Epi Info version 5. Atlanta: Centers for Disease Control, 1990.
7. ENGELKES, E. Process evaluation in a Colombian primary health care programme. Health Policy Plan. 5(1990)327-335.
8. FORSBERG, B. C, BARROS, F. C. and VICTORA, C. G. Developing countries need more quality assurance: how health facility surveys can contribute. Health Policy Plan. 7(1992)193-196.
9. GARNER, P., THOMASON, J. and DONALDSON, D. Quality assessment of health facilities in rural Papua New Guinea. Health Policy Plan. 5(1990)49-59.
10. MATHEMA, P. Primary Health Care in Nepal . Kathmandu: V. R. Mathema, 1987.
11. MAYNARD-TUCKER, G. Indigenous perceptions and quality of care of family planning services in Haiti. Health Policy Plan. 9(1994)306-317.
12. ROOS, B. R., VA N BRAKEL, W. H. and CHAURASIA, A. K. Integration of leprosy control into basic health services; an example from Nepal. Int. J. Lepr. 63(1995)422-429.
13. WARDEKAR, R. V. Integration of leprosy work into general medical service. Int. J. Lepr. 37(1969)308-309.
14. WORLD HEALTH ORGANIZATION. Tuberculosis Control as an Integral Part of Primary Health Care. Geneva: World Health Organization, 1988.
15. ZEITZ. P. S., HARRISON, L. H., LOPEZ, M. and CORNALE, G. Community health worker competency in managing acute respiratory infections of childhood in Bolivia. Bull. PAHO 27(1993)109-119.
1. M.Sc, Research Asociate; International Nepal Fellowship, Leprosy Control Project, Pokhara, Nepal.
2. M.D., M.Sc, Ph.D., Director, International Nepal Fellowship, Leprosy Control Project, Pokhara, Nepal.
Reprint requests to Ms. B. R. Roos, % P. van Hoist, de Patrijs 27, 2761 SX Zevenhuizen, The Netherlands.
Received for publication on 6 June 1995;
Accepted for publication on 5 September 1995.