Theoretical and Methodological Aspects of the External Evaluation of the Improvement , Access and Quality of Centers for Dental Specialties Program

Objective: To introduce the main theoretical and methodological aspects of the external evaluation of the 1st cycle National Program for Quality Evaluation (AEPMAQ/CEO). Material and Methods: This is an evaluative and quantitative research carried out in all Centers for Dental Specialties CEO of Brazil, and macro-geographical regions were taken into account for analysis. The general AE-PMAQ/CEO coordination was from the Collaborative Centre for Oral Health Surveillance of the Ministry of Health Federal University of Pernambuco and Department for Primary Health Care of the Ministry of Health. A collaborative network was established to offer a scientific and technical support for the Project among different higher education institutions around the country, state oral health coordination and quality researchers of AE-PMAQ/CEO. Data collection was carried out through interviews with managers, dentists and users. In addition, researchers used an observation template to check for infrastructure and a questionnaire to register previously discussed quality standards. Conclusion: The external evaluation of the 1st cycle National Program for Quality Evaluation offered data to demonstrate and give recognition to CEO services and municipalities’ managers to assure quality for specialized dental care.


Introduction
The Brazilian National Oral Health Policy (PNSB), called Brasil Sorridente, received an exponential increase in the use of financial resources from the Ministry of Health, proven in the following periods: more than R$ 1.2 billion was invested between 2003 and 2006, reaching more than R$ 2.7 billion from 2007 to 2010, being therefore considered a priority for the federal government.There was evidence of improved access to and reduction of dental problem indicators, expansion of basic care based on the Family Health Strategy, feasibility of adding fluoride to public water treatment plants, and increase in the supply and access to oral health secondary care via Centers for Dental Specialties (CEO) [1].
CEOs represented an innovation in the National Oral Health Policy developed by Ministry of Health up to that time, being the focus of some official and academic studies.A panorama of the main researches developed has shown advances such as: expansion of the offer, coverage and use of these services [2][3][4][5][6][7][8], and high degree of satisfaction of users [9,10].However, the achievement of production goals, as evaluated by the CEO's performance and related factors, persists as a challenge [11][12][13][14].
Studies on the process and organization of services, need for training and human resources required for the management of these services, as well as on the observation of professionals with little commitment to this strategy have been conducted [15][16][17].In addition, some studies aimed at the maturation of what would become an ideal interface model between Primary and Secondary Oral Health Care in the Brazilian case have also been performed [18][19][20][21].
In the last decades, initiatives have been growing in Brazil to institutionalize health assessment, as a result of the intention to give rationale to sectoral interventions, although this activity is still in an incipient way, little incorporated into practices, almost always having a prescriptive and bureaucratic character [22].
The evaluation of actions and services is a permanent challenge for administrators and health authorities.An evaluative process in health should be based on basic premises inherent to the social context and on the Brazilian health system, which are: the understanding of health as a right; the recognition that oral health is part of general health; the understanding that the process of generating information should serve for action, therefore for the improvement of the decisionmaking process; and, finally, the search for evaluation to improve the quality of services and oral health programs [23].Evaluation and 4th phase -Re-contractualisation.The aim of this article will be to present the main theoretical and methodological aspects of the External Evaluation of the 1st Cycle Program for the Improvement of Access and Quality of Centers for Dental Specialist (AE-PMAQ / CEO) [24].

Theoretical Aspects
The concept of evaluation of health services and programs should consider health actions not only as an intervention, but in a broader view, as a field of social practices, the evaluation being the judgment of value on the field or any of its components in order to assist in decision making.These judgments may be results of the application of criteria and norms (normative evaluation) or be elaborated based on scientific procedure (evaluative research).Evaluation can be external if conducted by a team that is not part of the organization; or internal, if performed by the organization itself [8,23,[25][26].
The theoretical bases used in the research go back to the systemic model of Donabedian: structure-process-result, and can be considered a quality evaluation.Conceptually, quality will always be a social construction, produced based on the references of the subjects involved -who attribute meaning to their experiences, privileging or excluding certain aspects according to a hierarchy of preferences.Thus, it will always be a great challenge to approach the concept of quality in health, considering the plurality of its dimensions (political, economic, social, and technological) and the subjects involved in its construction (individuals, communities, groups, managers, users and professionals) [27][28].In addition, as specific objectives, it was also sought to analyze the evaluation process based on external evaluators' reports, their field diaries and the state coordinators' report; observe and verify the CEO's infrastructure; identify compliance with quality standards for the PMAQ / CEO process component; verify the existence of documents proving the quality standards identified in interviews; know the users' perception and satisfaction regarding the CEO access and use; and, evaluate the quality components in relation to the PMAQ / CEO structure, process and result.

Material and Methods
For the operation of AE-PMAQ / CEO, a collaborative network was created as a technical and scientific reference for discussion, implementation and execution of this study, given that the development of actions relied on the participation of several Teaching and Research Institutions.The construction of this collaborative network recognized CEO as an important initiative of the National Oral Health Policy, as this research was able to provide subsidies for both the academic community and for the public service, particularly for managers, professionals and CEO users in the planning, management and evaluation for scientific evidence-based oral health decision-making.
Figure 1 shows the operational flowchart of the research, demonstrating how the relationship between the research team and management and the CEO team is established.
The description of the methodological aspects is divided into 2 stages: 1) Quality evaluation of the Brazilian CEOS and 2) Analysis of the evaluation process of Brazilian CEOs.

Quality Evaluation of Brazilian CEOS
An evaluative study was carried out in which the evaluator's position was external to intervention, with quantitative, observational and cross-sectional character, which included he following dimensions: I -Certification of CEO performance and municipal administrations participating in PMAQ / CEO; II -Evaluation of the access to CEO: consisting of an evaluation process that contemplates the evaluation of the network of local health units, with on-site verification of the quality standards associated to PMAQ / CEO [30].
This verification required an increase in the logistics capacity to carry out, in a short period of time, the process of visiting CEOs throughout the country, since it was associated with the identification of the infrastructure conditions, work process, degree of user satisfaction and use of services.

Figure 1. Operational Flowchart of PMAQ / CEO External Evaluation
As an inclusion criterion to AE-PMAQ / CEO, it was expected that all health units, CEO, would participate in the PMAQ / CEO external evaluation.Therefore; for the 1st cycle PMAQ / CEO, CENSO was performed to CEOs implanted in the country, totaling 932 evaluated services.
Municipal health units that until the date established for evaluation were still undergoing reforms and CEOs where the access of quality assessors were not allowed after three attempts to schedule the visit were excluded from the AE-PMAQ / CEO.
Prior to the external evaluation itself, pre-field operating stages were described as follows:  2 shows the distribution of quality evaluators according to states of the federation.AE-PMAQ / CEO had 73 evaluators for the whole national territory.Manuals were produced by PMAQ / CEO to operate the program, highlighting the Instructional Manual for the Center for Dental Specialties [24], which detailed its operational stages; the Self-Assessment manual for Quality Improvement (AMAQ / CEO), which brought the self-assessment forms and worksheets to intervention plans [29]; and the External Evaluation Tool for the Center for Dental Specialties, containing the details of the evaluation modules for the verification of access and quality standards [30].
The PMAQ / CEO External Evaluation consisted of the collection of information through on-site visits conducted by the AE-PMAQ / CEO quality assessors at health units that met the inclusion requirements.The External Evaluation Instrument [30] was organized into three modules, according to the method of information collection: • Module I -Observation in the CEO: the aim was to evaluate the conditions of the structure, equipment, instruments and inputs.
• Module II -Interview with the CEO Manager and a Dentist of any specialty: aimed at obtaining information about work processes, organization of the service and care for the users.
• Module III -Interview with CEO users: aimed at verifying users' satisfaction and perception regarding specialized oral health services regarding access and use of the service.

Data Collection
All data collection was performed through Tablets, which were programmed based on questionnaires validated by the research group.These questionnaires comprised the evaluation modules of the Instrument External Evaluation.The use of tablets ensured the quality of information by eliminating the need for double typing, and using filters to secondary responses, making data collection easier.Another form of data validation was the presentation of supporting documents to prove answers, since some questions were about normative aspects and legal requirements for the operation of CEOs.

Data Analysis
The obtained data were analyzed from a descriptive statistical analysis, with presentation of frequency distributions of variables; and, whenever appropriate, measures of central tendency and dispersion were calculated by summarizing the results in tables and charts.Statistical analyses were conducted using the following programs: Tab for Windows -TABWIN; Microsoft Office Excel; and Statistical Package for Social Sciences (SPSS) version 17.

Analysis of the Evaluation Process of Brazilian CEOs
This was a qualitative, exploratory and descriptive study, which promoted the analysis of the perceptions of quality assessors on the execution process of AE-PMAQ / CEO, and was operated using two strategies: 1) Use of research field diary data and 2) Performance of focal groups.
The inclusion of the external evaluator's view and perception in the process is justified, as unlike PMAQ / AB, in the PMAQ / CEO, all evaluators were dentists, registered with their respective class council.
In PMAQ-CEO, after specific training, external evaluators were instructed to use a field diary at the end of each visit.This instrument allowed the recording of factors that the objective instrument was not able to measure, but in addition, the diary recorded the perceptions and impressions of evaluators in the field, and full of subjectivity, the diaries revealed the surprises, anguishes, joys, frustrations and learning experiences of the external evaluator, that when arriving at the service, evaluated its physical structure, talked with managers and professionals and, on hearing users, could perceive the reality under his perspective.
For the focal groups, the study was voluntary and occurred in places and states interested in developing it, provided that researchers and quality evaluators agreed and signed the ICF (Table 3).For data collection, voice recorders were used and the contents of speeches were transcribed in full, typed in a WORD ® program of the Office ® package and kept under strict confidentiality in a computer with a password of each institution belonging to the multicentric research group, keeping secrecy and anonymity of participants in both official reports and scientific publications.
Considering the current situation, guided by the initiative of the Department of Primary Care -Ministry of Health (DAB / SAS / MS) focused on Primary Care qualification, strengthening the need to expand and support actions already implemented under the National Program for Improvement of Primary Care Access and Quality (PMAQ / AB), the National Policy on Oral Health through Administrative Rule 261 / GM / MS, of February 21, 2013, implemented the Program for the Improvement of Access and Quality of Centers for Dental Specialist (PMAQ / CEO) and the Financial Incentive (PMAQ / CEO), called the Quality Component of Specialized Oral Health Care.The Program for the Improvement of Access and Quality of Centers for Dental Specialist was organized into four phases that complement each other and form a continuous cycle, established in: 1st phase -Adherence and Contractualisation; 2nd phase -Development; 3rd phase -External Program for the Improvement of Access and Quality of Centers for Dental Specialist [29] consisted of the collection of information to analyze the conditions of access and quality of Centers for Dental Specialist (CEOs) participating in the Program.It sought to recognize and value the efforts and results of CEOs and the State, Municipal and Federal District managers in the qualification of Specialized Oral Health Care.The process of External Evaluation also allowed: • Reinforcing care, management and education practices that contribute to the permanent improvement of the Specialized Oral Health Care offered to the population; • Strengthening actions and strategies of SUS management that qualify working conditions and relations and that seek to support the development of the CEO work process; • Subsidizing the re-contractualization of CEOs in a singular way, respecting their potentialities • Considering the evaluation of users and strengthening their participation in the permanent qualification effort of SUS; • Revising in scale and depth the realities and singularities of CEOs in Brazil, recording the weaknesses and potentialities of each unit, contributing to planning and constructing actions to improve services; and • Developing strategies appropriate to differences of territories, promoting greater equity in the investments of the Federal, State, Municipal and Federal District Governments.It was sought as a general objective of the research to verify in loco a set of quality standards of structure and work process of Centers for Dental Specialties (CEO), within the scope of PMAQ / CEO, aiming to subsidize the process of quality certification and decision making in the definition of quality parameters for improvement and expansion of care and prevention actions throughout the national territory.
All the actions were coordinated by AE-PMAQ / CEO, representatives of the Collaborating Center in Oral Health Surveillance of the Ministry of Health -Federal University of Pernambuco (CECOL / MS / UFPE) and the General Oral Health Coordination / Department of Primary Care (CGSB / DAB), with the other institutions participating, which coordinated the research in the following macro regions of Brazil: • Federal University of Amazonas -Northern Macroregion • Federal University of Minas Gerais -Mid-western and Minas Gerais Macroregions • Federal University of Paraíba -Northeastern Macroregion • University of São Paulo (School of Dentistry) -Southeastern Macroregion • Conceição Hospital Group / Porto Alegre -Southern Macroregion The representatives of each institution and the general coordination of the study constituted the research manager group (GG-AE-PMAQ / CEO), however, to better operationalize the actions, state coordinators nominated by GG-AE-PMAQ / CEO and evaluators of AE-PMAQ / CEO were also aggregated.The network of state coordinators was composed of professors and / or researchers with notorious knowledge of the collective oral health area linked to the universities that are described below:

1 ) 2 )
State seminars for the presentation of the PMAQ / CEO External Evaluation results: the GG-PMAQ / CEO articulated with the state health secretariats seminar presenting the operational research model itself, with the participation of municipal representatives of health units to be evaluated; Selection and qualification of quality evaluators: the selection was made by the GG-PMAQ / CEO, based on the curricular analysis of candidates enrolled via digital platform.The training of selected ones was managed by the macroregional coordination and state coordinators of PMAQ / CEO, occurred in three national poles, namely: São Paulo (macroregions: Southeastern and Southern), Brasilia (macroregions: Northern, and Minas Gerais), and Recife (Northeastern macroregion, respectively in the spaces of the School of Dentistry of São Paulo (FO-USP), Oswaldo Cruz Foundation of Brasília (Fiocruz Brasilia) and Department of Health Sciences of the Federal University of Pernambuco./ CEO was responsible for the selection and qualification of CEO quality evaluators.Table