A Web-Based Tool for Monitoring and Evaluating Health Care Services : An Analysis of Centers for Dental Specialties Webpage

Objective: To evaluate the quality of Centers for Dental Specialties (CEO) using an electronic tool the CEO webpage. Material and Methods: Evaluative research was carried out through the use of a web-based tool, which has two modules with forms for the evaluation of quality of CEO components to Managers and Professionals. The tool generates classification scores and recommendation letters according to the score obtained. Satisfactory classification was used for scores equal to or above 7.0 and unsatisfactory for scores lower than 7.0. Representatives of manager and professionals of CEOs who attended the invitation of the research were qualified to use the CEO webpage. Portal data were analyzed in a descriptive way and the average scores were tested according to service and context variables. Results: Thirty-eight health facilities were evaluated. The average score for CEOs was rated as satisfactory for both the Manager and Professional modules. However, when quality components were evaluated, there was higher concentration of unsatisfactory scores to those related to the work process, particularly to the quality components of Personnel Management; Social control and Financing; and, Organizational Criteria. Conclusion: The CEO webpage, for instantly generating evaluation and recommendations for change, is easy to handle and lacks minimal technological resources (computer with access to the internet network), has become a tool for information management that allows immediate decision making. In addition, they can make a major contribution to planning / management support in identifying critical aspects of the service that impair quality, with strong potential to serve as supplementary institutional support to PMAQ / CEO. Initiatives such as the CEO webpage should be encouraged and disseminated for use within Unified Health System.


Introduction
Centers for Dental Specialties (CEO) are a strategic action of the National Oral Health Policy, Brasil Sorridente, in the Unified Health System (SUS) to expand the offer of specialized dental actions with a view to ensuring the integrality of oral health care.Since its implementation, official and academic studies on CEOs have been carried out [1].
An overview of the main researches developed has shown advances such as: expansion of the offer, coverage and use of these services [2][3][4][5][6], as well as the degree of user satisfaction [7,8].
Nonetheless, the achievement of production goals, such as CEO performance evaluation and related factors, persists as challenge [9][10][11][12].The process and organization of services, the need for training and qualification of human resources necessary for management have been studied [13][14][15].In addition, some studies searching for the maturation of what would become an ideal interface model between Primary and Secondary Care in Oral Health in the Brazilian case have also been performed [16][17][18][19].It is also necessary to evaluate their actions in order to guarantee the effectiveness and impact of their interventions.
The evaluation for improving the quality of services appears in this scenario as a resource to be used as it is considered as an exercise of value judgment about something from the joint elaboration among participants in a process involving learning and development of people and organizations, which also includes the perspective of changing elements recognized by them as problematic, being able to assist them in making conscious decisions [20].
The institutionalization of health evaluation can give rationale to sectoral interventions, although this activity is still performed in an incipient way, little incorporated into practices, almost always having a prescriptive and bureaucratic character [21].
Among the actions of health evaluation institutionalization, the Ministry of Health (MS) instituted the Program for Improving Access and Quality of Centers for Dental Specialties (PMAQ / CEO) through the General Oral Health Coordination.The PMAQ / CEO organization (first cycle) was composed of four phases that complement and form a continuous cycle of improving access and quality, namely: 1st phase: Adhesion and Contractualization; 2nd phase: Development; 3rd phase: External Evaluation; 4th phase: Re-contractualization.The first and the last phase are particularly related to normative and legal processes, and the evaluation processes are carried out in the second and third phases [22].
The second phase of the Program, called Development, consists of conducting crosssectional and systematic actions during the entire period of participation of the CEO team and management in the Program.These are translated into self-evaluation processes, characterized by the self-identification of problems within the scope of the service, as well as the implementation of interventions to overcome them.In addition, indicators are monitored based on the production goals established for each CEO specialty, with consequent use of information in order to promote improvement and increase of access and quality of services [22].
The External Evaluation, the third phase of PMAQ / CEO, carried out in partnership with Teaching and / or Research Institutions, consists of the application of instruments for on-site verification of standards of access and quality achieved by professionals and management, where observations related to aspects included in the self-assessment of services are made [22].
At the end of the program cycle, the certification process of CEOs is carried out.The final score of each service is composed of the following dimensions: I -implementation of self-assessment processes, comprising 10% of the final score, regardless of the result achieved; II -verification of the performance achieved for the set of indicators, constituting 30% of the final score; III -verification of evidence for a set of quality standards, comprising 60% of the final score [22].
Due to its characteristics regarding the cross-section aspect of data collection and analysis, and especially for the cycle duration, PMAQ / CEO could be considered punctual and time consuming to give timely results, particularly those originating from the external evaluation.The actions of the Development phase might show opposing results, but it seems that their compliance is merely normative.However, the financial incentive agreed with participating municipalities seems to be the factor that most encourages the participation in the program.
The use of electronic tools could be considered as an alternative to meet the needs of the management and planning for more agile and opportune decision making, since health evaluation based on technology opens ways for proposals to rationalize decisions and practices [23].
In health services, the use of technologies through the application of information technology (IT), can be used to obtain improved quality of care.However, IT is still little used as a management tool and as a way of promoting a closer approximation and integration of small municipalities with national health policies [24].
The aim of this study was to evaluate the quality of CEOs based the use of an electronic tool called CEO website.

Study Design
A quantitative evaluative research was carried out to evaluate the quality of CEOs of Pernambuco in the initial period of use of the CEO website (November and December 2014), which is a by-product of the dissertation titled "Initial evaluation of the use of a web-based electronic tool for planning, management and monitoring of Centers for Dental Specialties (CEO) of Pernambuco, 2014" [25].

Data Collection
The electronic portal for evaluation and monitoring, the CEO website, is a web-based tool that allows the automated processing of data to evaluate the health facility at local level; remote monitoring of services; and assists in the decision-making process for its quality improvement.It follows the universally accepted theoretical model [26,27], focusing on the structure and process components for evaluation.
It contains 2 quality assessment modules, which are called Manager and Professional, because they are answered by these social CEO actors.The Manager module has a form with 7 Quality Components (QC): Service Coverage; Service Management; CEO structure; People management; Organizational Criteria; Social Control and Financing; Evaluation and Monitoring of CEO actions.However, for the classification of services, the first quality component was not included.The Professional module has a form with the following quality components: Structure and Process.After completing the modules, answers were saved and sent by the software, and an evaluation with QC scores and the General Average Service Score (NGS) were generated, resulting from the arithmetic mean of the others, obtaining the service classification.
In addition to direct access by Managers and Professionals to their classifications and scores, evaluation can also be performed at municipal level (municipal manager module), state level (state manager module) and national level (GestBucal module), and, for such, the website offers several types of searches: by state, municipality, month, year and directly by CEO.The execution of these analyses and searches is done by users previously registered to the CEO website.
Additional resource was introduced to the CEO website from May 2015, which was the provision of recommendation letter from the ratings obtained.These recommendations are based on the legal-normative framework of CEOs and can assist the information management, planning and the decision-making process.
Participation for this study was voluntary.The number of 54 CEOs implanted in the state of Pernambuco until 2014 was taken as reference according to the National Register of Health Units (CNES).
Schematically, the research was divided into two stages: Implementation and Operationalization of the CEO website.In the implementation phase of the CEO website, a pilot study with duration of 3 months was carried out with the objective of testing the tool, and was attended by a state manager; municipal managers, CEOs managers and professionals from five municipalities in order to establish the decision rules to enable the inclusion of automated data processing as a way to determine the evaluation / classification version of CEOs of operating modules.
Subsequently, 2 events were organized for the implementation of the CEO website.One During workshops, participants were enrolled in the Portal website of health facilities, their operation was presented for inclusion of data and ability to generate scores for monitoring and evaluation of services.Instructional Manual of the CEO website was distributed to CEO management and professionals.Manager and Professional module forms were filled out and, consequently, the CEO's first quality assessment was carried out through CEO website.
In the second stage, CEO website Operation, there was remote monitoring of the evaluations through CEO website, obtaining scores for the Quality Components and General Service Scores through the automated inclusion and data processing, where it was possible to classify CEOs for Manager and Professional Modules.Satisfactory classification was established for evaluations with scores ≥7.0 and unsatisfactory for scores <7.0.The decision for this categorization was based on the understanding that score 7.0 would correspond to the minimum score required to comply with the quality component, as well for being the General Average Score of Services found for CEOs investigated in the state.
This article investigated CEO website data from the initial phase of its operation, referring to the months of November-December / 2014, directly obtained from the CEO website accessed by the tool administrator user (called GestBucal).However, this research was continued with data monthly reported to the manager module and semiannually to the professional module.

Data Analysis
Descriptive and exploratory analyses were performed.In the descriptive phase, the frequency distributions of variables were presented and, when appropriate, the central tendency and dispersion measurements were also calculated, and results were arranged in tables.In the exploratory analytical phase, differences between proportions were analyzed with the Chi-square for Pearson tendencies.

Results
Through the CEO website: Manager module, 38 quality evaluations were obtained, which corresponded to the participation of 38 CEOs, or 70% of implemented CEOs.The characterization of these health facilities investigated showed that 55.3% of CEOs were Type II, 39.5% Type I and 5.3% Type III.Their implementation took an average of 5 years (minimum 1 and maximum 10 years), and when categorized, 55.3% took time ≥ 5 years.
Regarding the coverage of oral health teams in the Family Health Strategy (ESB / ESB), 52.6% of municipalities presented coverage greater than 50% of ESB / ESF.The General Average Service Score for the state was 7.0, with median of 7.1 (minimum of 5.15 and maximum of 8.3), and was classified as Satisfactory.The average scores according to Quality Component, presented in Table 1, show this panorama in detail.The components that deserve more attention for obtaining average unsatisfactory scores were in descending order: People Management (6.12), Organizational Criteria (5.84) and Social Control and Financing (5.11).The highest average score obtained was for Assessment and Monitoring (8.42).3).However, the structural characteristics of services did not seem to influence the results, since for in both analyses, no statistical significance was found.Regarding the relationship with contextual characteristics, it was shown that 55.6% of CEOs classified as satisfactory were in municipalities with lower ESB / ESF coverage (<50%).According to the population size, attention should be given to the municipalities of the 1 st and 2 nd extracts (small cities), since they had 57.1% and 60.0%, respectively, of unsatisfactory CEOs, while larger municipalities had better results.For HDI, 58.3 of CEOs classified as satisfactory were found in municipalities with HDI ≥ 0.7 (Table 4).However, the contextual characteristics also appeared to be unrelated to the observed classifications, demonstrated by the non-significance to the statistical test.
Through the CEO website: Professional module, 24 quality evaluations were obtained, which corresponded to the participation of 24 CEO, lower than that observed for the Manager module.Of the characteristics of these services, 66.7% of CEOs were Type II, followed by Type I (29.2%) and Type III (4.2%).It was observed that they had been implanted for an average of 5 years (minimum of 2 and maximum of 10 years) and, when categorized, half of units were in each category.The General Average Service Score for the state was 7.83 (minimum of 6.0 and maximum of 9.0), and was considered Satisfactory.However, the average score of QC Structure 8.87 (minimum of 6.0 and maximum of 10.0) was better evaluated than the average score of QC Process 6.95 (minimum of 5.0 and maximum of 10.0).

According to the CEO classification according to Quality Components and General Average
Service Score in the professional module, 95.8% of services were classified as satisfactory for both NGS and QC Structure.However, a worrying factor was that verified for QC Process, whose percentage of unsatisfactory establishments was 33.3% (Table 5).The analyses of structural exploratory variables of the health units and contextual of municipalities according to the General Average Service Score in the professional module did not Pesq Bras Odontoped Clin Integr 2018, 18(1):e3385 obtain statistical significance, but it was verified that the only unsatisfactory service was Type II and those with more than 5 years of implantation; of municipality with population size between 50 and 100 thousand inhabitants, with ESB / ESF coverage above 50% and HDI lower than 0.7.

Discussion
The study using the CEO website contributed to the strengthening of the knowledge area in evaluation of health programs and services both by the introduction of technological innovation and by the expansion of the discussion of evaluations on secondary oral health care.
In the Brazilian context, Unified Health System is the subject of questions about the quality of services and their efficiency and effectiveness and to be charged for the use of new technologies, which can thus be considered a great impeller of evaluation [28] and really seems to be ideal to host this type of initiative.
Regarding the use of Information Technology, some considerations must be taken into account: it is a factor that can positively influence the production and management of institutions, and it has advantages such as cost reduction, improved communication and agility, facilitating the development of actions and the productive process itself, in addition to the production of knowledge from information [29].Thus, it is important to invest in technologies that help increase control and improve the quality of health services [30].In addition, the use of automated data collection methods has been advocated for showing to be superior in comparison to manual data collection methods, requiring lower time and cost, providing greater efficiency and effectiveness [24].
In relation to health evaluation, the restriction in its use has been attributed to factors such as limitation of resources and the long time required, without the timely availability of information when the decision is made [20].However, the CEO website can be an evaluation tool in which these limiting factors are minimized, also because it meets the requirements for evaluation of service quality, using quality criteria (quality components in the CEO website) and follows a theoretical model universally accepted that is the systemic model: quality components structure and process [26,27].
The best evaluated QC was Evaluation and Monitoring of actions.This result seems to indicate the use of the evaluation activity by the services, which is a positive aspect, but it is expected that this process can be matured within institutions, leading to greater reflection and dialogue at local level.CEO evaluation actions should include monitoring, analysis, discussion on performance indicators and their use in planning and management, and studies have demonstrated the challenge to overcome the achievement of these normative goals [2,[9][10][11][12].
The structure was also well evaluated by both managers and professionals, and this is a fundamental component of the evaluation of health facilities, particularly in the systemic model [26,27].Consequently, the necessary resources for the proper functioning must be guaranteed and maintained, but a good structure will not necessarily imply a good process or result, so more attention should be given to the quality components linked to the process [1,2,9], which in this research obtained unsatisfactory results, specifically People Management; Organizational Criteria; and Social Control and Financing.
People Management is relevant, as it points out the issues related to employment and professional satisfaction.These aspects can influence the production and supply of quality in the service to the user.However, this is an aspect of low local governance, lacking greater managementlevel decision to make changes.Contractual relationships and the profile of Brazilian CEO's professionals identified a skilled workforce, however, not necessarily corresponding to wage incentives [15].
Regarding QC Organizational Criteria, there seems to be difficulty of access and use to the service, but the reference and counter-reference relation or Primary and Secondary Oral Health Care interface is particularly fragile.A large demand for specialized dental services associated with low supply capacity ends up compromising the establishment of an adequate referencing system and, consequently, also compromising the integrality of care, which are being studied in the understanding of access and use as well as interface aspects, with a view to solving the barriers imposed [1][2][3][4]6,9,[13][14][15][16][17][18][19].
The results on QC Social Control and Financing demonstrate the low participation and listening of the users' needs, contrary to the understanding that the user should be seen as an important social actor of evaluations of programs and services, since his vision can modify health practices, making it most humane, welcoming and resolute.It is also worth noting that the evaluation from users is considered evaluation of the results of health interventions, with instruments validated for this purpose and developed studies [7,8].In spite of financing problems, the difficulty of managing resources received and the insufficient investment of resources stand out.
Only the investment from the federal sphere is not enough, and the failure of all federated entities to comply with their due contribution may lead to a compromise in the quality of services [30].
It is perceived that it is important to obtain better quality from evaluations of the CEO website if investing in a better offer of specialized oral health services that guarantee access; with an adequate profile of managers and professionals, so that they feel professionally valued and satisfied with their work, and that the bond weakness is not verified, and that the workload is also fulfilled; user access and service provision with interdependence, integration and complexity should be facilitated in the basic / specialized care interface; dialogue should be established among social actors and that there should be a specific space for discussion; to discuss issues that improve the work process; and in order to improve financing, and that better counterparts among federative entities should be establish, but above all, the political will to comply with what is determined by law.
As a limiting factor of this study, the small number of participating health units to generate data more representative of quality assessments can be highlighted, mainly in the professional module, despite the fact that 70% of health units were enrolled.The incipient evaluative culture in municipalities and health facilities may also have influenced participation, as well as the difficulty of accessing health technologies (internet network and computers) in some units.
However, despite the study limitations, the proposed objectives were achieved and showed that the CEO website can be considered a tool of daily use, allowing users to visualize and monitor the results of the quality evaluation, to reflect on the recommendations automatically issued by the program and to make decisions in a timely manner with a view to the rapid resolution of problems and the improvement of access and quality of secondary oral health care.
The differences between CEO website and PMAQ-CEO, which is currently the main strategy for the institutional evaluation of CEOs, is that the CEO website provides evaluation and feedback to those evaluated for timely improvement of management and BH practices, based on monthly normative-legal literature.Although the tool is of academic use, it has great potential for incorporation by public health services.
The use of the CEO website could be seen as an institutional support to CEOs, a constitutive activity of the PMAQ / CEO development phase, with a view to improving evaluation processes and organizing the work process, including commitments made between the CEO teams and the different levels of system management.In addition, the regulatory processes of Evaluation and Monitoring, monitoring and disclosure of the CEO results, assumed by the State, can also be tied to the role of the CEO website, given the functions of this electronic tool.

Conclusion
Quality assessments of Centers for Dental Specialties of Pernambuco participating in this research classified, in general, the services as Satisfactory in both manager and professional modules.
This result demonstrates that it was possible to obtain the necessary minimum conditions for the provision of quality service, since score 7.0 is considered the minimum threshold for a service to be considered as satisfactory.However, as the footnote was at the limit, the intention is that this result can be improved so that there is no risk of altering the average score obtained as satisfactory.
For instantly generating evaluation, recommendations for change, being easy to handle and lacking minimal technological resources (computer with access to the Internet network), the CEO website has become a tool for information management that allows immediate decision making.In addition, it can make major contributions to the planning / management support to identify critical aspects of the service, which hinder quality, and have the potential to serve as additional institutional support to PMAQ / CEO.Initiatives such as the CEO Website should be encouraged and disseminated for use within SUS.

Financing
event occurred in Recife and another in Serra Talhada, Pernambuco State.Each one relied upon the accomplishment of a Seminar: The ways of CEOs in Pernambuco and 3 Implementation Workshops, two of them being held in Recife, in the facilities of the information management laboratory in Oral Health (November 14); and one in the municipality of Serra Talhada, in the headquarters of the Regional Health Management (January 15).There was support and collaboration from the State Oral Health Coordination -Department of Health of the state of Pernambuco (SES / PE) and all representatives of municipal managers and professionals of municipalities with CEOs were invited.
Structural variables of health facilities and contexts of municipalities were studied to help in understanding the results.Structural variables were: Type of CEO (I, II and III), established by ordinance and Implantation Time (<5 years and ≥ 5 years), found in the National Registry System of Health Facilities (SCNES).Contextual variables were: Coverage of the Population by Oral Health Teams in the Family Health Strategy (ESB / ESF), which were collected on the website of the National Oral Health Coordination; Municipal population, divided into 4 extracts (<30 thousand inhabitants, 30-50 thousand inhabitants, 50-100 thousand inhabitants and >100 thousand inhabitants) and municipal HDI (less and equal to or equal to 0.7), collected from the site of the Brazilian Institute of Statistical Geography; and the division in Health Macro-regions in Pernambuco -Metropolitana, Agreste, Sertão, Vale do São Francisco and Araripe, which were collected from the Regionalization Master Plan of Pernambuco on 19/09/2011.
For all steps described, 5% significance level was considered and statistical analyses were performed by the Statistical Package for the Social Sciences (SPSS) version 17. Ethical Aspects Participation in the research was voluntary, and all participants signed the Informed Consent Form (TCLE) and a Consent Letter.The research was approved by the Ethics Research Committee of the Department of Health Sciences Center -Federal University of Pernambuco, (Protocol No.. 399.931 -(CAAE: 13381813.8.0000.5208).The project complied with the requirements of Resolution 466 of the National Health Council.
This article is a by-product of the research entitled "Evaluation of effectiveness of a webbased tool for planning, management and monitoring of Centers for Dental Specialties (CEO)" which was funded by the National Council for Scientific and Technological Development (CNPq) -Research in Oral Health (Process 403419 / 2012-3) and Foundation of Support to Science and

Table 2
Organizational Criteria; and Social Control and Financing accounted for 57.9%, 86.8% and 78.9% of CEOs considered as unsatisfactory.For NGS, 52.6% of CEOs were rated as satisfactory.
shows the CEO Classification according to Quality Components and General Average Service Score.It was observed that in the QC assessments Management; Structure; and Evaluation and Monitoring of actions obtained were respectively 84.2%, 71.1% and 84.2% of the health facilities evaluated as satisfactory.On the other hand, QC People Management;

Table 2 . Classification of CEO according to quality components and general service average score.
municipalities was performed according to the CEO's General Average Scores.It was found that 61.9% of Type II CEOs were classified as satisfactory, but 60.0% of Type I CEOs were classified as unsatisfactory.Regarding implantation time, 64.7% of CEOs with time ≥ 5 years were classified as satisfactory (Table

Table 4 . Relation between contextual characteristics of municipalities according to the General Average Classification of Dental Specialties Centers -Manager module.
Regarding the municipal characteristics of the services investigated, they were distributed according to the Health Macro-region as follows: Metropolitan (79.2%);Agreste (16.7%); and Vale do São Francisco and Araripe (4.2%), and a fact that drew attention was that there was no evaluation of CEOs included in the Sertão region.About 62.5% of health facilities were in large municipalities (> 100 thousand inhabitants), but the other distributions for this category were: 50-100 thousand (25.0%); 30-50 thousand (8.3%); and <30,000 (4.2%).Regarding the ESB / ESF coverage, 54.2% of municipalities had coverage ≥ 50%, where 58.3% were in municipalities with HDI below 0.7.