Original Article

Examining the quality of the competency-based evaluation program for dentistry based on the CIPP model: A mixed-method study

Introduction

Continuing program evaluation and identifying its strong and weak points are essential to having an efficient educational system. ¹ Educational program evaluation is an activity aiming to determine the quality and value of educational programs or processes, with the primary goal of making a value judgment about the implemented programs and presenting a model for educational planners and policy-makers to modify and, therefore, promote the quality of programs. ² The evaluation is valid when it is performed based on clear and comprehensive indices. ³ The context, input, process, and product (CIPP) evaluation model is a management-based evaluation model that enables program evaluation during and after implementation in four dimensions of context, input, process, and output. In the dimension of context, the needs, goals, and conditions of achieving the program outputs are evaluated. ⁴ In the dimension of input, all the factors affecting the achievement of educational goals, such as human resources, financial sources, strategies, and the program execution timetable, are evaluated. ⁴ With this dimension, one can examine various solutions to realize educational goals and prevent using unnecessary facilities for achieving these goals. ⁵ In the dimension of process, the program execution flow in the real world and its compatibility with the pre-determined goals and timetable are evaluated. Finally, in the dimension of output, the effectiveness and efficiency of the program in achieving the pre-determined goals are evaluated. ⁴ The results of the evaluation provide authorities and decision-makers with information about continuing or aborting the program. ⁶ In various departments of dentistry, educational programs have been evaluated based on the CIPP model. For instance, in a study by Tabari, the realization of educational goals was evaluated in three departments of operative dentistry, pediatric dentistry, and orthodontics. Based on this model, Babol University of Medical Sciences had realized its educational goals at an optimal level. ⁷ According to Pakdaman et al, the realization of educational goals based on CIPP in the periodontics and oral health department requires revision in two input and process dimensions. ⁸ In the study by Makarem et al in Mashhad Dental School, the educational program of the department of oral health needs revisions in the domain of process based on the CIPP model. ⁹ Based on the evaluation of the dentistry program at the University of Birmingham, the dentistry students had poor capabilities in terms of root canal treatment and were not quite familiar with the equipment and techniques of this discipline. ¹⁰ Henzi et al ¹¹ in the United States also evaluated dentistry educational programs from the learners’ viewpoint in some departments and identified the problematic areas. Student evaluation is an integral part of educational programs. In the Faculty of Dentistry, Tabriz University of Medical Sciences (Iran), modifications have been made to the dentistry evaluation program in line with a reform plan, and competency-based assessment methods have replaced traditional methods that relied on multiple-choice tests and professors’ opinions. The review of the literature showed that the evaluation program of dentistry using a competency-based approach has not been evaluated yet. Therefore, this study aimed to assess the dentistry competency-based evaluation program based on the comprehensive CIPP model to identify the strong and weak points of the program and provide insights for planners to promote the quality of exams.

Methods

The data in the present mixed-methods (quantitative-qualitative) study were collected via a reliable and valid researcher-made questionnaire based on the CIPP model and open-ended questions, with the professors’ voluntary participation Departments of Orthodontics and Prosthodontics, Faculty of Dentistry, Tabriz University of Medical Sciences.

This study was conducted in three stages.

Stage 1 consisted of designing and extracting the evaluation questions of the “implementation of professional competency-based evaluation program based on the CIPP model” in four dimensions of context, input, process, and output. Stage 2 involved a review by an expert panel to determine the content and face validity indices of the instrument. Stage 3, as a pilot study, evaluated the dentistry evaluation program in the two departments of orthodontics and prosthodontics, Faculty of Dentistry, Tabriz University of Medical Sciences, based on reliable and valid researcher-made instruments.

Step 1: designing the items

In this step, the questions were designed based on the CIPP model in the four domains. There were six items in the domain of context, six in input, six in process, and seven in output. The items were scored on a five-point Likert scale (very low, low, moderate, high, and very high). At the end of these items, some open-ended questions asked the participants to recommend methods for facilitating novel learner evaluation programs for the general dentistry program.

Step 2: evaluating the reliability and validity of the instrument

To determine the validity of the questionnaire, its face and content validity was assessed upon a review by the expert panel. As there was no similar questionnaire for this purpose, criterion validity was not examined.

Face validity

Face validity was assessed by two methods. In the quantitative method, to examine the content validity index (CVI), the method introduced by Waltz and Bausell was adopted, and the judgment of each member of the expert panel about each item was checked by three criteria of simplicity, clarity, and relevance on a four-point Likert scale. The minimum acceptable value for CVI, based on the expert panel size (n = 8), was 0.79, and if the CVI of an item was < 0.79, it was modified based on the experts’ opinion. ¹²

To determine the content validity ratio (CVR), each member of the expert panel’s judgment about each item was evaluated on a three-point scale: 1 = It is essential, 2 = It is useful but not essential, 3 = It is not essential. Based on Lawshe’s table and the number of evaluators (n = 8), items whose CVR was > 0.75 were deemed essential. ¹³

To examine face validity, the participants were asked to check the degree of importance of each item on a five-point Likert scale from 1 (not important at all) to five (very important). Only items with scores > 1.5 were deemed as having acceptable face validity.

The qualitative phase included a 90-minute focused group discussion with eight expert academicians, including two medical education experts and six dentistry faculty members. The feedback given on the items was discussed in the session, and some items were modified or deleted.

The reliability of the instrument was checked with Cronbach’s alpha. The finalized questionnaire was completed by 10 dentistry faculty members. Two weeks later, the same questionnaires were completed by the same group, and the internal consistency of the instrument was determined based on Cronbach’s alpha.

Step 3: Piloting the questionnaire

The Departments of Orthodontics and Prosthodontics were selected for the pilot study. With the permission of the Dean of the Faculty, the questionnaires were completed by the faculty members of these departments and analyzed by descriptive statistics (mean and SD) in SPSS 19. Open-ended questions were analyzed by content analysis. In this analysis, sentences with semantic content were highlighted; their semantic content was extracted and coded, and similar codes were subsumed under categories.

Results

The content and face validity of the tools were assessed by CVI, CVR, and the item impact score by seeking assistance from eight faculty members. The reliability of the tools was measured by Cronbach’s alpha. The content and face validity and the reliability of the program evaluation tool for the “implementation of novel clinical skills’ evaluation methods” from the viewpoint of dentistry faculty members based on the CIPP model were estimated and confirmed at CVR = 0.89, CVI = 0.92, and α = 0.76.

Twenty-five faculty members of the departments mentioned above completed the questionnaires. Table 1 presents the demographic data of the participants.

The participants’ level of satisfaction with the context indices in the professional competency-based evaluation program was examined in two domains of “contextual factors related to the educational institution” and “contextual factors related to human resources.” The general level of satisfaction with the context indices was above average (55%). The highest satisfaction level belonged to the “compatibility of the department’s educational atmosphere for implementing novel evaluation methods” and “novel methods’ acceptance by and attraction for the students.” In contrast, the lowest satisfaction level belonged to the “physical compatibility of the department for administering the exams” (Table 2).

The participants’ satisfaction with input indices in the professional competency-based evaluation program was examined in three domains of “physical infrastructure of the faculty,” “human infrastructure,” and “professor empowerment.” The general satisfaction with input indices was below average (48%). The highest degree of satisfaction belonged to the “physical infrastructure of the faculty.” The participants had the highest level of satisfaction with the “quality of the designed evaluation instruments” and the “effect of content familiarity workshops to familiarize the faculty members with competency-based assessment methods,” while they had the lowest level of satisfaction with the “professors’ familiarity and mastery” (Table 3).

The participants’ level of satisfaction with the process indices of the evaluation program was examined in two domains of “design and implementation of performance-based evaluation methods” and “feedbacks on the implementation of performance-based evaluation programs.” The general level of satisfaction with process indices was above average (55%). The highest satisfaction level belonged to “cooperation between faculty members in implementing performance-based assessment methods” and “cooperation in designing clinical skills’ evaluation methods.” The lowest degree of satisfaction belonged to “receiving feedback from students to promote performance-based evaluation methods” (Table 4).

The participants’ satisfaction with the output of implementing performance-based evaluation methods was examined in three domains of “teaching and learning outcomes,” “promotion of professional and academic ethics,” and “implementation of the educational curriculum.” The general satisfaction level of faculty members with the outputs of performance-based evaluation methods was above average (58%). The highest degree of satisfaction belonged to “teaching and learning outcomes,” and more specifically, with “application of performance-based evaluation methods in teaching the practical and clinical aspects of the discipline” and “application of novel evaluation methods in forming learners’ learning experiences.” The lowest level of satisfaction belonged to the “application of performance-based evaluation methods in promoting professional ethics for learners” (Table 5).

The faculty members’ general satisfaction level was the highest with output indices and the lowest with the input indices ().

Close

Figure 1. Faculty academic staff members’ general level of satisfaction with implementing the professional competency-based evaluation program based on CIPP indices.

The analysis of open-ended questions yielded two general categories of “providing the human and physical infrastructure” and “spiritual support and encouragement of educational innovation.”

In “providing the human and physical infrastructure,” two sub-categories of “dentistry professors’ empowerment” and “preparing physical facilities” were obtained.

Dentistry professors’ empowerment

The empowerment of human resources is the driving force for changing the educational system. To bring about a paradigm shift from traditional educational methods to novel evaluation methods, human resources should be prepared for accepting such a change. The university should design professor empowerment courses based on the timetable. A participant from the Department of Prosthodontics said, “It is better to hold short but continuous and practical empowerment courses in the faculty. We don’t need theoretical concepts but workshops and practice. This is the best method for teaching busy learners.”

According to a participant from the Department of Orthodontics, “It’s better to hold a workshop for our discipline on designing checklists and writing scenarios. We attend most workshops with medical professors, where they become the center of attention, and we do not get to see tangible examples.”

Preparing physical facilities

Preparing the inputs of program implementation is critical for program implementation. A change in assessment methods should be made in the faculty’s strategic plans, and its implementation requires a sufficient budget. “We do not have a test center to design standard stations or administer standard exams,” said a participant from the Department of Periodontics. According to a participant from the Department of Orthodontics, “For this number of students, we have few facilities and professors. We need many professors if we want to design Direct Observation of Procedurals Skills (DOPs) test for students, pay attention to the accurate implementation of procedures by the learners during the course, and ensure their authentic learning.”

In the category of “spiritual support and encouragement of educational innovation,” two sub-categories of “promotion of innovative professors” and “valuing educational innovations” were extracted.

Promotion of innovative professors

The promotion system in medical sciences universities is mostly research-based, and educational innovations are not defined in the promotion system or have an insignificant role. Therefore, professors prefer to publish scientific articles in ISI-index journals and get promoted instead of innovating educational activities that are not reflected anywhere and do not lead to promotion.

A participant from the Department of Orthodontics said, “If we want innovation in education, we must change the promotion regulations. Educational and research faculty members are viewed as being the same in Iran. The university’s policy determines the professors’ performance.”

According to a participant from the Department of Periodontics, “The mission of the university should change towards innovation in education, and we should move in this direction. A change in our attitude towards education and evaluation needs a change in the university’s attitudes and regulations.”

Valuing educational innovations

Educational innovations have not found their proper place in medical sciences universities, and professors come up with such innovations merely based on their personal interests. If the university modifies its values towards a student-centered approach and recruits innovative professors, it can motivate the professors to bring about changes and value the services of creative professors instead of regarding them as rebels who cause trouble for other professors.

A participant from the Department of Periodontics commented, “Other professors think novel evaluation and teaching methods are an extra burden, maybe because they receive no feedback from the Dean of the Faculty. If our feedback system is functional, positive feedback will serve as spiritual support, which encourages other professors to cooperate in the implementation of novel methods.”

According to a participant from the Department of Orthodontics, “Sometimes we don’t even receive written positive feedback, so we lose interest over time. The universities think a professor who gives only one final score to learners based on observing their performance over the semester is the same as one who designs DOPs forms and writes scenarios. That is why innovative professors become de-motivated.”

Discussion

This study aimed to evaluate the “dentistry competency-based evaluation program” based on the CIPP model from dentistry faculty academic staff members’ viewpoint. The faculty members of periodontics and orthodontics departments exhibited moderate satisfaction with the professional competency-based evaluation program. To promote the quality of exams, the infrastructure should be prepared in all four domains. In terms of context, the faculty members believed that the university does not provide a supportive context for the implementation of performance-based evaluation methods, and the physical facilities of the department are not suitable. Still, the atmosphere of the department and the novel methods’ acceptance by and attraction for the students were better compared to the other items. Therefore, faculty members and students are ready to accept novel evaluation methods. In terms of input, faculty members held that the educational facilities of the department are not appropriate. They also declared in open-ended questions that the department does not have an examination room or facilities for administering such tests. Similarly, in a study by Rezapour Mirsaleh et al, ¹⁴ professors and students at Ardakan University (Iran) believed that the resources and equipment were not optimal. Based on studies on dental schools in Tehran and Babol University of Medical Sciences, the equipment of periodontics and oral health departments was insufficient. ^8,15 Of course, the fact that these evaluation methods are new might explain the lack of sufficient equipment for administering these exams. Also, professors have not yet mastered the design and implementation of such exams. The participants mentioned in open-ended questions that there is no separate professor empowerment course for dentistry; therefore, they do not become familiar with objective scenarios appropriate for their discipline and test design techniques in the existing educational workshops. They believed that it would be better to hold workshops in the Faculty of Dentistry and between the class times so that they could participate in them. In the study by Makarem et al, ⁹ a major problem in educational inputs of oral health and social dentistry was professors’ insufficient skills and education. Another problem was the students’ lack of awareness. The use of novel evaluation methods without raising the learners’ awareness leads to stress and confusion in their studies. ¹⁶ Compared to written exams and multiple-choice tests, novel evaluation methods are stressful for students. According to Zartman et al, ¹⁷ in dentistry, performance examination increases learners’ hand tremors and changes their voice; in some cases, these have prevented the learner from taking the exam. In a study by Hassell ¹⁸ on rheumatology, the OSCE (Objective structured clinical examination) exam caused excessive stress in learners.Unfamiliarity with these exams can exacerbate this stress. Various studies have shown that the expression of expectations from the learners and the evaluation method at the beginning of each course directs the learners’ method of learning and studying. ¹⁹ In the study by Makarem et al, vague objectives and expectations from learners were a weakness of educational input. ⁹ In the domain of process, the highest satisfaction level was with the cooperation between faculty members in designing and implementing the exams.

The professors are ready for a paradigm shift from traditional to novel evaluation methods, but they need support and encouragement from the faculty. Professors become de-motivated because they do not receive feedback from the faculty and students. They mentioned in open-ended questions that they do not receive any positive feedback from the faculty, and their innovation does not affect their promotion; the promotion regulation is research-based and does not pay enough attention to educational innovations. ²⁰

According to the professors, novel evaluation methods help the learners’ practical learning and the formation of their learning experiences. These exams aim to ensure the learners’ mastery of the capabilities required by the course. ²¹ However, the professors believed that the implementation of these exams was weak in terms of the complete curriculum implementation as it does not provide enough facilities for designing exams based on the capabilities required by the course. Moreover, the professor-to-student ratio is low, and professors cannot evaluate all the capabilities with these methods. In the qualitative study, by examining the educational problems of the school of dentistry, Qazvin University of Medical Sciences, a major challenge was the low professor-to-student ratio in most departments. ²² Another reason for the poor performance of these exams in implementing the curriculum of dentistry could be that these exams are new, and professors are still in the first stages of exam design and adaptation to ensure the acquisition of capabilities. The overall results of the evaluation revealed that preparing the infrastructure, internalizing the use of novel evaluation methods, and supporting and encouraging the professors by giving them educational incentives and promotion can help implement these exams.

Conclusion

The results indicated that, according to the faculty academic staff members, the implementation of competence-based evaluation methods in two departments of Periodontics and Orthodontics has infrastructural problems and is far from the standard administration of these exams. Some modifications, such as preparing the standard setting and facilities for administering the exams, empowering the professors, and defining the status of these exams in the curriculum and their role in ensuring the learners’ capabilities can enhance the quality of practical exams to approach the standards of advanced countries.

Authors’ Contributions

RN, HH, SF, and NB prepared the manuscript. NB was responsible for coordinating the study. RN and SF distributed the questionnaires, coordinated FGDs, and interviewed the participants. RN and HH commented on the coding process and analyzed the questionnaires. HH wrote the first draft of the paper. RN, SF, and NB reviewed the first draft of the paper and improved it. All authors have read and approved the final version of the manuscript.

Acknowledgments

The authors would like to thank Tabriz University of Medical Sciences for supporting and granting this research as a part of an MS thesis in medical education and the dental faculty members who participated in this study.

Funding

This research was supported and granted by the Medical Education Research Center of Tabriz University of Medical Sciences, Tabriz, Iran. The funder had no responsibility for the design of the study, data collection, analysis, and writing of the manuscript.

Competing Interests

No competing interests.

Ethics Approval

This study was approved by the Research Ethics Committee of Tabriz University of Medical Sciences (IR.TBZMED.REC.1397.254). Data collection was carried out after obtaining verbal and signed informed consent from the participants following the Deceleration of Helsinki. The informed consent process provided participants the opportunity to ask questions and consider all options. Participants could refuse to participate or withdraw from the study at any time.

References

1. Ziaee V, Ahmadinejad Z, Morravedji AR. An evaluation on medical students’ satisfaction with clinical education and its effective factors. Med Educ Online 2004; 9(1):4365. doi: 10.3402/meo.v9i.4365 [Crossref]
2. Lippe M, Carter P. Using the CIPP model to assess nursing education program quality and merit. Teach Learn Nurs 2018; 13(1):9-13. doi: 10.1016/j.teln.2017.09.008 [Crossref]
3. Anderson VR, Rich AM, Seymour GJ. Undergraduate dental education in New Zealand: 2007-2009 final-year student feedback on clinical learning environments. N Z Dent J 2011; 107(3):85-90.
4. Stufflebeam DL. The CIPP model for evaluation. In: International Handbook of Educational Evaluation. Springer; 2003. p. 31-62.
5. Bazargan-Harandi A. Educational Evaluation: Concepts, Models and Operational Processes. Tehran: SAMT Publishers; 2018. [Persian].
6. Mohamadi R, Fathabadi J, Yadgarzadeh G, Mirzamohamadi M, Parand K. Quality Assessment in Higher Education: Concepts, Principles, Methods, and Criteria. Sanjesh Organization Publications; 2008. [Persian].
7. Tabari M, Nourali Z, Khafri S, Gharekhani S, Jahanian I. Evaluation of educational programs of pediatrics, orthodontics and restorative departments of Babol Dental School from the perspective of the students based on the CIPP model. Caspian J Dent Res 2016; 5(2):8-16. doi: 10.22088/cjdr.5.2.8.[Persian] [Crossref]
8. Pakdaman A, Soleimani Shayesteh Y, Kharazi Fard MJ, Kabosi R. Evaluation of the achievement of educational objectives of the community oral health and periodontics departments using the CIPP model of evaluation–students’ perspective. J Dent Med 2011; 24(1):20-5.
9. Makarem A, Movahed T, Sarabadani J, Shakeri MT, Asadian Lalimi T, Eslami N. Evaluation of educational status of oral health and community dentistry department at Mashhad Dental School using CIPP evaluation model in 2013. J Mashhad Dent Sch 2014; 38(4):347-62. doi: 10.22038/jmds.2014.3312.[Persian] [Crossref]
10. Stewardson DA, Shortall AC, Lumley PJ. Endodontics and new graduates, Part 2: undergraduate experience and course evaluation. Eur J Prosthodont Restor Dent 2003; 11(1):15-21.
11. Henzi D, Davis E, Jasinevicius R, Hendricson W. In the students’ own words: what are the strengths and weaknesses of the dental school curriculum?. J Dent Educ 2007; 71(5):632-45.
12. Lawshe CH. A quantitative approach to content validity. Pers Psychol 1975; 28(4):563-75. doi: 10.1111/j.1744-6570.1975.tb01393.x [Crossref]
13. Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar AR. Design and implementation content validity study: development of an instrument for measuring patient-centered communication. J Caring Sci 2015; 4(2):165-78. doi: 10.15171/jcs.2015.017 [Crossref]
14. Rezapour Mirsaleh Y, Atri Ardakani SA, Behjati Ardakani F. Evaluating the educational performance of Ardakan University based on CIPP model. Higher Education Letter 2017; 9(36):7-30.
15. Tabari M, Nourali Z, Jahanian I, Khafri S. Evaluation of educational programs in endodontics, periodontics and oral & maxillofacial surgery departments of Babol Dental School from students’ perspective based on CIPP model. Caspian J Dent Res 2018; 7(2):8-15. doi: 10.22088/cjdr.7.2.8.[Persian] [Crossref]
16. Noohi E, Motesadi M, Haghdoost A. Clinical teachers’ viewpoints towards objective structured clinical examination in Kerman University of Medical Sciences. Iran J Med Educ 2008; 8(1):113-20.
17. Zartman RR, McWhorter AG, Seale NS, Boone WJ. Using OSCE-based evaluation: curricular impact over time. J Dent Educ 2002; 66(12):1323-30.
18. Hassell AB. Assessment of specialist registrars in rheumatology: experience of an objective structured clinical examination (OSCE). Rheumatology 2002; 41(11):1323-8. doi: 10.1093/rheumatology/41.11.1323 [Crossref]
19. Blakely AW, Broussard LG. Blueprint for establishing an effective postbaccalaureate medical school pre-entry program for educationally disadvantaged students. Acad Med 2003; 78(5):437-47. doi: 10.1097/00001888-200305000-00004 [Crossref]
20. Tootoonchi M, Yamani N, Taleghani F. Evaluation of administrative performance indicators used for academic promotion: viewpoints of faculty members of Iranian medical universities. J Med Educ Dev 2014; 7(14):20-30.
21. Bindal N, Goodyear H, Bindal T, Wall D. DOPS assessment: a study to evaluate the experience and opinions of trainees and assessors. Med Teach 2013; 35(6):e1230-4. doi: 10.3109/0142159x.2012.746447 [Crossref]
22. Gharaei SH, Kargoza S, Amirchakhmaghi M, Gholami H. Students’ viewpoints of Mashhad Dental School about educational problems a qualitative study. Educ Strategy Med Sci 2015; 8(2):123-30.