Informatica

Real life problems are involved in multiple factors with complex relationships. To deal with these problems, we should apply a realistic point of view which considers the facts that problems need multi-valued assessments instead of binary ones. Education has been playing an important role in governmental and personal dimensions and a fair evaluation of students has traditionally been something of an Achilles heel for the education systems. In recent years, this has led governments to improve education systems and encouraged researchers to explore new educational tools and methods. For example, No Child Left Behind (NCLB) was an U.S. Congress Act to expand public education, which activated many researchers to design decision making frameworks and develop practical education models (Mandinach et al., 2006; Wohlstetter et al., 2001; Wayman, 2005). Later, the rise of social networks led American researchers to apply social network data and data mining in education (Romero and Ventura, 2010; Daly, 2012; Romero and Ventura, 2013).

In the absence of government endorsement or support, academicians have also developed different methods to make valuable contribution toward improving evaluation in education. Using appropriate mathematical modelling and sophisticated methods of artificial intelligence, researchers have conducted many researches from student project evaluation using fuzzy TOPSIS (Pejic et al., 2013) to measuring the real knowledge of examinees through calculating of Amo-Salas et al. (2014) and automated essay evaluation (Zupanc and Bosnic, 2015).

Evaluation methods in education are mainly classified into two main categories: evaluation of lecturers by students and evaluation of students by lecturers. To our knowledge, the latter one is mostly focused and frequently studied by researchers. Among infrequent lecturer evaluation studies, Hristova and Sotirova (2008) used generalized net model to algorithmization of multifactor method to assess teaching quality at universities. Chu (1990) applied a multi-criteria decision making model to grade teachers. The potential of fuzzy logic techniques to evaluate academic performance (Kakani et al., 2016) drew some researchers to develop fuzzy methods for educational evaluation. Considering the framework of Chu’s (1990) study, Othman (2016) established fuzzy rules in the form of If-Then to discriminate lecturers of 5 courses by 35 respondent students. Liu (2015) applied multi attribute decision making method using intuitionistic fuzzy information to measure the effectiveness of teaching in foreign language courses.

To better the evaluation of students by lecturers, Hameed et al. (2016) proposed to exploit fuzzy sets and replace the sharp criteria of traditional evaluation of students with fuzzy ones. Sakthivel et al. (2013) applied fuzzy numbers and fuzzy rules using Mamdani fuzzy decision technique to infer the performance of students. Johanyak and Kovacs (2014) used fuzzy arithmetic operations to evaluate student assignments and developed a software tool to support the user. To evaluate the English academic writing, Chai et al. (2015) developed a peer assessment method which establishes a combination of a Per-C and a fuzzy ranking algorithm that uses fuzzy preference relations. Identifying 20 evaluators/experts from different schools and using 5 linguistic expressions including very poor, poor, average, good, and very good, Salunkhe et al. (2016) evaluated the performance of 237 secondary school students using fuzzy classification based on fuzzy similarity relation. Li et al. (2015) applied preference-based fuzzy numbers and TOPSIS method to assess the development levels of higher vocational education.

Many evaluation methods for classical detailed exam questions have been developed using fuzzy logic. Chang and Sun (1993) proposed a fuzzy assessment method of junior high school students. Chiang and Lin (1994) applied fuzzy theory to teaching assessment. Biswas (1995) developed fuzzy evaluation method (fem) and generalized fuzzy evaluation method (gfem). Law (1996) focused on the precision, i.e. correctness and falseness, of the scores. This method presented a systematic approach to aggregate scores and produced linguistic grading.

Chen and Lee (1999) extended Biswas’s (1995) method and proposed two new methods. These methods used a fuzzy model to calculate each question’s score by linguistic terms, then evaluator achieves a final score for the student. They claimed that their methods were faster and fairer than Biswas’s. Ma and Zhou (2000) proposed a student-centered method to assess students themselves. Students and lecturers should determine a couple of criteria through brainstorming, and then weigh the criteria. Finally, this approach forms an evaluation matrix for each student. Based on the eigenvectors and using fuzzy envelopes, they allocate a letter grade for each student.

Wang and Chen (2006) extended Biswas’s (1995) and Chen and Lee’s (1999) methods. The authors used fuzzy numbers as the degrees of confidence of the evaluator. Then, α-cuts of these fuzzy evaluations and arithmetic operations between α-cuts evaluate the answerscripts of students. Wang and Chen (2008) presented a new evaluation method using type-2 fuzzy sets. They considered the degree of optimism of evaluator and provided a more flexible and intelligent method. Ibrahim and Kim (2009) also considered importance, complexity, and difficulty to evaluate the answerscripts of the students. They developed a fuzzy controller by Mamdani’s max-min inference mechanism and center of gravity defuzzification method to assist the evaluation process. Chen and Wang (2009) applied interval-valued fuzzy sets to evaluate the answerscripts. These intervals are between zero and one and the similarity of interval-valued fuzzy marks and a standard interval-valued fuzzy sets are used for evaluation of students. This method provides more stable evaluations than Biswas’s (1995) method.

Fuzzy rules and fuzzy reasoning methods are widely focused by researchers like Darwish (2016) who applied fuzzy rules to evaluate student performance. Baba et al. (2015) developed a rule-based assessment system based on a fuzzy group decision support system (FGDSS). Using fuzzy numbers and fuzzy rules, Kakani et al. (2016) developed an evaluation method calculating the degree of confidence and the degree of satisfaction of the evaluator which measure the confidence of examiner in assigning the marks and the satisfaction of examiner by given answers. Akbay et al. (2016) created a fuzzy rules-based system to maximize the achievement of secondary school students through finding the optimal sleeping hours and study time. Bai and Chen (2008a) proposed a new evaluation method using three criteria including difficulty, importance, and complexity, to develop fuzzy rules and fuzzy reasoning system. Later, Bai and Chen (2008b) developed a fuzzy rule-based method which automatically constructs grade membership functions. Their method regarded three types of grade membership functions, namely lenient-type grades, normal-type grades, and strict-type grades for students’ evaluation. Chen and Li (2011) extended Ibrahim and Kim’s (2009) method by considering accuracy, time rate (i.e. time consumed by a student to solve a question divided by a predefined maximum time), difficulty, complexity, answer-cost, and importance as fuzzy rules of their models.

The extensions of fuzzy sets and fuzzy prediction techniques are also considered by academicians (Rodriguez et al., 2012, 2014; Xu, 2007; Herrera et al., 2009; Chiclana et al., 2001; Zeng et al., 2016; Cabrerizo et al., 2015; Urena et al., 2015; Yu et al., 2017; Morente-Molinera et al., 2015). In order to increase the quality and consistency of assessment of students’ answer scripts, Hameed et al. (2016) applied interval type-2 fuzzy sets and fuzzy inference system to achieve higher transparency. Parmar and Kumbharana (2015) developed a text pattern recognition method to automatically evaluate multiple choice question (MCQ) with one word answer or fill in blank type question. To predict the performance of students, Arora and Saini (2016) implemented a user-friendly personalized performance monitoring system based on a hybrid fuzzy neural network model using 760 samples. Considering two grouping criteria: (1) the understanding levels of the students and (2) the interest levels of the students, both with respect of the topics of a given course, Yannibelli et al. (2016) proposed a steady-state evolutionary algorithm for building well-balanced teams and enhance students’ performance.

Measuring the knowledge of examinees in MCQ exams is a challenging problem, which affects the evaluation process during the preparation of questions and choices by examiner, selection of the correct choice by examinee, and the scoring by examiner. Classical MCQs cannot generally detect the differences between the knowledge of examinees accurately, e.g. an examinee with medium level of knowledge and an examinee with less than medium level of knowledge can be evaluated as equal examinees through classical MCQs. These tests provide a strict structure for examinees to select one of the choices. This choice would be the correct answer and other choices are considered as false answers. This structure of MCQs has been criticized for encouraging surface learning and unfair evaluation (Hameed, 2016). However, we believe that crisp structure of classical MCQs cannot evaluate the students properly, and fuzzy sets can improve this evaluation. Fuzzy sets have been recently applied to the evaluation of MCQs. Shahbazova and Kosheleva (2014) proposed fuzzy multiple choice quizzes, in which a student explicitly describes his/her degree of confidence in each possible answer. However, the practicability of this fuzzy approach is low since it is based on logarithm and entropy calculations which require many parameters such as integration constants. Fahim and Dehghankar (2014) proposed a fuzzy MCQ considering a degree of correctness for semi-correct choices to award partial knowledge of examinees. Rather than just selecting a particular choice, examinees are supposed to provide test-givers with the information on the reason why they think other choices are distractors. The idea of correctness degree provides fairer evaluation but writing further explanation on answer sheets complicates the evaluation process. Hameed (2016) also developed a fuzzy MCQ evaluation system using linguistic variables, Gaussian membership functions with fixed mean and variable variance or standard deviation for fuzzification of inputs, and Mamdani’s fuzzy inference system. Although this system can automatically and fairly discriminate examinees, the complexity of fuzzifying inputs and running the Fuzzy Logic Toolbox of MATLAB complicates the usage of this system.

Our literature review shows that most of the fuzzy evaluation studies are focused on detailed examination rather than MCQs. Those rarely developed fuzzy MCQ provide poor performance or results mainly because of their complexity and impractical structure. In this study, we present a new evaluation method through MCQs to achieve more accurate and fairer evaluation which can be easily used by examiners. Here, we propose five approaches to measure the performance of examinees in multiple choice exams. Two main approaches are punishing approach (PA) and awarding approach (AA). The third one is a mixed approach (MA) which is the arithmetic mean of the AA and PA (Fahmi and Kahraman, 2015). Classical approach (CA) defuzzifies examinees’ answers and provides defuzzified evaluation. Finally, joint approach (JA) combines above-mentioned approaches and provides the terminal evaluation.

The rest of the paper is organized as follows. In Section 2, a brief background, required definitions of fuzzy sets, and our proposed approaches are presented. Section 3 is devoted to the application of the proposed fuzzy MCQ examination and the simulation of the proposed approaches. In Section 4, the results of the application are discussed. Lastly, the conclusion and future works are provided in Section 5.

Zadeh (1965) proposed fuzzy sets and defined a membership degree for each member of a fuzzy set. This value is not restricted to be zero and one; however, values between zero and one are also considered as degrees of membership. Fuzzy logic enables us to judge fairly by providing values between zero and one as membership degrees of a particular set. Using degrees of membership afford fuzzy sets to deal with uncertainty in a proper manner. Examination involves the uncertainty of examiners and examinees. Fuzzy examination considers the uncertainty of examinees and examiners and evaluates examinees using fuzzy sets. In the following part, required definitions of fuzzy sets are presented.

Definition 1 is used in all the proposed approaches to calculate their total scores, while Definition 2 is only used in the classical approach to defuzzify the evaluations.

There are two main structural differences between classical and proposed MCQs. First, during preparation of questions, examiner should assign an adequate degree of correctness for each choice of questions, i.e. each choice embeds a degree of correctness and the most correct choice accompanies the full score for the question. Second difference is that our proposed MCQ obliges examinees to assign a degree of reliability to all choices of a particular MCQ, where the summation of these reliability degrees must be equal to one. Two main fuzzy approaches, AA and PA, as well as MA, CA, and JA are presented below.

We gave three exams in the Engineering Economics course of the BSc program in Industrial Engineering at the Istanbul Technical University. In the first exam, we applied AA and CA. Then, in the second and third exams, we added PA, MA, and JA. Here, the total scores of the exam and the corresponding ranking of students are presented in Tables 1 and 2, respectively. These tables distinctly represent the differences and similarities between the scores and rankings of different approaches.

Using Eq. (11), the Spearman rank correlation coefficients between any two approaches have been calculated and recorded in Table 3.

Table 3 indicates that there is not a strong correlation between the approaches, which means that each approach has a different point of view to the evaluation of examinees. The largest negative correlation exists between MA and CA while the largest positive correlation is between AA and MA. The least negative correlation is between AA and CA while the least positive correlation is between PA and AA.

We simulate the results of the proposed approaches using randomly generated numbers in MATLAB. First, the matrix of answer key is formed based on the answer key of the exam, i.e. the correctness degrees of the choices. Later, using the matrix of answer key and the uniformly distributed random numbers, we obtained random reliability degrees to form a matrix of answers. As shown in Fig. 1, by doing arithmetic operations on both answer key and answer matrices, random results of AA, PA, CA, MA, and JA in 100 iterations are attained.

In Fig. 1, the simulation of the exam’s results is depicted using randomly generated numbers. As shown in Fig. 1(a), blue graph or punishing results are lesser than red graph or awarding results in almost all the iterations. The simulation of AA, PA, and CA are presented in Fig. 1(b). Obviously, the red graph or the simulations of AA are greater than PA and CA, and black graph or mixed results are mostly between AA and PA. The simulations of AA, PA, MA, and CA are also gathered together in Fig. 1(c). The graphs of MA and CA, i.e. cyan and black graphs are overlapped in most of the iterations. This shows the close results of MA and CA. In Fig. 1(d), green colored graph or joint results are also added. This graph represents that although JA is a weighted arithmetic mean of AA, PA, MA, and CA, as JA simulation by random numbers is generally positioned somewhere at the middle of other approaches. Especially, it is generally lower than CA and higher than PA. This can imply that the JA is neither similar to CA nor like PA. It’s remarkable that JA does not behave like CA, and is able to punish examinees who randomly assign the reliability degrees for the choices.

We applied SPSS software to compare the total scores of different approaches. Statistical analyses are used to check if there is a significant difference among the total scores of punishing, awarding, mixed, classical, and joint approaches. Analysis below are the outputs of the analysis of variance (ANOVA) of the scores, where the assumptions of ANOVA are satisfied in the analysis. In order to obtain the differences between each pair of approaches, LSD post-hoc tests are applied to the one-way ANOVA. Table 4 presents the multiple comparisons of the results of post-hoc tests. In Table 4, I and J represent the numbers of the approaches, which 1, 2, 3, 4, and 5 represent ${\mathit{TS}_{P}}$, ${\mathit{TS}_{A}}$, ${\mathit{TS}_{M}}$, ${\mathit{TS}_{C}}$, and ${\mathit{TS}_{J}}$ respectively.

The multiple comparisons demonstrate significant difference between all the scores of the five approaches, except the differences between PA and CA ($p=0.692$), PA and JA ($p=0.572$), MA and CA ($p=0.099$), and CA and JA ($p=0.337$). Hence, the approaches of exception pairs have equal means. PA and CA have close total scores because a large number of students in the exam assigned equal reliability degree for each choice, e.g. 0.25 for four choices. This decreased the ${\mathit{TS}_{C}}$ of the students and approached the results of CA and PA. Additionally, the reason of equal means of PA and JA is that examinees who randomly assign the reliability degrees decreased the difference between ${\mathit{TS}_{P}}$ and ${\mathit{TS}_{J}}$. The results of other pairs of approaches, i.e. MA and CA, as well as CA and JA can be compared similarly.

The scatter plot of ${\mathit{TS}_{A}}$, ${\mathit{TS}_{P}}$, ${\mathit{TS}_{M}}$, ${\mathit{TS}_{C}}$, and ${\mathit{TS}_{J}}$ provides a deep insight into the total scores of students which is shown in Fig. 2. These total scores show that ${\mathit{TS}_{A}}$ of examinees is almost higher than the total scores of other approaches, and ${\mathit{TS}_{P}}$ is lower than the total scores of other approaches.

In Fig. 2, some students like number 4, 6, 11, and 15 have close scores with variances equal to or less than 10 points fluctuating between 60 and 70, 50 and 70, 40 and 50, and 30 and 40, respectively. These results imply that although they are positioned in different levels of knowledge, close scores of each of them uncover their similar sureness about their knowledge and assigned reliability degrees. However, Students number 10, 12, 22, and 25 have gained distinct scores in different approaches fluctuating by at least 25 points which are between 30 and 80, 30 and 70, 20 and 50, and 55 to 95.

The result comparison between Student 6 and Student 10 represents that Students 6 has greater joint score than Students 10, although their classical scores are almost equal. The compact set of scores of Student 6 shows that (s)he is neither unnecessarily awarded by AA nor dramatically punished by PA. But distinct scores of Student 10 reveals that (s)he is neither gratuitously awarded by AA nor drastically punished by PA. This means that Student 6 is surer on her/his knowledge in comparison with Student 10.

According to Fig. 2, we can state whoever achieves high total scores is knowledgeable, however, sureness is a different criterion. Knowledgeable examinees do not necessarily achieve close total scores in the five approaches, and examinees are not necessarily knowledgeable and sure simultaneously. Some examinees are sure about their knowledge and they are not trapped in the random assignment of reliability degrees for the choices. On the other hand, some examinees are not sure about their knowledge and assign reliability degrees for the choices in an unsure way. PA catches this type of examinees and leads to a low score in PA, however, AA assigns a high score for these examinees. Regardless of their level of knowledge, they are not sure about their knowledge. Finally, JA considers the knowledge and sureness of examinees at the same time. This approach can reveal any unsure assignment of reliability degree which might be considered as fully correct answer in classical MCQ tests. Consequently, JA is authorized to punish any unsureness of examinee in their choice evaluation process.

If students were to be ranked based on CA, Student 10 would be superior to Student 6 with a tiny margin. However, our proposed approaches, specifically JA which aggregates the scores, uncover the real knowledge of students. Table 2 shows the ranking difference between CA and JA in details. While Student 6 is the 5th and Student 10 is 17th in the ranking of JA, they are respectively 6th and 7th in the ranking of CA. Whilst CA is obviously incapable of evaluating properly, proposed approaches, particularly JA, provide fairer scores and ranking.

In this study, we challenge the fairness of classical MCQs and propose a modified structure for MCQs with novel evaluation method to find fairer results and ranking of students. Proposed method is based on fuzzy logic and consists of five approaches, namely punishing, awarding, mixed, classical, and joint. These approaches are applied to evaluate students through MCQs. Two primary approaches, AA and PA, could perfectly deal with the uncertainty embedded in the examinees’ evaluation of the choices. AA and PA regard the uncertainty of examinees and examiners to properly score the examinees, and then fairly rank them. Close AA, PA, MA, and CA results also represent the sureness of the examinee about their knowledge. While classical MCQs cannot perceive that the questions are answered surely, the proposed JA, which is a combination of other approaches, can provide an intelligent scoring method and reveal both knowledge and sureness of examinees. Accordingly, the ranking of students using this approach is fairer than CA, which can overcome the most important shortcoming of the classical MCQs, i.e. the random selection of the choices.

The main drawback of our proposed evaluation method is that it forces examinees to assign reliability degree which can take more period of their time during exam. It can also confuse examinees how to deal with the new structure of MCQs. So that examinees should be clearly informed about the structure of our MCQs. Despite new evaluation methods like Shahbazova and Kosheleva (2014) or Hameed et al.’s (2016) which consist complex mathematics including logarithmic operations and interval type-2 fuzzy sets, the preparation and scoring of the proposed MCQs can be conducted easily and quickly. In addition, although Fahim and Dehghankar’s (2014) study is logically similar to our proposed method, the time consuming process of answering and scoring of their method is eliminated in the proposed method.

These intelligent approaches do not equalize the examinees’ knowledge like crisp MCQs. They can improve the quality of knowledge measurement considering sureness of examinees in the examinations. Consequently, the results and ranking of examinees by using our proposed five approaches are more precise and fairer than classical MCQs. This method not only can be used by lecturers and teachers to accurately evaluate their students, but also can be applied by large institutes or organizations to fairly assess their examinees. This can be conducted by computerizing the proposed method by creating a graphical user interface and analysis tools. Future studies can focus on categorization of questions based on the examiner’s evaluation criteria, and by weighing criteria to evaluate the examinees. As future work, extensions of fuzzy sets such as type-2 fuzzy sets, intuitionistic fuzzy sets, or hesitant fuzzy sets can be investigated to check against our current baseline evaluations.