The reliability of scoring refers to the degree of consistency in the ratings given by multiple raters to the same group of people's answer sheets. For example, if there are only two raters, the correlation coefficient between their ratings can be calculated to obtain the rater reliability. It is mainly applicable when the test is a subjective question, and there may be differences in the ratings of different evaluators. In this case, the role of evaluators becomes prominent
1、 Data preparation
Here, two judges are used to rate 10 athletes for a certain project. Assuming two judges rate these ten athletes, the following figure shows the results. Here, we will use these two judges to evaluate the consistency of athlete ratings.
Figure 1 Data Display
2、 Analysis steps
Click on the top menu bar of SPSS, select "Analysis" - "Correlation" - "bivariate". Load judges A and B into the variable text box, and check the correlation coefficients for "Pearson" and "Spearman".
Figure 2 bivariate correlation
On the left side of the bivariate correlation window, the options, styles, boot pulling method, and confidence intervals remain default, but can be selected according to personal needs.
Figure 3 Default Parameters
3、 Result analysis
Due to the selection of Pearson and Spearman for the correlation coefficient in the analysis process, the results will be analyzed separately below. The Pearson correlation coefficient is 0.816, with explanations at the bottom of the table. A significance greater than 0.01 indicates a significant correlation, while a value of 0.04 indicates a significant correlation, indicating high consistency among raters.
Figure 4 Pearson analysis
Spearman conducted a similar analysis, with a correlation coefficient of 0.890 and a significance (two tailed) of 0.03. If 0.03 is greater than 0.01, it indicates a significant correlation and high consistency among raters.
Table 5 Spearman analysis
The above is a test using IBM SPSS Statistics for rater reliability analysis. You can conduct your own experiments as needed.
