Inter-rater analysis (IRR)
The results of the IRR tests, showed high strength of agreement, for
every question in every domain in the four practice guidelines among the
four raters. As well as the percent agreement of the first overall
assessment (OA1) in Figure 2. Most of the Kappa values were between
(0.50-1.00) denoting good to excellent agreement. Two evaluations only
shown in Figure 3, revealed poor strength of agreement (K=0.0). ACOG,
D2Q3 and D3Q2. ACOG evaluation showed one question out of 24 with
excellent agreement (K=1), 16 questions with good agreement (K=0.5), 5
questions with very good agreement (K=0.6-0.8), two questions with poor
agreement (K=0.00) and the overall assessment (1) showed good agreement
(K=0.5). RCOG 2011 evaluation showed no questions out of 24 with
excellent agreement, 15 questions with good agreement (K=0.5), nine
questions with very good agreement (K=0.6-0.8) and the overall
assessment (1) showed good agreement (K=0.5). NICE 2012 evaluation
showed one question out of 24 with excellent agreement (K=1), no
questions with fair agreement, 16 questions with good agreement (K=0.5),
seven questions with very good agreement (K=0.6-0.8), and the overall
assessment (1) showed good agreement (K=0.5). NHLBI evaluation showed no
questions out of 24 with excellent agreement, 15 questions with good
agreement (K=0.5), nine questions with very good agreement (K=0.6-0.8)
and the overall assessment (1) showed good agreement (K=0.5). Table 6.
Intra class correlation coefficient (Kappa value) among raters for the
four guidelines regarding the second Overall Assessment (OA2), showed
the following; Number of observed agreements: 6 (37.50% of the
observations). Number of agreements expected by chance: 4.0 (25.00% of
the observations). Kappa= 0.167. SE of kappa = 0.138. 95% confidence
interval: From -0.103 to 0.437, Weighted Kappa= 0.077.