A · F1
45
55
65
75
85
LLM 79.1
Annotators 81.7
Jev
tuned to 0.2–0.6
Jev, default 0.5: 76.8
76.8
Jev, tuned, held out: 75.6
75.6
djev
tuned to 0.005–0.2
djev, default 0.5: 54.5
54.5
djev, tuned, held out: 76.7
76.7
Luna
tuned to 0.01–0.5
Luna, default 0.5: 71.3
71.3
Luna, tuned, held out: 70.4
70.4
B · KRIPPENDORFF’S α
45
55
65
75
85
LLM 75.6
Annotators 78.5
Jev, default 0.5: 73.3
73.3
Jev, tuned, held out: 71.9
71.9
djev, default 0.5: 49.4
49.4
djev, tuned, held out: 73.0
73.0
Luna, default 0.5: 67.4
67.4
Luna, tuned, held out: 65.6
65.6
Default threshold 0.5, all 238 queries
Tuned on one half, scored on the other (mean of 2,000 splits)