Sent to the LLM (0.2 ≤ P < 0.8)IN THE BANDJevJev: P = 0.03, annotators splitJev: P = 0.03, annotators splitJev: P = 0.04, annotators splitJev: P = 0.05, annotators splitJev: P = 0.05, both annotators disagreeJev: P = 0.06, annotators splitJev: P = 0.08, annotators splitJev: P = 0.09, both annotators disagreeJev: P = 0.12, annotators splitJev: P = 0.12, annotators splitJev: P = 0.13, both annotators disagreeJev: P = 0.16, annotators splitJev: P = 0.22, both annotators disagreeJev: P = 0.31, both annotators disagreeJev: P = 0.33, both annotators disagreeJev: P = 0.39, annotators splitJev: P = 0.4, annotators splitJev: P = 0.5, annotators splitJev: P = 0.52, both annotators disagreeJev: P = 0.56, both annotators disagreeJev: P = 0.8, annotators split5 of 8clear mistakes caught7.6% of queries sent00.20.50.81djevdjev: P = 1e-06, annotators splitdjev: P = 6e-06, annotators splitdjev: P = 1.1e-05, annotators splitdjev: P = 1.6e-05, annotators splitdjev: P = 0.000289, annotators splitdjev: P = 0.000308, both annotators disagreedjev: P = 0.000625, annotators splitdjev: P = 0.000642, annotators splitdjev: P = 0.000806, annotators splitdjev: P = 0.000923, annotators splitdjev: P = 0.00429, both annotators disagreedjev: P = 0.00591, annotators splitdjev: P = 0.00656, both annotators disagreedjev: P = 0.0179, annotators splitdjev: P = 0.0194, both annotators disagreedjev: P = 0.022, annotators splitdjev: P = 0.039, both annotators disagreedjev: P = 0.0442, both annotators disagreedjev: P = 0.0974, both annotators disagreedjev: P = 0.136, both annotators disagreedjev: P = 0.219, both annotators disagreedjev: P = 0.232, both annotators disagreedjev: P = 0.294, both annotators disagreedjev: P = 0.422, both annotators disagreedjev: P = 0.447, both annotators disagreedjev: P = 0.477, both annotators disagreedjev: P = 0.486, both annotators disagreedjev: P = 0.488, both annotators disagreedjev: P = 0.524, annotators split8 of 16clear mistakes caught6.7% of queries sent00.20.50.81LunaLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, both annotators disagreeLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0, annotators splitLuna: P = 0.01, both annotators disagreeLuna: P = 0.01, annotators splitLuna: P = 0.02, both annotators disagreeLuna: P = 0.02, both annotators disagreeLuna: P = 0.02, both annotators disagreeLuna: P = 0.03, both annotators disagreeLuna: P = 0.04, annotators splitLuna: P = 0.06, both annotators disagreeLuna: P = 0.25, both annotators disagreeLuna: P = 0.28, annotators splitLuna: P = 0.47, both annotators disagreeLuna: P = 0.6, both annotators disagreeLuna: P = 0.94, annotators split3 of 10clear mistakes caught4.2% of queries sent00.20.50.8112 mistakes at P ≤ 0.01Each dot is a query the model got wrong at threshold 0.5, placed at the probability it gave.Both annotators disagree with the modelAnnotators split (half a mistake)