A · PREDICTED VS. OBSERVED
0
0
0.25
0.25
0.5
0.5
0.75
0.75
1
1
Model’s probability
Share of queries that were yes
perfectly calibrated
djev: 211 queries scored 0–0.05; mean P 0.001, 6% yes
djev: 3 queries scored 0.05–0.1; mean P 0.079, 33% yes
djev: 5 queries scored 0.1–0.3; mean P 0.217, 80% yes
djev: 5 queries scored 0.3–0.5; mean P 0.464, 100% yes
djev: 6 queries scored 0.5–0.7; mean P 0.594, 92% yes
djev: 7 queries scored 0.7–0.9; mean P 0.85, 100% yes
djev: 1 queries scored 0.9–1; mean P 0.908, 100% yes
Luna: 207 queries scored 0–0.05; mean P 0.001, 6% yes
Luna: 3 queries scored 0.05–0.1; mean P 0.06, 33% yes
Luna: 5 queries scored 0.1–0.3; mean P 0.226, 30% yes
Luna: 1 queries scored 0.3–0.5; mean P 0.47, 100% yes
Luna: 3 queries scored 0.5–0.7; mean P 0.557, 67% yes
Luna: 6 queries scored 0.7–0.9; mean P 0.777, 100% yes
Luna: 13 queries scored 0.9–1; mean P 0.956, 96% yes
Jev: 149 queries scored 0–0.05; mean P 0.029, 1% yes
Jev: 34 queries scored 0.05–0.1; mean P 0.064, 10% yes
Jev: 21 queries scored 0.1–0.3; mean P 0.144, 17% yes
Jev: 7 queries scored 0.3–0.5; mean P 0.376, 43% yes
Jev: 5 queries scored 0.5–0.7; mean P 0.562, 50% yes
Jev: 17 queries scored 0.7–0.9; mean P 0.84, 97% yes
Jev: 5 queries scored 0.9–1; mean P 0.918, 100% yes
Dot size = number of queries.
Jev
djev
Luna
B · WHERE THE PROBABILITIES LAND
Jev
Jev: 149 of 238 queries scored <0.05
63% below 0.05
Jev: 34 of 238 queries scored 0.05–0.1
Jev: 21 of 238 queries scored 0.1–0.3
Jev: 7 of 238 queries scored 0.3–0.5
Jev: 5 of 238 queries scored 0.5–0.7
Jev: 17 of 238 queries scored 0.7–0.9
Jev: 5 of 238 queries scored ≥0.9
5 queries at 0.9 or above
djev
djev: 211 of 238 queries scored <0.05
89% below 0.05
djev: 3 of 238 queries scored 0.05–0.1
djev: 5 of 238 queries scored 0.1–0.3
djev: 5 of 238 queries scored 0.3–0.5
djev: 6 of 238 queries scored 0.5–0.7
djev: 7 of 238 queries scored 0.7–0.9
djev: 1 of 238 queries scored ≥0.9
1 query at 0.9 or above
Luna
Luna: 207 of 238 queries scored <0.05
87% below 0.05
Luna: 3 of 238 queries scored 0.05–0.1
Luna: 5 of 238 queries scored 0.1–0.3
Luna: 1 of 238 queries scored 0.3–0.5
Luna: 3 of 238 queries scored 0.5–0.7
Luna: 6 of 238 queries scored 0.7–0.9
Luna: 13 of 238 queries scored ≥0.9
13 queries at 0.9 or above
Probability range
<0.05
0.05–0.1
0.1–0.3
0.3–0.5
0.5–0.7
0.7–0.9
≥0.9