Community project / Evaluations and independent research

Confident Where People Disagree

Preregistered, bias-corrected test of whether jev-1.13.0 lowers its confidence where people disagree, scoring Choice and Noul answers against the share of ChaosNLI's 100 annotators who chose the model's label on 750 low- and 750 high-disagreement items, with the paper and preregistration on Zenodo; Choice was close to calibrated on the agreed items but overconfident on the contested ones (bias-corrected ECE gap 0.264), Noul's gap was inconclusive, and only aggregate results are committed, with ChaosNLI text fetched separately.

Open the original resource ↗ See it in the full directory Start a reading list → Browse all Evaluations and independent research

Download share card ↓

This is an independent community listing. Check the source, license, data handling, and evaluation caveats before relying on a project. Inclusion is not an endorsement or security review.

Found an outdated or inaccurate detail? Report a correction →

Explore more in Evaluations and independent research

• 

  Convex Decision Evals

  Convex-run leaderboard comparing Jev with structured-output LLMs on 108 four-option questions about the Convex platform; its open runner sends questions and shuffled options to OpenRouter, uses native Jev Choice, and publishes per-answer evidence. This tests one platform's API knowledge, not whether a model can write working code.

  Read listing →
• 

  DecisionBench

  Open benchmark runtime for document-grounded decision models, with pinned tasks and a native OpenRouter adapter for Jev.

  Read listing →
• 

  Janus

  Independent calibration measurement of Jev on two labelled datasets, Banking77 and Web of Science, with a Jev to frontier cascade priced per row from measured tokens, now also packaged as an installable tool (pip install janus-decide) that measures a threshold on your own data and ships none by default; the protocol was frozen before any result and the raw JSONL and figures are committed, and no routing parameter transferred between the two datasets, as the optimal threshold, the sign of the accuracy gap between the two models, and whether routing paid for itself all changed; the Web of Science labels come from publication metadata rather than per-document annotation, so part of the error measured there is label ambiguity.

  Read listing →