Community project / Evaluations and independent research

jev-does-not-play-dice

Calibration check on inputs whose true probability is fixed by construction (hidden fair dice, coins, four-way spinners) and on synthetic forecast documents, with raw responses and offline recomputation; Choice put 82.9% on face 1 across all 400 rolls at 19.0% accuracy and turned a stated 30% risk into 5.3%, while Noul stayed near the truth for 2 to 4 options and reported 15 to 17% for 8 to 20 options; no comparison with other models and no realistic task set, so it shows how far the probabilities can drift when the input carries no evidence rather than how Jev ranks against alternatives, run in launch week against jev-1.13.0.

Open the original resource ↗ See it in the full directory Start a reading list → Browse all Evaluations and independent research

Download share card ↓

This is an independent community listing. Check the source, license, data handling, and evaluation caveats before relying on a project. Inclusion is not an endorsement or security review.

Found an outdated or inaccurate detail? Report a correction →

Explore more in Evaluations and independent research

• 

  jev-fanout-bench

  Billing study comparing batched and one-question-per-call Jev requests through OpenRouter's TypeSafe-compatible endpoint, with synthetic English tickets sent to the provider, raw responses and charge records, and repeated requests to measure answer stability; the tickets are unlabelled, and separate-call latency was measured serially.

  Read listing →
• 

  jev-measured

  Reproducible OpenRouter measurements of Jev's response shapes, cost, and latency across eight use cases, plus a small head-to-head on 27 author-written support tickets; the author publishes raw data and corrections to earlier comparison errors.

  Read listing →
• 

  jev-ood-calibration

  Independent Jev calibration study with raw Gateway responses from three public benchmarks and 900 rule-generated support tickets; probabilities were near calibrated on the public sets but overconfident on an unseen priority rule, while the Boolean question showed a different error direction. The synthetic task is one family and the Gateway did not expose a fixed model version.

  Read listing →