Community project / Evaluations and independent research

WaterSheep

Apache-2.0 open-weight decision model, fine-tuned from ModernBERT-base, that answers Noul, Choice, Score, and multi-label questions with a probability for every option; after downloading the weights it can run inference locally in Python or a browser through ONNX, and its local /v1/systemone server works with TypeSafe's Python SDK via TYPESAFE_BASE_URL. It is English-only, and its author reports 61.2% accuracy on held-out datasets, with rating-scale answers weaker than the other types.

Open the original resource ↗ See it in the full directory Start a reading list → Browse all Evaluations and independent research

Download share card ↓

This is an independent community listing. Check the source, license, data handling, and evaluation caveats before relying on a project. Inclusion is not an endorsement or security review.

Found an outdated or inaccurate detail? Report a correction →

Explore more in Evaluations and independent research

• 

  WebJev

  Apache-2.0 open-weight decision model for browser agents (a Qwen3.5-35B-A3B fine-tune) that answers Choice, Noul, and Score through a Jev-compatible /v1/systemone API and ships its training recipe, live-web training data, and a deterministic-verifier evaluation on 125 hand-picked real-website tasks. Its authors report 47/122 gradable successes versus 20/120 for Jev 1.13 inside the same agent, and 83.72% versus 84.70% mean accuracy on eight single-step typed-decision benchmarks; the live-web runs happened on different days and had different gradable subsets. It is a separate Jev-style alternative, not hosted Jev: self-hosting the weights needs one 80 GB GPU, and page observations go to the configured decision endpoint and text model.

  Read listing →
• 

  When a Judgment Layer’s Self-Reported Fields Lie

  Independent measurement of Jev’s typed decisions as one of three judgment layers run on identical items (Laya locally, Jev and a frontier model remotely), with per-call ledger-based cost accounting and Murphy/Brier/ECE calibration against binary ground truth; the Jev-specific findings are a verdict vocabulary reaching three values where six are documented, and a sufficient field that does not separate thin evidence from contradictory evidence, and the claim about contradiction rests on seven live readings that have no JSON artifact behind them, which the write-up states in its own errata rather than leaving to be discovered.

  Read listing →
• 

  Awesome Jev Robustness

  Curated index of independent Jev tests grouped by calibration, consistency, perturbation, injection, abstention, failure modes, and language; it links original sources, reports their numbers as author findings, and keeps task benchmarks without a robustness angle in a separate data file.

  Read listing →