Community project / Evaluations and independent research

Decision Models as Guardrails

raxIT Labs benchmark of Jev 1.13.0 and 1.14.0, 16 other decision models, and Amazon Bedrock Guardrails on the same 9,975 labelled guardrail checks (content, direct and indirect prompt attacks, denied topics, profanity, PII, and grounding); each check is a typed yes/no question blocked at one fixed 0.5 threshold, scored on balanced accuracy with 95% bootstrap intervals and USD per 1,000 checks at list price, and version 1.0.0 reports 88.7 for Jev 1.14.0, 90.4 for the Perplexity Decisions API, and 79.0 for Bedrock Guardrails. Per-row results, the scoring contract, and run scripts are public; the code is Apache-2.0, a 2,269-row contamination-check slice is held back, reproducing sends the rows to each paid provider, no model gets a calibrated threshold, and raxIT Labs built and ran the benchmark itself.

Open the original resource ↗ See it in the full directory Start a reading list → Browse all Evaluations and independent research

Download share card ↓

This is an independent community listing. Check the source, license, data handling, and evaluation caveats before relying on a project. Inclusion is not an endorsement or security review.

Found an outdated or inaccurate detail? Report a correction →

Explore more in Evaluations and independent research

• 

  DecisionBench

  Open benchmark runtime for document-grounded decision models, with pinned tasks and a native OpenRouter adapter for Jev.

  Read listing →
• 

  Drex v2.0

  Open-weight decision model from Nace.AI on Qwen3.6-35B-A3B (MoE, about 3B active per token) with a Kev pointer head that answers the same POST /v1/systemone Choice, Noul, and Score shape with states up to 131,072 tokens, self-hosted through Nace's llama.cpp or Ollama forks or through Nace's hosted API; the weights use the custom Nace.AI Open RAIL-M license, and its published scores are self-reported.

  Read listing →
• 

  Dwight 4B

  Apache-2.0 open-weight Qwen3-4B fine-tune for sales call transcripts that answers one yes/no, Choice, or 0-10 Score question as a single option token with a probability per option, and also writes JSON insights with verbatim quotes; runs self-hosted (vLLM or GGUF), so no call data leaves your machine; a separate Jev-style alternative, not hosted Jev, trained on English B2B software calls and weaker on companies unlike its training data.

  Read listing →