Community projects / 28 listings
Evaluations and independent research
Independent tests and open decision-model alternatives provide methods and results to inspect. Compare task data, calibration, and costs before carrying a result into your own workflow.
Download topic card
Janus
Independent calibration measurement of Jev on two labelled datasets, Banking77 and Web of Science, with a Jev to frontier cascade priced per row from measured tokens, now also packaged as an installable tool (pip install janus-decide) that measures a threshold on your own data and ships none by default; the protocol was frozen before any result and the raw JSONL and figures are committed, and no routing parameter transferred between the two datasets, as the optimal threshold, the sign of the accuracy gap between the two models, and whether routing paid for itself all changed; the Web of Science labels come from publication metadata rather than per-document annotation, so part of the error measured there is label ambiguity.
jeff
Self-hosted GLiFormer 400M server for Choice, Score, and Noul through a Jev-compatible API, with public benchmark code and per-item results. Its documented JevBench comparison finds weaker accuracy than Jev on reasoning-heavy items; hosted cost figures are estimates and depend on deployment throughput.
Jev Capability Atlas
Bilingual English and Chinese map of Jev use cases and failure modes that separates the author's small, raw-response API suites from cited third-party results and TypeSafe claims; its case studies help frame task-fit questions, but they do not establish population-level accuracy or a reusable confidence threshold.
Jev Enterprise Decision Fabric
Experimental .NET decision architecture with a 111-case Jev-versus-Claude agent-action evaluation, public labels and raw JSONL, report-rebuild tooling, and a decision inspector; one annotator revised labels after reviewing a Jev pilot.
Jev Judge vs Dimension Scores
Independent measurement on three classification tasks: one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights, 5,477 test rows and 34.1M input tokens for $1.43; decomposition reached 0.9076 against 0.8373 on Japanese NLI but flagged about 25× more hard benign rows as attacks, and four repair attempts failed, on dimensions the author wrote himself.
Jev Rerank Bench
Reranking comparison with raw provider responses, scoring code, dataset-level results, uncertainty intervals, and documented limitations.
Jev Spam Eval
Exploratory zero-shot spam study against trained TF-IDF baselines, including results and explicit post-hoc-tuning caveats.
jev-calibration-audit
Independent Jev-1.13.0 audit with reproducible code and per-call JSONL: tests abstention options, matched Korean and English items, question-shape interference, and option order; its strong abstention finding is on the KoBBQ dataset, not a universal calibration guarantee.
jev-certify
Early Python toolkit and CLINC150 study of conformal routing thresholds and prediction-powered audits over Jev 1.13 through OpenRouter, with request plans, 2,412 journalled answers and usage records, analysis code, and tests; its per-incoming-query risk bound depends on matching calibration and deployment traffic, and the study shows the scope gate missing its target after out-of-scope prevalence shifts while fallback accuracy remains unmeasured.
jev-measured
Reproducible OpenRouter measurements of Jev's response shapes, cost, and latency across eight use cases, plus a small head-to-head on 27 author-written support tickets; the author publishes raw data and corrections to earlier comparison errors.
jev-ood-calibration
Independent Jev calibration study with raw Gateway responses from three public benchmarks and 900 rule-generated support tickets; probabilities were near calibrated on the public sets but overconfident on an unseen priority rule, while the Boolean question showed a different error direction. The synthetic task is one family and the Gateway did not expose a fixed model version.
jev-orderby-bench
Independent measurement of whether ORDER BY over a Jev probability is defensible, with a pre-registered gate on pairwise inversion, Score ordinality against a graded target, calibration, and negation and paraphrase invariants; jev-1.13.0 passes on 20 Newsgroups topic membership and fails four of six conditions on Amazon ESCI human-graded product relevance, the DuckDB integrations' request shapes are shown to change the numbers (a 40-row batched state fails the ranking gate that one row per request passes), and two-decimal output leaves 53 of 360 rows tied at the top so LIMIT k cuts inside a tie; one seed and 30 ESCI queries, aggregates committed, corpus and cached responses regenerated locally for about a cent.
jev-sec-bench
Blind Jev-1.13.0 security evaluation with Go runner, raw per-sample results, and a TUI: 662 public prompt-injection messages and 200 matched vulnerable-code pairs; its reported classification scores use the study's stated context and a fixed 0.5 threshold, so deployment policy and corpus labels matter.
Jevals.com
Independent benchmark of hosted Jev and six LLMs on PubMedQA, Banking77, and HelpSteer2, with human labels, proper scores, calibration, cost, and latency; suite files and per-decision logs and the methodology are public, while the run harness is not published, LLM probabilities come from a prompt adapter, and the Gateway does not expose Jev's exact version.
JevBench
Independent cross-model benchmark for typed decisions with 534 frozen cases per complete entrant, source adapters, scoring code, public per-task outcomes, and aggregate result artifacts. Its four-axis score includes accuracy, calibration, latency, and cost; some self-hosted latency is adjusted by assumption, hosting costs can be estimates, and withheld cases are exposed to the services being measured.
jevmlx
MIT-licensed local decision layer for Apple Silicon that scores constrained Boolean, enum, and multi-select fields from MLX model logits in one prefill, returns schema-valid JSON, and offers a System One-compatible endpoint and benchmark runner; probabilities are over the supplied options and need calibration against labeled task data.
Kev
Apache-licensed, locally runnable Jev-style Choice, Score, and Noul models at 0.8B, 4B, and 9B, with released weights, training code, a System One-compatible server, frozen evaluation suites, and a playground. Its author reports a 0.822 new-source development accuracy for Kev-9B against 0.857 for hosted Jev on that suite, but Jev's training data is unknown, so this is not a controlled architecture comparison; test calibration and option order on your own data before setting a decision threshold.
Laya
Open local Choice, Score, and Noul decision models with published checkpoints and a router for English and multilingual inputs; its Jev comparisons use different prompts and sample sizes rather than a controlled head-to-head, and raw calibration and some languages remain weak.
LLM2Jev
Apache-licensed local toolkit that reads causal-model logits for Choice, Score, and Noul answers through Transformers or SGLang, with a System One-shaped HTTP endpoint, web and Snake demos, and shared-prefix cache measurements. Its published performance benchmark compares cache modes on one model and GPU, not decision quality against Jev.
Luce
Open recipe for Jev-style decision models: a task description, an LLM teacher that writes the data, then LoRA plus a decision head on Qwen3-4B-Base returning calibrated choice, score, and boolean probabilities on a 12 GB GPU; the README reports accuracy and ECE against Jev on identical test items, including where training does not help, with a GPU-free replay demo.
NanoJev
Open 0.6B Jev-style model for one-pass action probabilities in Maze, Snake, and ViZDoom, with public checkpoint, training data, and replayable game comparisons; it is specialized to those games and is not a general Jev API replacement.
Open Alternative to Jev
Apache-licensed Python library that reads option-token probabilities from open models through Transformers or vLLM, with packed and separate question modes, temperature scaling, benchmark scripts, and raw result files; its measurements compare modes of the same open model rather than Jev, packed answers can change with question order, and the package is not a drop-in System One API server.
OpenJev (DiffusionGemma)
Apache-licensed decision server that reads Choice, Score, and Noul probabilities from DiffusionGemma 26B through a System One-shaped API, with NVIDIA and Apple silicon backends plus optional image input. Its NVIDIA path pins an unmerged vLLM branch, some limits differ from Jev, and answer quality needs evaluation on the reader's own tasks.
poorjev
Open, local reproduction of the Choice/Score/Noul interface on commodity zero-shot NLI models with temperature scaling and conformal abstention; ships a reproducible calibration eval (ECE 0.170 to 0.071 on its own small labelled set, cross-validated) and runs offline with no API key.
Rizzo Flow
Local, open-model decision server that reads answer-token probabilities instead of generating text, with a Jev-compatible Choice, Score, and Noul API plus its own numeric primitive. Its authors explicitly make no quality-parity claim with Jev; probabilities need calibration on the user's own data.
SemIf (formerly OpenJev)
Independent open-model research baseline for direct typed option scoring; it reproduces the interface pattern, not Jev's undisclosed model or training.
TypeSafe AI Benchmark
Side-by-side Jev and Qwen-on-Cerebras comparison with raw exports, cost accounting, methodology, and task-specific limitations.
Von
Open local Choice, Noul, and Score model with public weights, training code, and a System One-shaped API. Benchmark comparisons are author-reported: its README gives a 91.23% accuracy headline without a matching result artifact in the repo, while its 49-task table reports 71.5% macro accuracy.
No listings match that search. Try another term or clear the filter.
Community entries are independent unless their source says otherwise. Inclusion is not an endorsement or security review. Report an outdated detail →