Community projects / 61 listings
Evaluations and independent research
Independent tests and open decision-model alternatives provide methods and results to inspect. Compare task data, calibration, and costs before carrying a result into your own workflow.
Download topic card
Artemis Edge decision models
Apache-2.0 GGUF builds of Clef, Clef-Flash, pplx-decider, Kev, and d3 for local /v1/systemone inference on compatible recent llama.cpp, with no request data leaving the machine; model cards report agreement with BF16 originals on 2,000 decisions, held-out results, and public-quantization comparisons, while their recommended files disagree with the originals on about 2 to 8% of test decisions and smaller variants can differ more.
Awesome Jev Robustness
Curated index of independent Jev tests grouped by calibration, consistency, perturbation, injection, abstention, failure modes, and language; it links original sources, reports their numbers as author findings, and keeps task benchmarks without a robustness angle in a separate data file.
blink
Self-hosted typed-decision models with Apache-2.0 code that return Choice, Noul, and Score option probabilities in one forward pass per batch through a TypeSafe-compatible API; the weights are non-commercial research and evaluation only, and there is no hosted TypeSafe-compatible endpoint.
Bosun v3.1
Apache-2.0 open-weight Qwen3 decision model family with released 0.6B and 1.7B weights and shipped choice, score, and noul inference; a separate Jev-style alternative, not hosted Jev.
Confident Where People Disagree
Preregistered, bias-corrected test of whether jev-1.13.0 lowers its confidence where people disagree, scoring Choice and Noul answers against the share of ChaosNLI's 100 annotators who chose the model's label on 750 low- and 750 high-disagreement items, with the paper and preregistration on Zenodo; Choice was close to calibrated on the agreed items but overconfident on the contested ones (bias-corrected ECE gap 0.264), Noul's gap was inconclusive, and only aggregate results are committed, with ChaosNLI text fetched separately.
Convex Decision Evals
Convex-run leaderboard comparing Jev with structured-output LLMs on 108 four-option questions about the Convex platform; its open runner sends questions and shuffled options to OpenRouter, uses native Jev Choice, and publishes per-answer evidence. This tests one platform's API knowledge, not whether a model can write working code.
Decision Models as Guardrails
raxIT Labs benchmark of Jev 1.13.0 and 1.14.0, 16 other decision models, and Amazon Bedrock Guardrails on the same 9,975 labelled guardrail checks (content, direct and indirect prompt attacks, denied topics, profanity, PII, and grounding); each check is a typed yes/no question blocked at one fixed 0.5 threshold, scored on balanced accuracy with 95% bootstrap intervals and USD per 1,000 checks at list price, and version 1.0.0 reports 88.7 for Jev 1.14.0, 90.4 for the Perplexity Decisions API, and 79.0 for Bedrock Guardrails. Per-row results, the scoring contract, and run scripts are public; the code is Apache-2.0, a 2,269-row contamination-check slice is held back, reproducing sends the rows to each paid provider, no model gets a calibrated threshold, and raxIT Labs built and ran the benchmark itself.
DecisionBench
Open benchmark runtime for document-grounded decision models, with pinned tasks and a native OpenRouter adapter for Jev.
Drex v2.0
Open-weight decision model from Nace.AI on Qwen3.6-35B-A3B (MoE, about 3B active per token) with a Kev pointer head that answers the same POST /v1/systemone Choice, Noul, and Score shape with states up to 131,072 tokens, self-hosted through Nace's llama.cpp or Ollama forks or through Nace's hosted API; the weights use the custom Nace.AI Open RAIL-M license, and its published scores are self-reported.
Dwight 4B
Apache-2.0 open-weight Qwen3-4B fine-tune for sales call transcripts that answers one yes/no, Choice, or 0-10 Score question as a single option token with a probability per option, and also writes JSON insights with verbatim quotes; runs self-hosted (vLLM or GGUF), so no call data leaves your machine; a separate Jev-style alternative, not hosted Jev, trained on English B2B software calls and weaker on companies unlike its training data.
Janus
Independent calibration measurement of Jev on two labelled datasets, Banking77 and Web of Science, with a Jev to frontier cascade priced per row from measured tokens, now also packaged as an installable tool (pip install janus-decide) that measures a threshold on your own data and ships none by default; the protocol was frozen before any result and the raw JSONL and figures are committed, and no routing parameter transferred between the two datasets, as the optimal threshold, the sign of the accuracy gap between the two models, and whether routing paid for itself all changed; the Web of Science labels come from publication metadata rather than per-document annotation, so part of the error measured there is label ambiguity.
jeff
Self-hosted GLiFormer 400M server for Choice, Score, and Noul through a Jev-compatible API, with public benchmark code and per-item results. Its documented JevBench comparison finds weaker accuracy than Jev on reasoning-heavy items; hosted cost figures are estimates and depend on deployment throughput.
Jev Capability Atlas
Bilingual English and Chinese map of Jev use cases and failure modes that separates the author's small, raw-response API suites from cited third-party results and TypeSafe claims; its case studies help frame task-fit questions, but they do not establish population-level accuracy or a reusable confidence threshold.
Jev Enterprise Decision Fabric
Experimental .NET decision architecture with a 111-case Jev-versus-Claude agent-action evaluation, public labels and raw JSONL, report-rebuild tooling, and a decision inspector; one annotator revised labels after reviewing a Jev pilot.
Jev IDS
MIT-licensed intrusion-detection prototype that asks Jev one Noul and one five-way Choice per network flow, comparing it with GPT-5.6 Luna and a Random Forest on NSL-KDD; it publishes run records and evidence limits, but the reported results are a 300-flow pilot and the gateway masks Jev's exact version.
Jev in Search: Three Practical Evaluations
Engineering report on search stopping, coding-memory reranking, and multi-hop relationship selection in DeepSearcher, MemSearch, and Vector Graph RAG, linking implementation and evaluation artifacts; some datasets are private, some comparisons use different sample counts, and the animated speed illustration is simulated.
Jev in the Wild
Data-driven survey of 2,170 public GitHub Jev projects, mapping early ecosystem growth, application domains, and decision-use patterns; it measures repository activity rather than production use.
Jev Judge vs Dimension Scores
Independent measurement on three classification tasks: one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights, 5,477 test rows and 34.1M input tokens for $1.43; decomposition reached 0.9076 against 0.8373 on Japanese NLI but flagged about 25× more hard benign rows as attacks, and four repair attempts failed, on dimensions the author wrote himself.
Jev Rerank Bench
Reranking comparison with raw provider responses, scoring code, dataset-level results, uncertainty intervals, and documented limitations.
Jev Spam Eval
Exploratory zero-shot spam study against trained TF-IDF baselines, including results and explicit post-hoc-tuning caveats.
jev-calibration-audit
Independent Jev-1.13.0 audit with reproducible code and per-call JSONL: tests abstention options, matched Korean and English items, question-shape interference, and option order; its strong abstention finding is on the KoBBQ dataset, not a universal calibration guarantee.
jev-certify
Early Python toolkit and CLINC150 study of conformal routing thresholds and prediction-powered audits over Jev 1.13 through OpenRouter, with request plans, 2,412 journalled answers and usage records, analysis code, and tests; its per-incoming-query risk bound depends on matching calibration and deployment traffic, and the study shows the scope gate missing its target after out-of-scope prevalence shifts while fallback accuracy remains unmeasured.
jev-does-not-play-dice
Calibration check on inputs whose true probability is fixed by construction (hidden fair dice, coins, four-way spinners) and on synthetic forecast documents, with raw responses and offline recomputation; Choice put 82.9% on face 1 across all 400 rolls at 19.0% accuracy and turned a stated 30% risk into 5.3%, while Noul stayed near the truth for 2 to 4 options and reported 15 to 17% for 8 to 20 options; no comparison with other models and no realistic task set, so it shows how far the probabilities can drift when the input carries no evidence rather than how Jev ranks against alternatives, run in launch week against jev-1.13.0.
jev-fanout-bench
Billing study comparing batched and one-question-per-call Jev requests through OpenRouter's TypeSafe-compatible endpoint, with synthetic English tickets sent to the provider, raw responses and charge records, and repeated requests to measure answer stability; the tickets are unlabelled, and separate-call latency was measured serially.
jev-measured
Reproducible OpenRouter measurements of Jev's response shapes, cost, and latency across eight use cases, plus a small head-to-head on 27 author-written support tickets; the author publishes raw data and corrections to earlier comparison errors.
jev-ood-calibration
Independent Jev calibration study with raw Gateway responses from three public benchmarks and 900 rule-generated support tickets; probabilities were near calibrated on the public sets but overconfident on an unseen priority rule, while the Boolean question showed a different error direction. The synthetic task is one family and the Gateway did not expose a fixed model version.
jev-orderby-bench
Independent measurement of whether ORDER BY over a Jev probability is defensible, with a pre-registered gate on pairwise inversion, Score ordinality against a graded target, calibration, and negation and paraphrase invariants; jev-1.13.0 passes on 20 Newsgroups topic membership and fails four of six conditions on Amazon ESCI human-graded product relevance, the DuckDB integrations' request shapes are shown to change the numbers (a 40-row batched state fails the ranking gate that one row per request passes), and two-decimal output leaves 53 of 360 rows tied at the top so LIMIT k cuts inside a tie; one seed and 30 ESCI queries, aggregates committed, corpus and cached responses regenerated locally for about a cent.
jev-sec-bench
Blind Jev-1.13.0 security evaluation with Go runner, raw per-sample results, and a TUI: 662 public prompt-injection messages and 200 matched vulnerable-code pairs; its reported classification scores use the study's stated context and a fixed 0.5 threshold, so deployment policy and corpus labels matter.
Jevals.com
Independent benchmark of hosted Jev and six LLMs on PubMedQA, Banking77, and HelpSteer2, with human labels, proper scores, calibration, cost, and latency; suite files and per-decision logs and the methodology are public, while the run harness is not published, LLM probabilities come from a prompt adapter, and the Gateway does not expose Jev's exact version.
JevBench
Independent cross-model benchmark for typed decisions with 534 frozen cases per complete entrant, source adapters, scoring code, public per-task outcomes, and aggregate result artifacts. Its four-axis score includes accuracy, calibration, latency, and cost; some self-hosted latency is adjusted by assumption, hosting costs can be estimates, and withheld cases are exposed to the services being measured.
JevGPT
Apache-2.0 code and per-item run records for the paper JevGPT: Thinking Fast and Slow in Autonomous Research, a research agent in which GPT-5.5 plans searches and writes answers while, in BrowseComp-Plus question answering, jev-1.13.0 answers parallel Choice questions that locate answer lines and Noul questions that check whether each retrieved document region addresses the request, with code returning the original passages to GPT; the authors report 90.48% accuracy on 830 BrowseComp-Plus questions and 0.5390 nDCG@20 on 105 NeuCLIRBench topics, the released records rebuild those rows without model calls, and live runs need GPT-provider credentials, a TypeSafe key and benchmark indexes; evidence queries and source regions go to TypeSafe, and research questions and selected original passages go to the configured GPT provider.
jevmlx
MIT-licensed local decision layer for Apple Silicon that scores constrained Boolean, enum, and multi-select fields from MLX model logits in one prefill, returns schema-valid JSON, and offers a System One-compatible endpoint and benchmark runner; probabilities are over the supplied options and need calibration against labeled task data.
jevos
MIT-licensed 1B yes/no model released as a q4_k_m GGUF that runs on CPU with llama.cpp behind a Jev-compatible /v1/systemone endpoint; it answers only Noul questions, and Choice and Score requests return 422. The authors report 0.815 accuracy against 0.927 for hosted Jev on 2,000 questions from unseen policies, but the training code and evaluation set are not published, so that result cannot be reproduced from the repository.
Kev
Apache-licensed, locally runnable Jev-style Choice, Score, and Noul models at 0.8B, 4B, and 9B, with released weights, training code, a System One-compatible server, frozen evaluation suites, and a playground. Its author reports a 0.822 new-source development accuracy for Kev-9B against 0.857 for hosted Jev on that suite, but Jev's training data is unknown, so this is not a controlled architecture comparison; test calibration and option order on your own data before setting a decision threshold.
Laya
Open local Choice, Score, and Noul decision models with published checkpoints and a router for English and multilingual inputs; its Jev comparisons use different prompts and sample sizes rather than a controlled head-to-head, and raw calibration and some languages remain weak.
LeJudge
MIT-licensed PushT study and cost module that turns LeWorldModel's imagined latents into closed-vocabulary facts, asks Jev typed questions about natural-language constraints, and adds the probabilities to a CEM planner's cost; ships oracle and keyword baselines, cached responses, and an offline paper rebuild; live runs send constraint text and probe facts to TypeSafe, and results cover one world model and twelve constraints, with near-miss false positives and rollout drift as material limitations.
LLM Memory Audit: Jev earnings test
Pre-registered test of whether jev-1.13-20260917 remembers how 12,533 US earnings announcements turned out, comparing each company only with itself; it found no detectable memory (within-company AUC 0.506 on beats, against 0.587 pooled for Claude Sonnet 5 as a positive control) and publishes code, per-call probabilities and a freeze proof, but withholds the licensed FMP earnings and price data, and the verdict stays provisional until 757 prospective announcements are scored in January 2027.
LLM2Jev
Apache-licensed local toolkit that reads causal-model logits for Choice, Score, and Noul answers through Transformers or SGLang, with a System One-shaped HTTP endpoint, web and Snake demos, and shared-prefix cache measurements. Its published performance benchmark compares cache modes on one model and GPU, not decision quality against Jev.
Love-Language Arena
Experimental local reproduction of the Choice, Score, and Noul pattern on Ollama logprobs, with order-reversed and negated re-asks to expose position bias, used to test three open models as fully crossed judges of Chinese and English rewrites, with self-preference correction, per-judge Platt calibration, dev/holdout prompt selection, and raw per-call JSONL. It does not call Jev, and its validation labels are Claude-generated rather than human, so the judge pass rates and model ranking show that the pipeline runs, not that the judges are valid.
Luce
Open recipe for Jev-style decision models: a task description, an LLM teacher that writes the data, then LoRA plus a decision head on Qwen3-4B-Base returning calibrated choice, score, and boolean probabilities on a 12 GB GPU; the README reports accuracy and ECE against Jev on identical test items, including where training does not help, with a GPU-free replay demo.
NanoJev
Open 0.6B Jev-style model for one-pass action probabilities in Maze, Snake, and ViZDoom, with public checkpoint, training data, and replayable game comparisons; it is specialized to those games and is not a general Jev API replacement.
OneJev
Apache-2.0 open multimodal System One models in four sizes (0.8B to 27B, plus a 27B FP8 build) that answer choice, score, and noul about screenshots, photos, video, and text in one forward pass through a /v1/systemone-compatible server; independent of hosted Jev.
Open Alternative to Jev
Apache-licensed Python library that reads option-token probabilities from open models through Transformers or vLLM, with packed and separate question modes, temperature scaling, benchmark scripts, and raw result files; its measurements compare modes of the same open model rather than Jev, packed answers can change with question order, and the package is not a drop-in System One API server.
Open Medical Jev
MIT-licensed independent reproduction of the Jev decision loop with two frozen open-weight readers, a fit-free confidence gate, and split-conformal candidate sets; its four-reading configuration lands within 2 points of hosted Jev 1.13.0 on three 600-item medical licensing papers without training anything, running locally on one 24 GB GPU; only the China paper is a real exam (the US and India papers are fixed-seed draws), the exam items are not redistributed, and its coverage–accuracy curve still trails hosted Jev at the highest precision tiers.
OpenJev (DiffusionGemma)
Apache-licensed decision server that reads Choice, Score, and Noul probabilities from DiffusionGemma 26B through a System One-shaped API, with NVIDIA and Apple silicon backends plus optional image input. Its NVIDIA path pins an unmerged vLLM branch, some limits differ from Jev, and answer quality needs evaluation on the reader's own tasks.
poorjev
Open, local reproduction of the Choice/Score/Noul interface on commodity zero-shot NLI models with temperature scaling and conformal abstention; ships a reproducible calibration eval (ECE 0.170 to 0.071 on its own small labelled set, cross-validated) and runs offline with no API key.
Rizzo Flow
Local, open-model decision server that reads answer-token probabilities instead of generating text, with a Jev-compatible Choice, Score, and Noul API plus its own numeric primitive. Its authors explicitly make no quality-parity claim with Jev; probabilities need calibration on the user's own data.
RSI-Jev
Jev-like 4B and 27B open models (v6.1-VL) from a self-improving research loop with published version notes and failed-experiment summaries (MIT code, Apache-2.0 weights; some training-source rights unclear); they return choice, score, and noul probabilities about text and images in one forward pass through a /v1/systemone-compatible server, with auto effort choosing depth per question. The authors report 50.98 (4B, pre-release kit) and 65.64 (27B, auto) on Decision Index 0.3's public set, with an official full score pending; the 27B was selected using that benchmark, needs a 96 GB GPU in bf16, and its training corpora are not public.
ruling
MIT-licensed System One-compatible server for Apple Silicon that scores typed questions from a stock model's logits with no training, accepts the official SDK unchanged, and re-scores the 256 public judgments that carry Jev's own answers (231 vs 238, no significant difference); serves MLX checkpoints or any OpenAI-compatible endpoint, with no native CUDA path.
SemIf (formerly OpenJev)
Independent open-model research baseline for direct typed option scoring; it reproduces the interface pattern, not Jev's undisclosed model or training.
Simple Jev
Apache-licensed local server that scores Choice, Score, and Noul with open-model logits; tokenizer support and decision calibration vary by model.
stuntd
Experimental local Jev-compatible proxy on Laya that records typed decisions in local SQLite, trains a head per decision site, and forwards uncertain requests to a configured upstream; its reported demos use rule-based teachers rather than hosted Jev, and proxy mode sends requests to the chosen upstream.
TetraJev
MIT-licensed, locally-deployed decision layer for complex bounded decisions: two frozen open-weight readers, four readings (letter + per-candidate yes/no) fused fit-free and routed by agreement into auto-release vs human review; eight decision suites and the RAG reranking pass benchmarked — including DecisionBench's 35 real-world task categories — with coverage–accuracy curves. It does not call the TypeSafe API, and its equal-weight fusion still trails the strongest single reading at some high-precision operating points.
TypeSafe AI Benchmark
Side-by-side Jev and Qwen-on-Cerebras comparison with raw exports, cost accounting, methodology, and task-specific limitations.
Verdict
Apache-2.0 118M multilingual bi-encoder for Choice, Score, and Noul on a System One-compatible endpoint, with temperature scaling and optional split-conformal abstention whose coverage depends on its calibration assumptions; it can run locally on CPU or in a browser through ONNX, and the author's reported calibration and typed-decision results are task-specific, including a lower score than Laya on the published typed-decision suite.
Vev
Independent Jev-style decision server with vision input, fine-tuned from Qwen3.5 4B and 9B, that answers Choice, Score, and Noul questions about text, JSON, and images, with screenshots or photos placed directly in state, through a /v1/systemone-compatible server; the repository has Apache-2.0 code and per-set results on public text and image judgment sets, requests are processed on your own GPU after the weights download from Hugging Face, and the weights are CC BY-NC 4.0, for non-commercial use only.
VideoJev
Apache-2.0 open System One video models that locally score temporal candidates and multiple-choice answers over a shared prefix without text generation; video inference requires no hosted API, and the models use their own task interface rather than a TypeSafe-compatible endpoint.
Von
Open local Choice, Noul, and Score model with public weights, training code, and a System One-shaped API. Benchmark comparisons are author-reported: its README gives a 91.23% accuracy headline without a matching result artifact in the repo, while its 49-task table reports 71.5% macro accuracy.
WaterSheep
Apache-2.0 open-weight decision model, fine-tuned from ModernBERT-base, that answers Noul, Choice, Score, and multi-label questions with a probability for every option; after downloading the weights it can run inference locally in Python or a browser through ONNX, and its local /v1/systemone server works with TypeSafe's Python SDK via TYPESAFE_BASE_URL. It is English-only, and its author reports 61.2% accuracy on held-out datasets, with rating-scale answers weaker than the other types.
WebJev
Apache-2.0 open-weight decision model for browser agents (a Qwen3.5-35B-A3B fine-tune) that answers Choice, Noul, and Score through a Jev-compatible /v1/systemone API and ships its training recipe, live-web training data, and a deterministic-verifier evaluation on 125 hand-picked real-website tasks. Its authors report 47/122 gradable successes versus 20/120 for Jev 1.13 inside the same agent, and 83.72% versus 84.70% mean accuracy on eight single-step typed-decision benchmarks; the live-web runs happened on different days and had different gradable subsets. It is a separate Jev-style alternative, not hosted Jev: self-hosting the weights needs one 80 GB GPU, and page observations go to the configured decision endpoint and text model.
When a Judgment Layer’s Self-Reported Fields Lie
Independent measurement of Jev’s typed decisions as one of three judgment layers run on identical items (Laya locally, Jev and a frontier model remotely), with per-call ledger-based cost accounting and Murphy/Brier/ECE calibration against binary ground truth; the Jev-specific findings are a verdict vocabulary reaching three values where six are documented, and a sufficient field that does not separate thin evidence from contradictory evidence, and the claim about contradiction rests on seven live readings that have no JSON artifact behind them, which the write-up states in its own errata rather than leaving to be discovered.
No listings match that search. Try another term or clear the filter.
Community entries are independent unless their source says otherwise. Inclusion is not an endorsement or security review. Report an outdated detail →