Community project / Evaluations and independent research

LLM Memory Audit: Jev earnings test

Pre-registered test of whether jev-1.13-20260917 remembers how 12,533 US earnings announcements turned out, comparing each company only with itself; it found no detectable memory (within-company AUC 0.506 on beats, against 0.587 pooled for Claude Sonnet 5 as a positive control) and publishes code, per-call probabilities and a freeze proof, but withholds the licensed FMP earnings and price data, and the verdict stays provisional until 757 prospective announcements are scored in January 2027.

Open the original resource ↗ See it in the full directory Start a reading list → Browse all Evaluations and independent research

Download share card ↓

This is an independent community listing. Check the source, license, data handling, and evaluation caveats before relying on a project. Inclusion is not an endorsement or security review.

Found an outdated or inaccurate detail? Report a correction →

Explore more in Evaluations and independent research

• 

  LLM2Jev

  Apache-licensed local toolkit that reads causal-model logits for Choice, Score, and Noul answers through Transformers or SGLang, with a System One-shaped HTTP endpoint, web and Snake demos, and shared-prefix cache measurements. Its published performance benchmark compares cache modes on one model and GPU, not decision quality against Jev.

  Read listing →
• 

  Love-Language Arena

  Experimental local reproduction of the Choice, Score, and Noul pattern on Ollama logprobs, with order-reversed and negated re-asks to expose position bias, used to test three open models as fully crossed judges of Chinese and English rewrites, with self-preference correction, per-judge Platt calibration, dev/holdout prompt selection, and raw per-call JSONL. It does not call Jev, and its validation labels are Claude-generated rather than human, so the judge pass rates and model ranking show that the pipeline runs, not that the judges are valid.

  Read listing →
• 

  Luce

  Open recipe for Jev-style decision models: a task description, an LLM teacher that writes the data, then LoRA plus a decision head on Qwen3-4B-Base returning calibrated choice, score, and boolean probabilities on a 12 GB GPU; the README reports accuracy and ECE against Jev on identical test items, including where training does not help, with a GPU-free replay demo.

  Read listing →