---
title: "Field reports: independent evaluations, critiques, open replicas"
type: community
source_tier: community
tags: [field-reports, evaluations, critiques, replicas, benchmarks]
created: 2026-09-20
updated: 2026-09-20
confidence: medium
sources:
  - raw/x/LangChain-2101454284927959080.md
  - raw/x/0xidanlevin-2100937437325205568.md
  - raw/x/trycua-2101437979180904640.md
  - raw/x/jpschroeder-2100952337904767258.md
  - raw/x/cyrilXBT-2101515130798297448.md
  - raw/x/skeptrune-2101209390992994570.md
  - raw/x/0xCarnagee-2101456270909690202.md
  - raw/x/0xMovez-2101325703635435523.md
  - raw/x/kunchenguid-2101032677940117875.md
  - raw/x/swill1ams-2100421326389354624.md
  - raw/x-repos/TheoLeeCJ__SemIf.md
  - raw/x-repos/TianyuCodings__NanoJev.md
  - raw/x-repos/ekzhang__openjev-sglang.md
  - raw/x-repos/r-ms__mini-jev.md
  - raw/x-repos/vinnylarouge__jevlike.md
  - raw/x-repos/tamaratran__fast-jev-compaction.md
  - raw/x-repos/nekuda-ai__WindTunnel.md
  - raw/x-repos/danielgshea__jev-as-a-judge.md
jev_version: "jev-1.13.0"
summary: "What independent testers measured in Jev's first week: strong repeatability and cost on bounded classification, weak standalone browser/spatial control, and a wave of open replicas."
---

# Field reports: independent evaluations, critiques, open replicas

> **TL;DR** Independent tests (2026-09-16→20, all narrow, none peer-reviewed) agree on one shape: Jev is excellent at **bounded, pre-enumerated decisions over a state you already filtered**, and mediocre whenever it must build its own option menu or act over many steps. The cost and repeatability wins are real in the published tests; "200x/400x" is a workflow ceiling, not a per-call result. Re-measure everything below on your own data ([[guides/testing-and-evaluation]]).

## What held up

| Finding | Who | Evidence quality | Note |
|---|---|---|---|
| Score repeatability far above LLM judges: quality-score variance **92–913x lower** than GPT-5.6 Luna/Terra and Claude Sonnet 4.6 | [@LangChain](https://x.com/LangChain/status/2101454284927959080) (Shea & Roche) | measured, repo published | `verified` in direction — docs claim "extremely consistent" ([[concepts/jaggedness-jev-1-13]] §8). n=5 traces × 100 reps. |
| Binary pass/fail matched a human oracle on **500/500** repeats (Terra 99.8%, Luna 96.4%, Claude 80.0%) | @LangChain | measured | Same 5-case set; does not generalise. |
| Tool **selection** from a menu the app already built is the sweet spot | [@0xidanlevin](https://x.com/0xidanlevin/status/2100937437325205568), [@trycua](https://x.com/trycua/status/2101437979180904640) | measured | Matches [[patterns/intent-routing]], jaggedness §4. |
| Parallel questions are nearly free; add questions, not round trips | [@sydneyrunkle](https://x.com/sydneyrunkle/status/2100754364545761643) | anecdotal | `verified` — one `state` ingest, 64k/32k budgets ([[patterns/fan-out]]). |
| Output can be wrong but never malformed | [@akshay_pachaar](https://x.com/akshay_pachaar/status/2101309986156712025) | anecdotal | `verified` — "Schema matching is guaranteed" ([[concepts/workflow-evals]]). Not "cannot be wrong". |
| Compaction by relevance-scoring tool calls (keep messages verbatim, drop stale ones) ships | [@0xCarnagee](https://x.com/0xCarnagee/status/2101077261407412732) on tamaratran/fast-jev-compaction (4,918★) | anecdotal | Caveats below. |
| Explainers (@mvanhorn, @shannholmberg, @chddaniel, @Layton_Gott, @de1lymoon, @0xCodila, @grok, @moritzkremb) only restate launch numbers | — | restatement | Not evidence; useful as idea lists. |

## Where it disappointed or broke

| Finding | Who | Evidence quality | Failure condition |
|---|---|---|---|
| Standalone browser control solved only **25/49** tasks; adding WebMCP tool menus took it to **49/49** | [@0xidanlevin](https://x.com/0xidanlevin/status/2100937437325205568) | measured, open repo (nekuda-ai/WindTunnel) | Jev must pick *the right next step*, not just a valid control. Author's own framing: their harness, not a limit on Jev. |
| Cannot generate tool arguments; needed Mercury 2.5 alongside it to write search queries | same | measured | `verified` — jaggedness §9 (no generation). Budget for a second small model. |
| Form-fill decision accuracy **83.6%** hosted Jev vs **99.7%** for a task-trained local specialist | [@trycua](https://x.com/trycua/status/2101437979180904640) | measured, narrow | Specialist was fine-tuned for that exact convention; Jev was not. Decision-level, not end-to-end. |
| Hosted round trip **260–280 ms** vs **7–9 ms** for a local forward pass | same | measured, different boundaries | Network is the latency floor. Kills sub-100 ms control loops. |
| Four overlapping Choice options ("high priority"/"urgent"/"needs review"/"flag") returned confident answers that were internally inconsistent "about a third of the time" | [@cyrilXBT](https://x.com/cyrilXBT/status/2101515130798297448) | anecdotal, self-reported (~6 h to fix) | `verified` as a schema bug, not a model bug — jaggedness §1 and §7. Options must be mutually exclusive. |
| A 0.6B open replica beat Jev on ViZDoom Basic (**128/128** vs **56/128**) and tied on Predict Position (11/128) | [TianyuCodings/NanoJev](https://github.com/TianyuCodings/NanoJev) | `unverified` — replica-run comparison, harness for the Jev side not independently checked | Spatial/game control is not a System One strength; a task-trained tiny model can win. |
| **Too expensive for high-frequency control.** $0.042/Mtok is ~7x a DeepSeek V4.1 Flash cache read ($0.006); 10 Hz at 10k context ≈ **$0.0042/s ≈ $15/h** | [@jpschroeder](https://x.com/jpschroeder/status/2100952337904767258) | measured arithmetic | Price `verified` ([[reference/models-and-pricing]]); the arithmetic checks out. He wants ~10x cheaper and on-device for robotics. |
| Jev-scored compaction caveats, per the plugin's own README | [tamaratran/fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction) | author-stated | Full state re-sent **per request**, so a long history costs one request per handful of questions; token sizes are character estimates, not a tokenizer; "a probability is not a proof that a result is safe to delete"; falls back to the built-in summary on failure. |
| `/compact` timing needed a **private, hand-labelled eval set** (40 sessions) plus prompt hill-climbing before it behaved | [@kunchenguid](https://x.com/kunchenguid/status/2101032677940117875) | anecdotal | Thresholds are yours to find; he shifts precision→recall as the window fills. |

No post in this corpus reports a *reproducible* correctness failure of Jev itself; the failures above are schema design, missing generation, harness scope, latency and price.

## Measured numbers

Never quote these as facts. Each is one poster's run.

| Number | Reported by | Setup | Caveat |
|---|---|---|---|
| $0.00035/call, 0.44 s avg; $0.34 total vs $28.17 for Claude Sonnet 4.6; mean per-case variance 0.0000149 | @LangChain | Deep Agents weather agent, 5 fixed traces, 100 reps/judge, human oracle, `langchain-typesafe==0.0.1a2` | 5 cases; Jev service version not recorded |
| 112x lower model cost than GPT-6 Astra (code-execution computer use); 245x vs screenshot computer use; WebMCP cut cost a further 18% | @0xidanlevin | WindTunnel, 49 tasks × 8 sites, Jev + Mercury 2.5 | Open, reproducible; compares harnesses, not models |
| 1,284 tool calls in one session, 938 yes/no; bill $64.77 → $26.16; $0.000041/decision; 38 ms vs 1.84 s | [@0xCarnagee](https://x.com/0xCarnagee/status/2101456270909690202) | One session, replayed side by side | Promotional, single run |
| 100,000 X posts in 20.4 s for $0.67, 14 Nouls each; Claude Opus 5 did 214 posts for $0.98 on the same clock | [@0xMovez](https://x.com/0xMovez/status/2101325703635435523) | Self-built analyser, one parallel pass | Opus leg is truncated and extrapolated to $458, not equal work |
| Second-hand, all `unverified`: 1,018 papers for $0.08 at 256 ms median; 500 emails for 3.5¢; Vercel fx 5–18x faster safety classification than GPT-5.6-Luna; ~1M-token session compacted to 86K in ~1 s | relayed in @0xMovez / @0xCodila articles | not first-hand | Rumour until you see the harness |

TypeSafe's own 193.6x / 444.6x is a *workflow-average ceiling* ([[concepts/workflow-evals]]). @cyrilXBT calls them "a ceiling, not a typical result": the multiplier comes from a workflow's decision-to-generation ratio, not a head-to-head call.

## Open replicas and Jev-like alternatives

| Project | Approach | Claims | Maturity |
|---|---|---|---|
| [TheoLeeCJ/SemIf](https://github.com/TheoLeeCJ/SemIf) (ex-openjev) | typed decisions from a frozen open ~4B, plus a WebGPU browser demo | reproduces the *interface pattern*, explicitly not Jev's model or training | 2,310★, MIT; its Jev side is read from TypeSafe's published records (102 aligned rows, not the 711-row aggregate), not a live endpoint |
| [ekzhang/openjev-sglang](https://github.com/ekzhang/openjev-sglang) | Jev-compatible HTTP API, Qwen3.6-35B-A3B, prefill-only on SGLang/B200 via Modal | drop-in endpoint for the documented API | 214★, no license; infra-heavy |
| [@skeptrune](https://x.com/skeptrune/status/2101209390992994570) `deepseek-v4.1-flash-jev` | sglang `/v1/score` instead of `/generate`, plus a forced closing think-tag | his words: "not as good as jev" | demo endpoint |
| [TianyuCodings/NanoJev](https://github.com/TianyuCodings/NanoJev) | 0.6B trained replica + training pipeline | beats/ties Jev on 2 of its 4 game tasks, loses a third | 1,322★, MIT |
| [vinnylarouge/jevlike](https://github.com/vinnylarouge/jevlike) | option-attention head; train your own scorer over changing option lists | starter model, same I/O shape | 1,059★; its chess checkpoint lost 48/50 to Stockfish level 0 |
| [r-ms/mini-jev](https://github.com/r-ms/mini-jev) | read option-letter logits on a frozen Qwen3-4B; preregistered | same accuracy as grammar-constrained JSON, 4x faster on short texts | 29★; **its shares are normalized candidate scores, not calibrated probabilities** |
| Qwen-2.5-1B-RLCD (via [@swill1ams](https://x.com/swill1ams/status/2100421326389354624)); `convaiinnovations/laya` (via [@0xCVYH](https://x.com/0xCVYH/status/2101171688683585622)) | on-device parallel constrained decoding | "5x faster on-device"; "already above Jev" | no methodology; `unverified` |

**For a build decision.** The replicas show the *interface* is commodity — one forward pass, read scores over a closed option set — so the API shape is not lock-in and porting later is realistic. They do not show that the *calibration* is commodity: mini-Jev says outright its scores are not calibrated probabilities, and [[patterns/confidence-routing]] rests on calibration ([[concepts/confidence]]). So prototype on the hosted API, keep questions and thresholds in your own code, and self-host only once a labelled set proves a local model matches on *your* decisions. Replica claims are unverified too.

## Access, waitlist and practicalities

All `unverified`; confirm against [[reference/models-and-pricing]] and [[entities/typesafe-console]] before relying on any of it.

- Official waitlist approved in roughly a day ([@studio_yebisu](https://x.com/studio_yebisu/status/2100686990090047569), 2026-09-17).
- Reachable without the waitlist via Vercel AI Gateway as model `typesafe-ai/jev` ([@ai_agent_dev](https://x.com/ai_agent_dev/status/2100529814289678514)), and via OpenRouter from 2026-09-18 ([@studio_yebisu](https://x.com/studio_yebisu/status/2101065176069886152)). Gateways add their own pricing, latency and data terms.
- A third-party `langchain-typesafe==0.0.1a2` package exists (@LangChain). Not a TypeSafe SDK; see [[reference/python-sdk]].
- **Legal:** MCA §2.3(f) prohibits publishing benchmarks or performance information about the Services ([[reference/legal-and-data]]). Most numbers here were published by people apparently doing exactly that. Measure privately; do not publish yours.

## What this means when advising on a project

- Re-measure on the project's own data. Every number here is n≈1 on someone else's workload.
- Expect the workflow multiplier, not the headline one: gains scale with the decision-to-generation ratio. Audit the loop first.
- Prefer many narrow, mutually exclusive questions over one broad one — overlapping options produced confident nonsense.
- Keep Jev out of anything that must generate (tool arguments, queries, summaries); pair it with a small fast LLM for that half.
- Don't put Jev in a control loop faster than ~4 Hz or with a sub-250 ms budget: hosted round trips measured 260–280 ms.
- Don't ship Jev-scored compaction without an eval set and a fallback; both published compaction tools have one, hand-labelled.
- Watch benchmark circularity: TypeSafe scores against an Astra+Fable average, replicas score against TypeSafe's published records, and most X numbers just restate the launch post.
- Cost is low per decision only. At high frequency, price it per hour against a cached open model.

## Related

- [[ideas/consult]] — the "could Jev help this project?" walkthrough
- [[ideas/patterns]] — community decision patterns with fit verdicts
- [[ideas/community-repos]] — the projects these reports came from
- [[concepts/jaggedness-jev-1-13]] — the official failure-mode list these confirm
- [[concepts/workflow-evals]] — where 193.6x / 444.6x comes from
- [[guides/testing-and-evaluation]] — running your own measurement

## Sources

- X posts (URLs inline above, all captured 2026-09-20 in `raw/x/`): @LangChain, @0xidanlevin, @trycua, @jpschroeder, @cyrilXBT, @skeptrune, @0xCarnagee, @0xMovez, @kunchenguid, @swill1ams, @0xCVYH, @akshay_pachaar, @sydneyrunkle, @studio_yebisu, @ai_agent_dev
- Repos (`raw/x-repos/`): TheoLeeCJ/SemIf, TianyuCodings/NanoJev, ekzhang/openjev-sglang, r-ms/mini-jev, vinnylarouge/jevlike, tamaratran/fast-jev-compaction, kunchenguid/compact-adviser, nekuda-ai/WindTunnel, danielgshea/jev-as-a-judge