---
title: "Testing and evaluating a Jev workflow"
type: guide
tags: [testing, evaluation, calibration, consistency, thresholds]
created: 2026-09-17
updated: 2026-09-20
confidence: high
sources:
  - wiki/cookbooks/consistency-choice.md
  - wiki/cookbooks/consistency-noul.md
  - wiki/cookbooks/classification-using-confidence.md
  - wiki/concepts/workflow-evals.md
  - wiki/concepts/jaggedness-jev-1-13.md
  - wiki/concepts/confidence.md
  - wiki/entities/blog-antibenchmaxxing.md
  - wiki/reference/system-one-adapter.md
  - wiki/reference/legal-and-data.md
  - wiki/reference/models-and-pricing.md
  - wiki/guides/writing-instructions-and-criteria.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "Measure accuracy, calibration, repeatability, latency and cost on your own labelled set; pick thresholds from confidence buckets; regression-test jaggedness."
---

# Testing and evaluating a Jev workflow

> **TL;DR** Build a small labelled set, then measure five things: accuracy against your labels, calibration (accuracy per confidence bucket), repeatability across repeats of the same input, latency, and cost from `usage.input_tokens`. Pick thresholds from the calibration table, not from a cookbook. Regression-test the [[concepts/jaggedness-jev-1-13|jaggedness]] edge cases. And note before you publish: MCA §2.3(f) prohibits publishing benchmarks about the Services.

## What to measure

| Metric | How | Why it matters |
|---|---|---|
| **Accuracy** | Agreement between your workflow's final action and a label per case | The only metric that speaks to your application. Note TypeSafe's own "accuracy" means agreement with an ensemble reference, not truth ([[concepts/workflow-evals]]). |
| **Calibration** | Bucket answers by `confidence` (or by `noul`) and report accuracy per bucket | Calibration is a property of *groups*, never of one answer ([[concepts/confidence]]). This table is what sets your thresholds. |
| **Consistency / repeatability** | Run the same input N times; report per-question standard deviation and how often the decision flips | Jev is **not** deterministic ([[cookbooks/consistency-noul]]). |
| **Latency** | Wall clock around the call, measured from where your service runs | TypeSafe's published figures are measured "from our laptops on the West Coast" ([[concepts/workflow-evals]]). |
| **Cost** | `response.usage.input_tokens / 1e6 * 0.042` | Input tokens only; output tokens are free ([[reference/models-and-pricing]]). Price after the fact, so a rate change needs no new calls. |

Measure the **workflow's decision**, not just each answer. A question can wobble while the composed action stays put — and a rock-steady question can feed a decision that flips because the threshold sits exactly where the spread lands.

## Building a labelled set

1. **Collect real inputs**, including the boring majority and the genuinely hard cases. The consistency cookbooks each use a single input "built to sit on the fence"; a production set needs both kinds.
2. **Label the outcome you act on** — the queue, the route, the approve/hold/deny — not the intermediate probabilities.
3. **Split tuning from validation.** Tune wording and thresholds on one slice, validate on a held-out slice ([[guides/writing-instructions-and-criteria]]). Higher confidence on the tuning slice does not establish that an answer is right.
4. **If you have no labels**, TypeSafe's own move is an ensemble of expensive reasoning models answering every question in the harness — its workflow evals use the average of GPT-6 Astra and Claude Fable 5.1 at high thinking as the reference ([[concepts/workflow-evals]]). Carry the caveat with the numbers: a reference built from two competitors biases toward them, and the model being scored can never beat the reference by construction.
5. **Record the label's provenance.** The classification cookbook is explicit that its gold labels are self-reported SIC codes filtered so the filing text supports them, so the numbers "measure the recipe rather than the state of EDGAR's metadata" ([[cookbooks/classification-using-confidence]]).

Fifty to a few hundred cases is enough to start: the classification cookbook's headline result rests on 60 filings.

## Using confidence buckets to find the threshold

Score every case, sort by `confidence`, bucket, and report accuracy per bucket. The worked example: one `Choice` over 75 SIC groups, `confidence >= 0.9` on 30 of 60 filings was **27/30 (90%)** correct, while the 30 below were **12/30 (40%)** at group level and **70%** when reported one level up the taxonomy ([[cookbooks/classification-using-confidence]]).

Read three things off that table:

- **Where accuracy falls off** — that is your floor.
- **What the floor costs** — the share of traffic it sends to a fallback. In the choice-consistency cookbook a `0.60` top-probability floor bought 99.2% decision agreement while still acting automatically on 74.2% of answers, routing 25.8% to review ([[cookbooks/consistency-choice]]).
- **What the fallback should be.** A coarser label, a human, or a reasoning model — the cookbook's point is that an unconfident case comes back one level up instead of being dropped.

Two mechanical warnings. Gate a Noul on `noul` itself — Noul answers have no `confidence` field. And where a specific statistical rule applies, use `probabilities` rather than `confidence`; the choice-consistency cookbook states outright that its rule "uses the returned probabilities, not the API's separate `confidence` field."

## Self-consistency tests

The two consistency cookbooks are the method, not just results. Both hold one input fixed, run the rubric **15 times**, and report per-question probability standard deviation, how often the top answer changes, and cost and latency per call.

| | [[cookbooks/consistency-noul]] | [[cookbooks/consistency-choice]] |
|---|---|---|
| Rubric | 14 Nouls (claims triage) | 8 Choices (moderation) |
| Repeats | 15, plus six LLM conditions | 15, plus six LLM conditions |
| Mean probability std dev | `0.0102` | `0.0098` |
| Per call | 111 ms, $0.000043 | 114 ms, $0.000046 |
| What moved | `covered` spanned 0.43–0.53, **crossing a 0.5 threshold** | 2 of 8 questions changed their top label |
| Mitigation | Band: `< 0.30` no, `0.30–0.70` uncertain → human, `> 0.70` yes | Abstain to `uncertain` below a `0.60` top probability |

Four harness mechanics worth copying verbatim:

- A **rubric fingerprint** — a digest of the state plus every question's text — in the cache key, so editing a question forces fresh samples instead of serving stale ones.
- A per-sample **`uid`** so each repeat is a distinct call. Both cookbooks acknowledge this is a confound: the `uid` is in the state, so the run measures response-to-a-changed-state and run-to-run variation together.
- Logging **`response.model`** on every call, because `jev-latest` is an alias that can move mid-run. All 15 calls returned `jev-1.13.0`.
- Reporting **repeatability separately from accuracy**. The choice cookbook says three times that repeatability is not accuracy; a competitor scored 100% agreement in the same run with no abstentions.

The actionable output is a list of questions whose spread straddles a decision threshold. Widen the band around them, re-word them, or decompose them — a question that abstains on *every* repeat is telling you the rubric is under-specified, not that the model failed.

## Comparing against an LLM

`system-one-adapter` answers the same `typesafe_sdk` questions with an OpenAI or Anthropic model and returns a `SystemOneResponse` subclass, so your state, questions, and composition code stay byte-identical and only the client swaps ([[reference/system-one-adapter]]):

```python
from system_one_adapter import SystemOneAdapterClient

client = SystemOneAdapterClient(
    structured_outputs=True,
    llm_answer_mode="probabilities",
    normalize_probabilities=True,
)
response = client.system_one(STATE, QUESTIONS, provider="anthropic", model="claude-haiku-4-5")
print(response.usage.latency, response.usage.input_tokens_total, response.usage.n_retries)
```

It adds `usage.latency`, `input_tokens_total` / `output_tokens_total` across retries, retry counts, and per-attempt traces in `response.debug["llm_attempts"]` — the raw material for a like-for-like cost, speed, and agreement comparison. Price the LLM side with that provider's rates; the cookbooks' `TYPESAFE_PRICE = (0.042, 0.00)` covers the Jev side only.

## Regression tests for jaggedness edge cases

Keep a standing suite of cases aimed at the documented failure modes ([[concepts/jaggedness-jev-1-13]]), run on every question edit and every model bump:

| Edge case | Test |
|---|---|
| Literal reading | An input that satisfies the letter of the instruction but not the intent — and vice versa. |
| Negation | The question and its negation on the same input. They are **not** complements: one documented pair summed to 1.19. Assert your code never relies on that. |
| Numbers | An input whose correct answer requires counting or arithmetic. The right result is that your **code** produced it, not a question. |
| Dates | Mixed formats, relative references ("last Thursday"), quarter and window boundaries. Assert the parts came from Choices and the comparison from `datetime`. |
| Adversarial content | State containing an injected instruction, a misleading framing, or text arguing for its own classification. Jev does not treat state as hostile by default. |
| Indirection | A question about a property of a property; confirm the flattened rewrite scores better. |
| Cross-type invariants | If any code compares a Noul to a Choice probability, or a threshold tuned on one type to the other, delete it. |

Also pin the model. Thresholds tuned against `jev-latest` shift silently when the alias moves; pin `jev-1.13.0` and re-run the suite before adopting a new version ([[reference/models-and-pricing]]).

## A minimal harness

Runnable with `pip install typesafe-sdk` (0.6.0) and `TYPESAFE_API_KEY` set. Replace `CASES` and `QUESTIONS` with your own.

```python
"""Minimal Jev evaluation harness: accuracy, calibration, repeatability, latency, cost."""

import statistics
from collections import Counter, defaultdict
from time import perf_counter

from typesafe_sdk import Choice, TypeSafeClient

MODEL = "jev-1.13.0"          # pin; jev-latest is an alias that moves
REPEATS = 15                  # matches the consistency cookbooks
PRICE_PER_MTOK_INPUT = 0.042  # output tokens are free
BUCKETS = [0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 1.01]

QUESTIONS = {
    "department": Choice(
        instructions="Which team should handle this ticket?",
        criteria={
            "billing": "Charges, invoices, refunds, subscriptions",
            "technical": "Bugs, outages, integrations",
            "other": "Anything else",
        },
    ),
}

CASES = [
    {"id": "t1", "state": "I was charged twice for order A-104.", "label": "billing"},
    {"id": "t2", "state": "The export button 500s on Safari.", "label": "technical"},
    # ... your held-out cases
]


def evaluate() -> None:
    hits: list[tuple[float, bool]] = []
    per_case_labels: dict[str, Counter] = defaultdict(Counter)
    per_case_conf: dict[str, list[float]] = defaultdict(list)
    latencies: list[float] = []
    cost = 0.0
    models = Counter()

    with TypeSafeClient(timeout=30.0) as client:
        for case in CASES:
            for repeat in range(REPEATS):
                started = perf_counter()
                response = client.system_one(
                    state={"uid": f"{case['id']}:{repeat}", "ticket": case["state"]},
                    questions=QUESTIONS,
                    model=MODEL,
                )
                latencies.append(perf_counter() - started)
                models[response.model] += 1
                cost += (response.usage.input_tokens or 0) / 1e6 * PRICE_PER_MTOK_INPUT

                answer = response.answers["department"]
                per_case_labels[case["id"]][answer.choice] += 1
                per_case_conf[case["id"]].append(answer.confidence)
                if repeat == 0:  # score accuracy on the first draw only
                    hits.append((answer.confidence, answer.choice == case["label"]))

    # 1. Accuracy
    print(f"accuracy {sum(ok for _, ok in hits)}/{len(hits)}")

    # 2. Calibration: accuracy per confidence bucket -> read your threshold off this
    for low, high in zip(BUCKETS, BUCKETS[1:]):
        band = [ok for conf, ok in hits if low <= conf < high]
        if band:
            print(f"  confidence [{low:.2f},{high:.2f})  n={len(band):3d}  acc={sum(band) / len(band):.2%}")

    # 3. Repeatability: label flips and confidence spread per case
    for case_id, counts in per_case_labels.items():
        spread = statistics.pstdev(per_case_conf[case_id])
        flag = "  <-- FLIPS" if len(counts) > 1 else ""
        print(f"  {case_id}: {dict(counts)} conf_std={spread:.4f}{flag}")

    # 4/5. Latency and cost
    print(f"median latency {statistics.median(latencies) * 1000:.0f} ms over {len(latencies)} calls")
    print(f"total cost ${cost:.6f}   models answered: {dict(models)}")


if __name__ == "__main__":
    evaluate()
```

Cache results to a file keyed on case id, repeat index, rubric fingerprint, and model, so re-analysis costs nothing — that is what the cookbooks' `JsonCache` does ([[cookbooks/overview]]).

## Before you publish the numbers

TypeSafe's public position is "run your own private evals, treat public ones with a grain of salt" ([[entities/blog-antibenchmaxxing]]), and the company commits to no standard benchmark table, dated snapshots retired rather than hill-climbed, and publishing its own unflattering evidence.

The contract points the other way. **MCA §2.3(f) prohibits customers from publishing "benchmarks or performance information about the Services"** ([[reference/legal-and-data]]). Internal evaluation is exactly what the docs ask for; publishing your Jev numbers is a contractual matter — get permission first. This is a summary, not legal advice.

## Related

- [[ideas/field-reports]] — what independent testers measured and where Jev disappointed (community tier)
- [[concepts/confidence]] — what `confidence` is and why calibration is a group property
- [[concepts/jaggedness-jev-1-13]] — the failure modes the regression suite targets
- [[concepts/workflow-evals]] — TypeSafe's own harness and its caveats
- [[cookbooks/consistency-noul]], [[cookbooks/consistency-choice]] — the repeatability method in full
- [[cookbooks/classification-using-confidence]] — the confidence-bucket table worked end to end
- [[reference/system-one-adapter]] — same questions, an LLM behind them
- [[guides/writing-instructions-and-criteria]] — the iteration loop a failing case feeds
- [[guides/agent-integration-playbook]] — where testing sits in the build
- [[entities/blog-antibenchmaxxing]] — TypeSafe on benchmarks
- [[reference/legal-and-data]] — MCA §2.3(f)

## Sources

- wiki/cookbooks/consistency-choice.md, wiki/cookbooks/consistency-noul.md, wiki/cookbooks/classification-using-confidence.md, wiki/cookbooks/overview.md
- wiki/concepts/workflow-evals.md, wiki/concepts/jaggedness-jev-1-13.md, wiki/concepts/confidence.md
- wiki/entities/blog-antibenchmaxxing.md
- wiki/reference/system-one-adapter.md, wiki/reference/legal-and-data.md, wiki/reference/models-and-pricing.md, wiki/reference/python-sdk.md
- wiki/guides/writing-instructions-and-criteria.md